PracticePilot
AI role-play training for insurance agents.
PracticePilot is an MVP app designed around focused AI-assisted rehearsal for new insurance agents. It turns a broad training brief into a voice-first practice loop with transcript-grounded coaching, daily challenges, and clearer artefacts for product, training, and compliance review.
10 min read
Impact
A clear path to scalable, compliance-ready sales coaching.
PracticePilot enabled agents to build muscle memory through repeated practice with AI-simulated customers. The transcript-grounded feedback system made coaching credible and inspectable for compliance teams, while gamification mechanics like challenges and milestones established a sustainable practice habit that drove measurable engagement and skill improvement.
- 150% increase in practice sessions with AI-simulated customers versus traditional training methods.
- 90% of feedback anchored to transcript evidence, enabling compliance teams to verify coaching insights.
- 4x higher return rate when agents were offered daily challenges, driving sustainable practice habits.
A quick summary
Connect habit, practice, and feedback
Agents move from a daily prompt into live role-play, then back into a coaching report.
Making practice a beneficial habit
Challenges reward consistency while the library keeps approved study material and references close.
Set up a focused session in three steps
training intensity, customer, and product choices make every session specific and allow the AI to play a bounded role.
Practise in a live simulation
ElevenLabs-powered: voice-first rehearsal lets agents practise with lifelike customers using their voice, not their notes.
Review with AI-assisted coaching
Users receive AI-powered feedback on five core skills, with every insight anchored to transcript evidence instead of generic summaries.
Track progress over time
The analysis view shows trends, session volume, and the next coaching focus so users can track their progress and plan future practice.
Have at least 10 minutes? Read the case study below.
PracticePilot helps new insurance agents turn training knowledge into rehearsed customer conversations.
New agents at a multinational insurer had extensive product references, scripts, and compliance guidance. However, knowing the material did not directly translate to confidence in the field during high-stakes client meetings or calls. PracticePilot bridges this gap by turning passive study into active conversation rehearsal with an AI-simulated customer they can always practise with. The product design focused on building trust in the AI feedback by anchoring coaching to transcript evidence, while gamified challenges and progress views encouraged ongoing practice beyond initial onboarding.
This was part of a larger initiative to enable sales readiness. The product used generative AI, but the core design challenge was building user and organisational trust: making practice realistic, making feedback auditable, and keeping training and compliance stakeholders in the loop.
Context & Problem
The overarching limiting factor was clear: a lot of new agents, limited trainers, limited time and space to practise, gain feedback, and build confidence.
The problem
Current training procedure was not turning into agent confidence efficiently.
New insurance agents struggled to apply academic training to real conversations. Standard onboarding offered reading materials and videos, but lacked a safe space to practice spoken objection handling and regulatory compliance.
Passive training
Agents could read material and attend sessions, but freeze when a customer misunderstood a benefit or raised an objection.
Thin feedback loops
Trainer feedback was hard to repeat at scale, and generic AI summaries are not credible in a regulated sales environment.
No return habit
There was no reason to practise after first-week onboarding, even though fluency needed repetition.
Limited training time
Trainers were also active agents, so they had limited time to provide personalised feedback at scale.
Inconsistent role-play quality
Live training varied based on trainer availability and expertise. The inconsistency made it hard for agents to gain consistent practice and feedback.
The Process: Behind the Scenes
To build a trustworthy AI-driven training experience, the team had to navigate several strategic constraints:
- Regulated parameters: AI agents had to play strict, realistic customer personas without hallucinating invalid policy options.
- Pre-approved scoring system: The insurance and compliance teams had an established rubric for evaluating core sales skills, and the AI’s coaching needed to align with this framework via product policies and prompt design.
- Clear feedback: Every piece of AI feedback had to cite direct transcript evidence rather than generic summaries.
- Stakeholder approval: Trainers required editorial control over scenario criteria and scoring, while compliance leads needed to review and approve all AI-generated content and coaching.
- Micro-learning fit: Rehearsal sessions had to fit easily within 10-15 minute blocks between active appointments.
My team then leveraged AI to accelerate content drafting and prompting while reserving product decisions for human reviewers. The process relied on a tight loop: draft scenarios with AI, refine against training guidelines, test functional prototypes, gather compliance approvals, and polish.
Interactive Prototyping
Since the design and engineering teams worked on separate schedules, I built an interactive browser-based HTML prototype with the help of ChatGPT to test prompts, conversation states, and voice interruption behaviour before committing to final UI designs. This allowed the product, engineering, and compliance teams to critique the same system interactions together, rather than relying on static mockups or separate engineering tests.
Note: The HTML prototype is hosted on the company’s private server, so the link cannot be included here. However, you can view a screen recording of the alpha version of PracticePilot, which is password protected (due to the company’s name and products being mentioned) and available upon request: PracticePilot Alpha Prototype Demo.
Solutions Deep Dive: Key Design Decisions
Calibrating Scenarios Before Practice
Rather than dropping agents into a blank microphone screen, the onboarding sequence guides them through training intensity levels, target products, and structured customer persona cards.
Optimising the Voice-First Simulation Loop
The live simulation is voice-first to mirror real sales calls and build fluency under pressure. A clean, focused interface keeps the agent oriented during the high-stress conversation, minimising screen-based distractions. The system processes speech-to-text dynamically as a form of subtitle and transcript, which allows the AI to analyse the conversation in real time and provide feedback anchored to specific moments in the dialogue.
Auditable AI Feedback with Transcript Evidence
PracticePilot maps feedback to five core skill dimensions through a bounded AI report-generation workflow that requires the model to anchor every suggestion to a direct timestamped quote in the transcript. This inspectability allows agents and human coaches/mentors to verify or challenge the AI’s conclusions.
Habit-Building via Gamified Challenges
The dashboard centres progress and daily challenges rather than static tutorials. Agents earn experience points (EXP) , complete seasonal milestones, and access a reference library containing approved sales scripts and compliance materials. Gamified challenges and progress views encourage ongoing practice beyond initial onboarding, while the library keeps approved study material and references close at hand.
History and analysis views show session volume, score trends, and skill breakdowns so agents can track their progress and plan future practice sessions. The analysis view also surfaces the next coaching focus to guide users on what to practise next.
Impact & Outcomes
The 20-week design sprint transformed a broad AI brief into a structured and auditable AI-powered sales practice MVP. The MVP gained traction with the company’s Thailand sales team, and further regional expansion was approved. A personal insurance agent friend of mine in Singapore from the same company has already been invited to test it with his team.
Project Summary
How the project transformed the training experience for new insurance agents.
Practice
from reading to role-play
New agents moved from passive study to guided spoken rehearsal.
Coaching
feedback grounded in evidence
Session feedback ties directly to transcript moments and score drivers.
Retention
daily reasons to return
Challenges, EXP, and milestones encouraged consistent repeat practice.
20
weeks to validated direction
The team progressed from open brief to a tested concept and stakeholder-ready narrative.
Procedures before and after PracticePilot
Skill-building
Learning products and scripts from PDFs, videos, and live sessions.
Practising with an AI-simulated customer and receiving coaching tied to the session transcript.
Engagement and return
Intermittent training sessions based on trainer availability, with no structured reason to return after onboarding.
AI-reinforced daily challenges, progress tracking, and a reference library encourage ongoing practice and skill development.
AI credibility
Generic AI summaries that lacked transparency and auditability, making them less trustworthy in a regulated sales environment.
AI feedback anchored to specific transcript evidence, allowing agents and trainers to verify insights and build trust in the system's coaching.
Outcome
PracticePilot gave the team a concrete product direction for AI-assisted sales training.
Agents are more likely to practise in their own time, owning their skill development rather than relying on mentors or trainers.
150%
increase in practice sessions
Agents practised more frequently with the AI-simulated customer than with live trainers or self-study materials when comparing similar time periods before and after PracticePilot's launch.
90%
of feedback anchored to transcript evidence
Training and compliance stakeholders could verify coaching insights instead of trusting generic AI summaries, significantly reducing review time.
4x
higher return rate with daily challenges
Agents returned to practice consistently when offered scenario challenges matching real objections, compared to generic practice prompts that didn't target specific skills or weaknesses.
100%
of stakeholder alignment on prototyped flows
Product, training, engineering, and compliance teams all agreed on the practice flow and report structure, eliminating scope misalignment.
Reflection
As I was designing PracticePilot, I found that in regulated spaces like insurance, the novelty of generative AI wears off quickly. For me, real value came from building inspectable systems. When I collaborated with the insurance and compliance team to configure the prompts, scoring criteria, and transcription evidence, I turned an “AI black box” into a credible training tool that agents and trainers could trust and verify together. The product design had to be built around the AI’s strengths and weaknesses, which meant designing a practice loop that made the most of the technology while keeping human reviewers in the driver’s seat. At this point, humans are still the best at making strategic decisions, and our role as designers is to create systems that leverage AI’s capabilities rather than let them control the end-to-end experience.
Contents