The pilot is accurate. Why does nobody use it?
Fieldgate’s service assistant passes its quality sample. Usage falls after its first week.
The question
You are reviewing the pilot with the service team. The launch report and live workflow tell different stories.
- Dashboard100 invited agents; 80 sign in once; 12 use it in week two.
- AgentI must copy the ticket into another tool, then copy the answer back.
- Team leadThe demo used easy tickets. The live queue has multilingual attachments and missing policy numbers.
“Separate trial from sustained use. Propose one change and the evidence needed to expand.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Read the denominator
What would you do here? Open the move you’d make.
Report 80% as sustained adoption
Tempting, but it misses something. One sign-in is trial, not sustained use; eligible work also changes the adoption denominator.
Separate invitation, trial and repeated use
The strong move. Different events answer different adoption questions.
See how a strong candidate says it
“Week-two use is 12 of 100 invited agents, not 12 of 80 sign-ins presented as general adoption. I would also check which agents had eligible tickets before inferring rejection.”
Why it works: Different events answer different adoption questions.
Step 2: Observe the work
What would you do here? Open the move you’d make.
Schedule mandatory training first
Tempting, but it misses something. Training alone does not remove copy-and-paste work or unsupported attachments.
Observe eligible work and its friction
The strong move. The packet identifies workflow and coverage gaps.
See how a strong candidate says it
“Watch an agent handle an eligible ticket. Measure copying, review, exceptions and completion. Ask about the missing inputs and attachments; do not assume the only issue is lack of training.”
Why it works: The packet identifies workflow and coverage gaps.
Step 3: Change one constraint
What would you do here? Open the move you’d make.
Add a more capable model immediately
Tempting, but it misses something. A stronger model may leave the queue integration and missing-input problem intact.
Test the workflow integration on the missing cases
The strong move. Better model output may leave copying and context gaps intact.
See how a strong candidate says it
“Test an in-queue draft on a permitted, representative ticket subset. Keep a visible path to source records and human correction. Compare end-to-end effort with the same tasks without the assistant.”
Why it works: Better model output may leave copying and context gaps intact.
Step 4: Choose the next signal
What would you do here? Open the move you’d make.
Use sign-ins as the only success metric
Tempting, but it misses something. Sign-ins cannot show whether agents completed correct work with less effort.
Track repeat useful work and quality by slice
The strong move. Adoption without useful outcomes is insufficient.
See how a strong candidate says it
“Track repeat use among agents with eligible work alongside correct completion, review time and rework. Report multilingual performance separately before broadening. The original sample cannot certify those missing cases.”
Why it works: Adoption without useful outcomes is insufficient.
The answer, in short
- Decision
- Repair the workflow and test the omitted work.
- Because
- The accuracy sample and sign-ins omit the live task’s main constraints.
- What would change it
- Evidence that eligible agents save time with correct outcomes would justify expansion.
Common traps
- Calling first-week sign-ins sustained adoption.
- Generalizing from easy text-only tickets to the live queue.
Check your answer
- Did you separate the 100 invitations, 80 trials and 12 repeat users?
- Did you observe an eligible live ticket?
- Did you test queue integration and the omitted input types?
- Did you track correct completion and review effort by task slice?
Last reviewed Nov 10, 2026