The sponsor wants a faster AI demo. What should you measure?
A warehouse sponsor asks Fieldgate to cut support handling time. The team wants an AI assistant next Friday.
The question
You are responsible for the pilot’s acceptance plan. The sponsor and operations lead disagree about success.
- SponsorCut average handling time from 8 to 4 minutes.
- OperationsAgents often close cases early. Customers reopen them when the replacement part is wrong.
- AnalystThe two work stages consume the same team’s capacity.
| Cases | Minutes per case | |
|---|---|---|
| Initial handling | 1000 | 8 |
| Reopened work | 200 | 10 |
Authored example. Reopened work is additional effort on the initial cohort.
“Choose the success measure, calculate current weekly effort, and propose a pilot with a release rule.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Choose an outcome
What would you do here? Open the move you’d make.
Measure first-close speed only
Tempting, but it misses something. First-close speed rewards closing a case even when the customer returns with the same problem.
Measure completion and repeat work
The strong move. The sponsor’s metric omits a documented failure.
See how a strong candidate says it
“I would track correct, completed resolutions, including reopened work. Handling time alone rewards closing the wrong case. I would agree a fixed follow-up window and verify completion with operations.”
Why it works: The sponsor’s metric omits a documented failure.
Step 2: Calculate the starting cost
What would you do here? Open the move you’d make.
Ignore the reopened cases
Tempting, but it misses something. The 200 reopened cases require 2,000 additional minutes from the same team.
Include 2,000 minutes of repeat work
The strong move. Both stages consume capacity.
See how a strong candidate says it
“The stated week uses 8,000 initial minutes plus 2,000 repeat minutes: 10,000 minutes. The 20% reopen rate is an event rate within the supplied cohort, not a prediction for every future customer.”
Why it works: Both stages consume capacity.
Step 3: Bound the first pilot
What would you do here? Open the move you’d make.
Automate every replacement immediately
Tempting, but it misses something. The existing evidence does not justify unattended orders across all part categories.
Pilot one category with reviewed recommendations
The strong move. The pilot tests useful work before broad side effects.
See how a strong candidate says it
“I would start with one common part category, draft recommendations with their source, and keep an agent responsible for the final selection. Operations would audit a held-out sample for correct parts.”
Why it works: The pilot tests useful work before broad side effects.
Step 4: Agree the release rule
What would you do here? Open the move you’d make.
Launch if the demo is impressive
Tempting, but it misses something. An impressive demonstration does not establish correct customer resolutions.
Compare completion and errors in the same cohort
The strong move. A release condition must measure the customer’s actual goal.
See how a strong candidate says it
“Proceed only if completed resolutions improve without more wrong-part orders. I would name the operations owner and compare the same category and observation window with the baseline.”
Why it works: A release condition must measure the customer’s actual goal.
The answer, in short
- Decision
- Pilot for correct resolutions, not faster closures.
- Because
- The stated baseline includes repeat work.
- What would change it
- A validated change in how operations defines a completed resolution would change the metric.
Common traps
- Halving first-close time while increasing repeat work.
- Comparing different categories or observation windows.
Check your answer
- Did you count reopened work in the 10,000-minute baseline?
- Did you define a completed resolution and observation window?
- Did you keep first-pilot scope to one tested category?
- Did you gate expansion on completion and wrong-part outcomes?
Last reviewed Nov 10, 2026