02 / Verification
The number is right. The question is not.
Answers now arrive faster than anyone can check them, and a plausible number is the hardest mistake to spot. Show what an answer was checked against, together with the answer.
Being able to say it is right.
Scroll to explore. Use the controls to try it.
Read the question. Then check the answer.
The head of sales must decide whether to take a large order with a tight delivery date. First they ask a data Q&A agent: what was last month's on-time delivery rate? The agent answers 94%. The same question went wrong once before, and the correction is still in memory. Whether this answer can be used is yours to check.
- 02Check the calculation: recount the 100 orders by the answer's definition.
- 03Check the definition: compare it with the correction in memory.
- 04Count both ways, and rewrite the answer.
- 05A person decides whether to take the order, and the decision is kept on record.
Every step waits for you, and opens only when the one before it is done.
Fictional scenarioThe orders, people and figures were made up to show a design mechanism. They are not real customer data, they show no measured efficiency gain, and no AI runs on this page.
01The same question. 94% again.
A week later, for another large order, the same question came up. The agent answered 94% again. Memory still holds last time's correction, but that does not mean the answer follows it.
Before anyone decides, check two things and keep them apart: is the calculation right, and does it answer the question?
02Is the calculation right?
Recount last month's 100 orders by the definition the answer used.
Delivered by the factory's promised date: 94 / 100
94%, the same as the answer.
Pass · the calculation holds
03Does it answer the question?
Compare the definition the answer used with the correction held in memory.
The calculation has not been checked yet. Check it first
- The answer's definition
- Counted against the factory's promised dates
- Correction in memory
- Answer this question by the customer's requested dates (head of sales)
Fail · the calculation is right, but it answers another question
04Count it both ways.
Count the same 100 orders against the factory's promised dates and against the customer's requested dates.
The calculation has not been checked yet. Check it first
- Factory's promised date
- 94 / 10094%
- Customer's requested date
- 79 / 10079%
The 15 in between met the factory's date and missed the customer's. 6 missed both.
See the 15 orders
- O-005requested day 9promised day 10delivered day 10
- O-012requested day 14promised day 18delivered day 18
- O-018requested day 12promised day 14delivered day 14
- O-024requested day 10promised day 14delivered day 14
- O-031requested day 15promised day 18delivered day 18
- O-037requested day 13promised day 14delivered day 14
- O-043requested day 11promised day 14delivered day 14
- O-050requested day 16promised day 18delivered day 18
- O-056requested day 14promised day 18delivered day 18
- O-062requested day 12promised day 14delivered day 14
- O-069requested day 17promised day 18delivered day 18
- O-075requested day 15promised day 18delivered day 18
- O-081requested day 13promised day 14delivered day 14
- O-088requested day 18promised day 22delivered day 22
- O-094requested day 16promised day 18delivered day 18
The answer, rewritten
Last month's on-time delivery rate was 79% by the customer's requested dates (94% by the factory's promised dates).
05Whether to take it is a person's call.
The answer goes to the head of sales with its definition and its evidence. Take the large order with the tight delivery date?
The calculation has not been checked yet. Check it first
Held, on the grounds of the 79%. The decision is on record.
- Question
- Last month's on-time delivery rate?
- Answer
- 79% by requested date (94% by promised date)
- Checks
- Calculation pass · definition fail → counted both ways
- Decision
- Hold it
- Decided by
- Head of sales (you)
Who carries this decision out, and how far? Next: deciding what the agent may do.
03 / On to collaborationA correct calculation is not a correct answer.
Split checking in two: is the calculation right, and does it answer with the definition the question asks for? When the definition is unclear, say so. A person decides, and the decision is kept on record too.
What changed
The answer went from 94% to 79%, and the arithmetic was never wrong. The definition changed: the same 100 orders, counted against the customer's requested dates instead of the factory's promised dates. 15 orders changed status.
Why it matters
An answer built on the wrong definition looks as trustworthy as the right one: a precise number, a calculation that checks out. A check of the arithmetic alone lets it through, and the decision on the order rests on that number.
How it transfers
For any number an AI gives you, ask two things apart: is the calculation right, and does it answer this question? When the definition is unclear, say so. List the possible definitions with their results, and let the person who asked choose.
The same gap is everywhere. Are active users counted by sign-in or by action? Does revenue include refunds? Which date makes a delivery on time?
What does this answer rest on?
It follows your steps above. What you checked becomes evidence. What is taken as true without a check stays an assumption. What this data cannot answer is listed as unknown.
Evidence
0Checked on this page
Nothing yet. Until it is checked, 94% is only a claim.
Assumptions
3Taken as true, not checked
- The calculation is right.Step 01
- The answer's definition is the one the question asks for.Step 01
- These 100 orders are all of last month's orders, and their dates are recorded correctly.Step 01
Unknowns
0This data cannot answer
Not listed yet. Only after counting both ways can you see what this data cannot answer.
A fictional, limited example: 100 made-up orders and one question. Both rates are counted from the order data on this page. Real verification also has to check where a definition comes from and what the count covers.
Questions
Is this a real company's data?
No. The orders, people and figures are fictional. 94% and 79% are not typed in either: the page counts them from these 100 made-up orders every time.
Is an AI running on this page?
No. The agent's answer is a written scenario. The page demonstrates a design mechanism: split checking in two, and hand over the answer with what it rests on.
Does this show gains in efficiency or accuracy?
No. The page measures neither. It shows one thing: what a check of the arithmetic alone lets through.
The definition was corrected once. Why did the answer go wrong again?
A correction kept in memory does not make the next answer follow it. So check every answer against its question, rather than trusting that the last fix held. See how memory brings results back
What if the question's own definition is unclear?
Say that it is unclear. List the possible definitions with their results, and let the person who asked choose. The system does not make that choice for them.
How do I apply this to my team's AI answers?
Start with the last number someone used to decide something. Which definition does it count by? Does that match what was asked? Which parts were checked, which are assumptions, and which are still unknown? Hand over those three columns with the number.