Module 4 · Check before you trust
Checking the output before you use it
In short
Your time is limited, so first check the parts that hurt most if wrong: figures and commitments, names and references, then general facts, and only then tone. Ask the AI to list the claims to check, then you match them against sources.
After this lesson you can
- Check AI output starting from the parts that do most harm if wrong
- Ask the AI to flag every claim that needs checking
Terms in this lesson
Understand
Hardly anyone has time to check every word of every AI output. The good news is you do not need to. What you need is an order: check the parts that hurt most if wrong first, then move down to lighter parts until you run out of time or risk.
The order we use has four layers.
Layer one, things that bind: money amounts, quantities, dates, deadlines and commitments. "Price includes tax", "we deliver by Friday at the latest", "the discount applies to every branch". Sentences like these can become obligations once sent. One wrong figure in a quotation costs more than ten awkward sentences.
Layer two, names and references: people's names, job titles, company names, document numbers, rule titles, links, quotes. Getting a client's name wrong makes the reader stop trusting the rest. An invented reference makes you look as if you did not check.
Layer three, facts about the world: "this rule took effect in ...", "the market grew by so many percent", "most customers prefer ...". This is where hallucination hides most often: it sounds reasonable with nothing behind it.
Layer four, flow and tone: does the order make sense, is it polite, does it suit the reader. Important, but mistakes here rarely cost as much as the three layers above, and they usually show as soon as you read.
One thing is easy to miss in every layer: what is not there. A model can drop an important condition from a contract it summarises, or forget an exception in a policy. When checking a summary, also ask "what should be here that does not show up?".
For important figures, re-reading is not enough. How to recalculate a figure by another method is covered in the lesson "Check the numbers and measure the result" in the Produce real work module. Here, just remember that figures always belong to layer one.
AI can help with checking, as long as its role is clear: it helps find what needs checking; it does not decide what is true.
The way to do it is to ask the model for a list of claims. A claim is any sentence that can be true or false: a figure, a name, a date, a cause and effect, a commitment. For each claim, ask it to note its type, where it sits in the text, whether it came from the material you gave or not, and how much harm it would do if wrong. A list like this helps because people easily miss a claim tucked into a long sentence, while a model is diligent enough at breaking text into small pieces.
Two things make the list more honest. First, run it in a new conversation, not the one that wrote the draft. A model tends to defend its own writing. Second, do not ask the model to declare claims "true" or "verified". Ask it only to flag them and give a risk level. The matching against sources is still yours, starting with high-risk claims.
This is also a small evaluation: you test the output in a structured way, not on the feeling that "it looks right".
Before
The prompt
Please check whether this quotation email is correct. [draft quotation email]
What came back
This email is good and professional. Its structure is clear, the tone is polite and the information is complete. You could add a closing line such as "We look forward to hearing from you" to make it warmer.
After
The prompt
I am about to send this quotation email to a prospective client. Do not judge right or wrong. Make a table of every claim in the email: every figure, date, name, commitment and statement of fact. Columns: claim, type, the sentence it comes from, whether it is in the price sheet I attached, and risk if wrong (high, medium, low). Sort by highest risk. Also add anything from the price sheet that is missing from the email. Email: [draft] Price sheet: [price table and terms]
What came back
1. "Total Rp48,500,000 including tax" - figure and commitment - the price sheet has Rp48,500,000 excluding tax - high risk. 2. "Delivery within 14 working days at the latest" - deadline commitment - the price sheet says 21 working days - high risk. 3. "Two-year warranty" - commitment - not in the price sheet - high risk. 4. Recipient's name and title - name - not in the material - medium risk. 5. "Used by more than 100 companies" - general fact - not in the material - medium risk. Missing from the email: the 30 percent down-payment term from the price sheet.
What changed
- "Check whether it is correct" invites a general verdict, and the model tends to agree.
- The second request asks only for flagging, not a verdict of right or wrong.
- The price sheet is the comparison, so three high-risk errors show at once.
- You still open the price sheet to confirm rows 1 to 3 before fixing the email.
Common mistake
- Mistake
- Asking the same conversation "anything wrong?" and accepting "nothing".
- Why it hurts
- A model that just wrote the draft tends to treat it as right, and a general question does not make it look at figures or names one by one. You feel you have checked, yet nothing has been matched.
- Do this instead
- Open a new conversation, ask for a claims table with risk levels, then match the high-risk claims to their sources yourself.
Try
Sorting card
An AI draft announcement to customers contains the parts below. Sort them: check first, or can wait?
The order of the second and third paragraphs
A slightly stiff opening line
A promise of compensation for affected customers
The customer service number
The date the service pauses
5 cards left
Ask for a list of claims to check
Works with any AI assistant.
Your prompt
I will use the text below for ‹pemakaian›. Do not judge right or wrong, and do not rewrite the text. Make a table of every claim in the text: every figure, date, name, reference, commitment and statement of fact. Columns: claim, type, the sentence it comes from, whether it is in the comparison material, risk if wrong (high, medium, low). Sort by highest risk. Below the table, list anything important from the comparison material that does not appear in the text. Text: ‹teks› Comparison material: ‹bahan›
Open the cards: what is most often missed when checking an AI summary?
The contract summary states your obligation but drops the condition "once payment is received". The sentence is true, but without the condition its meaning changes.
The policy summary says "every employee is entitled", while the original excludes employees on probation.
"About 50 percent" in the summary came from 43 percent in the data. Rounding like this changes the reader's impression, so it belongs in layer one.
Take home
Checking order before you send
- I have matched figures, dates, deadlines and commitments to their source.
- I have checked names, titles, document numbers, links and quotes.
- I have removed or sourced facts about the world that are not in my material.
- I asked: what should be here but is missing?
- The claims table was made in a new conversation, and the final call stays with me.
What went wrong for us
Error pages kept by the CDN
- Context
- The evidence gallery in our legal app calls an API through a proxy. One routing rule in the proxy was wrong, so gallery requests got 404.
- What happened
- The CDN in front of the app stored those 404 answers with a long cache lifetime, so the gallery still looked broken once the server was right.
- Impact
- Users saw an empty gallery.
- Root cause
- The proxy routing rule was wrong, and the CDN gave a cache lifetime to every answer, errors included.
- What we did
- The proxy routing rule was fixed, a routing test added, and the gallery's API requests now use no-store.
- The guard we keep now
- A routing test guards the proxy rules, and data that must always be fresh is fetched without the cache.
The lessonA fix is not done until users actually see it.
What it means for your workAfter fixing something, look at the result from the receiver's side: open the link, check what they see, and do not stop at "I changed it".
