InfraLoka AcademyPathwayAI
PathwayAI
Masuk
InfraLoka Academy
IDEN

Module 2 · Know the tool

Why the answers keep changing

3 min read5 min readReviewed 26 October 2026

In short

A model picks words by sampling, so the same prompt can give different answers. Temperature sets variety, not truth.

After this lesson you can

  • Explain why the same prompt can produce different answers
  • Understand what temperature controls and what it does not

Terms in this lesson

Understand

You run the same prompt twice and get two different answers. That is not a sign something is broken. It is how the model is designed to work.

Remember that the model writes token by token. At each step it does not have one fixed continuation but a spread of possibilities: this word is quite likely, that one a little less, another rare. The app then picks one at random, with odds that follow that spread. This is called sampling. Once an early pick differs, say the first sentence starts with "Here" instead of "Sure", the rest of the answer turns down another path.

How wide that choice is gets set by temperature. A low temperature makes the model pick the top choices more often, so answers are more alike from one run to the next. A high temperature gives less common choices a bigger chance, so answers vary more. In an ordinary chat app the temperature is usually set by the provider and hidden. In developer services it is one of the settings you can change.

What to hold on to: temperature controls variety, not truth. A low temperature does not make a model more honest or more knowledgeable. If the model has no basis for a figure, a low temperature only makes it repeat the same invented figure more consistently. Even at the lowest temperature, some services can still give slightly different answers.

Temperature is not the only reason answers change either. A different conversation history, memory the app keeps, attached files, changing search results and a model the provider has updated can all change the answer. So one good answer does not prove your prompt is reliable. How to test an important prompt on 5-10 real samples is covered in the lesson "Improving in rounds" in the module Ask clearly.

This variety is not always the enemy. For creative tasks, such as finding a title, an opening line or a fresh angle, varied answers are useful. You can run the same prompt several times, or ask for several options at once, then pick the best.

For tasks that must be consistent, such as categorising customer complaints, filling table columns or scoring documents against fixed criteria, variety is a risk. Usually its biggest cause is not temperature but a request open to many readings. When the rules for choosing are clear, the reasonable options narrow and the answer steadies with them, whatever the temperature.

Worked example

Before

The prompt

I need a category for this complaint: "The parcel came two days late and the box was dented, the contents cracked."

What came back

First run: "Category: Delivery." Second run, same prompt: "Category: Product damage."

After

The prompt

I need a category for a customer complaint, for the service team's weekly report. Choose exactly one of: Late, Damaged item, Wrong item, Other. Rule: if an item is damaged, choose Damaged item even if it was also late. If unsure, write Needs a human check. Write the category, then one sentence of reason. Complaint: "The parcel came two days late and the box was dented, the contents cracked."

What came back

First and second runs match: "Damaged item. The contents cracked; there was also a delay, but the rule puts damage first."

What changed

  • Both first answers were reasonable. The difference came from an open request, not a broken model.
  • A closed list and a priority rule narrow the plausible continuations, so reruns give the same result.
  • The "Needs a human check" option gives an honest way out for complaints that really are unclear.

Common mistake

Mistake
Lowering the temperature to fix answers that get facts wrong.
Why it hurts
Temperature only sets how varied the word choice is. If the model has no basis for the fact, a low temperature makes the wrong answer appear more consistently, and that consistency is easily mistaken for truth.
Do this instead
For facts, give the material and its source, let the model say it does not know, then check. Use a low temperature only for tasks that need to be uniform, such as categorising.

Common mistake

Mistake
Saving a prompt as "tested" because one answer looked good.
Why it hurts
The next answer is drawn from the same spread of possibilities and may take another path. A colleague using that prompt tomorrow may get a very different result and think the fault is theirs.
Do this instead
Run it several times and try several different samples, hard ones included, before calling it ready to share.

Try

The same prompt run twice: "I need one sentence announcing the office is closed next Friday for electrical maintenance." Open the cards.

The IT team offers to lower the temperature of the model that answers staff questions about leave rules, because it is sometimes wrong. What is the best response?

Sorting card

For these tasks, does varied output help or is it a risk?

  1. Scoring job applications against fixed criteria

  2. Finding name ideas for an office event

  3. Filling a status column from email content

  4. Simplifying a paragraph for lay readers

  5. Writing three versions of a presentation opener

5 cards left

A prompt to ask for several options at once

Works with any AI assistant.

Fill in your parts
For example: three
For example: subject lines for an email announcing a new schedule
For example: all warehouse staff

Your prompt

I need ‹jumlah› options for ‹hal› for ‹pembaca›. Make the options genuinely different from each other, not small variations. For each option, write one sentence on when it fits best.

Take home

When the answers keep changing

  • Does this task need uniform answers, or does variety help?
  • Is my request open to many readings?
  • Have I given a closed list and a rule for unclear cases?
  • Am I judging the prompt from a single answer?

What went wrong for us

Incident, October 2026

Filters that dropped every row

Context
We were building a psychometric testing feature in a recruitment app with AI agents. An audit page lets HR filter results by group.
What happened
The group filters compared a key in JSON data with the text "true". Rows without that key give NULL, and NULL never equals anything, so those rows were dropped. Every filter but one returned nothing.
Impact
Found before release; no HR user saw the empty pages.
Root cause
Every test fixture always set that key, so the tests never met a row without it. Real rows often lack it.
What we did
A reviewer ran the queries in Postgres against rows shaped like real data, found the empty results, then the filters were fixed and fixtures gained rows without the key.
The guard we keep now
Reviewers check queries against rows shaped like real data, not only fixtures, and fixtures include missing and empty values.

The lessonGreen tests only prove the code matches the examples you gave it.

What it means for your workOne good result only proves the result for that sample. Try empty, odd and hard samples too.