How to run a model comparison you can trust

In August 2026 we wrote “tested, it passes” three times during one round of model tests. Each time, the same request failed when we ran it again. Every one of those passes rested on a single run.

The tests were our own: three rounds, run by hand between 18 and 31 August 2026 on the prompts of a design product, and recorded in a ledger with the raw token counts and our mistakes. LLMCompare is the product version of that procedure. The ten rules below are what the ledger taught us about running a model comparison on your own task, whether you compare AI models with LLMCompare or with a script.

What goes wrong when you compare models by hand?

The first round, dated 18 to 19 August 2026, compared Claude Sonnet 5, Gemini 3.7 Flash and GPT-5.6 Terra on three design briefs. The ledger lists four mistakes we made while reaching the conclusions of that round, and calls all four reproducible.

1. We treated one lucky run as a result. We did this three times.
2. We changed the pass criterion halfway through an experiment. “Has content” first meant a title that is not empty, then a title and a copy plan that are both not empty. Results from before and after the change could not be compared.
3. We changed two variables at once. We cut the number of design directions from 3 to 1 while testing the effect of a second variable, then announced that we had found the root cause. Setting the number back to 3 overturned the finding immediately.
4. We verified with a request that production never sends. The three passes were hand-written HTTP calls without the system prompt, and the 72 KB system prompt was the only difference between those calls and production.

None of the four mistakes is about a model. All four are about procedure, which a tool can enforce.

How many times should each test case run?

More than once. Model output varies between identical requests, so a single success is not evidence.

The second round, on 30 August 2026, ran GPT-5.6 Terra twice at each of five reasoning_effort levels. The input was identical: all ten calls reported 1,261 input tokens. At the xhigh level the first run used 458 reasoning tokens and the second used 813. One run per level would have reported one of those two numbers as the cost of xhigh.

Rule 1: run each cell several times and show the spread. Every repeat is another call. LLMCompare shows the estimated credits before anything runs, and the pricing page explains how credits are priced.

Rule 2: put the sample size next to every conclusion. The first round ran each model once on each brief, nine data points in total, and the ledger labels every quality judgement from that round an observation that still has to be checked on real traffic. A report on a small sample should say “for reference only”.

Which numbers are measurements, and which are opinions?

Token counts, response time and cost are measurements. A quality rating is an observation by whoever did the rating. In the second round the quality comparison was one person reading five outputs, with no blind review and no scoring, and the ledger says so.

Rule 3: report measurements and judgements in separate places, and never merge them into one score. A combined score hides which part was measured and which part was judged.

Did every model run under the same conditions?

In the first round the three models did not. Each model produced structured output through a different mechanism. Claude Sonnet 5 wrote JSON as plain text, with the schema placed in the user message. Gemini 3.7 Flash used responseSchema. GPT-5.6 Terra used response_format with a JSON schema and strict set to false. With one mechanism for all three, two of the models would not have produced a result.

Because the three models ran through three mechanisms, the ledger restates the question of the first round. The question is not which model is stronger. The question is which combination of a model and the best mechanism available to that model is stronger.

Rule 4: when conditions differ, say so in the result. The second round shows the opposite case: one configuration line changed between runs, and the identical input token count across ten calls shows that the input did not.

Why can’t token counts be compared across providers?

Providers count tokens differently. In the first round Claude Sonnet 5 reported about 4,000 input tokens per call, while Gemini 3.7 Flash and GPT-5.6 Terra reported about 20,000. All three received the same system prompt of 72,705 bytes. Anthropic’s input_tokens field excludes the cached part of the prompt, and the prompt_tokens field of the other two providers includes the cached part.

Output counts in the first round differed too. OpenAI’s output count already included reasoning tokens, while Anthropic and Gemini reported output and reasoning as two numbers. All three billed reasoning tokens as output.

Rule 5: convert usage to one definition before comparing, and keep the raw numbers available.

Rule 6: list reasoning tokens on their own line. Reasoning tokens never appear in the visible output, and they are often the main cause of a cost difference. In the second round, raising reasoning_effort on GPT-5.6 Terra from none to xhigh raised reasoning tokens from 0 to as many as 813, while the visible output stayed between 252 and 288 tokens across all ten calls. The ledger puts the output bill at xhigh at 3.8 times the output bill at none. Effort was not a monotonic scale either: in the second run the high level used 159 reasoning tokens and the medium level used 317.

What counts as a failure?

A timeout, a refusal and an empty output are results, and they count against the model that produced them. A rate limit at the provider or an outage on the platform is a different event: the call did not complete, and it says nothing about the model.

Rule 7: record every failure, and keep the failures of the model apart from the calls that never completed. Our own tooling broke this rule in the second round. A script that waited for results recognised only success. When a call failed with HTTP 400, the usage log stayed empty because it was written only on success, and the script kept waiting.

Where do the prices and parameters come from?

Rule 8: give every price a source and a date. The prices of the first round were checked against the pricing pages of the three providers on 19 August 2026. On that date Gemini 3.7 Flash cost $0.75 per million input tokens and $3.75 per million output tokens, introductory prices due to rise to $1.50 and $7.50 after 31 December 2026. A cost comparison that does not name its prices cannot be checked once they change. LLMCompare shows the date each price was last checked on its list of models and prices.

Rule 9: freeze the conditions at the moment of the run. Store the model version, the parameter values that took effect, the prompt and the price table as they were when the run started. Documentation is not a record of what took effect: it listed seven values for reasoning_effort, and GPT-5.6 Terra rejected minimal with an HTTP 400 that named the five values it supports, which are none, low, medium, high and xhigh.

Rule 10: say where speed was measured. Response time depends on the region of the machine that sends the request and on the network at that moment. LLMCompare measures response time from its own servers and states the location in each report.

Should the reviewer know which model wrote which output?

No. The third round, on 31 August 2026, was a blind A/B test of a prompt change on GPT-5.6 Terra: a “best for” field added to every line of a font catalogue in the prompt. Four briefs ran with and without the change. The reviewer saw the two groups side by side without being told which was which, and picked the group with the field. The measured cost of the change was 6,725 more input tokens per call, from 20,975 to 27,700, an increase of 32%.

The ledger limits the conclusion of the third round: four briefs, two groups, one run each, one reviewer, one pass. That sample supports a decision to switch the field on or off, and no statement about how much the quality improved.

Human review in LLMCompare is blind by default: model names are hidden and the outputs of each test case appear in random order.

The ten rules as a checklist

1. Run each cell several times and show the spread.
2. Put the sample size next to every conclusion.
3. Report measurements and judgements separately, never as one score.
4. When conditions differ between models, say so in the result.
5. Convert token usage to one definition and keep the raw numbers.
6. List reasoning tokens on their own line.
7. Record every failure, apart from calls that did not complete.
8. Give every price a source and a date.
9. Freeze model version, parameters, prompt and prices at the moment of the run.
10. Say where response time was measured.

What this article does not show

The ledger covers three rounds on the prompts of one product, with one or two runs per cell. That record does not support a ranking of the three models, and this article gives none. Every number quoted here describes August 2026, and both model versions and prices change.

Each test article in our model comparisons links to a public report where the test cases, the outputs and the numbers can be checked. To apply the ten rules to your own task, start an evaluation with your own test cases.