Why Gemini and Claude Answer Long, and What a Cut-Off Answer Does to a Visibility Score

In August 2026, 92% of the Gemini answers our tracker collected stopped before they finished. Claude's stopped 42% of the time. Nothing was wrong with the questions or the engines. The output budget we gave them was sized for ChatGPT, and reasoning models spend that budget before they start writing. This post is about how that happened, what it did to scores, and what a tracker should do about it — because it is not specific to us.

Reasoning models think inside the output budget

Every API call sets a maximum number of output tokens. For a plain chat model, output tokens are the answer. For a reasoning model with web search on — Gemini's pro tier, Claude Sonnet with search — output also includes the model's hidden reasoning: the searches it decides to run, what it made of the pages, the plan for the answer. That reasoning can run to a couple of thousand tokens before the first visible word.

Set the cap at 1,500 tokens, which is generous for a five-item answer, and Gemini returns 55 visible tokens and a finish reason of MAX_TOKENS. Raise it to 8,000 and the same prompt produces a complete, ranked answer of 500 visible tokens. The cost difference is small — you only pay for tokens produced — and the difference in information is everything.

What a cut-off answer does to a score

A tracker has two bad options and one good one. Score the truncated answer as "not mentioned", and every long-answering engine reports the brand as absent when it might have been named in the paragraph that never arrived. Score it as neutral, and it dilutes the real answers. The good option is to exclude it: a truncated answer that contains no mention is inconclusive, and the engine simply did not contribute to that check.

That is what our tracker did, correctly. The consequence was invisible for a month: Gemini contributed 44 usable answers out of 560 calls, its weight silently dropped out of almost every score, and the dashboard still said "5 engines". The scores were built from ChatGPT, Perplexity and a shrinking share of Claude and Grok. Because those engines have their own opinions about which brands to name, every customer's number was tilted towards them.

How it was found

Not by the engine health page, which counted HTTP errors and showed Gemini as "degraded" at 29%. Truncation is stored as a successful call with a flag, not an error, so it was not on the health page at all. It was found by reading the data: the share of Gemini rows with the truncated flag set and no mention, month by month. That column now feeds the health page, and an engine whose unusable share crosses a threshold is written to the same alert surface as a failed job.

What changed

• Gemini and Claude have their own output budgets (8,192 and 4,096 tokens), sized for a reasoning pass plus a full answer. Verified live: both now finish.
• Engine health counts an answer cut off before naming the brand as unusable, alongside errors, and a daily check files any engine below threshold where it will be seen.
• Historical scores are not rewritten. They stay as they were computed, marked by scoring version, and the trend chart will show the step when the engines come back.

What to check in any tracker

Ask the vendor three questions. What output budget do you give reasoning models, and is it per engine? What do you do with a truncated answer — count it as a miss, or exclude it? And can I see, per engine, how many of my checks were usable last month? A tool that cannot answer the third question cannot tell you whether the score it shows is built from the engines it lists.