Google's Gemini 4 Carbon leads refreshed AI hallucination benchmark

Three days after Google launched Gemini 4 Carbon, the model has a new independent result. Artificial Analysis has refreshed AA-Omniscience, its benchmark for knowledge reliability and hallucination, to version 1.1. Gemini 4 Carbon (high) leads with a Net Score of 54, six points ahead of Claude Opus 5.5. The refresh is part of the broader Artificial Analysis Intelligence Index v5 release.

The result matters because one of Carbon’s headline launch claims, first place on the Intelligence Index at 59, came from Google DeepMind’s own account. Artificial Analysis’s 27 October announcement had Opus 5.5 on top at 58 and did not list Carbon. The new AA-Omniscience table is a score from the benchmark maker itself, and it puts Carbon clearly ahead on this one test.

How AA-Omniscience scores a model

AA-Omniscience asks a model factual questions and checks three things. A model gets credit for correct answers. It is penalised for confidently wrong ones. It is not punished for simply admitting it does not know. That balance is captured in the Net Score, the benchmark’s primary metric, which ranges from -100 to 100. A score of zero means a model gets as many answers right as wrong. A positive score signals reliable knowledge rather than lucky guessing.

The benchmark also reports two supporting numbers. Accuracy is the raw percentage of questions answered correctly. Hallucination Rate measures how often a model guesses rather than admits uncertainty. A model can lead on accuracy and still trail on Net Score if it guesses wrong too often, and v1.1 shows exactly that pattern at the top.

Artificial Analysis uses the same test elsewhere in its work. Its Search Index, launched in August, includes AA-Omniscience as one of three component benchmarks, using a private subset of 600 factual questions balanced across six domains and scored on accuracy. In that setting the score shows how much a search tool adds to a model’s internal knowledge.

Where Gemini 4 Carbon lands against Claude, GPT-6 and open models

Carbon’s lead is wide by the standards of a frontier leaderboard. It sits six points clear of Claude Opus 5.5 and eight points clear of Claude Fable 5.1. OfficeChai notes that Opus 5.5 had itself opened one of the biggest leads in months when it first took the top spot on the Intelligence Index, and AA-Omniscience was among the evaluations it led at the time.

Rank	Model (setting)	AA-Omniscience v1.1 Net Score
1	Gemini 4 Carbon (high)	54
2	Claude Opus 5.5 (max with fallback)	48
3	Claude Fable 5.1 (max with fallback)	46
4 (tie)	GPT-6 Astra (max)	42
4 (tie)	GPT-6.1 Sol (max)	42
Next	Grok 4.7 (xhigh)	31
Next	Meta’s Muse Wonder Full (max)	31
Top open weights	Muse Spark Open (xhigh)	26
Lower	GLM-5.3 (max)	6
Lower	Mistral Large 4 (high)	5

Scores as reported by OfficeChai from Artificial Analysis’s v1.1 results.

Among open-weights models, Muse Spark Open (xhigh) leads at 26, 28 points behind Carbon. Further down, scores fall away sharply. GLM-5.3 (max) scores 6 and Mistral Large 4 (high) scores 5. A cluster of models posts negative Net Scores, meaning they get more answers wrong than right once hallucinations are counted. That cluster includes DeepSeek V4.1 Flash, Nemotron 3.5 Super VL, MiMo-V2.6-Pro, K2 Horizon 375B A23B, Qwen3.8 27B, MiniMax-M3.1-Flash and GPT-6 Luna (max).

Carbon leads Net Score, Claude leads raw accuracy

The top of the table splits by metric. Claude Fable 5.1 (max with fallback) posts the highest accuracy on the updated set at 68%, with Claude Opus 5.5 (max with fallback) at 66%. The lowest hallucination rate belongs to Gemini 4 Argon (high), the first Gemini 4 model, released ahead of Carbon, at 4%. The available coverage does not give Carbon’s own accuracy or hallucination rate. It is therefore not possible to say from these sources how much of Carbon’s lead comes from answering more questions correctly and how much from declining to guess.

What Artificial Analysis changed in v1.1

Version 1.1 is a rebuilt dataset, not a re-run of the old one. The main changes, as OfficeChai reports them:

• Half the questions. The set drops from 6,000 questions to 3,000.
• Contaminated and faulty items removed. Artificial Analysis stripped out questions that had become public and saturated, meaning models could have been trained on them, and questions with errors or ambiguities.
• New questions added. Fresh items backfill the set, which still spans the same six domains and 42 topics: Science, Engineering and Mathematics; Software Engineering; Finance; Health; Humanities and Social Sciences; and Law. The distribution across domains shifts slightly.
• A calibrated public set. A new 300-question public set on Hugging Face is designed to match the difficulty of the private set.

The public set is the most practical change for builders. At launch, public and private Net Scores correlate at 0.99 on both Pearson and Spearman measures, with a median difference of 0.0 points. In plain terms, running the 300 public questions should give a close read on how a model would score on the full private benchmark.

Scores fall across the board, but rankings mostly hold

Because v1.1 removes questions models may already have seen, most models score lower than on v1. The average drop is 1.9 points and the median drop is 0.6 points. Artificial Analysis says this reflects a harder, cleaner dataset rather than any regression in the models. The rank order between versions is largely preserved, with a 0.98 Spearman rank correlation, a measure of how closely two rankings agree.

A few models fall much further:

Model	v1 Net Score	v1.1 Net Score	Change
MiniMax M3.1 Flash	16	-6	-22
MiMo V2.6 Pro	8	-5	-13

Both swing from clearly positive to negative Net Scores.

Across models, accuracy slips by a median of 1.5 percentage points, while the median Hallucination Rate improves by 1.4 percentage points. The sources do not explain why those two models fell so steeply. One reading consistent with how v1.1 was built is that they had benefited more from the removed public questions, but neither OfficeChai nor Artificial Analysis says so in the material available.

What the lead means for developers building on Gemini

The key limit is access. At launch, Google DeepMind said Carbon goes first to a subset of cyber defenders and then to more trusted partners through a programme it calls Fairwind. It is not yet on the developer, business or consumer surfaces, and the launch post does not say when it will be. Gemini 4 Argon is the Gemini 4 model developers can call today. Minerka’s advice from launch week still applies: do not migrate a workflow to Carbon until it is on a surface you can call.

For teams choosing a model now, the v1.1 table gives some concrete signals:

• Abstention-heavy workloads. Retrieval-free question answering, support bots and internal knowledge tools often need a model to say “I don’t know” rather than guess. Argon’s 4% hallucination rate, the lowest on the table, is relevant here, and it is a model developers can already reach.
• Raw recall. If a downstream checker catches errors and coverage matters most, the Claude models’ 68% and 66% accuracy figures are the highest reported.
• Open-weights deployments. The best open model scores 26, and several popular open and small models score below zero. Teams self-hosting for knowledge-heavy tasks should plan for retrieval or verification rather than relying on the model’s internal knowledge.
• Your own evaluation. The 300-question public set lets teams test candidate models, including fine-tunes, against a set that tracks the private benchmark closely.

Open questions about the result

Several caveats limit how far the headline can be stretched.

Settings differ between entries. Carbon is listed at “high”, the Claude models at “max with fallback”, the GPT-6 models at “max” and Grok at “xhigh”. These are each lab’s reasoning-effort settings, so the table compares each model at the setting Artificial Analysis tested, not on an identical configuration.

Benchmark figures move with the source. Frontier Log found that Google’s Argon and Carbon launch tables print different Terminal-Bench 4.0 scores for the same rival models. Claude Opus 5.5 appears at 66.4% in the Argon table and 59.6% in the Carbon table, which uses Artificial Analysis’s runs. Carbon’s document gives no harness or effort setting for those Artificial Analysis figures. That analysis concerns a different benchmark, but it shows how much a reported number can depend on who ran it and how.

One benchmark, not the whole index. AA-Omniscience measures knowledge reliability specifically. OfficeChai notes that the gap between proprietary and open models remains wide here, even as it narrows on other benchmarks. The available sources do not confirm Carbon’s overall Intelligence Index v5 position from Artificial Analysis directly. The 59 figure from launch is Google DeepMind’s report.

Google’s other claims are still internal. Carbon’s cybersecurity scores, its WebDev Arena gain and its reported drop in unsupported claims versus Argon all come from DeepMind’s own account, and some were run without guardrails.

What to watch next

• A developer release date. Google has not said when Carbon reaches the public API or the developer, business and consumer surfaces.
• Carbon’s full AA-Omniscience profile. Its accuracy and hallucination rate on v1.1 would show what drives the 54.
• Responses from Anthropic and OpenAI. OfficeChai describes a leaderboard where a new model takes the lead roughly every few weeks. A new Claude or GPT-6 release would be tested on the harder set.
• Community runs on the public set. If outside results on the 300 public questions keep tracking the private scores, the benchmark becomes a practical pre-deployment check for builders.