Skip to content
Bernhard GötzendorferAI automation that runs on its own · Vienna
MeasuredAI Deep Dives

Gemini 4 Neon, Measured: My Test Broke Before Any Model Did

Gemini 4 Neon against Opus 5.5 and GPT-6.2 Astra on my own code: 155 runs, five of eight tasks dropped out. What holds: cost and speed, not a ranking.

Painted illustration: a measuring rule snapped in the middle on a shelf before an orange disc, with five weights beside it.

On the morning of 20 November the first results come back, and for most tasks they are red. Every model fails, run after run, on the same hidden tests. Two days earlier Gemini 4 Neon had come out. The word was: quality like Opus or Fable, even better than Astra. I wanted to measure that on my own work. And now the first thing to break was not a model. It was my test.

By the end of the morning there were 155 runs. Five of eight tasks could not be solved by any model, because the hidden tests asked for something the task description did not contain. What was left does not answer the hype question. It does show four things that are worth more for daily work with agents: what a task costs, that the higher thinking level bought nothing, that the cheapest model is enough once the tests exist, and that a review verdict is not a measure.

What Was Said About Gemini 4 Neon and What Google Itself Writes

That line is hearsay. It is only what got me started. What Google writes can be looked up.

The announcement from Google gives the price: 3 USD per million input tokens and 15 USD per million output tokens. It also says Neon is third in the Artificial Analysis index at its highest thinking level, "just behind larger, slower models", and first in that provider's index for coding agents. It says nothing about "better than Astra or Fable".

One statement from it matters later. Google itself names a large gap depending on the environment: on one coding benchmark 12.8% in a third-party environment and 39.0% in its own.

For orientation, the list prices of the others, per million input and output tokens: Claude Opus 5.5 is at 4 and 20 USD, GPT-6.2 Astra at 8 and 40, GPT-6.2 Sol at 2 and 10, Muse Wonder at 4 and 20. I read those, I did not measure them.

The Setup: Eight Real Changes, Seven Arms, Three Environments

The material is my own repository, lead-session-orchestrator. It is the tool I use to lead several agent sessions, written in TypeScript, around a thousand commits without merges in the last five weeks. The source code is private, the built package has been public since October, with the latest release on 9 November 2026. Five of the eight changes are older than that release and are part of the package in built form, among them two of the three tasks that counted in the end. The tasks are changes from the last four weeks, the newest from 16 November, two days before Neon was released. That a model saw them in training is unlikely, but I cannot rule it out. There is one sign against it: on the five tasks that dropped out, no model hit the wording of the real commit.

I call the task "fix to plan". The model gets the state before a real commit, and the commit message as its task description. The tests of the real commit stay hidden and decide. The folder has no Git history, so the solution cannot be found. I know the principle from my earlier model comparison on real receipts: my own material, and an answer that is fixed beforehand.

There were eight tasks, all real changes across 2 to 5 files with 74 to 464 changed lines. Each model got two passes. Seven arms ran, in three different agent environments. I call the environments A, B and C here.

Environment A ran Gemini 4 Neon high, Gemini 4 Neon medium, Claude Opus 5.5 high and Muse Wonder high. Environment B ran GPT-6.2 Sol medium and GPT-6.2 Astra high. Environment C ran Claude Opus 5.5 high once more, so the same model as in A in a different environment.

That sets the limit for everything below. Only arms in the same environment are strictly comparable. Neon against Opus in environment A is a clean comparison. Neon against Astra is one across two different environments, and I say so every time.

One more limit. Seven of the eight real changes were made with Claude Opus 5.5, one with Claude Fable. So the hidden tests also carry the handwriting of Opus. Opus in environment C still solved only 2 of 6. A large home advantage does not show in that, but I cannot rule one out.

The rules for what counts as better were fixed before the first scored run and committed beforehand. Around the run there were twelve documented deviations. I name two of them. I had to tighten the checking rules for tool calls five times. And my first cost table was wrong. It compared unequal tasks and left out one cost item, so I calculated it again.

I checked every tool call of every run afterwards. No model read files that were not its own, searched for Git history or used the network. Outside its folder a model only read the places where its environment stores long output of the running task. The checking rules themselves raised several false alarms, on search patterns and on exactly those places. Real violations: zero.

Of the 155 runs, 86 were fix runs with hidden tests, 21 ran with visible tests and 48 were review runs. Seven arms times eight tasks times two passes would be more. Once it was clear that five tasks measure nothing, I no longer ran them to the end for every arm. On the three usable tasks every arm has both passes.

The Test Broke First, Not the Model

On five of eight tasks the same hidden tests failed in every run of every model. When four to six arms from two to four model families get stuck in the same places, it is rarely the arms.

The reason: the tests check the exact wording of messages, field names, function signatures or a file format. All of that is only fixed by the real commit. The commit message does not contain it. A model would have had to guess what I named a field.

An example: in one task all models delivered the right behaviour, with the right exit code. All six remaining failures were comparisons of a message text. The work was done, only the sentence read differently from mine.

I ran the cross-check the same morning. With the same tests lying visible in the folder, GPT-6.2 Sol medium solves all five tasks in 88 to 207 seconds. So the tasks were not too hard. They were underspecified.

What dropped out: a guard against rolling back an authority state (the largest task, at 464 lines), English output texts without German abbreviations, a report on gaps in continuous operation, counting aborted handovers, and accepting a handover only through a secured path.

That is my mistake, not a model's. My pre-filter had not caught it. It only checked whether the tests were red before and green after, and whether new names get imported. Two days earlier, on smaller tasks, 2 of 10 had the same problem. On the larger ones it is 5 of 8.

Gemini 4 Neon High and Medium Against Opus and Astra

Three tasks work as a measurement. In the first, the conditions of an agent session should be matched through a unique identifier and not through a name that can occur twice. The task description says a hint names the key "when it is unambiguous". What "unambiguous" means exactly, it leaves open. The second is about a log that is only ever appended to. In three places it counted as valid although its end had been cut off. All checkers should compare it against a stored anchor point. In the third, a question to me that would delete something should be checked when it is created and not only after my click.

That gives six attempts per arm: three tasks, two passes. Cost is reported tokens times list price. It is not an invoice. With six runs I take the third value as the median.

ArmEnvironmentSolvedSeconds (median)List price per run
GPT-6.2 Astra highB5 of 61671.09 USD
Claude Opus 5.5 highA4 of 61980.97 USD
Gemini 4 Neon mediumA3 of 63501.01 USD
Gemini 4 Neon highA2 of 67402.39 USD
Muse Wonder highA2 of 62591.36 USD
GPT-6.2 Sol mediumB2 of 61310.30 USD
Claude Opus 5.5 highC2 of 62380.96 USD

For Opus in environment C the environment itself reports 1.15 USD per run on these three tasks. The table shows my figure by list price.

And the same runs per task, as hits out of 2:

TaskNeon mediumNeon highMuse WonderOpus (A)SolAstraOpus (C)
1 Matching conditions0000020
2 Log protection2222122
3 Deletion questions1002110

This is not a ranking. Six attempts are few, and a gap of less than three attempts is noise.

The second table shows what the first one depends on. Almost every arm solves task 2, so it does not tell them apart. Only Astra solves task 1, in environment B, and that task depends on the ambiguous word in the task description. Astra covered both readings, the others one. The reading Opus and Neon chose is defensible.

If I leave task 1 out, four attempts remain and the order changes: Opus in environment A 4 of 4, Astra in environment B 3 of 4, Neon medium 3 of 4, Neon high 2 of 4, Muse Wonder 2 of 4, Sol 2 of 4, Opus in environment C 2 of 4. So the order at the top depends on a single, ambiguous task.

Now to the two claims from the start. "Better than Astra": my data gives no indication of that. Neon high is three attempts behind Astra, and the two ran in different environments. "Like Opus": not decidable. In the same environment Neon is one to two attempts behind Opus, and that is noise. Fable was not in the run.

One more limit belongs here. Neon did not run in Google's own environment, and Google itself names clearly higher values for its own environment. My run says nothing about what Neon does there.

Muse Wonder high came to 2 of 6, at a median of 259 seconds and 1.36 USD list price per run. That is no lead over Opus in the same environment, at a higher price per run.

Same Model, Two Environments

The finding I keep coming back to is about Opus. Claude Opus 5.5 solves 4 of 6 in environment A and 2 of 6 in environment C. Same model, same tasks, same thinking level.

The gap comes from task 3 alone, 2 of 2 against 0 of 2, and it is two attempts. By my own threshold that is noise, just like Neon against Opus in environment A. It does not prove that the environment matters as much as the model. It is a reason to compare models only in the same environment.

What Holds: Cost and Speed

On hits, a lot stays open. On time and cost it does not.

Neon high was the slowest and most expensive arm: a median of 740 seconds and 2.39 USD per run, counted as reported tokens times list price. Opus in the same environment took 198 seconds and 0.97 USD. That is around 3.7 times as long and 2.5 times as expensive.

Neon medium solved no less than Neon high, 3 against 2 of 6. At 350 seconds it took half the time and came to 1.01 USD per run. That is the price of Opus, not below it.

The output shows where that comes from. Neon writes the most output tokens: Neon high a median of 62,095 per run, Opus in the same environment 14,630. Around three quarters of what Neon outputs are thinking tokens according to the environment, 74 to 78% of the output tokens. The lower token price of Neon is eaten up by more tokens.

At the other end is GPT-6.2 Sol medium, in environment B. It is by far the cheapest and fastest, 0.30 USD and 131 seconds, but solves only 2 of 6 on these tasks. Astra has the most hits in the same environment, for 1.09 USD and 167 seconds.

With Finished Tests, the Cheapest Model Is Enough

I ran the five dropped tasks once more, this time with the tests visible in the folder and the instruction: make these tests green, do not change any test file.

Arm, tests visibleEnvironmentSolvedSeconds (median)List price per run
GPT-6.2 Sol mediumB5 of 51480.29 USD
GPT-6.2 Astra highB5 of 51311.04 USD
Claude Opus 5.5 highC5 of 51931.31 USD
Gemini 4 Neon highA2 of the 2 that ran341 and 7122.04 USD

Here too the arms ran in different environments. None of the three complete models touched a test file. The 1.31 USD for Opus in environment C applies to these five tasks with visible tests. The 0.96 USD further up applies to the three usable tasks.

For Neon three of the five runs are missing. In the middle of this section the quota of one of the environments was used up, and the run ended there. Of the 21 runs I started, 17 are in the table: three broke off at the quota, and a single run of Opus in environment A is not worth a row on its own. That was my mistake. I had not read the quota beforehand and had set no cost cap.

A tie at the ceiling does not mean "equally good". The finding is a different one: for work against finished tests, the cheapest model is enough. Astra and Opus cost three and a half to four and a half times as much for the same result. On the three usable tasks, without visible tests, the same model solved 2 of 6.

A Review Verdict Is Not a Measure

The third part of the morning was reviews. Per task, two submissions went to a reviewer. Once a diff that takes a real fix back out. The right answer would be "reject". And once the real fix that I had merged. The right answer would be "let through".

GPT-6.2 Sol and GPT-6.2 Astra catch 8 of 8 reverts. But they also reject 8 of 8 real fixes. In 16 of 16 cases the verdict is the same. Claude Opus 5.5 catches 6 of 8 reverts and lets 1 of 8 real fixes through.

Two limits on that. The reasons given on the real fixes name edge cases in the substance. Whether those are right, I did not check. It may be that the reviewers are right and my merged fixes have gaps. And on three of the eight counter-samples "reject" is defensible, because the real fix contradicts an old test assumption in the folder. Without those three, Sol let 0 of 5 through, Astra 0 of 5 and Opus 1 of 5.

I did not measure Neon as a reviewer, the quota was gone.

Why this stings: two days earlier a review test without counter-samples looked perfect. All models caught all errors. Only the clean counter-samples show that the verdict tells nothing apart. A reviewer that always says "reject" finds every error.

Part Two: New Decision Models Against Jev

On the same day a second measurement ran. For small typed decisions (pick a category, yes or no, book automatically or send to review) I have been measuring a decision model called Jev against language models since September. The write-up of that comparison exists in German only. Here Jev is the reference point against which I measure new decision models. By now there is a whole class of such models from several providers.

I measured six new ones, from Microsoft, Liquid AI in two sizes, HeyGen, Google and Perplexity, against two Jev versions. The material was six decisions from my products: 14,592 calls, no technical error, 0.65 USD for the scored calls. The rules for when a model counts as a replacement come from my runbook from September.

None of them replaced Jev under those rules. The reason is the booking decision with 200 cases. There the hit rate is not what counts. What counts is whether a model wrongly waves a booking through as "automatic".

On two smaller tasks, support routing with 67 cases and role mapping with 96, single candidates were level with Jev. With that few cases my rules only allow a separate check.

ModelHit rateWrong automatic bookings in the first pass
Jev, pinned84.0%1
Jev, newest version78.5%0
Microsoft84.0%10
Liquid AI small89.0%6
Liquid AI large98.0%2
HeyGen90.0%3
Google85.0%0
Perplexity98.0%1

The sentence "Jev is still the best" is not true as it stands. Two candidates get the booking decision right clearly more often, 98.0 against 84.0%. My pinned Jev itself made one wrong automatic booking in the same run, on a borderline case with 50 to 49% confidence. The model from Google came to zero wrong ones at 85.0% and counts as undecided against Jev. And the newest Jev version is right less often than the old one, 78.5 against 84.0%, but made no wrong automatic booking and sends more cases to review that did not need it. So two versions of the same model behave differently. That is why I pin the version.

Where Jev still stands: it is the fastest on every task, with a median of 187 to 208 milliseconds against 228 to 364 for the others. And on the booking decision none has pushed it out under my rule "no wrong automatic booking". That rule applied to the candidates only. My pinned Jev would have missed it itself in this run.

The hit rate alone would have made me switch. The rule that counts the expensive error stops me. Liquid AI large and Perplexity deserve a measurement of their own, with a confidence threshold on fresh cases. One observation on that, explicitly not evidence: none of the wrong automatic bookings came with 90% confidence or more. Price is no longer an argument in this class, it is 0.004 to 0.16 USD per thousand calls.

What This Means for My Setup

From here on this is my interpretation, not a measurement. My setup looks like this: a leading model plans and decides, cheaper models do the supporting work, and deterministic checks (tests, typecheck, lint) decide whether something is finished. The tool for that is my open-source plugin session-orchestrator. Where it sits is in the overview of how the foundation works together.

Seven things I take from the morning:

  1. The task description is the bottleneck, not the model. Five of eight tasks failed for every model on missing information. A task for a supporting model names the interface (field names, signatures, wordings) or brings the tests along.

  2. Tests first, then the cheapest model. With finished tests it was 5 of 5 for 0.29 USD list price per run, on five tasks with one pass each. That measures the second half of the division of work "expensive model writes plan and tests, cheap one implements". Whether the expensive model writes the tests well, I did not measure here.

  3. The expensive model goes where the task needs interpretation. The one task with an ambiguous word was solved only by Astra, in environment B. That is a hint from one task, not proof.

  4. No merge on a review verdict. The verdict "reject" comes almost always. What is usable are the single findings that somebody checks, and the deterministic checks.

  5. The higher thinking level is not a default. With Neon, high bought nothing but double the time and 2.4 times the cost of medium. Two days earlier the same picture with GPT-6.2 Sol: medium solved the same as high with half the output.

  6. Read the quota and set a cap before a run. That cost me part of the measurement that morning.

  7. Compare models only in the same environment. Opus came to 4 of 6 in A and 2 of 6 in C. With six attempts that proves nothing. But Google itself reports a large gap for Neon depending on the environment, 12.8% against 39.0%.

And Neon? I am not buying anything on this basis. Neon stays on the watch list. The next honest test is Neon in Google's own environment, with a repaired task set.

To Try It Yourself

This is how I would do it today:

  1. Take a private repository of your own. The model may already know public tasks.
  2. Pick real commits from the last few weeks that bring tests with them.
  3. Per task, export the state before the commit without Git history.
  4. Check that the tests of the commit are red on the old state and green on the new one.
  5. The step I was missing: do a reference run with the task description alone. If it fails on names or wordings, extend the task description or drop the task.
  6. Before the run, write down what counts as "better" and from which gap, and commit it.
  7. Give each model at least two passes, better three. Compare only models in the same environment directly.
  8. After the run, check every tool call: did the model read outside the folder?
  9. Compare cost only on tasks that every model had.
  10. For reviews, always include clean counter-samples.

If you try this on your own repository: write to me about what your run showed.

lead-session-orchestrator

Many sessions, one lead, and whatever only you can decide waits in one inbox.

The language model
runs the lead session: it reads the situation, orders the merges and bundles your decisions.
The code
sets the limits: only ever one lead, one rule file per repository, and whatever it cannot measure, it leaves alone.

session-orchestrator.com/lead (opens another website)Product page for lead-session-orchestratorWhat I build for you