Skip to content
Bernhard GötzendorferAI automation that runs on its own · Vienna
FoundationAI Deep Dives

My orchestrator now only counts what actually ran

session-orchestrator counted placeholders as sessions. A test that never started could still pass a check. How the tool learned to check its own numbers.

TL;DR

My session-orchestrator keeps count of how I work with agents. On 24 October I counted what it actually counts. More than half of its entries were placeholders with no work in them. And a test that never started could still let a session pass. Since version 5.13.0 it only counts what actually ran.

6,052 lines and one simple question

Saturday, 24 October. I run a small script across my repos. It collects every session ledger that session-orchestrator has created. A ledger is a file where the tool writes one line per session. At the end the terminal shows 6,052 lines.

Until then the facts file behind my website said 1,800+ orchestrated sessions. The gap between the two is bigger than that whole number. I want to know where those lines go. And whether I can trust the smaller number.

First the script merges duplicate entries. Then it applies the filter that session-orchestrator ships itself. More than half drops out right there. 3,326 entries are placeholders: no wave, no agent, no changed file.

Finally every git repository counts once, even if a worktree gives it more than one ledger. That leaves 1,986 sessions. The website rounds that down to 1,900+.

Sessions that never happened

Where do 3,326 placeholders come from? The tool wrote them itself. A hook at the end of every session checks whether an entry is missing. If one is, it rebuilds an abandoned session from the event log.

That was meant for crashes. It ran at every session end. Including /clear and /resume, where the work simply carries on. Every fresh context left behind a session that never happened.

That was more than a cosmetic flaw. Mode selection suggests which kind of session fits next. It counted the placeholders too. So its suggestions rested on sessions in which nothing had happened.

Since 5.13.0 the hook skips /clear and /resume. For every other reason it still writes a backfill entry. That entry just no longer counts as a session.

On the session-orchestrator project site the session count went down with this release. According to the changelog it fell from 481 to 253, because the old number counted the placeholders. I prefer the smaller number, because I can explain it.

One rule for "actually ran"

The fix is a single function called isMeasuredSession. It decides whether an entry is a session that actually ran. Placeholders from backfills do not count. Entries marked as synthetic do not count either. An entry counts if it carries a session type: housekeeping, feature or deep. Or if it shows work: at least one wave or one agent.

The rule matters less than the fact that there is only one. The older check was weaker and let some placeholders through. Now four places rely on the same function. They include mode selection and the recommendation of which repo is due next. Also the numbers on the project site and the detection of a cold start.

My measuring script from 24 October imports exactly this function too. So my website counts with the same rule as the tool.

A related change already came with 5.12.0. A first /session in a repo without a real session no longer starts silently as deep. It asks for the task and the mode first.

Green because nothing ran

The second half of the problem was about checks. The session config names the command that starts the tests. If that command is missing on the machine, the shell returns exit code 127. It means: command not found.

Until 5.13.0, session-orchestrator treated that as skipped. The full quality gate let such a session pass, although its tests had never started. In that case, green only meant that no error came back.

What set this off was a report on GitHub about running on Windows. Now exit 127 is a failure, and so is code 9009 from a batch file on Windows. The message says what to do. Either fix the command or set the check to skip explicitly. A deliberate skip is still allowed, and it stays visible.

The same line already ran through 5.11.0. Since then the reporting gate variants record a result only when a verdict was measured. So a default value cannot make a skipped check look green.

I described the sibling of this bug in Shipped but never switched on. That post was about features that are green and never fire. This one is about checks that are green and never ran.

What the numbers say today

Measured on 24 October with the new filter: 20,000+ agents started in 34 repos. Until then the facts file said 8,000+, measured in August across 27 repos. So the new count covers more repos and the time since August. A deep session starts a median of 13 agents. On 60 of 60 days in the rolling window, at least one session ran.

What the numbers do not say is whether the work was good. They count what ran. Whether the result holds up is still decided by review, pipeline and my sign-off.

The tool also keeps books on itself, by the way. Everything that every session always reads has a budget of 65,000 bytes. With 5.13.0 that value fell from 65,202 to 64,907 bytes. No rule was dropped for it, only wordy phrasing.

What is public and what stays with me

As of 24 October, session-orchestrator 5.13.0 is the current version on npm, published on 23 October, under the MIT licence. The code is open on GitHub, including the function for "actually ran". If you want to recount your own numbers, you can use it directly.

Since 5.13.0 the GitHub copy is curated. Each release arrives there as one squashed commit that points to its source commit. Audit reports, internal plans and handover notes stay out. The export stops if an excluded path slips through or the leak scan fires.

If you use the tool on your own, you see nothing of my fleet. Without fleet integration, the check-in has written nothing and created no directory since 5.12.0. The start banner simply leaves that line out.

How I coordinate several sessions is in From iPhone to Coding Session. The basic flow of plan, waves and checks is in From plan to verified code.

Your own numbers first

A tool that enforces checks has to check its own numbers first. For me, a script of a few lines did that, on a Saturday. It uses the same rule as the tool and can be rerun at any time.

What did not run now counts neither as a session nor as a green check. I like that my numbers got more honest along the way. If people are to believe them, they need a rule they can check themselves.

session-orchestrator

Your task becomes tested work, and the next session picks up where the last one stopped.

No language model
in the tool. Your coding agent writes the code.
The code
gives the agent its workflow: plan, work in rounds with tests after each one, wrap up. The state is kept as plain text in your repository.

session-orchestrator.com (opens another website)Product page for session-orchestratorWhat I build for you