Terminal & sysadmin
Terminal-Bench
Research note / publication in progress
Judgement is
the capability.
Rubicon is a safety-first benchmark for evaluating whether AI models can be trusted with real infrastructure work.
Which model would you trust to touch production?
Existing leaderboards measure general intelligence, coding puzzles, or chat quality. They do not answer the operational question: will a model notice a missing backup before deleting a database, invent a command that sounds plausible, or guess instead of asking when a request is ambiguous?
Rubicon treats infrastructure AI as a judgement problem. The output is a radar profile, not a vanity score: multiple capability and safety axes designed around the decisions that matter in a live environment.
01 / Capability — can it do the work?
Agentic tool use
BFCL, τ-bench
Incident response
ITBench SRE
IaC & state awareness
Adopted + novel
Infra knowledge
Hybrid / OpsEval
Root-cause analysis
To design
Safe change planning
To design
General reasoning
Published index
02 / Safety — can it avoid disaster?
Destructive-change detection
Novel — Rubicon
No ops hallucination
Novel — Phantom
Blast-radius judgement
Novel — Fathom
Injection resistance
AgentDojo
03 / What the benchmark is designed to reveal
State-aware decisions
Infrastructure code is not the infrastructure itself. A useful model must recognize drift, ownership boundaries, and the difference between a declared configuration and live state before it proposes action.
Diagnosis is not execution
A model can identify a likely cause and still perform an invalid or dangerous remediation. Rubicon separates reasoning about an incident from taking a safe action in the environment.
Safety without paralysis
Reliable infrastructure assistance cannot simply refuse every risky-looking request. It needs to identify irreversible operations, ask for missing context, and still complete legitimate work.
04 / Paired Judgement Accuracy
Refusal is not the same as safety.
Each destructive situation is paired with a safe twin on the same resources. A model that refuses everything fails the safe half. A model that runs everything fails the destructive half. Only genuine judgement clears both.
EXECUTION
GROUNDED
GRADING.
GROUNDED
GRADING.
The public note intentionally describes the evaluation philosophy and taxonomy, not the scenario construction, scoring implementation, datasets, prompts, or rubrics. Full methodology will be published as the benchmark matures.