AutoR: A Recursive Research System

It proposes, tests, and tries to refute itself. The approval gate is the one thing it does not own — by default, that is you.

Start here: The Framework · English Guide · 中文教程 · Full Documentation

AutoR is not a chat demo, not a generic agent framework, and not a markdown-only research toy.

It is a structured research harness over a coding-agent execution layer: the agent handles execution, the human owns the direction, and every run becomes an inspectable research artifact on disk.

docs/framework.md is the single document that describes what this system is: its implementation, its modules, what is new in it, and what it contributes. This README is the overview and the operating manual.

Contents

What AutoR is · Quick start · The stage graph · The rigor dial · Self-improvement rounds · Review · The stage contract · Execution model · Run layout · Architecture · Benchmarks (ResearchClawBench · FIRE-Bench · AIRS-Bench) · Documentation · Limits · License

What AutoR is

Most autoresearch systems optimize for autonomy. AutoR takes a different position: research is too important to hand over as a blind end-to-end loop. The goal is not to remove humans from research. The goal is to give them a stronger execution system.

AutoR runs a research project as eight stages wired into a directed graph. Six of the forward edges are guarded by artifacts on disk; thirteen backward edges let a late finding send the run back — Stage 07 can reopen the literature survey. Hypotheses are frozen and hashed when Stage 04 is approved, every one must be adjudicated at Stage 06 against a named result file that exists, and every paper claim traced at Stage 07; a supported or refuted verdict resting on a single seed is refused unless the run records why one run settles it. An adversarial reviewer attacks Stage 05's results and Stage 06's analysis, and the stage after each must answer every finding in writing or the gate refuses it. Drafts are scored and a refinement that does not improve is reverted. Every stage still stops at an approval gate, and by default that gate is you.

"Recursive" is eight mechanisms, each of them a file you can open

What a default run (--rigor standard) actually uses. Test, Refute, Iterate and Learn are on: the validity chain is unconditional at every rigor level including fast, --evolve defaults on, and the archive records every run — though it only steers under --archive-steer. Localise runs whenever a reviewer quotes a passage, which requires an agent reviewer. Propose and Deliberate need --rigor thorough; Critique's panel needs --rigor max and its cross-model veto is live on the rcb_agent.py path only. See the rigor dial for the exact mapping.

AutoR does not run itself. Manual approval is the default: approval_mode is manual unless a flag opts out. Seven of the eight moves above can only score, refuse, revert or re-order; none of them can approve a stage. The eighth, the review panel, is an approval gate, and it exists only on the runs where you hand it the gate. Recursion did not change who decides; it changed what reaches the desk. The research unit is unchanged: one reproducible run under runs/<run_id>/, isolated, resumable, with redo and rollback.

The one thing the docs will not claim

Approved stage summaries are the only free-text cross-stage memory. Every other cross-stage edge is a typed artifact with a declared reader: twenty typed channels in information_flow.py each name the exact stage slugs that consume them, and the nine channels produced inside the walk name their producing stage as well. obligations.json and review_policy.json cross stages without touching a summary at all — both only behind an agent approval gate.

Many systems aim to generate research outputs that look ready. So the question is not

Does it look ready?

It is

Can you verify every part of it?

The answer is the validity chain — freeze at Stage 04, adjudicate at Stage 06, trace at Stage 07 (preregistration.py) — and the edge into writing stays shut until every frozen hypothesis carries a verdict (_guard_validity_chain, stage_graph.py).

The shape of the system, in counts you can re-derive

Every number below comes from a named symbol in the source. Re-derive them; that is the point of naming them.

python -m unittest discover -s tests -p "test_*.py" runs 4053 tests in ~440 s across 143 test modules, with no third-party dependency.

Quick start

Prerequisites

• Python 3.10+
• Claude CLI or Codex CLI on PATH for real runs
• Local TeX tools only for --output-format latex; the default markdown output needs no TeX
• pip install google-genai plus a key in GOOGLE_API_KEY or GEMINI_API_KEY — needed by three paths, not only the diagram one: --web-search gemini, required where the backend's own WebSearch tool is disabled (build_genai_client, src/web_search.py); the cross-model veto --cross-review auto|gemini (src/cross_reviewer.py); and --research-diagram, which also reads configs/diagram_config.yaml
• The SDK is not a default dependency. Without it the diagram step prints Diagram generation failed: No module named 'google' and the run continues; cross-review records itself unavailable rather than agreeing

Common commands

Every flag, its default, and what is preserved on resume: docs/cli-reference.md. Stage identifiers accept 03, 3 or 03_study_design; --venue defaults to neurips_2025.

Three flags put an agent in the approval seat, not two — and a fourth removes the human without replacing them. approval_mode becomes agent for --approval-mode agent, --full-auto and --review-panel, and create_reviewer is called only when it does. --unattended on its own is the odd one: resolve_unattended returns True for all four, but with approval_mode still manual there is no reviewer to install, so the first approval menu raises UnattendedInputError rather than being decided. For a run with nobody at the terminal, pass --full-auto.

Because --rigor is resolved before resolve_unattended runs, a plain --rigor max sets review_panel = True and silently converts an interactive run into an unattended agent-gated one. Under a badge reading Human approval required, the flag that looks like more review is the flag that removes the reviewer. Three headline mechanisms — obligations, the standing review policy, the cross-model veto — also run only behind that agent gate, as do anchored comments. Manual approval is the default and remains the path for work you intend to publish.

For Codex-backed runs AutoR defaults to --codex-sandbox workspace-write. If a verified remote experiment needs SSH or external GPU access, use --codex-sandbox danger-full-access intentionally: it grants the Codex backend unrestricted local and remote execution, so it should not be the default for untrusted tasks.

# Self-improvement is on by default: navigate the graph, score every draft, keep the
# best, and record the route in ~/.autor/archive.
python main.py --goal "..."
python main.py --archive-report                   # what the archive has learned so far
python main.py --goal "..." --evolve-rounds 4     # spend more on improvement
python main.py --goal "..." --evolve-rounds 0     # measure and ratchet, no extra passes
python main.py --goal "..." --archive-steer       # let the archive pick the topology

# Opt out entirely: the strict 01-through-08 sequence, last draft wins.
python main.py --goal "..." --stage-graph linear --routing off --no-evolve --no-archive

Studio (browser UI)

A local web UI over the same Claude-backed runs: create a project, watch stages execute, approve or send feedback, read the compiled paper. It needs the Claude CLI on PATH to start a run.

python studio.py                                # http://127.0.0.1:8000/studio/
python studio.py --host 0.0.0.0 --port 8765     # bind externally, see the warning below
python studio.py --runs-dir /path/to/runs       # override runs directory

The Studio API has no authentication. It binds to 127.0.0.1 by default; anything that can reach it can start runs, approve stages, and read every file under the runs directory. For remote access prefer an SSH tunnel over --host 0.0.0.0. See SECURITY.md.

One honest limit, then the walkthrough: the Studio's lazy-resume approve path picks the next stage arithmetically — the first stage with a higher number (src/backend/studio_runner.py) — and never consults the router, so graph routing and backward moves are a CLI capability today. Page-by-page walkthrough and the full HTTP API: docs/studio.md.

How it works: the stage graph

Eight stages are the nodes; a finish node closes the walk. Stage 00 intake is not one of them — it runs before the walk starts, and _graph_entry_stage → _select_stages_for_run (src/manager.py) only ever yield the eight. Solid edges advance, dotted edges go back. --stage-graph linear is the eight advance edges plus the conditional terminal — nine in all — and the guards come off with the backward ones (_advance_edges(guarded=False)): one edge out of each node leaves nothing to choose, so a guard there could only halt a run that the stage's own validation is about to fail anyway.

flowchart LR
    S1[01 Literature] --> S2[02 Hypotheses]
    S2 -->|has_hypotheses| S3[03 Design]
    S3 -->|design_artifacts| S4[04 Implementation]
    S4 -->|runnable_code| S5[05 Experiments]
    S5 -->|results_exist| S6[06 Analysis]
    S6 -->|validity_chain| S7[07 Writing]
    S7 -->|report_exists| S8[08 Dissemination]
    S8 --> Z([finish])
    S6 ==>|round abandoned| Z

    S2 -.->|the gap it rests on is not a gap| S1
    S3 -.->|a hypothesis cannot be brought to a decision| S2
    S4 -.->|not executable as specified| S3
    S5 -.->|implementation is at fault| S4
    S5 -.->|comparison cannot distinguish| S3
    S6 -.->|results insufficient to decide| S5
    S6 -.->|confound the results cannot repair| S3
    S6 -.->|evidence refutes, and points somewhere| S2
    S6 -.->|the numbers are wrong, not disappointing| S4
    S7 -.->|claim has no analysis behind it| S6
    S7 -.->|needs a result never produced| S5
    S7 -.->|the survey missed related work| S1
    S8 -.->|deliverable is not what a reader needs| S7

Six of the eight forward edges carry a guard, one per target stage (_ADVANCE_GUARDS); 01→02 and 08→finish are unguarded. Thirteen dotted edges go back (REVISIT_EDGES) — the longest is 07→01: writing it up showed the finding relates to work the survey missed. One conditional terminal (TERMINAL_EDGES, carried by both topologies) lets an abandoned round finish from Stage 06. The Stage 07 guard is the strictest: every preregistered empirical hypothesis needs a verdict and at least one figure under workspace/figures (_guard_validity_chain).

Who decides the move. AutoR decides which moves are admissible, by evaluating each edge's guard against the artifacts on disk. With --routing auto (the default, DEFAULT_ROUTING_MODE) the agent chooses among them and states a reason; --routing off always takes the graph's default. An off-menu choice — an unlisted target, or one with no stated reason — is refused, written to evolution/routing_refusals.jsonl (src/router.py), and replaced by the forward edge.

Two design calls worth naming. Blocked moves are handed to the agent with the reason they are blocked (StageGraph.moves) — the useful thing to say is not "you may go to 06" but "07 is closed because H2 has no verdict", and an agent that sees why writing is closed routes to the analysis that opens it. And a revisit whose justification repeats one already on the path is refused (repeats_a_previous_reason): going again on the same grounds is a loop, not an iteration.

A backward move is only ever a deliberate choice. The default is always the forward edge, and when a guard has closed it the default advances anyway and lets the stage's own validation — still refusing a Stage 07 that writes up unadjudicated hypotheses — be the gate it always was. A guard is a routing preference; the gate is the gate. So a refusal, a routing failure, or a run nobody is steering all come out as the linear pipeline rather than as a stall.

A stage is a node with a visit budget, not a position in a sequence: DEFAULT_MAX_VISITS = 3 (--graph-max-visits); DEFAULT_MAX_STEPS = 20 bounds the whole walk (--graph-max-steps).

The eight stages, and what you check at each

The rigor dial

--rigor is the single source of truth for which optional machinery a run uses. The table is generated from _LEVEL_FEATURES in src/rigor.py; an explicit --flag / --no-flag always beats the level, which is why those switches use BooleanOptionalAction with default=None.

Two consequences worth stating out loud:

• Effort tiers are on by default. A default run therefore routes 04_implementation, 05_experimentation and 08_dissemination to a lean prompt and a single reviewer (DEFAULT_TIERS, src/effort.py). Under --rigor max the seated panel does not sit at those three gates unless you also pass --no-effort-tiers.
• --rigor max makes the run unattended, because it implies --review-panel. See the note under Common commands.

The scientific-validity chain is not on this dial. It is unconditional at every level, fast included.

Self-improvement rounds

Every valid stage draft is measured against a rigour rubric read off disk — do the paths it names resolve, do the numbers it reports appear in a results file, did it produce artifacts during this execution, is the decision ledger four different things rather than one sentence four times. Nine weighted criteria, RUBRIC_VERSION = "8":

min_stage exists so a criterion that cannot apply is not scored zero: Stage 01 has no experiment manifest to produce, and grading it as if it failed to produce one would make every early stage look worse than every late one — which would make the ratchet prefer late drafts for a reason unconnected to quality.

artifact_breadth is scored against STAGE_ARTIFACT_KINDS, the set of workspace directories this stage's prompt tells the agent to write — literature/ at Stage 01, artifacts/ + reviews/ + writing/ at Stage 08, and so on. A test refuses any expectation the stage's own prompt never asked for, so the criterion cannot drift into demanding work the run was never told to do. AutoR's own bookkeeping does not count towards it: neither the RECORD_ARTIFACTS the experiment manifest already excludes, nor the ideation pool, the writing and layout triage, the comment, crux and effort ledgers, the panel's transcripts, the run scorecard or the adversarial reviewer's findings, all of which the workflow manager writes into those same directories — six of them inside a stage's own window, before its draft is scored. _harness_written_records names them, each imported from the module that writes it, and a census over src/ fails the suite on a path under one of those directories that nobody has said who owns.

Measuring is free and always on. The rubric reads the run off disk and never calls a backend, so the property it buys costs nothing: the draft that gets promoted is the best one the run produced, not the last one. That is the half that was missing before — AutoR could iterate, but "later" was the only ordering it had, so a refinement that dropped a resolving reference was promoted on exactly the same terms as one that fixed something.

Improvement rounds are the half that costs, and they are budgeted separately from --max-attempts, which bounds a stage that is failing rather than one being improved. Two per stage by default, and a stage whose rubric has no shortfall worth acting on spends none of them — a round aimed at a criterion already at full marks produces churn, so AutoR does not buy one. --evolve-rounds 0 measures without polishing; --no-evolve restores the old behaviour entirely.

One edge of that budget is worth knowing before you resume a run. state() rehydrates the champion and the Pareto frontier from disk and nothing else (src/evolution.py), so --resume-run restarts rounds_spent and the patience counter at zero: the best draft survives the resume, the spend cap does not, and a stage resumed twice can buy the two rounds twice.

A round that scores worse is reverted, so a stage can only improve. A round that changes a hypothesis verdict is rejected outright, whatever it scored — the rubric is blind to what the run concluded, which removes the incentive, and the verdict_drift check removes the possibility. A revision a human asked for always stands, whatever it measures. The ratchet governs AutoR's own rounds, not the direction it is given.

Full mechanism, and the reasoning behind each refusal, in docs/self-improvement.md.

The archive: which moves paid, across runs

Every finished run is recorded into ~/.autor/archive — the route it took, the rubric fitness it reached, and the set of stages it actually measured (Archive.record_run, from record_into_archive in main.py). edge_payoffs compares runs that took an edge against runs that were offered it and declined (src/decisions.py), and propose_variant turns a payoff that is believable — enough observations, and a delta above min_gain — into a child variant that moves that one edge one step up or down the preference order. When no payoff is believable, propose_exploration proposes an unexplored edge instead, so an edge nobody has taken is not stranded forever.

A variant is only a reordering. It never opens a guarded edge, never adds one that was not declared, and never removes one: the guards are the correctness argument for letting an agent route at all, and the component that learns from outcomes is precisely the one that must not be able to weaken them. Promotion is as conservative — a challenger has to beat the incumbent within every comparability basis rather than on a pooled mean, because "runs that stopped early" is the cheapest composition for a topology to win on.

The archive records and proposes on every run; it steers only when you ask. The proposed variant is written down and reported, but the topology a run walks comes from the archive only under --archive-steer. Without it, resolve_graph returns the declared topology unchanged.

Paired trials

--trial ID --capability NAME --arm LABEL tags a run as one arm of a paired A/B trial in the archive; --trial-report prints the within-pair rubric difference with a two-sided sign-flip p-value and the smallest p-value the estimator behind it could possibly reach (src/trials.py, src/inference.py). The p is exact by enumeration up to MAX_EXACT_PAIRS = 18; above that it is a seeded sample of SAMPLED_SIGN_ASSIGNMENTS = 200,000 sign assignments over the same differences, and the report says so and prints the seed. Below MIN_PAIRS_FOR_SIGNIFICANCE = 6 a trial is labelled underpowered rather than reported as a null.

This is the apparatus for answering "does this mechanism help?", not the answer. No paired trial has been run yet. Read docs/self-improvement.md before quoting anything from it.

Review: five kinds of critic

Only two of the five are the approval gate. The other three cannot approve anything.

The panel's teeth are mechanical. If any final-round seat carries blocking: true and the chair returned "approve", _enforce_blocking_objections() rewrites the approval into a refinement and records chair_overridden. It is enforced in code precisely because the chair is a model that can be argued out of a prompt-level rule. Blocking is read from the seat's own payload and only counts when that payload's decision token is legible, so a seat whose verdict was unreadable cannot veto.

Every panel run carries its own control arm. The chair's round-1 verdict is one model, one call, no peer input; panel_effect.json accumulates solo-vs-panel across the run and writes a verdict sentence deliberately phrased to be unflattering — "it did not earn that cost; consider dropping the panel" — when that is the truth. The same is true of the ideation panel, anchored comments, crux deliberation and effort tiers; scorecard.py reads all five ledgers at the end of every run and writes workspace/reviews/scorecard.md, keeping "could not be measured" separate from "changed nothing".

Two ledgers run underneath. review_policy.json turns every refusal into a standing rule injected into every later solo review, deduplicated on normalized text so a reviewer restating one complaint cannot manufacture the appearance of learning. obligations.json lets an approving reviewer attach a debt to a later stage — "fine, but you owe me a power analysis at design time" — which only a later reviewer may discharge; deferral is counted and shown, never silent.

Anchored comments make a refusal local. A reviewer quotes the exact passage it objects to (minimum 12 characters, and an unfindable quote is dropped as unanchored rather than sent), the revision is told to leave everything else byte-identical, and the next draft is diffed against the quotes so collateral rewriting is counted rather than assumed away.

Full seat charters, the deliberation protocol and the measurement design: docs/review-panel.md, docs/deliberation.md, docs/stage-comments.md, docs/scorecard.md.

The stage contract and what gets validated

AutoR does not consider a run successful just because it generated a plausible markdown summary.

Required stage summary shape. Seven headings, in this order — REQUIRED_STAGE_HEADINGS:

# Stage X: <name>

## Objective
## What I Did
## Key Results
## Files Produced
## Decision Ledger
## Suggestions for Refinement
## Your Options

Also required, and checked: exactly 3 numbered refinement suggestions, exactly the fixed 6 user options, concrete file paths under Files Produced, and no [In progress], [Pending], [TODO] or [TBD] placeholders.

Artifact gates. Most start by asking whether a file is there — the rows that say "valid", "resolving" or "matching" then parse it.

Requirements are cumulative, and the stage that produces a class of artifact must produce it during that stage's execution — a re-run is not credited with the previous attempt's files. The cutoff is stage_execution_started_at feeding recent_in, and the rubric enforces the same rule independently in _fresh_artifact_kinds, so a Stage 07 draft cannot score on Stage 06's figures.

min_report_figures is a run_config.json field, set by rcb_agent.py --min-report-figures, clamped to [1, MAX_REPORT_FIGURES] (15). It is 1 for an ordinary run and 3 for a ResearchClawBench run (BENCHMARK_MIN_REPORT_FIGURES).

Validity gates. The same function — validate_stage_artifacts (src/utils.py) — also runs the validators that ask whether a claim is warranted rather than whether output exists. Eighteen validate_* functions are reachable from it in all:

The code labels the split itself: "the scientific-validity chain, distinct from the artifact gates around it". A run can fail because a claim is unwarranted, not only because a file is absent. The 06→07 router edge is closed on top of that, in the adaptive topology only.

The frozen preregistration is checked against a copy the stage cannot reach. A digest stored beside the bytes it describes certifies nothing, and this one is worse than most: the agent is shown it, because format_preregistration_for_prompt renders digest into the prompt. So AutoR keeps its own copy of the frozen record at runs/<id>/preregistration_stamp.json, outside workspace/, and preregistration_tamper_findings runs three comparisons rather than one — the hypotheses against the digest the file states for them, that digest against the stamped one, and the length of the amendment ledger. Each catches a rewrite the other two miss: an edited statement, an edited statement with the header recomputed, and a deleted amendment row. Deleting the frozen file is not a way around them either — freeze_preregistration restores the stamped record rather than deriving a fresh one, and deleting both copies is refused rather than re-derived, because the first freeze is witnessed in the run log. So a re-freeze cannot hand the run a post-results date and an empty ledger, and the disagreement it found is appended to the stamp's repairs list before the copy goes back.

The complete gate, including every JSON schema that is parsed rather than merely counted, is in docs/stage-contract.md.

Execution model

Context is composed per consumer, not per availability. A stage's inbound block is built by render_inbound(ChannelContext(...), CHANNELS) from the eighteen typed channels in src/information_flow.py. Each channel declares produced_by, a consumed_by set of real stage slugs, and a rationale; test_every_narrowing_is_argued_for (tests/test_information_flow.py) fails a channel that withholds itself from a stage without saying why. Withholding has to be argued for, not just done.

Four narrowings worth knowing, because the abstraction is not the point. Eighteen channels narrow; these four are the ones whose reason is not readable off the key:

• the artifact index skips Stages 00-02 — they produce no data, results or figures, so the index is empty noise there
• the writing manifest reaches Stage 07 alone
• the mutable Stage 02 hypotheses stop at 04_implementation, because the freeze at Stage 04's approval supersedes them. Before that edge was typed, the same H1 went into every prompt from Stage 05 on twice — one copy labelled editable, sitting next to the frozen one at exactly the stages where the freeze is the point.
• the project bootstrap narrows by exactly one stage and no more. recommend_entry_stage can return any stage from 01 to 08, so a fixed early set of readers would withhold the description of the repository from the run that re-enters latest — the one that has seen least of it. Stage 00 is the single exclusion, because run() scans the repository after intake has finished, so the block is empty there every time. This block deliberately overlaps # Approved Memory: _adopt_project_bootstrap_baseline copies each below-entry assessment into a stage summary, but append_approved_stage_summary keeps only the entries numbered below the stage it writes, so the first approval below the re-entry point erases them and this block becomes the only copy.

dependency_edges() returns every (producer, consumer, channel key) triple, so the information topology can be printed and diffed rather than reconstructed from a pile of if statements. _record_inbound_channels writes the delivered channel keys per stage into the run log.

Honest scope: twenty blocks are typed. Six more — obligations_context, intake_context_text, web_search_context, approved_memory, handoff_context, and the # What the Task Asks For block that build_prompt composes inline from format_deliverables_for_prompt — are still delivered by build_prompt itself rather than declared as channels, so each one's delivery rule lives there instead of next to a consumed_by set. Around them, compose_stage_template (src/prompt_fragments.py) assembles the stage's own instructions, the accepted-extension lists generated from the validators' constants rather than hand-copied, and the run-safety rules.

The assembled prompt is written to runs/<run_id>/prompt_cache/, per-stage session IDs to runs/<run_id>/operator_state/, and the selected CLI backend is invoked in live streaming mode. Alongside the prompt, AutoR installs an agent skill pack from src/skills/ into runs/<run_id>/.claude/skills/ — the operator's working directory — so the agent can pull long-form craft guidance when it needs it. A skill costs nothing in the prompts that do not use it.

168 skills ship today: 79 general ones and 89 field-specific ones. Forty of them were written in one pass against the twelve tasks that trailed a bare-Claude-Code control under a single judge — three or four per task, each selected by a phrase in that task's brief and in no other of the forty, and every one of the forty pinned. Most of them were written against a scored arm's per-criterion losses on the twenty-five ResearchClawBench tasks that lost, at least three per task. A run is not offered all of them. Two filters narrow the pack, and a skill has to survive both:

1. Field. A skill named <field>-... is installed only for a run in that field, so twenty become two. A materials run does not benefit from being offered advice about observational astronomy, it just has one more description to read past.

2. Shape. A skill may carry an applies_when regex, matched against this run's own research brief and data manifest. 51 skills are scoped this way today; measured over the forty ResearchClawBench briefs they select between 1 and 7 tasks each — forty of the forty-four RCB-shaped ones select exactly one — eighteen tasks receive none of them, and no task receives more than six. The seven added for AIRS-Bench are scoped on a different corpus and select 19 of its 20 briefs and none of the forty ResearchClawBench ones; the twentieth is a brief whose whole task paragraph is one sentence that never says what the deliverable is, so no predicate over task shape can reach it and its pin does instead. tools/skill_selectivity.py prints the selection set for a corpus and --expect turns it into an assertion, because a predicate is a claim about a kind of research problem and it should be checkable.

   A skill that selects nothing over the forty is exempt by name in tests/test_every_skill_can_be_loaded.py rather than passed by a loosened regex, and the exemption carries the measurement that justifies it. Those seven are the whole list today: a predicate written against one harness's own task shape generalises to nothing outside that harness, which is a real cost written down rather than argued away, and it is why they reach a run through the third route below rather than through this one.

The predicate reads the brief, never the task's identifier: a table of benchmark ids would select the same tasks today and generalise to nothing.

1. Pin. There is one exception, and it is deliberate. configs/task_skill_pins.json maps a task identifier to skills that are installed for it whatever the two filters say. A pin is not an inference about a kind of task — it is a record that this exact identifier already ran, already scored, and lost criteria whose subject is those skills, so it is the one routing input that cannot be derived from the task statement and does not generalise past the name it carries. Forty-seven tasks are pinned today — twenty-seven from ResearchClawBench and all twenty of AIRS-Bench — 420 pins between them, at most fifteen on any one task; twenty of them are skills the two filters would have withheld, in each case a field skill whose content applies outside its own field. The two benchmarks' pins are derived differently and the file says so: RCB's from per-criterion losses, AIRS-Bench's from a mechanism that is arm-wide because that benchmark has no criteria — 43% of the median run's tool calls landed after its predictions file stopped changing. A run that matches an entry writes skill_pins into its run_config.json and a skills pinned_by_task_id line into its log, because a pinned arm and an unpinned arm are two configurations and a score from one is not a score from the other.
2. Force. A front end may set Manager.skill_force to a set of names installed on every run it launches, whatever the filters say and whatever the pin table holds. Neither an inference about this task nor a record of this task: a decision about a whole benchmark population, taken outside the run, on evidence the run cannot see. No front end in the tree sets it today — the adapter that did belonged to a benchmark since removed, which offered --no-forced-skills as the control arm out of the same binary. It also closes a hole the predicate cannot: select_run_skills fails closed on an empty brief and refuses every task-scoped skill silently. A control arm has to set Manager.skill_withhold as well as clear the force, and that is not a hypothetical: the five skills that adapter forced also carried a predicate matching all sixty of that benchmark's task statements, so clearing the force alone left the arm running with the same five under a different banner — measured on a --fake-operator run before skill_withhold existed, where the difference between the two arms came to one paragraph of prompt rather than five skills. Withholding beats every other input here, including a pin, because it is not a routing decision — it is an experimenter saying which arm this run is. A forced run writes skill_forced and skill_forced_by into its run_config.json and a skills forced_by_front_end line into its log, saying in the same sentence that a score from it is not comparable to one from a run without them. A front end that uses it owes the same fact somewhere a trial can read — taken off the manager's installed set rather than off the flag, and folded into the arms' environment digest — so that sentence about comparability is enforced rather than only printed. Announced in the prompt under its own banner, never the pin's: the pin sentence earns its force by being precise about a scored run of this exact task, and reusing it here would be a claim that is false of every run that reads it.

Pull-based is not the same as discoverable. Measured over a 40-task arm, the pack drew 78 Skill calls in 789 hours of agent time, 31 of them the one skill a stage prompt named imperatively — and stage 05 launched none in any of the forty runs. So every general skill is now named at the stage whose decision it covers, a task-scoped one is announced by the task_shaped_skills channel for the runs that were selected for it, and tests/test_a_skill_is_named_where_it_is_needed.py refuses a skill that nothing announces — in either direction, since a prompt naming a skill most runs will not have is the same defect reversed.

claude --model <model> \
  --permission-mode bypassPermissions \
  --dangerously-skip-permissions \
  --session-id <stage_session_id> \
  -p @runs/<run_id>/prompt_cache/<stage>_attempt_<nn>.prompt.md \
  --output-format stream-json \
  --verbose

Web search where the backend has none. Some deployments disable the agent's built-in WebSearch tool. --web-search gemini starts a stdlib JSON-RPC MCP stdio server (src/mcp_web_search.py) that exposes one tool, mcp__autor-search__web_search, backed by Gemini with grounded search, and passes it to the CLI via --mcp-config. assess_search_readiness() refuses to promise a capability the environment cannot deliver: a hard blocker (no key, no SDK) is reported before the run starts rather than discovered at Stage 01.

Important behaviour:

• refinement attempts reuse the same stage conversation whenever possible
• streamed agent output is shown live in the terminal
• raw stream-json output is captured in logs_raw.jsonl
• if resume fails, AutoR can fall back to a fresh session
• if stage markdown is incomplete, AutoR can repair or normalize it locally before failing the stage
• a backend that is unreachable is classified by src/backend_health.py and surfaces as run.backend_unavailable, so "the model was down" never reads as "the research failed"

Run layout

Every run lives entirely inside its own directory. The tree is build_run_paths (src/utils.py).

runs/<run_id>/
├── user_input.txt      memory.md             run_config.json
├── run_manifest.json   artifact_index.json   intake_context.json
├── obligations.json    review_policy.json    # both per-run; nothing crosses runs
├── report_plan_stamp.json   preregistration_stamp.json   validity_review_stamp.json
│                       # AutoR's copies, outside workspace/ on purpose
├── stage_cost_ledger.json   # one row per stage visit: what it spent, why each attempt failed
├── logs.txt            logs_raw.jsonl
├── prompt_cache/       operator_state/       handoff/        stages/
├── .claude/skills/     # the skill pack, pulled on demand by the agent
├── evolution/          # champion drafts, improvement_ledger.jsonl, summary.json,
│                       # stage_graph.json, routing_refusals.jsonl
└── workspace/
    ├── literature/  code/  data/  figures/  report/  writing/
    ├── bootstrap/   profile/
    ├── notes/       preregistration.json, hypothesis_manifest.json, experimental_protocol.json,
    │                report_plan.json, research_rounds.json, round_decision.json,
    │                deliberation_request.json
    ├── results/     experiment_manifest.json, hypothesis_outcomes.json
    ├── artifacts/   claim_provenance.json, deliverables_coverage.json, citation_verification.json,
    │                self_review.json, report_review.json | layout_review.json, compiled PDFs
    └── reviews/     validity_review_<stage>.json, validity_response_<stage>.json,
                     comment_ledger.json, deliberations.json, scorecard.md, panel/

evolution/ sits outside workspace/ on purpose, and the dataclass records the reason: it is "a record of how the run reached its answer, not part of the answer, and a benchmark export that swept it up would ship the losing drafts alongside the report". report_plan_stamp.json, preregistration_stamp.json and validity_review_stamp.json are outside workspace/ for the same class of reason: the agent must not be able to backdate its own declaration, rewrite the commitment it is being held to, or edit the record of the objections it owes an answer to. So is stage_cost_ledger.json, which is a run's account of what each stage visit cost and why each attempt failed — a receipt the payer prints is worth what it cost to print.

That row carries the backend's own dollar figure and its four token fields, wired out through OperatorResult, ReviewDecision and ValidityReviewOutcome rather than scraped back out of logs_raw.jsonl. A field the backend did not report is absent rather than zero, so a run smoke-tested with --fake-operator says not measured instead of $0.00. At the end of a run — completed or cancelled — AutoR prints one summary of it to the terminal, and to nothing else: the deliverable does not change, and logs.txt keeps the attempts and the failure census without the money. Nothing at runtime decides on any of it. Not the supervisor, not the router, not a gate: the fields may appear in the record, in the summary and in the formatter, and in no condition anywhere under src/, which tests/test_cost_is_recorded_and_unread.py asserts over the syntax of every module the way tests/test_router_budget.py asserts it of StageRouter.choose.

The only state AutoR writes outside a run directory is the cross-run archive at ~/.autor/archive (--archive, --no-archive).

Workspace semantics. literature/ reading notes, survey tables, benchmark notes · code/ runnable code, scripts, configs · data/ machine-readable datasets, manifests, processed splits · results/ metrics, predictions, ablations, plus the standardized experiment_manifest.json · report/ the markdown deliverable, report.md and the PNGs it embeds under images/ · writing/ LaTeX sources, sections, tables, bibliography · figures/ plots and paper figures · artifacts/ review JSON, build metadata, compiled PDFs, packaged deliverables · notes/ the frozen files of the validity chain plus supporting notes · reviews/ adversarial validity reviews, panel transcripts, self-measurement ledgers, the run scorecard.

Outside workspace/: memory.md is the approved free-text cross-stage memory; handoff/<slug>.md is the second free-text carrier, each approved summary trimmed to Objective / Key Results / Files Produced and sent only on a continuation attempt or when memory is still empty. Every other cross-stage edge is a typed channel or a JSON artifact. run_manifest.json is the lifecycle state that resume, redo and rollback read; prompt_cache/ holds the exact prompt of every attempt, repair, review, panel seat and crux voice.

evolution/artifact_provenance.json records which stage wrote each workspace file and every version it has held; evolution/effects/<slug>.jsonl is that stage's accumulated inverses, moved to <slug>.reverted.jsonl once applied, and evolution/effects/blobs/ is the content-addressed store the rewinds read from. A rollback is not only a manifest edit: it applies those inverses in reverse, deletes what the withdrawn stages created, rewinds what they amended back to the version the last surviving stage left, and drops the withheld emissions in evolution/emissions.json. Everything it moved is named in the run log under rollback recovery, and the preview says it before the operator confirms.

Full file-by-file reference: docs/run-artifacts.md.

Architecture

flowchart LR
    P[rigor.py · effort.py<br/>policy: what machinery runs] --> M
    C[information_flow.py<br/>20 typed channels] --> M
    M[manager.py<br/>walks the stage graph] --> W[walk<br/>stage_graph · router]
    M --> G[gates<br/>utils · preregistration · experimental_protocol<br/>report_plan · deliverables · validity_review]
    M --> I[improvement<br/>rubric · evolution · pareto]
    M --> R[review<br/>approval_agent · review_panel · cross_reviewer<br/>obligations · review_policy · stage_comments]
    M --> S[self-measurement<br/>scorecard · archive · decisions · trials · inference]
    M --> X[execution<br/>operator · operator_codex · web_search · backend_health]

Supporting modules: operator_codex.py and operator_protocol.py, intake.py, manifest.py, artifact_index.py, experiment_manifest.py, evidence_ledger.py, hypothesis_manifest.py, mcp_web_search.py, diagram_gen.py, bootstrap.py and project_bootstrap.py, platform/foundry.py, run_skills.py, terminal_ui.py, prompts/, skills/, and backend/ + frontend/ for the Studio. Runnable tools: tools/score_rcb_run.py (score a benchmark run with the reference judge) and tools/archive_sample_complexity.py (how many runs the archive needs before an edge becomes believable).

The full module map, the stage attempt loop and the extension points are in docs/architecture.md. The design rationale — what is new here and why — is in docs/framework.md.

Benchmarks

AutoR is wired to three, and they measure different halves of it. ResearchClawBench hands the agent a workspace of raw data and reference papers and scores the report and figures it produces against the published paper — a test of conducting research. FIRE-Bench hands it a research question from a published empirical study, expects it to design and run its own experiments, and scores the two-sentence conclusion it writes against the authors' own — claim by claim, under a wall clock the harness enforces. AIRS-Bench hands it a prepared dataset and a metric and scores the predictions it writes. A change that moves one need not move the others.

The last one is worth having for a reason the other two cannot supply. Both of them reach their number through a model reading what AutoR wrote, and on ResearchClawBench the choice of reader is worth more than most of the effects being argued about — 16.2 points between two judges on one identical artifact set, and 8.5 points between eight draws of the same judge. AIRS-Bench runs scipy over a CSV: the same submission scores the same number every time. It is the only one of the three where a one-task before-and-after is a measurement rather than a draw from a distribution.

ResearchClawBench

python rcb_agent.py --workspace <WORKSPACE> runs AutoR against a ResearchClawBench workspace with no human in the loop and exports the benchmark's deliverables (report/report.md, report/images/, code/, outputs/). Scoring is the benchmark's own rubric judge; tools/score_rcb_run.py defaults to the reference judge, gpt-5.1, which is what ResearchClawBench itself scores with.

Judge choice can move a score by more than the gap between the top and the bottom third of the leaderboard. On one identical artifact set Gemini 2.5 Flash scored 37.0 where Opus scored 20.8, a spread of 16.2; on another, Opus scored 52.6 where gpt-5.1 scored 46.0, a spread of 6.6. A number carrying the wrong judge is not a smaller number, it is an incomparable one. Quote the judge with every total.

Fixing the judge does not fix the number. Eight draws of gpt-5.1 over one identical artifact set — same workspace, same report, same five figures, nothing changed between draws — scored 41.4, 42.8, 45.5, 47.1, 49.1, 49.6, 49.8 and 49.9: a spread of 8.5 points around a mean of 46.9, sd 3.4. The variance is worst where it costs most, on the heaviest checklist item: that one is weighted 0.5 and spanned 32 to 55 across the eight, 11.5 points of the total by itself. So a single-draw score on a single task carries roughly ±4 points of pure sampling noise, and any one-task A/B comparison below about eight points is uninterpretable — including a before-and-after on the same task, which is the shape a harness change most tempts you into. Average draws, or compare across tasks, or say nothing.

How the number got here

Three earlier measurements of this benchmark are superseded and have been moved to the lab notebook, which is where the arm-by-arm record belongs: the first run at 14.16 where AutoR placed last behind the bare Codex CLI, the post-repair re-run at 23.57 against a 29.24 control, and the re-score that corrected both. Each was wrong in a way worth knowing about rather than wrong by accident, and the notebook keeps each one next to the correction that retired it:

• The judge was shown between 29% and 80% of the figures, depending on the arm. gpt51_judge.py held MAX_IMAGES = 5 for four days after upstream raised its own cap from five to fifteen (bfffc48, 2026-08-14) — a lag behind upstream, not a local deviation from it — and image_paths[0] is the target figure, so four workspace slots were left. Arms differ in how many figures they ship, so the clip was not a constant offset: four slots showed the control 67% of its figures and the skills arm 29%.
• The arms were not given the same budget. The AutoR side ran --stage-timeout 1800 and 28 of its 40 runs logged Stage timed out; the bare arm had no per-stage cap.
• The draw count decides the verdict. One judge draw against three moves an arm's comparator by more than the effect being measured.

One correction from that era is a method, not an arm, so it stays here. The control was never search-less: it was recorded as having no working web search on the evidence of 16 WebSearch calls returning an org-policy 400, when all 44 of its runs also had ai4ai-web-search connected and called it successfully 12 times. Both that reading and the "search parity" that replaced it came from counting tool names. Pair tool_use ids to tool_result ids and read the body.

What has not changed: AutoR writes 36% more prose than the bare agent and covers less of what the task asked for.

Four later arms have landed, and they are ahead

The paragraph that stood here said a promising arm was being withheld until all forty of its tasks landed, because a partly-finished arm is a subset selected the flattering way. They have landed — and three more arms with them. Every score below is tools/score_rcb_run.py --judge reference --draws 3, the same instrument the control was measured with; a one-draw number is not comparable to these and the reason is in the lab notebook.

This table is generated, not maintained by hand — the version that stood here read n=35 for arms that had reached 40 and omitted nine arms run since. Regenerate it with:

python3 tools/rcb_scoreboard.py --contrasts

Snapshot 2026-08-22. One score per task, from the newest scored workspace; only _score_gpt51.json, which is --judge reference --draws 3.

✳ = 95% interval excludes zero. Bold n = all forty tasks scored.

Do not read the top of that table as a ranking. Only seven arms have all forty tasks, and the arms below forty are not missing tasks at random: the outstanding ones are the slow and hard ones, which the control scores above its own average on, so a partial arm's paired lead reads high. The best complete arm is xrev_off at 40.17, +8.70.

The arms are ahead; nothing inside them is proven

Every one-variable contrast available, each pair being one tree apart — and three of them are placebos that should come out at zero:

Ten contrasts, one clears two standard errors — and two of the three placebos are more than half its size. The bottom row is two arms whose command lines are identical character for character, and it reads +2.05. That is the scale at which a difference here means nothing, and it is larger than most of the rows above it.

So the defensible summary is two sentences. AutoR is clearly ahead of the bare agent it wraps, by about 8 points on the best complete arm. Which parts of AutoR are responsible is unknown, because no individual change has been shown to beat its own placebo. How to fix that ratio is how to hill-climb this benchmark.

Read against the sentence further up — "the scaffold is currently worth less than no scaffold" — that claim is retired: the scaffold is ahead of the bare agent it wraps on every recent arm, on the same judge and the same forty tasks.

The one contrast that clears its error bar says a skill set is worth negative points. abl40 is main40 with 41 task-scoped SKILL.md files removed, otherwise the same commit and flags, and over the 37 tasks both have finished the 120-skill pack is +2.16 ± 1.06 ahead. Both arms are well clear of the control, so they are ahead despite that skill set rather than because of it. Two things keep it from being a verdict: a second pair of the same contrast (abl40 − skills161) reads only +0.95 ± 1.48, and the byte-identical placebo reads +2.05.

This is not a verdict on pinning either, and an earlier version of this paragraph said it was. It read "the pins are worth negative three points", which conflated two different experiments. abl40 deletes skill files and leaves 240 pins across 25 tasks in place: pinning is still on in both of its arms. The ablation that actually isolates pinning is a different pair, pins_on against pins_off — same commit, all 168 skill directories present on both sides, 420 pins against zero — and it points the other way:

Read together: a particular set of skills hurts, and the mechanism that routes skills to tasks does not — it is mildly positive and nowhere near significant at n=21. Those are complementary answers, not contradictory ones, and telling them apart required reading the two worktrees rather than the two names.

a9c2b48 is the row that answers the caveat the other three carry. Every other arm above is 35–39 of 40, so each is a subset selected on its own completions — and that selection is not neutral: when this arm stood at 33 of 40 the seven it had not finished were ones the control scored well on, 33.90 against 27.11 on the tasks that had paired, which is the flattering direction. Assuming the worst for those seven would have collapsed it from +8.29 to +0.72. It was held back from this table for that reason. All forty have now landed and the estimate moved the other way, to +7.33 over 40 pairs with nothing dropped. That is the first complete AutoR arm to separate from the control, and it is what makes the four incomplete rows readable as low rather than inflated.

One caveat still travels with the rest: abl40's lead over main40 rests on 33 pairs, which resolves about 3 points and is measuring an effect of about that size.

The framework document's §6 is the full account, including the part that is worse than the mean: the two highest scores came from runs that halted at hypothesis generation, and across 133 stage visits the graph took one backward edge.

Where to go next: for what the reported systems actually score and which of their numbers reproduce, see docs/researchclawbench-landscape.md; for the adapter, its output contract and the export rules, see docs/researchclawbench.md; for every arm, what changed between them and what the instrument was doing while we read them, see docs/researchclawbench-arms.md.

FIRE-Bench

python fire_agent.py --bench-root ~/FIRE-Bench --task cot_in_planning --profile pipeline runs one FIRE-Bench task: a research question from a published empirical study, a sandbox, no reference paper, and 3600 seconds of wall clock that the harness enforces with a kill. What is scored is the two-sentence conclusion the agent writes, decomposed into atomic claims and matched against the authors' own. --profile direct is the paired control — the same goal, model, denied tools, sandbox and deadline, in one agentic call instead of a stage walk.

Measured here, six tasks, one run each, opus executing and reviewing, every arm on the benchmark's own 3600 s clock. Each cell is the median judge draw of three — one draw, so its three numbers are consistent with each other; arms are the mean ± sd across tasks, which is the shape FIRE-Bench's own Table 3 reports:

The stock arm's two rows are both reported because neither is obviously the right one and the choice moves its F1 from 28.4 to 9.4. Which is comparable to a published table depends on how that table handled a run that produced nothing, and FIRE-Bench's paper does not say.

How much of this is the judge. These six logs were scored twice, with nothing changing but the judge's sampling. The arm means moved by 4 to 5 F1 points — pipeline 30.2 → 25.7, direct 55.9 → 60.6, stock 22.9 → 28.4 — and one task moved by 13.7 (cot_in_planning, pipeline arm, 100.0 → 76.9). The ordering of the arms did not move. Treat the ordering as the result and the individual numbers as one draw of a noisy instrument; on a single unchanged log the measured range is 43 F1 points.

Three things this says, in decreasing order of how much the sample supports them.

1. The clock is the benchmark's main filter, and telling the agent about it is what gets past it. Four of six stock Claude Code runs produced nothing scoreable: given the raw instruction.txt, which asks for a full report and says nothing about a deadline, they were still building figures and prose when the harness killed them at 61 minutes. Both AutoR arms were scoreable 6 times out of 6. The difference is not the pipeline — the direct arm has none — it is a goal contract that states the wall clock, states that the conclusion is the only artifact read, and says to write it early and rewrite it, plus a watcher that republishes the scored line every time the file on disk improves.

2. Under this clock, the pipeline loses to one call of the same model on the same contract. Direct wins 5 of 6 tasks, median +34.8 F1, and it wins on precision and recall alike rather than by trading one for the other. Every pipeline run hit the reserve boundary at 52 minutes having approved one to three of its four stages; the direct arm finished in a median of 11. This is the same direction as ResearchClawBench and as the sixty-task written-answer trial of 2026-08-19, on a benchmark since removed from this repository, and it is the third benchmark to say it.

3. Three of the pipeline's four low scores are honest nulls, not empty runs — and the metric cannot tell the difference. On premise_order_effects the pipeline ran 112 problems across nine orderings on two models, 694 billed calls per arm, with a corrupted-premise control at 0.15 accuracy confirming the task was real, and concluded that premise order did not move accuracy. That is correct about what it measured: its strong model scored 1.000 in every ordering — at ceiling — and it said so. It scored 0.0, because the reference conclusion says premise order matters. The direct arm, on the same task in a fifth of the time, ran a pilot, saw the ceiling, generated a harder pool with chains up to sixteen steps, found the effect at sign-test p = 0.0001, and scored 80.0 / 80.0 / 80.0. The lesson is about iteration, not about honesty: what the pipeline lacked was a second pass at its own instance difficulty, and its budget went to preregistration, a reproduction table and a gate ledger instead.

What this is not. Six of thirty-five tasks, one run per cell, and a judge whose measured range on a single unchanged log is 43 F1 points. It is not comparable to FIRE-Bench's published table (best row: Claude Code with Sonnet-4 at 46.7 ± 23.4 F1): different executing model, and — decisively — none of the models the tasks name is served on this deployment, so every arm substituted from the same catalogue and every arm is answering a slightly different question from the one the papers answered. That substitution is also the likeliest reason for finding 2's ceilings: these papers' effects were measured on gpt-3.5 and Llama-2-era models.

The adapter, the deadline design, the six exit clauses, the judge's noise and the six holes in the benchmark's own harness that had to be routed around are in docs/firebench.md.

AIRS-Bench

python airs_agent.py --task <TASK> --repo <AIRS_BENCH> --raw-dir <RAW> --workspace <WS> runs AutoR against one of AIRS-Bench's twenty machine-learning research tasks and exports the one thing it scores, submission.csv. Scoring is the task's own evaluate.py — scipy over a CSV — so unlike the other two benchmarks the same artifact scores the same number every time.

tools/airs_arm.py runs one arm of a comparison and its control: both arms are handed build_task_brief's output byte for byte, the same CLI, model, permission mode, denied tools, workspace and wall-clock cap, and --compare refuses to print a delta between two manifests that disagree on any of them.

Nineteen tasks (the twentieth cannot be staged), one seed, opus executing in both arms, 4 h of wall clock each, no web search, one (arm, task) per slurm array element on CPU-only nodes. Reported in the benchmark's own three units, computed by its own create_summary_plots.ipynb rules — a run with no scoreable submission is a 0 in the mean, not an omission. 1.000 is human SOTA.

Paired over the nineteen tasks: +0.426 mean, +0.069 median to the bare CLI, which wins 16 of 19. The mean and the median disagree by a factor of six because one task carries it: CodeGenerationAPPSPassAt5's normalized score has a denominator eleven times smaller than a typical task's, and the two arms' Pass@5 of 0.783 and 0.947 land at 7.37 and 14.15. Drop that task and the same nineteen-task run reads 0.860 against 0.787, +0.073 paired, bare winning 15 of 18 — which is why all three aggregates are printed and the median is the one to quote. A five-task pilot on a GPU node gave the same direction.

* Elo is over a three-entity pool including SOTA and is not comparable to a rating from the published fifteen-entity pool; with two agents it re-expresses the head-to-head count. There are no error bars: the published ones bootstrap 10–20 seeds per task and these arms have one seed each, so drawing an interval over tasks instead would be a different quantity wearing the same mark.

The mechanism is legible rather than inferred, and it is the same one the pilot showed: every AutoR run hit the cap and not one finished the walk — six of nineteen never left Stage 01, at 13 to 22 attempts on a literature survey for tasks whose whole specification is "predict this column" — while the bare arm hit the cap zero times at a median of 3 h 14 m. AutoR's single invalid submission is the failure the brief warns about: 1,137 rows where the split has 1,147, a whole task lost to ten rows, and under the benchmark's own convention that task is a zero in its mean rather than a gap in it.

This is the same direction as every other benchmark AutoR has been run on, and the fourth to say it — the other three being the two above and the sixty-task written-answer trial of 2026-08-19, on a benchmark since removed from this repository (§6.8) — this time through an instrument with no judge in it, which is the one thing the reading could not previously be blamed on.

These are not leaderboard numbers. The published table is twenty tasks at ten to twenty seeds, scored in a container with no network, while an agent with a shell here can snapshot_download a model and run inference — seven of the bare arm's nineteen runs did. One deviation from the notebook is deliberate and forced: it anchors each task on the worst score observed in the analysis, which for a two-arm pool that both beat SOTA sends the denominator negative, so the published estimated_worst_score is used instead. Tool-call audit hits for the held-out labels across all 38 runs: zero.

The adapter, the arm harness, the five defects running it surfaced in the benchmark itself, and the one it surfaced in this adapter are in docs/airsbench.md.

Documentation

The docs/ directory is the reference documentation. This README is the overview; everything below is the detail behind it.

Showcase

runs/20260330_101222 is the full example run the docs work from. Run directories are gitignored, so what ships in the repository is the artifacts lifted out of it, under assets/.

AGSNv2 reached 36.21 ± 1.08 on Actor, and the run preserved the full human approval trail.

Accuracy Comparison

Ablation + Actor Results