Anti-Autoresearch 🛡️

· · · · · · · ·

🏆 Built on a battle-tested foundation: ARIS (~12.5k★ · HuggingFace Daily Papers #1 · 78+ skills across 7+ platforms). Anti-Autoresearch points ARIS's production audit DNA (experiment-audit · paper-claim-audit · citation-audit · kill-argument) outward — auditing a third party's submission instead of your own.

Autoresearch has gone mainstream, and a fast-growing share of what reaches the review pile is machine-generated — and a lot of it doesn't hold up: tables that don't match the text, baselines that aren't there, open-sourced code that won't reproduce its own paper. Reviewers, area chairs, and honest authors increasingly need to verify that, not just suspect it.

Regardless of who or what wrote a paper, does the science hold together and reflect its own evidence? Anti-Autoresearch audits a submission for self-consistency and fabrication, and produces a span-anchored, reviewer-ready report. It is not an opaque AI-text classifier (no authorship probabilities, no "AI-written" verdict) and does not judge misconduct — it surfaces discrepancies a human reviewer should investigate. Separately, it lists transparent, itemized AI writing-style impressions in a quarantined, zero-verdict-weight section (a paper can be integrity-CLEAN while listing many), because reviewers react to them.

🧭 What's inside

46 integrity patterns across 8 families — the coverage vocabulary every finding cites — plus 13 zero-weight AI writing-style impressions and 2 advisories:

Delivered as 11 skills + 1 orchestrating workflow on a deterministic spine: a span-anchored, hashed evidence ledger → LLM auditors that only propose findings → a rules-only reporter that lists every proposal with what the auditor said about it, and summarizes — with 8 patterns eval-gated end-to-end (GRIM · GRIMMER · statcheck · delta arithmetic · hedge-density · …) and the whole gate in CI.

📰 News

• 2026-07-10 — 🧮 Math was invited to overturn false alarms — and honestly declined. We built exact calculators to re-check the arithmetic behind numeric criticals, hoping to auto-clear false alarms like rounding differences. Adversarial review broke every version — the killer was "dropout of exactly 50%" vs a table's 50.4%: a calculator can't know whether a number is exact or rounded. The calculators stayed, demoted: they attach their full working to the report for the human, and nothing gets auto-cleared.
• 2026-07-10 — 🛡️ Criticals must now survive their own defense. A critical has to state which innocent explanations it ruled out (blank → drops to major), face one fresh refutation thread whose only power is a ⚠️ CONTESTED marker for the human, and — if numeric — declare exactly which numbers it computes over. Reports stamp adjudicator: deterministic-rules-v2. ⚠️ Re-run the auditors on old findings files instead of re-adjudicating them.
• 2026-06-28 — ✍️ v0.5 — the AIS track. 13 itemized AI writing-style impressions now live in a separate, zero-verdict-weight report section — a paper can be integrity-CLEAN while listing many. Transparent impressions, never an authorship verdict; the 5 pure-style patterns moved out of family F.

🚀 Quickstart

Agent workflow (normal use)

Anti-Autoresearch runs as a Claude Code skill workflow — the Python tools are the deterministic spine inside that workflow, not the usual interface.

# 1) Install the skills + workflow (global, or pass a project's .claude/skills dir)
git clone https://github.com/wanshuiyin/Anti-Autoresearch.git
./Anti-Autoresearch/tools/install_anti_autoresearch.sh              # → ~/.claude/skills
# project-local instead: ./Anti-Autoresearch/tools/install_anti_autoresearch.sh ./.claude/skills

# 2) Wire the cross-model reviewer (end state: Claude Code exposes mcp__codex__codex)
claude mcp add codex -- codex mcp-server
claude mcp list

# 3) Audit a paper
claude
> /anti-autoresearch ~/papers/submission

The run writes REPORT.md + report.json + claims.json + per-skill *.findings.json into the paper directory. Put the code/result artifacts alongside the paper to unlock L2 checks; PDF/source-only runs are observability-limited by design.

Zero verdict weight — the AIS + advisory tracks (reported, never moves the verdict)

Three skills produce outputs that are reported but carry zero weight on the integrity verdict — the non-integrity categories that round out a report: the AIS writing-style track and the advisory memos. They matter to a human reviewer (a style impression, the worst-case rejection paragraph, prior-art overlap), so the report shows them in their own section — but the deterministic verdict stays driven only by the 46 integrity patterns. A paper can be CLEAN_GIVEN_EVIDENCE while listing many. /anti-autoresearch runs them automatically; to run one standalone, build the ledger first (next section) and invoke it like any auditor.

Single-skill use

Every auditor is also a standalone skill — the installer drops all of them plus the workflow, so you can run just the axis you care about. They share one contract, so run it in order:

claude
# 1) Build the evidence ledger ONCE — the spine every auditor anchors to. Skip it and
#    any auditor stops with:  NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST
> /evidence-ledger ~/papers/submission        # → claims.json + observability level (L0/L1/L2)

# 2) Then run any auditor below against that ledger → <skill>.findings.json

The verdict-bearing auditors — each takes the paper dir, reads the ledger, and proposes span-anchored findings the deterministic adjudicator turns into the verdict (the zero-weight AIS + advisory skills are in the section above):

A single skill only proposes span-anchored findings — it never returns a verdict. To get one, feed the findings to the deterministic adjudicator (the python3 tools/adjudicate_findings.py … --ledger … command in the next section); the model never grades. Two more notes: consistency-audit, presentation-signals, and ai-style-impressions also write a *.deterministic.findings.json (works with no cross-model reviewer wired); and /anti-autoresearch runs every auditor above in one shot, adding ingest (arxiv-id / pdf → workdir + pdftotext), automatic observability, auto-selection of which auditors apply, and the final cross-dimension verdict + REPORT.md.

Deterministic core (CI / offline / zero-dependency)

This bypasses the agent layer and exercises only the eval-tested deterministic checks — use it for CI, regression tests, or environments with no cross-model reviewer (Python 3 stdlib, nothing to install):

# Prove the pipeline on clean + corrupted fixtures (the regression gate)
python3 eval/run_eval.py
#   clean / delta_inflate / dup_table / headline_inflate  → all PASS
#   injected-defect recall: 100% (7 deterministic patterns) · clean FP: none
python3 tests/test_adjudicator.py        # gate unit tests (the anti-slop guarantee)

# Or run the spine by hand on a real paper:
python3 tools/build_claim_ledger.py --paper-id mypaper --latex main.tex sections/*.tex \
    --observability-level 1 --out claims.json
python3 tools/check_numeric_consistency.py --ledger claims.json --out findings.json
python3 tools/adjudicate_findings.py --findings findings.json --ledger claims.json \
    --paper-id mypaper --observability-level 1 --out report.json --md REPORT.md
#   --ledger is REQUIRED: a finding must quote a verbatim ledger span or it fails closed to info.

🎯 Why this exists

Machine-generated papers and reviews are now a measurable share of the literature, and the failure that matters for an area chair is rarely "was this text written by an LLM?" (a human can write a dishonest paper; an LLM can write an honest one). It is: does the paper contradict itself, and is it backed by its own evidence?

That is what autoresearch pipelines get wrong — they hallucinate local coherence: an abstract number that no table reports, a "16% improvement" that the operands say is 6%, a citation for a claim the cited paper never makes, a method described one way and evaluated another.

Those are checkable under a declared observability level. Concretely, taxonomy v0.5 names 46 integrity patterns across 8 families (numeric self-consistency · method / scope · baseline integrity · experiment integrity · citation integrity · presentation / surface signals · proof & derivation integrity · evaluation design & validity) — the repo's coverage vocabulary, not a detector benchmark — plus a 13-signal AI writing-style impression track (AIS) that carries zero verdict weight.

Shipped v0: the deterministic spine and the seven ✓-marked patterns (across the representative list below and the full catalog) are eval-tested; the other 39 integrity patterns are agent-layer contracts (a cross-model reviewer proposes span-anchored findings, the deterministic adjudicator scores or demotes them) — not bundled-eval detector claims.

The full catalog, with detection signals and false-positive cases, lives in the taxonomy. A representative ten (✓ = gated by the deterministic eval today):

• HP-NUM-INFLATE — abstract says 85.3%, but Table 2 never gets past 84.7%. ✓
• HP-DELTA-ERROR — a "16% improvement" from 73.1 to 78.0 is really 6.7%. ✓
• HP-DUP-TABLE — two tables carry the identical ordered numbers — usually copy-paste padding. ✓
• HP-METHOD-DRIFT — the method section says "no labels"; the eval quietly uses gold-label calibration.
• HP-SCOPE-INFLATE — "comprehensive" turns out to be two datasets, one domain, maybe one seed.
• HP-MISSING-BASELINE — SOTA is claimed while the obvious recent baseline never appears in the table.
• HP-FAKE-GT — (L2) the "reference" targets are model outputs, then reported as ground truth.
• HP-PHANTOM-RESULT — (L2) a headline number points at a result file or metric key that isn't there.
• HP-PROOF-CIRCULARITY — (L1) the "proof" restates the claim in different words and calls it done — it proves nothing.
• HP-CITE-HALLUC — the DOI / arXiv id / venue / author list simply doesn't exist.

This is not hypothetical. Paraphrased from a public reviewer account during the NeurIPS 2026 cycle (illustrative, not a citation), one batch maps almost one-to-one onto the taxonomy this repo encodes:

• Paper 1 — "data tables don't match the text; several rows are misaligned; there's an obvious add/subtract regularity across backbones — it doesn't look like it was actually run." → consistency · HP-SUSPICIOUS-REGULARITY
• Paper 2 — "two tables fill a page and are identical; the one figure is LLM-generated; and it still didn't fill 9 pages." → HP-DUP-TABLE · presentation signals
• Paper 3 — "formula derivations don't hold; the experiments look complete but the math can't give those results." → proof-derivation-forensics · HP-DERIVATION-INVALID
• Paper 4 — "open-sourced, beautifully written and drawn — but I ran the code and it gives completely different results from the paper." → experiment-forensics (L2)

The fourth case is this repo's thesis in one line: surface polish is not integrity.

🔒 How it stays honest (the anti-"LLM-slop" design)

The obvious dismissal of any such tool is "an LLM grading another LLM's paper is just noise." Three structural defenses, not just a disclaimer:

1. Evidence ledger. One deterministic pass turns the paper into claims.json — span-anchored, hashed claims. Every finding must cite a claim_id + verbatim span. No span → it cannot be a high-severity finding.
2. The LLM never grades. Auditors propose findings; a deterministic adjudicator (tools/adjudicate_findings.py, pure rules) computes the verdict. Same findings → same summary, by a fixed rule. The severities in it are what the auditors proposed, annotated with what the computation could and could not confirm.
3. Observability levels. A run declares what it could see (L0 PDF-only → L2 repo+results). A finding that needs code is reported with the level it needs and a mark that this run did not have it, so nothing shouts "fraud" from a PDF and nothing disappears either. See references/observability-levels.md.

Surface signals and AI writing-style impressions have firewalls. Family-F surface tells (duplicate tables, LLM-generated figures, page-padding, leftover pipeline strings) are reported only as high-false-positive context: the adjudicator hard-caps them at minor (SURFACE_ONLY_SKILLS / SURFACE_PATTERNS), so they reach at most SOFT_FLAGS. The pure AI writing-style impressions (the AIS track — defensive hedging, LLM phrasing tics, …) go further: they carry zero verdict weight — forced to info, excluded from overall_verdict / counts / dimensions, and rendered in a separate "NOT integrity" section. A paper can be CLEAN_GIVEN_EVIDENCE while listing many. Both caps are enforced in code (_is_zero_weight + the weight-1-only verdict in tools/adjudicate_findings.py), not just promised — and regression-tested.

And an eval harness (eval/) proves the deterministic core on clean + synthetically-corrupted fixtures every change — measured false-positive / recall, not vibes.

🏗️ Architecture

input (pdf | pdf+latex | pdf+repo+results)
   │
   ▼  [evidence-ledger]  artifact_manifest.json (+ observability level) + claims.json   ← deterministic
   │
   ▼  fan out auditors (each reads the ledger, emits span-anchored findings):
        consistency-audit          flagship · paper vs itself · ARIS paper-claim-audit
        citation-forensics         exists? correct? right context? · ARIS citation-audit
        baseline-comparison-audit  missing/weak/mistuned baselines · ARIS paper-claim-audit
        experiment-forensics       L2: fake GT / self-norm / phantom · ARIS experiment-audit
        proof-derivation-forensics L1: proof gap / circularity / invalid step · verdict-bearing · ARIS proof-checker
        eval-design-forensics      L0/L1: data leakage / conflicted-or-unvalidated LLM judge / selective reporting
        presentation-signals       checkable surface tells · auxiliary, capped at minor
        ai-style-impressions       AI writing-style impressions · ZERO verdict weight · separate section
        adversarial-case-builder   evidence-bound memo, no verdict · ARIS kill-argument
        novelty-duplication-advisory  prior-work overlap memo, no verdict · ARIS novelty-check
   │
   ▼  [adjudicate_findings.py]  rules, not a model → REPORT.md + report.json            ← deterministic

⚠️ Honest limitations

• Forensics ≠ proof of misconduct. Output is flags for a human, never an accusation.
• PDF-only (L0) catches inconsistency + tells, not all fabrication — it cannot verify external ground truth or run code.
• False positives exist (legitimate round numbers, single-seed pilots, deliberate scope choices). Hence tiers, FP-risk tags, and the eval harness — not a binary "guilty".
• The taxonomy is a living document. Adversaries who know a signal can route around it; this is a safety net, not a guarantee. See docs/limitations.md.

🧬 Provenance: derived from ARIS

ARIS — Auto Research in Sleep is an AI research-agent skill platform that runs end-to-end research pipelines (literature → idea → experiment → paper) — and does so with integrity guardrails built in, which is what makes it a credible base for the auditor:

• 🛡️ A three-layer audit stack keeps ARIS's own output honest: experiment-audit (fake GT / score-normalization / phantom results), result-to-claim (is the claim scientifically supported?), and zero-context paper-claim-audit + citation-audit (do the reported numbers and references hold up?). Anti-Autoresearch is these same audits pointed outward.
• 🔬 Cross-model adversarial review is the core doctrine: the executor and the reviewer must be different model families, so no LLM ever judges its own output. Anti-Autoresearch inherits this and hardens it — here the model only proposes findings; a deterministic adjudicator decides.

Two sides of one coin. ARIS is how to do autoresearch responsibly; Anti-Autoresearch is how to flag autoresearch that wasn't. A generator that publishes its own audit stack knows precisely how these pipelines fail — because it engineered against those failures from the inside. That is the perspective this repo brings.

👉 ARIS main repo: https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep

How the skills map — Anti-Autoresearch's skills are ARIS's audit skills, copied and reframed for a third party auditing an unknown submission rather than an author checking their own work: consistency-audit ← paper-claim-audit, experiment-forensics ← experiment-audit, citation-forensics ← citation-audit, baseline-comparison-audit ← paper-claim-audit, proof-derivation-forensics ← proof-checker, adversarial-case-builder ← kill-argument, novelty-duplication-advisory ← novelty-check, plus the new evidence-ledger spine, presentation-signals, eval-design-forensics, and the zero-weight ai-style-impressions (the AIS track).

🤝 Prior art & acknowledgments

Anti-Autoresearch's design borrows ideas — and in places, taxonomy structure — from a body of integrity, reproducibility, and evaluation-hygiene work that predates it. We credit it explicitly. Taxonomies and ideas are adapted with credit; no external code is vendored — where a tool is GPL/AGPL we reimplemented the method from its paper rather than copying its source, and where a tool is proprietary we credit the concept only.

Deterministic self-consistency (the closest methodological cousins).

• statcheck — Nuijten & Epskamp. Recomputes reported NHST p-values from their test statistics; the canonical "paper against itself" check. (GPL-3 — method reimplemented from the paper, source not vendored.) Informs HP-STAT-INCONSISTENCY.

• GRIM / GRIMMER — Brown & Heathers (GRIM); Anaya (GRIMMER). Tests whether reported means/SDs are arithmetically possible for the stated sample size. Informs HP-GRANULARITY-IMPOSSIBLE / HP-VARIANCE-IMPOSSIBLE.

• scrutiny — Jung. An R toolkit packaging GRIM/GRIMMER-style consistency tests. (MIT.)

  These inform taxonomy family A (numeric self-consistency); tools/check_stat_consistency.py is an independent, pure-stdlib reimplementation.

Evaluation integrity & LLM-judge validity (why the model never grades).

• Leakage taxonomy — Kapoor & Narayanan, Leakage and the Reproducibility Crisis in ML-based Science. The priority reference behind the planned family-H eval-integrity patterns.
• LLM-as-judge validity — Zheng et al. (judging LLM-as-a-judge), Panickssery et al. (evaluators favor their own generations), Wang et al. (position / identity bias). Precisely why our adjudicator is deterministic and the model only proposes findings.
• "Show Your Work" — Dodge et al. Reporting-hygiene discipline behind the planned selective-reporting checks.

Citation & claim integrity.

• Retraction Watch — the retraction-record project; conceptual basis for citation-status awareness.
• Problematic Paper Screener — Cabanac, Labbé, Magazinov. Corpus-scale screening for tell-tale strings; the basis for HP-PIPELINE-ARTIFACT.
• scite — supporting / contrasting citation context. (Proprietary — conceptual credit; informs HP-CITE-CONTEXT.)
• SciFact — Wadden et al. Scientific-claim verification dataset/model behind the claim–evidence framing.
• Fabricated-citation taxonomy — Ansari. Informs HP-CITE-HALLUC / HP-CITE-CONTEXT.

Reproducibility detection (prior art for planned L2 repro patterns — not yet implemented).

• ODDPub — Riedel et al. Detects open-data / open-code statements. (AGPL-3 — conceptual/method prior art; no code vendored; any implementation will be independent.)
• RTransparent — Serghiou et al. Large-scale data/code-sharing transparency detection. (GPL-3 — same.)
• SciScore — automated methods-rigor / reproducibility checker. (Proprietary — conceptual credit only.)

Author-side self-audit (evaluated, scope-bounded).

• academic-integrity-skill — 1anj. An author-side, wet-lab/biomedical self-audit skill (image-forensics-heavy). We evaluated its non-image deterministic screens — raw-data terminal-digit / exact-duplicate forensics (last-digit / Benford tradition) and reported-vs-source reconciliation — and adopt neither as a check: digit forensics need raw per-sample tables our reviewer-side tiers rarely see, and reported-vs-source overlaps family D. Credited as prior art and the author-side counterpart to this reviewer-side toolkit. (MIT.)
• anti-defensive-writing — Kiterlin. An author-side Codex skill that revises defensive writing (removes unnecessary caveats/hedges, strengthens prose). The clean dual of our AIS-DEFENSIVE-HEDGE: they fix it for the author, we flag it for the reviewer (zero-weight). We cross-referenced its discouraged-construction list to extend our deterministic hedge templates ("this is not to say", "this should not be taken to mean", "rather than arguing X, we argue Y"). (MIT.)

Anti-Autoresearch's own contribution is not any single one of these checks but their combination into an autoresearch-specific taxonomy under a deterministic adjudicator and explicit observability tiers (see docs/positioning.md).

🔭 Related projects

Where Anti-Autoresearch sits relative to neighboring tools (stars / last update as gathered 2026-06-27; not a ranking).

A few framing-relevant efforts have no open repository and are credited by name only: Pangram, GPTZero, and the Problematic Paper Screener.

💬 Community

The taxonomy grows with the community. Caught an autoresearch / AI-Scientist paper pulling a trick that isn't in the pattern catalog yet? That is the single most valuable contribution here — open an issue with the concrete example, or send a PR adding the pattern (with an eval fixture + a false-positive case so it doesn't over-fire). New auditor skills, adjudicator gates, and corruption fixtures are just as welcome. CONTRIBUTING.md explains how a pattern is structured and the honesty rules every flag must follow (describe a checkable discrepancy, never impute misconduct or authorship).

Join the WeChat group (shared with the ARIS community) to swap autoresearch failure modes:

(The group QR rotates weekly — if it's expired, open an issue and we'll post a fresh one.)

📖 Citation

Anti-Autoresearch is derived from ARIS and reuses its audit DNA. If this repository helped your research / paper / review, please cite the ARIS methodology paper:

@article{yang2026aris,
  title={ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration},
  author={Yang, Ruofeng and Li, Yongcan and Li, Shuai},
  journal={arXiv preprint arXiv:2605.03042},
  year={2026}
}

⚖️ License

MIT — see LICENSE.