OfficeBuddy

English | 中文

Most document agents write bytes into a file and hope. OfficeBuddy renders the document through Microsoft Word and Excel themselves after every single edit, diffs the pages, and hands the screenshot to a separate multimodal verifier that has to sign off before the next step begins. When the verifier objects, it says which page, which element, what is wrong — so the repair is targeted, not a blind retry.

The renderer is not a lookalike engine. It is Word.

Proof, not promises

Nothing below is a mockup. This is a real run on a real .docx, on a real Mac, rendered by real Word — instruction: "Make the title bold, 24pt and centered; then add a 2×3 table at the end with headers 任务/负责人/状态 and one data row, with black solid borders."

1 · before	2 · after the edit	3 · what the verifier sees

That red box is not decoration. It is drawn by a pixel diff against the last verified render, so the verifier is told exactly where to look instead of re-reading the whole page.

The complete evidence trail — the model's plan, every tool call, the rendered PDFs, and the verifier's structured verdicts — is checked into examples/harness-walkthrough/. Read it and grade the claim yourself.

Measured against another agent

Byte-level assertions and schema validation share a blind spot: a file can be structurally valid, pass every assertion, and still be visibly broken once a real layout engine renders it. Render-Truth Bench is 9 tasks built on exactly that gap — each pairs an ordinary edit instruction with a literal-execution trap, where doing what was asked, verbatim, yields a file that validates cleanly and renders wrong.

A 2×2 factorial over two systems × visual-feedback availability, 3 repeats per cell, 108 runs, every artefact rendered through real Microsoft Word and Excel:

DDR — defect delivery rate; share of runs whose delivered file carries the defect, decided by deterministic detectors with no model in the loop. SFR — silent failure rate; the file is broken and the user was never told. DAR — defect awareness rate. DAR* — DAR excluding the two tasks this project has already hardened against, so the number cannot be explained by a baked-in default.

Read the caveats before quoting any of this. The task set is deliberately aimed at the one dimension this architecture targets — it does not measure capability breadth, speed, or token cost, and on those the comparison runs the other way (see bench/REPORT.md). At n=3 per cell, differences of a few points are not distinguishable. The harness, the three-state calibration gate that every task had to pass, and the full report with rendered evidence are all in bench/ — including RESULTS.md and visual_report.html.

Why this is more than a retry wrapper

• The verifier is a separate, stateless call. It never sees the edit history or the model's own reasoning — only the screenshot and the step description. It cannot talk itself into believing an edit worked.
• The baseline ratchets. Each new render is diffed against the last render that passed, not merely the last render produced. A failed step cannot quietly become the new normal.
• Failure escalates instead of repeating. Errors are normalized into signatures; the same signature twice forces a different strategy, a third time asks you. Every step and every task has a hard budget ceiling.

Quick start

git clone https://github.com/richardChenzhihui/OfficeBuddy.git
cd OfficeBuddy
pip install -e .
export MINIMAX_API_KEY=...      # Anthropic-compatible endpoint, model MiniMax-M3
officebuddy doctor              # one-time automation-permission setup + self-check

Then just talk to it:

# one-shot, then drop into a REPL to keep going
officebuddy "把第一段改成 Times New Roman 12 号，并加粗标题" report.docx

# pure one-shot
officebuddy "add a totals row and bold it" sales.xlsx --one-shot

# interactive session
officebuddy

Useful flags:

Requirements: macOS, Microsoft Word / Excel (they are the renderer), Python 3.10+, and a MiniMax API key. office-agent still works as an alias of the officebuddy command.

What it can edit

Word — text editing and find/replace (paragraph- and run-level), character styling (font, size, bold/italic/underline, color, per-script CJK font slots), paragraph styling (alignment, indent, spacing), inserting and deleting elements (paragraphs, tables, page breaks), table and cell borders (tblBorders / tcBorders), and structure inspection.

Excel — cell read/write with type preservation, formulas, cell and range styling (font, fill, alignment, number format, borders), conditional selection (row[Salary>5000]-style predicates), row/column insertion and deletion, sheet management, freeze panes, charts, and a fidelity guard that inventories the workbook's parts before and after a save and reports exactly what the underlying reader would have dropped.

Safety, because it edits your actual files

• Your original is never written until you say so. All work happens on an isolated copy; the default output is <name>.edited.<ext>. Overwriting the original requires an interactive confirmation or --yes.
• Every change is snapshotted byte-for-byte with a persistent index — undo and restore work at any point.
• Document content is data, not instructions. Text read out of your files is never allowed to steer the agent (prompt-injection defense).
• No permission dialogs during normal use. Working copies live inside each Office app's own sandbox container, exports pre-delete their target, alerts are suppressed, and focus is never stolen. macOS asks for automation permission exactly once, and doctor walks you through it.
• The Excel fidelity guard warns you up front when a workbook contains parts the reader cannot round-trip (see Limitations).

Architecture

cli.py                  REPL / one-shot / doctor
agent/
  loop.py               main harness: plan → clarify → execute → render → verify → repair
  verifier.py           independent stateless visual verification (forced structured verdict)
  budget.py             error signatures + escalation ladder (retry → new strategy → ask)
  history.py            message history (images are kept out of the main loop's context)
tools/
  registry.py           pydantic models → tool schemas; uniform error envelope; auto-snapshot
  word_tools.py         word_edit_text / edit_style / insert_element / delete_element /
                        find_replace / read_content
  excel_tools.py        excel_write_cells / edit_formula / edit_style / conditional_select /
                        create_chart / manage_sheet / freeze_panes / fidelity_report / …
  interaction_tools.py  propose_plan / update_plan / ask_user / render_preview
render/
  applescript.py        Word/Excel → PDF (in-container, dialog-free, timeout + error classes)
  pdf_to_images.py      PDF → PNG (PyMuPDF, 144 dpi)
  page_diff.py          changed-page detection + red bounding-box annotation
  renderer.py           content-addressed render cache + verified-baseline ratchet
core/
  session.py            working-copy isolation (the original is touched only on explicit save)
  snapshot_manager.py   per-step byte snapshots + persistent index (undo / restore)
adapters/               stateless python-docx / openpyxl operation layer

A bug the screenshots caught by themselves

After the first demo run, plain body text showed words that looked randomly bolder than their neighbours — in a paragraph the agent had never touched. Chasing it down: the source document.xml had no per-character formatting at all, but pdffonts showed the exported PDF embedding both MS-Mincho and MicrosoftYaHei, and a per-span extraction confirmed Word was guessing a fallback font per CJK character, because the document's Normal style declared no east-asian font.

That is exactly the class of defect a byte-level assertion can never see and a rendered screenshot cannot miss. The fix — proper w:eastAsia font slots in the Word adapter — and the full investigation are written up in examples/harness-walkthrough/README.md.

Design notes

Longer write-ups on the edit layer live in docs/edit-layer-designs/: the native-ops router, the Excel fidelity guard, and the Word raw-XML patch modality.

There is also a self-contained visual walkthrough of the harness design at examples/harness-walkthrough/visualization/harness-design-manual.html — download it and open it in a browser.

Limitations

• Opening an .xlsx that already contains charts or images and saving it loses them. openpyxl's reader does not parse them. You are warned explicitly on open and told not to overwrite the original; pure data and styling edits are unaffected.
• Excel charts are generated by openpyxl: the first column of the data range becomes the category axis by default, and the styling is plain.
• Word tracked changes, comments, footnotes and TOC field updates are not supported yet (planned via a Word-internal automation escape hatch). Table and cell borders are supported.
• The first Word render of a session can take 1–2 minutes while Word itself cold-starts; warm renders within a session take about 0.6s. Word and Excel are deliberately left running afterwards to preserve that warm start — doctor only quits instances it started.
• Paragraph-level find/replace flattens run formatting within that paragraph when a match straddles a formatting boundary. The result carries a warning when this happens.
• macOS only, by construction. The whole premise is driving real Office as the renderer.

Development

pytest -q                    # offline suite (unit + tools + FakeLLM loop tests)
pytest -m mac_office -q      # integration tests that drive real Word/Excel
OFFICE_AGENT_LIVE_TEST=1 pytest -m live -q   # live MiniMax smoke test (costs money, skipped by default)

mac_office tests drive the real Office apps and may surface macOS permission dialogs — run them with a human present, never unattended.

License

MIT — see LICENSE.