Trace File Lineage

Trace where files came from, keep AI-agent output in the right place, and see what each task changed — locally, with evidence, and with honest uncertainty.

📖 Docs • 🎯 Views • ⚖️ vs DVC / Git • 📊 Real-world results

English • 简体中文

Built for Python and notebook work: research code, data analysis, and the piles of files AI coding agents now generate.

pip install trace-file-lineage
lineage demo

lineage demo builds a small project, records a run, and shows you the answer — under a second, nothing to configure, and it writes only into ./lineage-demo. The image above is that command's real output.

Two evidence modes

Being precise about this up front, because they give different kinds of answer:

For files you already have — reconstruct the most likely origins and show the evidence. No setup was needed, and nothing had to be running beforehand. These answers are ranked guesses with the reasoning attached.

For runs from now on — wrap a command and get verified provenance automatically. These answers are proof.

lineage demo shows both at once:

[4/4] Asking where figures/trend.svg came from.

  This is proof            @run/run:8aa88ebd
    assurance: verified   evidence: task-boundary-diff
    A command was recorded while it ran, and this file changed during it.

  This is a good guess     analysis/plot.py
    assurance: candidate   evidence: static-callsite at analysis/plot.py:16
    That line writes to this path, but nobody watched it happen.

Guesses are labelled as guesses. A matching filename, a nearby timestamp, a line of code that mentions a path — none of those are proof, and no amount of them stacked together ever becomes proof. When the evidence genuinely isn't there, the answer is insufficient rather than something plausible.

Common workflows

1. "Where did this old file come from?"

You have a chart from three weeks ago and no memory of making it.

lineage explain figures/final_panel.png

It reads your code, documents, image metadata, and Git history, then ranks the candidates and shows the evidence for each. Often the answer is obvious once you see it — that notebook, reading that CSV. Sometimes the honest answer is that the trail is gone.

2. "My agent just wrote 150 files"

lineage run --task "Parameter sweep" -- python sweep.py
lineage receipt
lineage views --view agent-run

Every file that run touched, recorded as proof, grouped rather than dumped as 150 separate mysteries. agent-run renders the complete manifest as a nested directory tree, so you can see both what changed and where the outputs landed.

Then, before you change an input:

lineage impact data/raw.csv     # what depends on this
lineage stale data/raw.csv      # what is now out of date

3. "Where should the agent put this report?"

lineage layout --suggest monthly.pdf

It first reuses a unique existing filename when one already establishes a stable path. Otherwise, if most existing PDFs live in reports/, the result suggests reports/monthly.pdf and shows the evidence behind that recommendation. If the workspace has no clear convention, it says so instead of inventing a folder.

lineage enable writes this placement check and an end-of-task structure report into AGENTS.md and CLAUDE.md. Placement analysis never moves, renames, or deletes files.

The interactive graph

lineage open

An interactive map in your browser: drag, zoom, click a file to focus on just its neighbourhood and read the evidence behind each link. Solid arrows are proof, dashed arrows are guesses. One self-contained page — offline, no server, nothing sent anywhere.

lineage export --format mermaid gives the same graph as a diagram. Here is the demo project, with the labels written in plain words:

flowchart LR
    n0["recorded run"]
    n1["figures/trend.svg"]
    n0 ==>|"proved it made this"| n1
    n2["analysis/plot.py"]
    n2 -.->|"probably made this"| n1
    n3["data/measurements.csv"]
    n3 -.->|"probably read by"| n2

The tool's own labels are was_generated_by · verified, can_generate · candidate, and declares_read · candidate.

How sure is it?

Five labels, always shown:

Order of trust: your own confirmation, then recorded runs, then imported provenance, then declarations, then static code, then content, then names and timestamps.

Where it's strongest

Best at: Python and Jupyter notebooks. Their code is genuinely parsed, so file reads and writes are understood rather than guessed at — including common Pandas, NumPy, Matplotlib, PIL, and pathlib patterns.

Good at: Word, PowerPoint, Excel, OpenDocument, EPUB, and PDF (text, structure, embedded images); PNG/JPEG/TIFF/WebP metadata; Git rename history; and searching about 50 text and code formats for file references.

Deliberately shallow: JavaScript and TypeScript get a cautious static scan, not real language understanding. Other languages are searched, not parsed. Spreadsheet formulas are recorded but not turned into links. Paths built at runtime can't be resolved.

So it fits research code, data analysis, notebook workflows, Python automation, and agent-generated artifacts. It is not a general-purpose lineage platform for any software project, and doesn't claim to be.

Run lineage doctor for what your own machine can read.

Everything stays local

• Nothing is uploaded, ever. No account, no API key, no AI service.
• Scanning and retrospective analysis never execute your code. lineage run executes only the command you explicitly place after --.
• Passwords, keys, and .env files are skipped automatically.
• Recorded commands have password-looking arguments stripped.
• From an AI agent, only a summary and the changed-file list is kept — never your conversations or prompts.

The .file-lineage/ folder holds text extracted from your files, so treat it like the project itself. It's Git-ignored by default. See SECURITY.md.

Speed

Measured on macOS with Python 3.14, reproducible with tests/benchmark.py:

The first scan reads everything and shows a progress counter; after that only changes are read. Individual questions answer in milliseconds. node_modules, virtual environments, caches, and build output are skipped by default, and each scan reports what it left out.

With an AI coding assistant

Ask Claude Code or Codex "where did this file come from?" and it will use this tool.

claude --plugin-dir .                                     # Claude Code

ln -s "$PWD/skills/trace-file-lineage" \
      "$HOME/.agents/skills/trace-file-lineage"           # Codex

Details: docs/install.md.

Why not Git, DVC, or OpenLineage?

Short version: Git records versions, DVC and OpenLineage record pipelines you declared in advance, and this records evidence — including for work nobody planned to track. Full comparison, including when not to use this: docs/comparison.md.

Integrations — experimental

These exist, are tested against fixtures, and are not part of the core promise: import and export W3C PROV, read dvc.yaml, read OpenLineage events, import an external code graph, export to Obsidian.

They have fixture-level coverage only and have not been validated against real third-party pipelines. Treat them as a starting point rather than a compatibility guarantee, and ignore all of it if you don't need it. The core — tracing files in your workspace — does not depend on any of them.

Status

Early release (0.7.0). Tested on Python 3.11–3.14 across macOS, Linux, and Windows — all twelve combinations green in CI, plus coverage, linting, and real PDF/OCR fixtures. Commands may still change.

Run against three externally-authored repositories, with the results and the defect it uncovered written up in docs/real-world-validation.md — including what those runs do not establish.

Known limits are written down rather than glossed over: docs/limitations.md.

Contributing

Issues and pull requests welcome, including from first-timers — CONTRIBUTING.md. Security reports go privately via SECURITY.md.

lineage demo                                          # see it work
python -m unittest discover -s tests -p 'test_*.py'   # run the tests

Two ways to use it

Manual — ask a question any time, nothing to set up:

lineage explain report.pdf     # where did this come from?
lineage views --list           # pick an angle: project map, one file, a run, duplicates…
lineage layout --suggest report.pdf  # where should this output live?

Continuous — for a project you are actively working in:

lineage enable

That writes a required instruction into the project's CLAUDE.md and AGENTS.md, so the agent records a boundary after every task instead of when it happens to remember, checks placement against the project's existing conventions, and reports changed files as a directory tree. A boundary proves that files changed during the task; only a captured command (lineage run) or other direct evidence proves which process produced them. lineage status shows whether it is on; lineage disable removes exactly that block.

It is an instruction, not an enforcement mechanism — more reliable than hoping the agent recalls a skill, less reliable than a lifecycle hook. Details and the full view list: docs/skill.md.

Authors

tianyiwei and Claudia Chen — see AUTHORS.md.

License

MIT. See LICENSE.