tokf

tokf.net — reduce LLM context consumption from CLI commands by 60–90%.

Commands like git push, cargo test, and docker build produce verbose output packed with progress bars, compile noise, and boilerplate. tokf intercepts that output, applies a TOML filter, and emits only what matters — so your AI agent sees a clean signal instead of hundreds of wasted tokens.

Before / After

cargo test — 61 lines → 1 line:

   Compiling tokf v0.2.0 (/home/user/tokf)
   Compiling proc-macro2 v1.0.92
   Compiling unicode-ident v1.0.14
   Compiling quote v1.0.38
   Compiling syn v2.0.96
   Compiling serde_derive v1.0.217
   Compiling serde v1.0.217
   ...
running 47 tests
test config::tests::test_load ... ok
test filter::tests::test_skip ... ok
test filter::tests::test_keep ... ok
test filter::tests::test_extract ... ok
...
test result: ok. 47 passed; 0 failed; 0 ignored
  finished in 2.31s

✓ 47 passed (2.31s)

git push — 8 lines → 1 line:

Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 10 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 312 bytes | 312.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:user/repo.git
   a1b2c3d..e4f5a6b  main -> main

ok ✓ main

Quickstart

brew install mpecan/tokf/tokf   # or: cargo install tokf
tokf setup                      # detect your AI tools and install hooks

That's it. Every command your AI agent runs is now automatically filtered.

Run tokf gain to see how many tokens you've saved, or tokf setup --refresh to re-run detection.

Installation

Homebrew (macOS and Linux)

brew install mpecan/tokf/tokf

cargo

cargo install tokf

Build from source

git clone https://github.com/mpecan/tokf
cd tokf
cargo build --release
# binary at target/release/tokf

How it works

tokf run git push origin main

tokf looks up a filter for git push, runs the command, and applies the filter. The filter logic lives in plain TOML files — no recompilation required. Anyone can author, share, or override a filter.

Set up automatic filtering

If you use an AI coding tool, install the hook so every command is filtered automatically — no tokf run prefix needed:

# Claude Code (recommended: --global so it works in every project)
tokf hook install --global

# OpenCode
tokf hook install --tool opencode --global

# OpenAI Codex CLI
tokf hook install --tool codex --global

Drop --global to install for the current project only. See Claude Code hook for details on each tool, the --path flag, and optional extras like the filter-authoring skill.

Usage

Run a command with filtering

tokf run git push origin main
tokf run cargo test
tokf run docker build .

Apply a filter to a fixture

tokf apply filters/git/push.toml tests/fixtures/git_push_success.txt --exit-code 0

Verify filter test suites

tokf verify                    # run all test suites
tokf verify git/push           # run a specific suite
tokf verify --list             # list available suites and case counts
tokf verify --json             # output results as JSON
tokf verify --require-all      # fail if any filter has no test suite
tokf verify --list --require-all  # show coverage per filter
tokf verify --scope project    # only project-local filters (.tokf/filters/)
tokf verify --scope global     # only user-level filters (~/.config/tokf/filters/)
tokf verify --scope stdlib     # only built-in stdlib (filters/ in CWD)
tokf verify --safety           # run safety checks (prompt injection, shell injection, hidden unicode)
tokf verify git/push --safety  # safety check a specific filter

Task runner filtering

tokf automatically wraps make and just so that each recipe line is individually filtered:

make check    # each recipe line (cargo test, cargo clippy, ...) is filtered
just test     # same — each recipe runs through tokf

See Rewrite configuration for details and customization.

Explore available filters

tokf ls                    # list all filters
tokf which "cargo test"    # which filter would match
tokf show git/push         # print the TOML source

Customize a built-in filter

tokf eject cargo/build            # copy to .tokf/filters/ (project-local)
tokf eject cargo/build --global   # copy to ~/.config/tokf/filters/ (user-level)

This copies the filter TOML and its test suite to your config directory, where it shadows the built-in. Edit the ejected copy freely — tokf's priority system ensures your version is used instead of the original.

Flags

Color passthrough

By default, filters with strip_ansi = true permanently remove ANSI escape codes. The --preserve-color flag changes this: tokf strips ANSI internally for pattern matching (skip, keep, dedup) but restores the original colored lines in the final output. When --preserve-color is active it overrides strip_ansi = true in the filter config.

tokf does not force commands to emit color — you must ensure the child command outputs ANSI codes (e.g. via FORCE_COLOR=1 or --color=always):

# Node.js / Vitest / Jest
FORCE_COLOR=1 tokf run --preserve-color npm test

# Cargo
tokf run --preserve-color cargo test -- --color=always

# Or set the env var once for all invocations
export TOKF_PRESERVE_COLOR=1
FORCE_COLOR=1 tokf run npm test

Limitations: color passthrough applies to the skip/keep/dedup pipeline (stages 2–2.5). The match_output, parse, and lua_script stages operate on clean text and are unaffected by this flag. [[replace]] rules run on the raw text before the color split, so when --preserve-color is enabled their patterns may need to account for ANSI escape codes, similar to branch-level skip patterns, which also match against the restored colored text.

Built-in filter library

Generic Commands

When no dedicated filter exists for a command, three built-in subcommands provide useful compression for arbitrary output:

tokf err — Error extraction

Scans output for error/warning patterns across common toolchains (Rust, Python, Node, Go, Java) and shows only the relevant lines with surrounding context.

# Show only errors from a build
tokf err cargo build

# Adjust context lines around each error
tokf err -C 5 cargo build

# Works with any command
tokf err python train.py
tokf err npm run build

Patterns matched: error:, warning:, FAILED, Traceback, panic, npm ERR!, fatal:, Python/Java exception types, and more.

Behaviour:

• Empty output: [tokf err] no errors detected (empty output)
• Short output (< 10 lines): shows [tokf err] header with full output
• No errors + exit 0: prints [tokf err] no errors detected
• No errors + exit ≠ 0: includes full output (something failed but no recognized pattern)

tokf test — Test failure extraction

Extracts test failure details and always includes summary/result lines.

# Show only test failures
tokf test cargo test
tokf test go test ./...
tokf test npm test
tokf test pytest

Patterns matched: FAIL, FAILED, panicked, assertion mismatches, Jest ✕ markers, Go --- FAIL:, and more.

Summary lines always included: test result:, Tests:, passed, failed counts, pytest/RSpec summary lines.

Behaviour:

• Output < 10 lines: passed through unchanged
• All pass + exit 0: prints [tokf test] all tests passed
• Failures detected: shows failure lines with context + summary

tokf summary — Heuristic summary

Produces a budget-constrained summary by identifying header, footer/summary, and repetitive middle sections.

# Summarize a long build log
tokf summary cargo build

# Limit to 15 lines
tokf summary --max-lines 15 make all

Algorithm:

1. Header (first 5 lines) and footer/summary (last lines matching keywords like "total", "finished", "result")
2. Middle section is sampled; highly repetitive content shows a count + samples
3. Extracted statistics (pass/fail counts, timing) appended as [tokf summary] line

Common flags

All three commands support:

Using generic commands with rewrites

Generic commands can be integrated with the rewrite system so they trigger automatically through the hook. Add rules to .tokf/rewrites.toml for commands that don't have dedicated filters:

# Route build commands without filters through tokf err
[[rewrite]]
match = "^mix compile"
replace = "tokf err {0}"

[[rewrite]]
match = "^cmake --build"
replace = "tokf err {0}"

# Route test runners without filters through tokf test
[[rewrite]]
match = "^mix test"
replace = "tokf test {0}"

[[rewrite]]
match = "^ctest"
replace = "tokf test {0}"

# Summarize long-running commands
[[rewrite]]
match = "^terraform plan"
replace = "tokf summary {0}"

Important: User rewrite rules are checked before filter matching. Don't add rules for commands that already have dedicated filters (like cargo build, npm test) — the dedicated filter will produce better output than the generic command.

To check whether a command already has a filter: tokf which "cargo build".

Tracking and history

Generic commands record to the same tracking database and history as tokf run, using filter names _builtin/err, _builtin/test, and _builtin/summary. Use tokf raw last to see the full uncompressed output.

Filters are TOML files placed in .tokf/filters/ (project-local) or ~/.config/tokf/filters/ (user-level). Project-local filters take priority over user-level, which take priority over the built-in library.

Minimal example

command = "my-tool"

[on_success]
output = "ok ✓"

[on_failure]
tail = 10

Command matching

tokf matches commands against filter patterns using two built-in behaviours:

Basename matching — the first word of a pattern is compared by basename, so a filter with command = "git push" will also match /usr/bin/git push or ./git push. This works automatically; no special pattern syntax is required.

Transparent global flags — flag-like tokens between the command name and a subcommand keyword are skipped during matching. A filter for git log will match all of:

git log
git -C /path log
git --no-pager -C /path log --oneline
/usr/bin/git --no-pager -C /path log

The skipped flags are preserved in the command that actually runs — they are only bypassed during the pattern match.

Note on run override and transparent flags: If a filter sets a run field, transparent global flags are not included in {args}. Only the arguments that appear after the matched pattern words are available as {args}.

Local environment wrappers — you don't need to do anything special for your filter to match through a local wrapper like nix develop -c cargo test. tokf strips the wrapper prefix and matches the inner command (cargo test) against your existing patterns. See Local environment wrappers for the configurable list.

Common fields

command = "git push"          # command pattern to match (supports wildcards and arrays)
run = "git push {args}"       # override command to actually execute
description = "Compact git push output"  # human-readable description (shown in `tokf ls`)

skip = ["^Enumerating", "^Counting"]  # drop lines matching these regexes
keep = ["^error"]                      # keep only lines matching (inverse of skip)

# Per-line regex replacement — applied before skip/keep, in order.
# Capture groups use {1}, {2}, … . Invalid patterns are silently skipped.
[[replace]]
pattern = '^(\S+)\s+\S+\s+(\S+)\s+(\S+)'
output = "{1}: {2} → {3}"

dedup = true                  # collapse consecutive identical lines
dedup_window = 10             # optional: compare within a N-line sliding window

strip_ansi = true             # strip ANSI escape sequences before processing
trim_lines = true             # trim leading/trailing whitespace from each line
strip_empty_lines = true      # remove all blank lines from the final output
collapse_empty_lines = true   # collapse consecutive blank lines into one
truncate_lines_at = 120       # truncate lines longer than N chars (with trailing …)

tail = 30                     # keep last N lines regardless of exit code (branch tail overrides)
on_empty = "git push: ok"     # message when filter produces empty output (all lines stripped)

show_history_hint = true      # append a hint line (`tokf raw <id>`) pointing to the full output in history
inject_path = true            # inject shims into PATH so sub-processes (e.g. git hooks) are filtered

passthrough_args = ["--watch", "--web", "-w"]  # skip filter when user passes these flags

# Lua escape hatch — for logic TOML can't express (see Lua Escape Hatch section)
[lua_script]
lang = "luau"
source = 'return output:upper()'    # inline script
# file = "transform.luau"           # or reference a local file (auto-inlined on publish)

match_output = [              # whole-output substring checks, short-circuit the pipeline
  { contains = "rejected", output = "push rejected" },
]

[on_success]                  # branch for exit code 0
output = "ok ✓ {2}"          # template; {output} = pre-filtered output

[on_failure]                  # branch for non-zero exit
tail = 10                     # keep the last N lines (overrides top-level tail)

The run override

run makes tokf execute a different command than the user typed. It is a sharp tool, and there is one rule:

run must not lose information. It may re-encode the same data more densely (--porcelain, --format json, -o json). It must never truncate, cap, or otherwise answer a narrower question than the user asked.

The reason is recoverability. tokf only ever sees the output of the command it actually ran, so that is what lands in history and what tokf raw <id> gives back. If run throws data away before tokf sees it, nothing can recover it — not tokf raw, not anything else. Reductions that drop content belong in the filter pipeline (skip, chunk, max_lines, templates), which runs after the full output has been captured.

Substitutions are recorded and shown. tokf history show prints an Executed: line, tokf raw prints a note to stderr (stdout stays pure output, so pipes are unaffected), and --verbose reports the substitution as it happens:

$ tokf run --verbose -- git status
[tokf] executing: git status --porcelain=v1 -b -uall --find-renames
[tokf]   (substituted by `run` for: git status)

$ tokf history last
Command: git status
Executed: git status --porcelain=v1 -b -uall --find-renames

Note that savings for a run-override filter are measured against the substituted command's output — that is the only baseline tokf ever observes. The Executed: line tells you which command the figure refers to.

The cost of capturing everything

Reducing in the pipeline rather than in run means tokf captures the command's full output, holds it in memory, and writes it to the history database. That is what makes tokf raw able to give it back, and it is not free: tokf run -- git log on a repository with a few thousand commits captures hundreds of KB per invocation, where git log --oneline -n 20 captured about a kilobyte.

History keeps history.retention entries per project (default 10) with no per-entry size cap, so the database grows with the largest output you filter. tokf history clear resets it. Prefer a passthrough_args entry over a run override when a flag means the user wants the unreduced output anyway — that skips both the reduction and the capture.

Passthrough args

Some filters inject flags like --json or --format via the run field. When users pass conflicting flags (e.g. --watch), the combined command fails. The passthrough_args field declares flag prefixes that trigger passthrough mode — tokf skips the filter entirely and runs the original command as-is.

command = "gh pr checks *"
run = "gh pr checks {args} --json name,state,workflow"
passthrough_args = ["--watch", "--web", "-w"]

Matching semantics: each user arg is checked with starts_with against each prefix. This handles --format=table matching --format, while -w does not match --watch (correct — they are different flags). Short-flag prefixes like -o also match concatenated forms like -oyaml (common in tools like kubectl). Empty-string prefixes are ignored. When any arg matches, no run override is applied and no filter pipeline runs.

Variant interaction: passthrough is checked on the resolved filter config after file-based and args-based variant detection. If a parent filter delegates to a variant (via file detection or args_pattern), the variant's own passthrough_args apply — not the parent's. Output-pattern variants (post-execution) are not resolved when passthrough is active.

Use --verbose to see when passthrough activates:

$ tokf run gh pr checks 142 --watch --verbose
[tokf] passthrough: user args match passthrough_args, skipping filter

Template pipes

Output templates support pipe chains: {var | pipe | pipe: "arg"}.

Example — filter a multi-line output variable to only error lines:

[on_failure]
output = "{output | lines | keep: \"^error\" | join: \"\\n\"}"

Example — for each collected block, show only > (pointer) and E (assertion) lines:

[on_failure]
output = "{failure_lines | each: \"{value | lines | keep: \\\"^[>E] \\\"}\" | join: \"\\n\"}"

Sections

Sections collect lines into named buckets using a state-machine model. They are processed on the raw output (before skip/keep filtering) so structural markers like blank lines are available.

[[section]]
name = "failures"
enter = "^failures:$"        # regex that starts collecting
exit = "^failures:$"         # regex that stops collecting (second occurrence)
split_on = "^\\s*$"          # split collected lines into blocks at blank lines
collect_as = "failure_blocks" # name used in templates: {failure_blocks}

[[section]]
name = "summary"
match = "^test result:"      # stateless: collect any matching line
collect_as = "summary_lines"

Stateful sections (with enter/exit) toggle on/off as the state machine hits the enter/exit patterns. Stateless sections (with match only) collect every matching line regardless of state.

Section data is available in templates:

• {failure_blocks} — the collected items
• {failure_blocks.count} — number of items (blocks if split_on is set, otherwise lines)
• {failure_blocks | each: "..." | join: "\\n"} — iterate over items

Aggregates

Aggregates extract numeric values from section items and produce named variables for templates.

Single aggregate (backwards compatible):

[on_success]
output = "{passed} passed ({suites} suites)"

[on_success.aggregate]
from = "summary_lines"
pattern = 'ok\. (\d+) passed'
sum = "passed"
count_as = "suites"

Multiple aggregates — use [[on_success.aggregates]] (plural) to define several rules:

[on_success]
output = "✓ {passed} passed, {failed} failed, {ignored} ignored ({suites} suites)"

[[on_success.aggregates]]
from = "summary_lines"
pattern = 'ok\. (\d+) passed'
sum = "passed"
count_as = "suites"

[[on_success.aggregates]]
from = "summary_lines"
pattern = '(\d+) failed'
sum = "failed"

[[on_success.aggregates]]
from = "summary_lines"
pattern = '(\d+) ignored'
sum = "ignored"

Each rule scans the named section's items. sum accumulates the first capture group as a number. count_as counts the number of matching lines. Both singular aggregate and plural aggregates can be used together — they are merged at runtime.

Chunk processing

Chunks split raw output into repeating structural blocks, extract structured data per-block, and produce named collections for template rendering. Use chunks when you need per-block breakdown (e.g., per-crate test results in a Cargo workspace).

Note: Like sections, chunks operate on the raw (unfiltered) command output. Skip/keep patterns do not affect chunk processing. This ensures structural markers are available for splitting.

[[chunk]]
split_on = "^\\s*Running "   # regex that marks the start of each chunk
include_split_line = true     # include the splitting line in the chunk (default: true)
collect_as = "suites_detail"  # name for the structured collection
group_by = "crate_name"       # merge chunks sharing this field value

[chunk.extract]
pattern = 'deps/([\w_-]+)-'  # extract a field from the split (header) line
as = "crate_name"

[[chunk.aggregate]]
pattern = '(\d+) passed'     # aggregates run within each chunk's own lines
sum = "passed"

[[chunk.aggregate]]
pattern = '(\d+) failed'
sum = "failed"

[[chunk.aggregate]]
pattern = '^test result:'
count_as = "suite_count"

Fields:

The resulting structured collection is available in templates as {suites_detail} and supports field access in each pipes.

Structured collections in templates

When a chunk produces a structured collection, each item has named fields. Use each to iterate with field access:

[on_success]
output = """✓ cargo test: {passed} passed ({suites} suites)
{suites_detail | each: "  {crate_name}: {passed} passed ({suite_count} suites)" | join: "\\n"}"""

Inside the each template, all named fields from the chunk item are available as variables ({crate_name}, {passed}, {suite_count}), plus {index} (1-based) and {value} (debug representation).

{suites_detail.count} returns the number of items in the collection.

Carry-forward fields

When a chunk's extract or body_extract rule has carry_forward = true, chunks that don't match the pattern inherit the value from the most recent chunk that did. This is useful when boundary markers (like Running unittests) identify a group, and subsequent chunks (like integration test suites) should inherit that identity.

[chunk.extract]
pattern = 'unittests.+deps/([\w_-]+)-'
as = "crate_name"
carry_forward = true

Tree-structured groups (children_as)

When children_as is set alongside group_by, the grouped collection preserves each group's original items as a nested collection. Inside an each template, the children are accessible by the children_as name and support their own each/join pipes:

[[chunk]]
split_on = "^\\s*Running "
collect_as = "suites_detail"
group_by = "crate_name"
children_as = "children"

[on_success]
output = """✓ {passed} passed ({suites} suites)
{suites_detail | each: "  {crate_name}: {passed} passed\n{children | each: \"    {suite_name}: {passed}\" | join: \"\\n\"}" | join: "\\n"}"""

This produces tree output like:

✓ 565 passed (2 suites)
  tokf: 565 passed
    unittests src/lib.rs: 550
    tests/cli_basic.rs: 15

JSON extraction

When commands produce JSON output (e.g. kubectl get pods -o json, gh api, docker inspect), use the [json] block to extract values via JSONPath (RFC 9535) instead of line-based parsing.

command = "kubectl get pods -o json"

[json]

# Array of objects → structured collection (usable with |each: pipe)
# Auto-generates {pods_count} with the number of matched items.
[[json.extract]]
path = "$.items[*]"
as = "pods"

# Sub-field extraction from each matched object (dot-path, not JSONPath)
[[json.extract.fields]]
field = "metadata.name"
as = "name"

[[json.extract.fields]]
field = "status.phase"
as = "phase"

[on_success]
output = "Pods ({pods_count}):\n{pods | each: \"  {name}: {phase}\" | join: \"\\n\"}"

Result mapping:

Pipeline position: JSON extraction runs after lua_script (step 2c) and replaces parse/sections/chunks — when [json] is configured, those line-based structural steps are skipped. The extracted vars and chunks flow into branch selection (on_success/on_failure) and template rendering.

Dot-path syntax for [[json.extract.fields]]: uses simple dot-separated paths (not JSONPath). Supports array indices: containers.0.name traverses obj["containers"][0]["name"].

Error handling: if the input is not valid JSON, extraction is skipped and tokf falls back to raw output (templates are not rendered). Invalid JSONPath or dot-path expressions are silently skipped.

Tree restructuring

When a filter emits a list of file paths, common directory prefixes are repeated on every line. The [tree] section restructures the output into a directory tree, writing each shared prefix once. Reusable across any path-shaped filter (git status, git diff --name-only, etc.).

command = "git status"

[tree]
# Regex with two capture groups: (1) decoration to keep on the leaf
# (e.g. "M  ", "?? "), (2) the path itself.
pattern = '^(.. )(.+)$'

# Lines that don't match (e.g. "main [synced]" branch headers) are kept
# verbatim: unmatched lines before the first matched path stay in place
# above the tree, and any later unmatched lines are emitted after the
# tree. Set to false to drop them.
passthrough_unmatched = true

# Engagement gates — when not satisfied, the original flat output is
# returned unchanged. Tuned per filter.
min_files = 3            # require at least N matched paths
min_shared_depth = 1     # require at least N common directory levels

# Visual style. "indent" is the cheapest in token count (plain 2-space
# indent, no connectors). "unicode" uses ├─ │ └─ box-drawing characters
# (prettier but each char is 3 bytes in UTF-8 — measurably more expensive
# on deep trees). "ascii" uses |- | `-.
style = "indent"

# Collapse single-child internal directories. Without this, narrow-deep
# paths like a/b/c/d/foo.rs render as four separate dir nodes.
collapse_single_child = true

# Sort children alphabetically. Off by default — source order is stable
# and predictable for LLMs.
sort = false

Example

Before (git status raw porcelain):

## main...origin/main
M  crates/tokf-cli/src/config/cache.rs
M  crates/tokf-cli/src/config/types.rs
M  crates/tokf-cli/src/main.rs
M  crates/tokf-cli/filters/git/diff.toml
M  crates/tokf-cli/filters/git/status.toml
?? crates/tokf-filter/src/filter/tree.rs

After (with [tree] enabled, indent style):

main [synced]
crates/
  tokf-cli/
    src/
      config/
        M  cache.rs
        M  types.rs
      M  main.rs
    filters/git/
      M  diff.toml
      M  status.toml
  ?? tokf-filter/src/filter/tree.rs

The shared crates/tokf-cli/ prefix is written once. The single-child chain tokf-filter/src/filter/ collapses into one leaf. The model sees at a glance which directories cluster work.

Pipeline position

The tree transform runs after dedup and before on_success.output / max_lines. Specifically: stage 2.6 in apply_internal, between dedup (2.5) and the lua/json/parse/section pipeline (2b–4).

Constraints

• Color restoration is bypassed when [tree] is active. Tree-rendered lines are synthesized from path components, so per-line ANSI color spans from the original output don't survive structural rearrangement. If you need both colored output and tree structuring, you'll have to pick one.
• Engagement is opt-in. Without a [tree] section, filters behave exactly as before — no magic detection.
• Engagement gates fail closed. If min_files or min_shared_depth aren't met, the original flat lines pass through unchanged. There's no half-rendered intermediate state.
• Rename arrows like R old.rs -> new.rs are handled: the path is split on -> and the suffix stays attached to the leaf. The trie key is the old path.
• [parse] takes precedence. A filter that declares both [parse] and [tree] will run parse and skip tree entirely. The two solve different problems (tree restructures path-list output, parse structures arbitrary text) and don't compose, so the precedence is fixed at parse-wins.

Filter variants

Some commands are wrappers around different underlying tools (e.g. npm test may run Jest, Vitest, or Mocha). A parent filter can declare [[variant]] entries that delegate to specialized child filters based on project context:

command = ["npm test", "pnpm test", "yarn test"]

strip_ansi = true
skip = ["^> ", "^\\s*npm (warn|notice|WARN|verbose|info|timing|error|ERR)"]

[on_success]
output = "{output}"

[on_failure]
tail = 20

[[variant]]
name = "vitest"
detect.files = ["vitest.config.ts", "vitest.config.js", "vitest.config.mts"]
filter = "npm/test-vitest"

[[variant]]
name = "jest"
detect.files = ["jest.config.js", "jest.config.ts", "jest.config.json"]
filter = "npm/test-jest"

Detection has three modes, checked in order:

1. File detection (Phase A, before execution) — checks if config files exist in the current directory. First match wins.
2. Args pattern (Phase A.5, before execution) — regex-matches the remaining command-line arguments (joined with spaces). Fires after file detection but before the passthrough_args check, so a matched variant's own passthrough_args apply instead of the parent's.
3. Output pattern (Phase B, after execution) — regex-matches command output. Used as a fallback when no file or args pattern matched.

Args-pattern example — route git diff --name-only to a tree-structured child filter:

[[variant]]
name = "name-list"
detect.args_pattern = '--(name-only|name-status)'
filter = "git/diff-name-list"

When no variant matches, the parent filter's own fields (skip, on_success, etc.) apply as the fallback.

The filter field references another filter by its discovery name (relative path without .toml). Use tokf which "npm test" -v to see variant resolution.

TOML ordering: [[variant]] entries must appear after all top-level fields (skip, [on_success], etc.) because TOML array-of-tables sections capture subsequent keys.

Filter resolution

1. .tokf/filters/ in the current directory (repo-local overrides)
2. ~/.config/tokf/filters/ (user-level overrides)
3. Built-in library (embedded in the binary)

First match wins. Use tokf which "git push" to see which filter would activate.

Writing test cases

Filter tests live in a <stem>_test/ directory adjacent to the filter TOML:

filters/
  git/
    push.toml          <- filter config
    push_test/         <- test suite
      success.toml
      rejected.toml

Each test case is a TOML file specifying a fixture (inline or file path), expected exit code, and one or more [[expect]] assertions:

name = "rejected push shows pull hint"
fixture = "tests/fixtures/git_push_rejected.txt"
exit_code = 1

[[expect]]
equals = "✗ push rejected (try pulling first)"

For quick inline fixtures without a file:

name = "clean tree shows nothing to commit"
inline = "## main...origin/main\n"
exit_code = 0

[[expect]]
contains = "clean"

Assertion types:

Exit codes from tokf verify: 0 = all pass, 1 = assertion failure, 2 = config/IO error or uncovered filters (--require-all).

Richness checks

Every assertion above is positive: it checks a string the author remembered to think about. Nothing observes what a filter dropped that nobody asserted. That is the failure mode that hurts — someone widens a skip regex to suppress a noisy line, it also swallows a panic backtrace, and every test still passes. Richness is the counterweight: a rarity-weighted measure of how much irreplaceable information survived filtering.

tokf verify prints it on every case line:

    ✓ rejected push (1240 → 88 tokens, 92.9% reduction, richness 0.31 [12/97 atoms])

kept/atoms are counts of distinct atoms, which is what makes the scalar interpretable — 12/97 on a small fixture means something quite different from 12/997.

How it is computed:

1. Split raw and filtered output on whitespace.
2. Trim non-alphanumeric characters from each token's edges ((hello_world), → hello_world); interior punctuation is kept, so src/main.rs stays intact.
3. Keep tokens of 6 or more characters (counted in characters, not bytes). Matching is case-sensitive — hashes and paths are case-significant.
4. Weight each distinct atom by its self-information, -log2(count / total), where total counts all atom occurrences including repeats. The 400th Compiling is worth almost nothing; a unique path, hash, or error code is worth a lot.
5. Score = surviving weight / total weight. An atom counts as surviving if it appears anywhere in the filtered output, either as a standalone token or as a substring of a rewritten line.

Empty or atom-free input scores 1.0 (nothing irreplaceable existed, so nothing could be lost). If the raw output contains only one distinct atom, its self-information is zero and the weighted ratio is undefined, so the score falls back to the plain kept / atoms ratio — dropping that atom still scores 0.0.

There is no default threshold, and richness never fails a build on its own. tokf is deliberately lossy. cargo check succeeding and collapsing to ✓ cargo check: ok scores near zero, and that is correct. A low score is information, not a defect.

The opt-in assertion. A case can declare a floor with the top-level min_richness field (a whole-case scalar, not an [[expect]] field), taking a value in 0.0–1.0:

name = "panic backtrace survives filtering"
fixture = "tests/fixtures/cargo_test_panic.txt"
exit_code = 101
min_richness = 0.4

[[expect]]
contains = "panicked at"

When the score falls below the declared floor, the case fails — exit code 1, and therefore CI via tokf verify --require-all. Cases that do not declare min_richness are never failed on richness grounds, no matter how low they score.

To pick a value: run tokf verify <filter> first, read the printed score, and set the threshold a little under the observed value. It then acts as a ratchet against future skip-pattern widening.

The same assertion is honoured by registry publish validation, so a published filter must satisfy its own declared min_richness.

Safety checks

Add --safety to detect potential security issues in your filter:

tokf verify --safety
tokf verify my-filter --safety --json

Safety checks scan for:

• Prompt injection — templates containing patterns like "ignore previous instructions", "you are now", "system prompt", etc. Both static config text and filtered output are checked (NFKC-normalized to handle compatibility/fullwidth forms; cross-script homoglyphs are not fully covered).
• Shell injection — run, step[].run, and rewrite replacement strings containing shell metacharacters ($(...), backticks, ;, &&, pipes, redirections). Known-safe templates like tokf run {0} are allowlisted.
• Hidden Unicode — zero-width spaces, RTL overrides, and other invisible characters that could smuggle content.

Safety warnings do not block publishing — filters with issues are published with safety_passed = false and the registry shows a warning badge. Use --safety locally to catch issues before publishing.

Determinism

tokf verify runs every test case's filter pipeline twice against the identical input and asserts the two outputs are byte-for-byte identical. This is not behind a flag — it always runs, for every case, in every tokf verify invocation. Determinism is a correctness invariant of a filter, not an opt-in preference.

If a filter fails this check, tokf verify reports it like any other assertion failure — it names the filter and shows the first differing byte offset with context from both runs:

✗ my-filter (0 → 12 tokens, 100.0% reduction)
    ✗ shows recent count
        my-filter: output is not byte-stable across repeated runs (first differing byte at offset 14)
            run 1: "3 files changed"
            run 2: "5 files changed"

The context window is 20 bytes on each side of the differing byte. A leading or trailing ... appears only when output was actually clipped there — short outputs, like the one above, are shown whole.

Why this matters more than it looks like it should. Tool results get resent on every subsequent turn of a session — the same filtered output is retransmitted as conversation history grows. The provider's prompt cache matches on the request prefix byte-for-byte. If a filter's output for the same input differs between invocations, the bytes at that point shift, and everything after it in the prompt misses cache and re-bills at full input rate instead of the cached discount. A filter that trims 200 tokens but knocks 40k tokens of suffix out of cache is a large net loss — and it's invisible in any single local test run, because a single run only ever sees one version of the output.

Input variance is fine. Output variance is not. cargo test genuinely printing Finished in 20.81s on one run and 21.04s on the next, with the filter passing that duration through unchanged, is correct behavior — not a bug. The invariant under test is that the filter is a pure function of its input: same bytes in, same bytes out, every time. The double-run check holds the input constant (the same fixture, fed through filter::apply twice) specifically so it isolates the filter's own behavior from legitimate variance in what the underlying command printed.

The HashMap-ordering trap. Rust's default HashMap/HashSet hasher is randomly seeded per process, so iteration order is stable within a single run — a filter can look perfectly deterministic in one cargo test or one tokf verify invocation and still vary from run to run of the compiled binary. This is why the check performs two independent filter::apply calls rather than comparing a value to itself, and why the filter engine avoids exposing HashMap/HashSet iteration order to rendered output: sorted collections (BTreeMap) or an explicit sort-before-join are used wherever a collected or keyed value reaches the final template. The same trap applies to the Lua escape hatch — a lua_script step that iterates a table with pairs() should build and iterate an explicit order array instead, the way crates/tokf-cli/filters/docker/images.toml and crates/tokf-cli/filters/cargo/clippy.toml already do, rather than relying on Luau's table iteration order.

The stdlib was audited against this check as part of introducing it. No stdlib filter needed changes — the check exists to hold the invariant going forward, not because a stdlib filter was found broken.

Enforced at publish time, too. The identical byte-stability check now runs server-side when you tokf publish a filter: during publish validation the server runs each test case's filter pipeline twice and rejects the upload — with the same failure-message shape shown above — if the two outputs differ. A nondeterministic filter therefore cannot enter the registry even if a contributor skipped running tokf verify locally, so nobody downstream silently pays the prompt-cache cost. The same across-process caveat applies (a single-process double run cannot see HashMap-seed drift), so the BTreeMap/explicit-ordering discipline above remains the real defence.

Configuration files

tokf uses TOML files for configuration. Files can live at two levels:

Paths shown are for Linux/macOS. On macOS, the global directory is ~/Library/Application Support/tokf. On Windows, it is %APPDATA%/tokf.

Set TOKF_HOME to redirect all user-level paths to a single directory (like CARGO_HOME or RUSTUP_HOME):

export TOKF_HOME=/opt/tokf   # all config, data, and cache under /opt/tokf

Run tokf info to see which paths are active.

config.toml

[history]

Controls how many filtered outputs are retained in the local history database.

[history]
retention = 10   # number of history entries to keep (default: 10)

[sync]

Settings for remote filter-usage sync (requires tokf auth login).

[sync]
auto_sync_threshold = 100   # sync after this many unsynced records (default: 100)
upload_usage_stats = true   # upload anonymous usage statistics (default: not set)

[shims]

Controls PATH-based shim injection for sub-process filtering. When filters use inject_path = true, tokf generates shim scripts and prepends them to PATH so that sub-processes (e.g. commands inside git hooks) are automatically filtered.

[shims]
enabled = true   # generate and use shims for inject_path filters (default: true)

Set to false to disable shim generation and PATH injection globally. This overrides any per-filter inject_path = true setting — no shims will be generated or used.

tokf config set shims.enabled false

Shim scripts are stored in ~/.cache/tokf/shims/ (or $TOKF_HOME/shims/). Disabling shims via config set immediately removes any existing shim scripts.

Note: shims.enabled is read from the global config only — project-local overrides are not checked, to avoid filesystem scanning on every command invocation.

[telemetry]

Export metrics via OpenTelemetry OTLP. Disabled by default.

[telemetry]
enabled = true
endpoint = "http://localhost:4318"   # OTLP collector endpoint
protocol = "http"                    # "http" (default) or "grpc"
service_name = "tokf"                # service.name resource attribute

[telemetry.headers]
x-api-key = "your-secret"

Default endpoints by protocol: HTTP uses port 4318, gRPC uses port 4317.

Environment variables override config-file values for telemetry — see the Environment variables section below.

Priority order

For all config.toml settings:

1. Project-local .tokf/config.toml (highest priority)
2. Global ~/.config/tokf/config.toml
3. Built-in defaults

Managing config via CLI

tokf config show              # show all effective config with source paths
tokf config show --json       # machine-readable JSON output
tokf config get <key>         # print a single value (for scripting)
tokf config set <key> <value> # set a value in the global config
tokf config set --local <key> <value>  # set in project-local .tokf/config.toml
tokf config print             # print raw config file contents
tokf config path              # show config file paths with existence status

Available keys: history.retention, shims.enabled, sync.auto_sync_threshold, sync.upload_stats.

rewrites.toml

Rewrite rules let tokf intercept and transform commands before execution — wrapping task runners, stripping pipes, and injecting baselines. See the Rewrite Configuration section for the full reference.

Environment variables

CLI flags

Global flags

Available on all tokf run invocations and most subcommands:

Run-specific flags

Filter resolution order

tokf searches for matching filters in three tiers, stopping at the first match:

1. Project-local — .tokf/filters/ in the project root
2. User-level — ~/.config/tokf/filters/ (or $TOKF_HOME/filters/)
3. Standard library — built-in filters shipped with tokf

To override a built-in filter, eject it to your project or user directory:

tokf eject cargo/test              # copy to .tokf/filters/ (project-local)
tokf eject --global cargo/test     # copy to ~/.config/tokf/filters/ (user-level)

The ejected copy takes priority on subsequent runs. See tokf eject --help for details.

Directory layout

~/.config/tokf/                    # global config directory ($TOKF_HOME overrides)
├── config.toml                    # history, sync, telemetry settings
├── rewrites.toml                  # shell rewrite rules
├── auth.toml                      # registry credentials (managed by tokf auth)
├── machine.toml                   # machine UUID for remote sync
└── filters/                       # user-level filter overrides
    └── cargo/
        └── test.toml

~/.local/share/tokf/               # data directory
└── tracking.db                    # token savings database ($TOKF_DB_PATH overrides)

~/.cache/tokf/                     # cache directory
├── manifest.bin                   # binary filter discovery cache
└── shims/                         # generated shim scripts for inject_path

<project>/
└── .tokf/                         # project-local overrides
    ├── config.toml                # project-specific settings
    ├── rewrites.toml              # project-specific rewrite rules
    └── filters/                   # project-specific filters
        └── custom/
            └── lint.toml

For logic that TOML can't express — numeric math, multi-line lookahead, conditional branching — embed a Luau script:

command = "my-tool"

[lua_script]
lang = "luau"
source = '''
if exit_code == 0 then
    return "passed"
else
    return "FAILED: " .. output:match("Error: (.+)") or output
end
'''

Available globals: output (string), exit_code (integer — the underlying command's real exit code, unaffected by --no-mask-exit-code), args (table). Return a string to replace output, or nil to fall through to the rest of the TOML pipeline.

Sandbox

All Lua execution is sandboxed — both in the CLI and on the server:

• Blocked libraries: io, os, package — no filesystem or network access.
• Instruction limit: 1 million VM instructions (prevents infinite loops).
• Memory limit: 16 MB (prevents memory exhaustion).

Scripts that exceed these limits are terminated and treated as a passthrough (the TOML pipeline continues as if no Lua script was configured).

External script files

For local development you can keep the script in a separate .luau file:

[lua_script]
lang = "luau"
file = "transform.luau"

Only one of file or source may be set — not both. When you run tokf publish, file references are automatically inlined (the file content is embedded as source) so the published filter is self-contained. The script file must reside within the filter's directory — path traversal (e.g. ../secret.txt) is rejected.

tokf records input/output byte counts per run in a local SQLite database:

tokf gain              # summary: total bytes saved and reduction %
tokf gain --daily      # day-by-day breakdown
tokf gain --by-filter  # breakdown by filter
tokf gain --json       # machine-readable output

How tokens are estimated

tokf does not run a tokenizer. Token counts are derived from byte counts with one constant:

tokens ≈ bytes / 3.5

That is why every token figure tokf gain prints is labelled est., and why it will keep being labelled that way for as long as this estimator ships. It is a heuristic, not a measurement.

Where 3.5 comes from

The constant used to be 4, and nothing had ever checked it. It was measured against a real cl100k tokenizer across the whole tokf corpus — every filter _test/ case (both the raw input and the filtered output) plus every fixture under tests/fixtures/ — which gives the implied divisor, i.e. bytes per real token:

3.5 is the combined figure rounded. The old 4 undercounted real cl100k tokens by roughly 8% on raw output and 25% on filtered output; 3.5 lands within 1% of the corpus aggregate.

Two honest caveats:

• cl100k is not Claude's tokenizer. Even the "real" numbers we calibrated against are an approximation of the thing users actually care about. The goal was removing a large systematic bias, not achieving exactness.
• The corpus is what it is — heavily weighted toward cargo, git, npm and docker output. The p10/p90 spread above shows the per-item divisor genuinely ranges from under 2.8 to over 4.6 depending on output shape: prose is cheap per byte to tokenize, dense symbolic output is expensive. One constant cannot capture that, and tokf deliberately does not try. It says est. instead.

Percentages are more reliable than absolute counts — except when filters rewrite

The savings percentage divides two counts that share the divisor, so most of the error cancels out. That holds well for filters that mostly delete lines:

It stops holding for filters that rewrite content — replacing English prose with a dense symbolic summary. Prose tokenizes cheaply per byte; the summary does not, so the byte ratio overstates the token ratio and the filter flatters itself:

Those are worst cases on tiny outputs, where a handful of tokens swings the percentage wildly — but the direction of the bias is consistent, and it is upward. Treat reduction percentages on rewriting filters as indicative, not as a claim.

Verifying the estimate yourself

The tokenizer is a contributor tool, not a runtime feature. It sits behind an optional cargo feature that is off by default, so normal builds take no tokenizer dependency at all:

cargo test -p tokf --features tokenizer --test calibration -- --ignored --nocapture

That prints the full per-item table and the aggregates above, and fails if the shipped constant drifts more than 25% from what the corpus implies. There is no way to enable a real tokenizer at runtime, and no plan to add one — carrying a vocabulary table in the shipping binary to serve a statistic is not a trade tokf wants to make.

Estimates changed: a deliberate discontinuity

Changing the divisor changes the numbers. Being explicit about what that means:

• Rows recorded before the change keep their old bytes / 4 token counts. They are not rewritten.
• Rows recorded after use bytes / 3.5.
• tokf gain totals spanning the changeover therefore show a step increase in absolute token counts. Nothing is being saved differently; the estimate simply got less wrong.
• Savings percentages are essentially unaffected, because both sides of the ratio scale together. That is the reason the discontinuity is tolerable.
• The same applies to server-side aggregates, which receive already-computed token columns via sync.

We deliberately did not: version the estimator in the SQLite schema (real surface area across three crates for a statistic), rewrite historical rows (local history would then diverge from already-synced server rows — worse than one honest step), or recompute tokens at read time (touches every aggregate query and still cannot fix the server side). One documented step change beat all three.

Remote gain

View aggregate savings across all your registered machines via the tokf server:

tokf gain --remote              # summary across all machines
tokf gain --remote --by-filter  # breakdown by filter
tokf gain --remote --json       # machine-readable output

Remote gain requires authentication (tokf auth login). The --daily flag is not available remotely. See Remote Sharing for the full setup workflow.

Output history

tokf records raw and filtered outputs in a local SQLite database, useful for debugging filters or reviewing what an AI agent saw:

tokf raw last                  # print raw output of last filtered command
tokf raw 42                    # print raw output of entry #42
tokf history list              # recent entries (current project)
tokf history list -l 20        # show 20 entries
tokf history list --all        # entries from all projects
tokf history show 42           # full details for entry #42
tokf history show --raw 42     # print only the raw captured output (long form)
tokf history search "error"    # search by command or output content
tokf history clear             # clear current project history
tokf history clear --all       # clear all history (destructive)

History hint

When an LLM receives filtered output it may not realise the full output exists. Two mechanisms can automatically append a hint line pointing to the history entry:

1. Filter opt-in — set show_history_hint = true in a filter TOML to always append the hint for that command:

command = "git status"
show_history_hint = true

[on_success]
output = "{branch} — {counts}"

2. Automatic repetition detection — tokf detects when the same command is run twice in a row for the same project. This is a signal the caller didn't act on the previous filtered output and may need the full content:

🗜️ ✓ cargo test: 42 passed (2.31s)
🗜️ compressed — run `tokf raw 99` for full output

The 🗜️ prefix appears on all filtered output (disable with tokf config set output.show_indicator false or TOKF_SHOW_INDICATOR=false). The hint line is appended to stdout so it is visible to both humans and LLMs in the tool output. The history entry itself always stores the clean filtered output, without the hint line, indicator or recovery marker.

Per-entry recovery markers

When a filtered command is recorded in history, the indicator carries that entry's ID directly:

🗜️#87 ✓ cargo test: 42 passed

🗜️#87 means the full, unfiltered output is one command away — tokf raw 87. Without an ID (🗜️ alone) the run was not recorded, so there is nothing to recover.

This is additive: the filtered body is byte-identical to what tokf printed before, and the ID rides the indicator that was already being printed. It costs roughly three tokens; there is no extra line and no extra newline. Disabling the indicator (output.show_indicator = false) removes the marker too — the ID is not smuggled back in on its own line.

Why the CLI rather than a tool call

Recovery is deliberately a shell command. tokf raw <id> composes:

tokf raw 87 | grep -n 'error\[' | head -20

A recovered entry can be enormous — a cargo metadata capture runs to hundreds of thousands of tokens — so being able to narrow it before it reaches the model matters. Output that arrives through a tool call lands in context whole, with no opportunity to filter it first. Piping also means the recovered text can itself be filtered by tokf.

Prefer tokf raw <id> | ... over reading an entry whole.

Why the ID is decimal

Decimal is the cheapest encoding, which is counterintuitive — a shorter string is not a smaller number of tokens. BPE tokenizers pack runs of digits (up to three per token) while mixed-case alphanumerics fragment. Measured against cl100k:

Decimal is never worse and sometimes better, so the ID is printed as-is.

Context injection

During tokf hook install, tokf creates a .claude/TOKF.md file and adds an @TOKF.md reference to .claude/CLAUDE.md. This gives LLMs a short context explaining what 🗜️ and 🗜️#<id> mean and how to retrieve full output (tokf raw <id>, or tokf raw last). Use --no-context to skip this step.

Setup

tokf auth login            # authenticate via GitHub device flow
tokf remote setup          # register this machine
tokf sync                  # upload pending usage events
tokf gain --remote         # view aggregate savings across all machines

Authentication

tokf uses the GitHub device flow so no secrets are handled locally. Tokens are stored in your OS keyring (Keychain on macOS, Secret Service on Linux, Credential Manager on Windows).

tokf auth login    # start device flow — prints a one-time code, opens browser
tokf auth status   # show current login state and server URL
tokf auth logout   # remove stored credentials

Machine registration

Each machine gets a UUID that links usage events to a physical device. Registration is idempotent — running it again re-syncs the existing record.

tokf remote setup    # register this machine with the server
tokf remote status   # show local machine ID and hostname (no network call)

Machine config is stored in ~/.config/tokf/machine.toml (or $TOKF_HOME/machine.toml).

Syncing usage data

tokf sync uploads pending local usage events to the remote server. Events are deduplicated by cursor — re-syncing the same events is safe.

tokf sync              # upload pending events
tokf sync --status     # show last sync time and pending event count (no network call)

A file lock prevents concurrent syncs. Both tokf auth login and tokf remote setup must be completed before syncing.

Viewing remote gain

View aggregate token savings across all your registered machines:

tokf gain --remote              # summary: total runs, tokens saved, reduction %
tokf gain --remote --by-filter  # breakdown by filter
tokf gain --remote --json       # machine-readable output

Note: --daily is not available with --remote. Use local tokf gain --daily for day-by-day breakdowns.

Backfill

Usage events recorded before hash-based tracking was added may be missing filter hashes. Backfill resolves them from currently installed filters:

tokf remote backfill             # update events with missing hashes
tokf remote backfill --no-cache  # skip binary config cache during discovery

Backfill runs locally — no network call required.

For discovering and installing community filters, see Community Filters. To publish your own, see Publishing Filters. For the full server API reference, see docs/reference/api.md.

Claude Code hook

tokf integrates with Claude Code as a PreToolUse hook that automatically filters every Bash tool call — no changes to your workflow required.

tokf hook install          # project-local (.tokf/)
tokf hook install --global # user-level (~/.config/tokf/)

Once installed, every command Claude runs through the Bash tool is filtered transparently. Track cumulative savings with tokf gain.

Custom binary path

By default the generated hook script calls bare tokf, relying on PATH at runtime. If tokf isn't on PATH in the hook's execution environment (common with Linuxbrew or cargo install when PATH is only set in interactive shell profiles), pass --path to embed a specific binary location:

tokf hook install --global --path ~/.cargo/bin/tokf
tokf hook install --tool opencode --path /home/linuxbrew/.linuxbrew/bin/tokf

tokf also ships a filter-authoring skill that teaches Claude the complete filter schema:

tokf skill install          # project-local (.claude/skills/)
tokf skill install --global # user-level (~/.claude/skills/)

Gemini CLI

tokf integrates with Gemini CLI as a BeforeTool hook that automatically filters run_shell_command tool calls.

tokf hook install --tool gemini-cli          # project-local (.gemini/)
tokf hook install --tool gemini-cli --global # user-level (~/.gemini/)

This registers a hook shim in .gemini/settings.json (or ~/.gemini/settings.json for --global). When --no-context is not set, it also creates .gemini/TOKF.md and patches .gemini/GEMINI.md with context about the compression indicator.

Cursor

tokf integrates with Cursor via a beforeShellExecution hook that automatically filters shell commands.

tokf hook install --tool cursor          # project-local (.cursor/)
tokf hook install --tool cursor --global # user-level (~/.cursor/)

This registers a hook in .cursor/hooks.json (or ~/.cursor/hooks.json for --global). When --no-context is not set, it also creates .cursor/rules/TOKF.md with context about the compression indicator.

Cline

tokf integrates with Cline via a rules file that instructs the agent to prefix supported commands with tokf run.

tokf hook install --tool cline          # project-local (.clinerules/)
tokf hook install --tool cline --global # user-level (~/Documents/Cline/Rules/)

This writes .clinerules/tokf.md (or ~/Documents/Cline/Rules/tokf.md for --global), which Cline auto-discovers. The rules file uses alwaysApply: true frontmatter.

Windsurf

tokf integrates with Windsurf via a rules file.

tokf hook install --tool windsurf          # project-local (.windsurf/rules/)
tokf hook install --tool windsurf --global # user-level (appends to global rules)

Project-local creates .windsurf/rules/tokf.md. Global mode appends a tokf section (with <!-- tokf:start/end --> markers for idempotent updates) to ~/.codeium/windsurf/memories/global_rules.md.

GitHub Copilot

tokf integrates with GitHub Copilot via instruction files. Copilot only supports repo-level instructions (no --global option).

tokf hook install --tool copilot

This creates .github/instructions/tokf.instructions.md (with applyTo: "**" frontmatter) and appends a tokf section to .github/copilot-instructions.md.

Aider

tokf integrates with Aider via conventions files.

tokf hook install --tool aider          # project-local (CONVENTIONS.md)
tokf hook install --tool aider --global # user-level (patches ~/.aider.conf.yml)

Project-local appends a tokf section to CONVENTIONS.md (which Aider auto-discovers). Global mode writes a conventions file and adds it to ~/.aider.conf.yml's read: list.

OpenCode

tokf integrates with OpenCode via a plugin that applies filters in real-time before command execution.

Requirements: OpenCode with Bun runtime installed.

Install (project-local):

tokf hook install --tool opencode

Install (global):

tokf hook install --tool opencode --global

This writes .opencode/plugins/tokf.ts (or ~/.config/opencode/plugins/tokf.ts for --global), which OpenCode auto-loads. The plugin uses OpenCode's tool.execute.before hook to intercept bash tool calls and rewrites the command in-place when a matching filter exists. Restart OpenCode after installation for the plugin to take effect.

If tokf rewrite fails or no filter matches, the command passes through unmodified (fail-safe).

OpenAI Codex CLI

tokf integrates with OpenAI Codex CLI via a Codex PreToolUse hook plus skills for guidance and discovery.

Install (project-local):

tokf hook install --tool codex

Install (global):

tokf hook install --tool codex --global

This writes a Codex PreToolUse hook to .codex/hooks.json (or ~/.codex/hooks.json for --global) and installs .agents/skills/tokf-run/SKILL.md (or ~/.agents/skills/tokf-run/SKILL.md for --global), which Codex auto-discovers.

Codex CLI 0.124.0 and newer enable lifecycle hooks by default. tokf's Codex integration uses PreToolUse, which first appeared in Codex 0.117.0.

If the installed hook does not run, upgrade Codex or check that hooks are enabled, then restart Codex. Codex 0.129.0+ prefers [features].hooks = true; older builds used the legacy alias [features].codex_hooks = true.

Codex CLI 0.131.0 and newer support PreToolUse updatedInput, so tokf transparently rewrites matching Bash commands in-place. During installation, tokf checks the local codex --version output and installs a conservative deny-and-rerun fallback for older or unknown Codex versions so the original command does not fail open. After upgrading Codex, rerun tokf hook install --tool codex so tokf can refresh the generated shim mode. Commands without a matching tokf filter pass through unchanged.

Permission engines

tokf supports pluggable permission engines that analyse commands and decide whether to allow, deny, or prompt. This is useful for auto-approving safe commands without manual confirmation.

Dippy is an open-source permission engine with deep semantic analysis of bash commands — 34 CLI handlers covering git, aws, kubectl, docker, and more. See the external permission engine section for configuration.

Hook handle command

All hook-based integrations (Claude Code, Gemini CLI, Cursor, Codex) call tokf hook handle internally. The --format flag tells tokf which response protocol to use:

tokf hook handle                    # default: claude-code
tokf hook handle --format gemini    # Gemini CLI protocol
tokf hook handle --format cursor    # Cursor protocol
tokf hook handle --format codex     # Codex CLI protocol

The hook scripts generated by tokf hook install set --format automatically — you don't need to pass it manually. The format also determines which JSON field the external permission engine should set (see hook JSON reference).

Creating Filters with Claude

tokf ships a Claude Code skill that teaches Claude the complete filter schema, processing order, step types, template pipes, and naming conventions.

Invoke automatically: Claude will activate the skill whenever you ask to create or modify a filter — just describe what you want in natural language:

"Create a filter for npm install output that keeps only warnings and errors" "Write a tokf filter for pytest that shows a summary on success and failure details on fail"

Invoke explicitly with the /tokf-filter slash command:

/tokf-filter create a filter for docker build output

The skill is in .claude/skills/tokf-filter/SKILL.md. Reference material (exhaustive step docs and an annotated example TOML) lives in .claude/skills/tokf-filter/references/.

Task runners

tokf also integrates with task runners like make and just by injecting itself as the task runner's shell. Each recipe line is individually filtered while exit codes propagate correctly. See Rewrite configuration for details.

Rewrite configuration (rewrites.toml)

tokf looks for a rewrites.toml file in two locations (first found wins):

1. Project-local: .tokf/rewrites.toml — scoped to the current repository
2. User-level: ~/.config/tokf/rewrites.toml — applies to all projects

This file controls custom rewrite rules, skip patterns, and pipe handling. All [pipe], [skip], and [[rewrite]] sections documented below go in this file.

Task runner integration (make, just)

Task runners like make and just execute recipe lines via a shell ($SHELL -c 'recipe_line'). By default, only the outer make/just command is visible to tokf — child commands (cargo test, uv run mypy, etc.) pass through unfiltered.

tokf solves this with built-in wrapper rules that inject tokf as the task runner's shell. Each recipe line is then individually matched against installed filters:

# What you type:
make check

# What tokf rewrites it to:
make SHELL=tokf check

# What make then does for each recipe line:
tokf -c 'cargo test'          → filter matches → filtered output
tokf -c 'cargo clippy'        → filter matches → filtered output
tokf -c 'echo done'           → no filter → delegates to sh

For just, the --shell flag is used instead:

just test  →  just --shell tokf --shell-arg -cu test

Exit code preservation

Shell mode (tokf -c '...') always propagates the real exit code — no masking, no "Error: Exit code N" prefix. This means make sees the actual exit code from each recipe line and stops on failure as expected.

Shell mode (tokf -c)

When invoked as tokf -c 'command' (or with combined flags like -cu, -ec), tokf enters string mode. The command string is passed through the rewrite system, which rewrites matching commands to tokf run --no-mask-exit-code .... The rewritten command is then delegated to sh -c for execution. If no filter matches, the command is delegated to sh unchanged.

When invoked with multiple arguments after -c (e.g. tokf -c git status), tokf enters argv mode. Each argument is shell-escaped and joined into a command string, which is then processed the same way as string mode. This form is used by PATH shims.

Shell mode is not typically invoked directly; it is called by task runners (make, just) and PATH shims.

Compound and complex recipe lines

Compound commands (&&, ||, ;) are split at chain operators and each segment is individually rewritten. This means both halves of git add . && cargo test can be filtered. Pipes, redirections, and other shell constructs within each segment are handled by the rewrite system's pipe stripping logic (see Piped commands) or passed through to sh unchanged.

Debugging task runner rewrites

Use tokf rewrite --verbose "make check" to confirm the wrapper rewrite is active and see which rule fired.

Shell mode also respects environment variables for diagnostics (since it has no access to CLI flags like --verbose):

TOKF_VERBOSE=1 make check     # print filter resolution details for each recipe line
TOKF_NO_FILTER=1 make check   # bypass filtering entirely, delegate all recipe lines to sh

Overriding or disabling wrappers

The built-in wrappers for make and just can be overridden or disabled via [[rewrite]] or [skip] entries in .tokf/rewrites.toml:

# Override the make wrapper with a custom one:
# "make check" → "make SHELL=tokf .SHELLFLAGS=-ec check"
# Note: use (?:[^\\s]*/)? prefix to also match full paths like /usr/bin/make
[[rewrite]]
match = "^(?:[^\\s]*/)?make(\\s.*)?$"
replace = "make SHELL=tokf .SHELLFLAGS=-ec{1}"

# Or disable it entirely:
[skip]
patterns = ["^make"]

Adding wrappers for other task runners

You can add wrappers for other task runners via [[rewrite]]. The exact mechanism depends on how the task runner invokes recipe lines — check its documentation for shell override options:

# Example: if your task runner respects $SHELL for recipe execution
[[rewrite]]
match = "^(?:[^\\s]*/)?mise run(\\s.*)?$"
replace = "SHELL=tokf mise run{1}"

Routing to generic commands

For commands that don't have a dedicated filter, you can route them through generic commands (tokf err, tokf test, tokf summary) via rewrite rules:

# .tokf/rewrites.toml

# Build commands → error extraction
[[rewrite]]
match = "^mix compile"
replace = "tokf err {0}"

# Test runners → failure extraction
[[rewrite]]
match = "^mix test"
replace = "tokf test {0}"

# Long-running commands → heuristic summary
[[rewrite]]
match = "^terraform plan"
replace = "tokf summary {0}"

Note: User rewrite rules fire before filter matching. Only add these for commands that don't already have a filter — check with tokf which "<command>". Commands with dedicated filters (e.g. cargo build, git status) produce better output through tokf run.

Piped commands

When a command is piped to a simple output-shaping tool (grep, tail, or head), tokf strips the pipe automatically and uses its own structured filter output instead. The original pipe suffix is passed to --baseline-pipe so token savings are still calculated accurately.

# These ARE rewritten — pipe is stripped, tokf applies its filter:
cargo test | grep FAILED
cargo test | tail -20
git diff HEAD | head -5

Multi-pipe chains, pipes to other commands, or pipe targets with unsupported flags are left unchanged:

# These are NOT rewritten — tokf leaves them alone:
kubectl get pods | grep Running | wc -l   # multi-pipe chain
cargo test | wc -l                        # wc not supported
cargo test | tail -f                      # -f (follow) not supported

If you want tokf to wrap a piped command that wouldn't normally be rewritten, add an explicit rule to .tokf/rewrites.toml:

[[rewrite]]
match = "^cargo test \\| tee"
replace = "tokf run {0}"

Use tokf rewrite --verbose "cargo test | grep FAILED" to see how a command is being rewritten.

Disabling pipe stripping

If you prefer tokf to never strip pipes (leaving piped commands unchanged), add a [pipe] section to .tokf/rewrites.toml:

[pipe]
strip = false   # default: true

When strip = false, commands like cargo test | tail -5 pass through the shell unchanged. Non-piped commands are still rewritten normally.

Prefer less context mode

Sometimes the piped output (e.g. tail -5) is actually smaller than the filtered output. The prefer_less option tells tokf to compare both at runtime and use whichever is smaller:

[pipe]
prefer_less = true   # default: false

When a pipe is stripped, tokf injects --prefer-less alongside --baseline-pipe. At runtime:

1. The filter runs normally
2. The original pipe command also runs on the raw output
3. tokf prints whichever result is smaller

When the pipe output wins, the event is recorded with pipe_override = 1 in the tracking DB. The tokf gain command shows how many times this happened:

tokf gain summary
  total runs:     42
  input tokens:   12,500 est.
  output tokens:  3,200 est.
  tokens saved:   9,300 est. (74.4%)
  pipe preferred: 5 runs (pipe output was smaller than filter)

Note: strip = false takes priority — if pipe stripping is disabled, prefer_less has no effect.

External permission engine

By default, tokf does no permission checking — the AI tool (Claude Code, Gemini, Cursor) handles its own deny/ask rules natively. tokf only rewrites commands that match a filter and auto-allows them.

You can optionally delegate permission decisions to an external process — a "sub-hook" that performs deeper semantic analysis of commands. When configured, the engine is consulted on every command, not just ones tokf has a filter for. This lets the engine auto-approve safe commands without the user being prompted.

Configuration

Add a [permissions] section to .tokf/rewrites.toml:

[permissions]
engine = "external"

[permissions.external]
command = "dippy"
args = ["hook", "handle", "--mode", "{format}"]
timeout_ms = 3000    # default: 5000
on_error = "builtin" # what to do if the engine fails

Tool format ({format})

The {format} placeholder in args is replaced with the AI tool identifier before spawning:

If the engine expects different names, add a format_map:

[permissions.external]
command = "my-engine"
args = ["check", "--tool", "{format}"]
format_map = { "claude-code" = "claude", "gemini" = "google" }

Protocol

1. tokf spawns the engine process (command + args, with {format} resolved)
2. The original hook JSON (from the AI tool) is written to the engine's stdin
3. The engine returns a standard hook response JSON on stdout — the same format the AI tool expects
4. tokf extracts the permission decision from the response and applies it to its own rewritten command

The engine sees the original command (not the rewritten one). tokf controls the rewrite; the engine controls the permission decision. Engines that exit with a non-zero status but still produce valid JSON will have their JSON verdict honoured — this supports engines like Dippy that use exit codes for signalling alongside a valid response.

Hook JSON reference (for engine developers)

The engine receives the AI tool's hook JSON verbatim on stdin. The format depends on the tool:

Claude Code (--mode claude-code):

{"tool_name": "Bash", "tool_input": {"command": "git push --force"}}

Expected response — set permissionDecision to "allow", "deny", or "ask" (or omit for ask):

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "allow",
    "updatedInput": {"command": "git push --force"}
  }
}

Gemini CLI (--mode gemini):

{"tool_name": "run_shell_command", "tool_input": {"command": "git push --force"}}

Expected response — set decision to "allow", "deny", or "ask":

{
  "decision": "allow",
  "hookSpecificOutput": {
    "tool_input": {"command": "git push --force"}
  }
}

Cursor (--mode cursor):

{"command": "git push --force"}

Expected response — set permission to "allow", "deny", or "ask":

{
  "permission": "allow",
  "updated_input": {"command": "git push --force"}
}

Error handling (on_error)

When the engine fails (crash, timeout, invalid output), the on_error field determines the fallback:

Example: Dippy integration

Dippy is a permission engine that performs deep semantic analysis of bash commands — auto-approving safe commands while blocking dangerous ones.

[permissions]
engine = "external"

[permissions.external]
command = "dippy"
args = ["hook", "handle", "--mode", "{format}"]
timeout_ms = 3000
on_error = "builtin"

Environment variable prefixes

Leading KEY=VALUE assignments are automatically stripped before matching, so env-prefixed commands are rewritten correctly:

# These ARE rewritten — env vars are preserved, the command is wrapped:
DEBUG=1 git status              → DEBUG=1 tokf run git status
RUST_LOG=debug cargo test       → RUST_LOG=debug tokf run cargo test
A=1 B=2 cargo test | tail -5   → A=1 B=2 tokf run --baseline-pipe 'tail -5' cargo test

The env vars are passed through verbatim to the underlying command; tokf only rewrites the executable portion.

Skip patterns and env var prefixes

User-defined skip patterns in .tokf/rewrites.toml match against the full shell segment, including any leading env vars. A pattern ^cargo will not skip RUST_LOG=debug cargo test because the segment doesn't start with cargo:

[skip]
patterns = ["^cargo"]   # skips "cargo test" but NOT "RUST_LOG=debug cargo test"

To skip a command regardless of any env prefix, use a pattern that accounts for it:

[skip]
patterns = ["(?:^|\\s)cargo\\s"]   # matches "cargo" anywhere after start or whitespace

Implicit skip rules

Three implicit skip rules are always active and can't be disabled. They cover cases where rewriting would silently corrupt the agent's data, so there's no plausible reading under which the rewrite would be correct.

Heredocs

Commands that contain a top-level heredoc (<<EOF, <<-EOF) are passed through unchanged. Wrapping them with tokf run would break the lexical binding between the command and its heredoc body.

# Not rewritten — heredoc body would be cut off:
cat <<EOF > /tmp/cfg.yaml
key: value
EOF

Heredocs inside command substitution

Commands that contain a heredoc anywhere inside a $(...) (or backtick) command substitution are also skipped — even when the substitution is buried deep inside an argument like git commit -m "$(cat <<'EOF' … EOF)". The heredoc body lives in a logically-separate region of the source from the surrounding command, and downstream re-tokenization (clap argv parsing in tokf run, second-pass shell parsers, byte-offset pipe stripping) can slice through it. The canonical failure mode this prevents is git commit -m "$(cat <<'EOF' … EOF)" 2>&1 | tail -10 mangling -m's value into git: error: switch 'm' requires a value.

# Not rewritten — multi-line commit messages, gh PR bodies, etc.:
git add foo && git commit -m "$(cat <<'EOF'
feat: a thing

with multi-line body
EOF
)" && git push

gh pr create -b "$(cat <<EOF
## Summary
…
EOF
)"

Unlike the output-redirect skip below, this rule fires for the whole compound: if any segment contains a substitution-nested heredoc, sibling git add / git push segments are also passed through. This is intentionally conservative — re-emitting any part of a compound that contains this construct risks downstream byte-offset slicing into the heredoc body.

Output redirection

Commands that redirect output to a file (>, >>, &>, &>>, >|, 1>, 2>, <>, etc.) are also passed through unchanged. The agent explicitly redirected to a file because they want the raw output for downstream processing — interposing tokf's filter would write filtered bytes into the file and silently corrupt what the agent reads back.

# These are NOT rewritten — tokf leaves them alone:
git diff > /tmp/diff.txt          # explicit output redirect — agent wants raw form
git log --all > history.txt       # for grep/awk processing on the file
cargo test > test.log 2>&1        # combined output to file
git status > /dev/null            # output discarded; nothing to filter
exec 3<> /tmp/sock                # read+write file open

# These ARE still rewritten — fd remap only, no file involved:
git diff 2>&1                     # merges stderr into stdout, both still feed the agent
git diff 1>&2                     # redirects stdout to stderr — still no file
git diff >&-                      # closes a file descriptor

# Compound commands are handled per-segment — only the segment with the
# redirect is skipped, the other segments are still rewritten:
git diff > foo.txt; git status    # → "git diff > foo.txt; tokf run git status"
git status && git diff > foo.txt  # → "tokf run git status && git diff > foo.txt"

tee in a pipeline (git diff | tee log.txt) is not currently treated as an output redirect because tee is a command argument, not a redirect operator. The current pipe-handling behaviour is preserved. This is a known follow-up.

Transparent-arg commands

Some commands take a shell-code payload that runs in a different environment — ssh HOST 'cmd' runs cmd on the remote host, where tokf is not installed and shouldn't be referenced. For these commands tokf must be especially conservative: it can still wrap the local invocation with tokf run (which preserves the argv byte-for-byte), but user [[rewrite]] regex rules are not applied because a sufficiently broad pattern can splice text into the opaque payload and break the remote call.

The built-in list — always active, can't be disabled — is ssh, mosh, slogin. Basename matching applies, so /usr/bin/ssh is treated identically to ssh.

You can extend the list via [transparent] for tools like kubectl exec, docker exec, etc.:

[transparent]
commands = ["kubectl", "doctl"]

Names are matched against the basename of the command's first word. commands extends — not replaces — the built-in list, so kubectl is added on top of ssh/mosh/slogin.

What still happens for transparent commands:

• The standard tokf run <cmd> wrap from a matching filter (argv-preserving prefix only).
• Pipe stripping with --baseline-pipe '<suffix>' (flags inserted between tokf run and <cmd>; the inner argv is untouched).
• The built-in ^tokf skip and the heredoc / output-redirect skips above.

What is gated:

• User [[rewrite]] regex rules in rewrites.toml. They run on the full command string and could splice text into the inner argv, so they're skipped when the first command word's basename is in the transparent list.

If you genuinely need a regex rewrite for an ssh-class command, invoke it explicitly: tokf run ssh … is preserved by the ^tokf skip rule, so it won't be re-rewritten.

This was added to address #338, where a long-output ssh HOST 'cmd' would land tokf on the remote bash and exit 127.

Local environment wrappers

Some commands wrap an inner command that runs in a local environment — the canonical example is nix develop -c cargo test, which runs cargo test inside a Nix devshell on the same machine. By default tokf only sees the outer command (nix), so a cargo test filter never matches.

Local wrappers are the mirror image of transparent-arg commands. Transparent commands (ssh) run their payload remotely, so tokf stays hands-off. Local wrappers run their inner command locally, so tokf can safely:

1. Inspect the inner command to pick a filter (nix develop -c cargo test → matches the cargo test filter), and
2. Wrap the whole command with tokf run and filter its combined output.

tokf uses the outer wrap — tokf run nix develop -c cargo test, not nix develop -c tokf run cargo test. Because tokf is the parent process and captures the wrapper's output, nothing requires tokf to be on PATH inside the devshell (which would reintroduce the remote-shell failure mode from #338 locally).

# What you type:
nix develop -c cargo test

# What tokf rewrites it to (outer wrap):
tokf run nix develop -c cargo test        # → cargo test filter applied to the output

# With a pipe, the pipe is stripped and re-expressed as a baseline:
nix develop -c cargo test | tail -5
  →  tokf run --baseline-pipe 'tail -5' nix develop -c cargo test

Built-in wrappers

The built-in list, active by default, is:

Any flags or attributes between nix develop and the marker are tolerated, so nix develop .#agent --impure -c cargo test unwraps correctly. Basename matching applies (/nix/store/…/bin/nix is treated as nix). A marker with nothing after it (a bare nix develop -c) is not a match.

Configuring wrappers ([local_wrapper])

Add your own wrappers, disable built-ins by name, or turn built-ins off entirely:

[local_wrapper]
# Turn off ALL built-in wrappers (user rules below still apply). Default: true.
builtins = true
# Disable specific built-ins by command basename.
disabled = ["nix"]

# Add your own wrappers. These extend the built-ins.
[[local_wrapper.rules]]
command = "distrobox"       # matched against the first word's basename
subcommands = ["enter"]     # must immediately follow `command`; omit if not needed
markers = ["--"]            # the inner command starts after the first marker

• command / subcommands decide which prefixes are treated as wrappers.
• markers decides which part of the command is parsed as the filterable inner command — everything after the first marker.
• disabled removes named built-ins; builtins = false removes all of them. User rules always apply.

Limitation: nested task runners

A task runner nested inside a local wrapper — e.g. nix develop -c make check — is not filtered per recipe line. Task-runner integration works by injecting SHELL=tokf so make runs each recipe line through tokf, which would require tokf inside the devshell (the exact thing outer-wrapping avoids). Such commands pass through unchanged.

This was added to address #403.

Debug settings

The [debug] section enables diagnostic logging for the rewrite system. All settings are off by default.

[debug]
log_parse_failures = true

tokf info

tokf info prints a summary of all paths, database locations, and filter counts. Useful for debugging when filters aren't being found or to verify your setup:

tokf info          # human-readable output
tokf info --json   # machine-readable JSON

Example output:

tokf 0.2.8
TOKF_HOME: (not set)

filter search directories:
  [local] /home/user/project/.tokf/filters (not found)
  [user] /home/user/.config/tokf/filters (not found)
  [built-in] <embedded> (always available)

tracking database:
  TOKF_DB_PATH: (not set)
  path: /home/user/.local/share/tokf/tracking.db (will be created)

filter cache:
  path: /home/user/.cache/tokf/manifest.bin (will be created)

filters:
  local:    0
  user:     0
  built-in: 38
  total:    38

Environment variables