semble_rs

semble_rs
Fast and Accurate Code Search for Agents — in Rust
_{Replaces grep / cat / read / ls and compresses build & CI output. Up to -99% tokens.}

semble_rs is a Rust port and superset of MinishLab/semble built for AI coding agents. It returns the exact code chunks an agent needs, prints a token-cheap codebase tree instead of ls -R, and compresses 3 MB CI logs into 35 KB. One single binary, no daemon, no API keys, no GPU. Hybrid BM25 + Model2Vec static embeddings with code-aware reranking, plus a dependency graph, AST chunking, and a digest pipeline for build / test / CI output.

Quickstart

# Install Rust if needed, then:
git clone https://github.com/johunsang/semble_rs.git && cd semble_rs
cargo install --path .

The binary lands at ~/.cargo/bin/semble_rs. On first run, the default embedding model minishlab/potion-code-16M (~60 MB) is downloaded from HuggingFace.

# Map the codebase (replaces ls -R)
semble_rs tree ./my-project --symbols

# Find code by what it does (replaces grep + cat)
semble_rs search "how is auth handled" ./my-project --outline

# Compress build / CI output before reading it
cargo build 2>&1 | semble_rs digest
gh run view <id> --log-failed | semble_rs digest

For agent integration (Claude Code, Codex, Cursor), see Agent integration.

Main Features

• Fast: indexes the local repo (22 files) in ~150 ms, ~10 s on 1,600 files. Static embedder — no transformer forward pass at query time.
• Token-efficient: tree collapses ls -R by 4×–747×; --outline is -47% vs full output; digest reaches -98.9% on real GitHub Actions logs.
• Hybrid retrieval: BM25 + Model2Vec embeddings fused with RRF, then reranked with definition / identifier-stem / file-coherence boosts and noise penalties.
• Dependency graph: deps / impact show what a file imports, defines, and what changes if you touch it. Optional Graphviz --dot output.
• Build / CI compression: digest auto-detects cargo, pnpm/npm/yarn/bun, tsc, pytest, go test, gradle, ruff, mypy, clang/gcc/cmake/make/swiftc, GitHub Actions.
• Single binary: no Python, no daemon, no API keys. Runs on CPU.

Search

semble_rs search "auth flow" ./my-project --outline    # pass 1: structural overview
semble_rs search "loginWithEmail" ./my-project --compact   # pass 2: matching lines
semble_rs search "save model" https://github.com/MinishLab/model2vec   # git URL

path defaults to the current directory; git URLs are accepted (cloned shallow).

Output modes

Recommended: --outline to overview → --compact to narrow → --json --strip only if the chunk body itself is needed.

find-related

Given a file:line from a previous search result, returns chunks semantically similar to that location.

semble_rs find-related src/auth.rs 42 ./my-project

plan

When the agent doesn't know where to start, plan runs a small search and prints a recommended sequence of --outline / --group / --compact / deps / impact commands.

semble_rs plan "fix auth flow bug" ./my-project -k 5

plan is a guardrail, not an oracle: low-confidence candidates are leads, not facts. Skip it when the symbol or feature name is already known.

--model

All search-side commands accept --model <hf-repo-or-local-path> to override the default embedder. Also honours the SEMBLE_MODEL_PATH environment variable.

Tree

semble_rs tree prints the codebase file tree using the same gitignore-aware index as search. It exists because ls -R on a real project explodes into tens or hundreds of thousands of tokens (.git/, target/, node_modules/ all included). Measured on real repos:

semble_rs tree                              # current directory
semble_rs tree -d                           # directories only
semble_rs tree --max-depth 2                # cap depth
semble_rs tree --symbols                    # append top-level symbols per file
semble_rs tree --lang rust,python           # filter by language

Digest

semble_rs digest collapses build / test / install / CI output. Errors, file:line:col, tracebacks, panic stacks, and failed-step bodies are always preserved — only progress lines collapse to counts.

cargo build 2>&1            | semble_rs digest
pnpm install 2>&1           | semble_rs digest
pytest 2>&1                 | semble_rs digest
gh run view <id> --log-failed | semble_rs digest

Measured on 15 real-world fixtures:

Auto-detection covers cargo, pnpm/npm/yarn/bun, tsc, pytest, go test, gradle, ruff, mypy, clang/gcc/cmake/make/swiftc, GitHub Actions. Force a handler with --format <name>; inspect with --show-format.

Dependency graph

semble_rs deps   src/auth.rs ./my-project                  # what this file imports / defines (flat)
semble_rs deps   src/auth.rs ./my-project --tree           # transitive imports as ASCII tree
semble_rs deps   src/auth.rs ./my-project --tree --max-depth 3
semble_rs deps   src/auth.rs ./my-project --dot | dot -Tpng > deps.png
semble_rs impact src/auth.rs ./my-project                  # who depends on this file (flat list)
semble_rs impact src/auth.rs ./my-project --tree           # reverse-dependency tree
semble_rs impact src/auth.rs ./my-project --dot | dot -Tpng > impact.png

--tree (v0.9.1+) renders forward (deps) or reverse (impact) dependencies as an ASCII tree with cycle detection (repeated nodes marked (cycle)) and --max-depth N truncation (…). No external tool required, agent-readable.

impact is intended to be run before edits to a shared module to avoid surprises.

find-pattern

Thin wrapper around ast-grep for structural queries that semantic search can't express:

semble_rs find-pattern 'fn $name($$$)' . --lang rust --compact

Requires ast-grep installed (brew install ast-grep or cargo install ast-grep).

Encode

semble_rs encode exposes the embedding model as a CLI for scripting and debugging:

semble_rs encode "search result scoring"            # one vector → JSON array
echo -e "auth\nlogin\ntoken" | semble_rs encode     # stdin, one sentence per line
semble_rs encode "x" --model minishlab/potion-multilingual-128M

Agent integration

Append a snippet like the following to your project-root CLAUDE.md or AGENTS.md. It works for Claude Code, Codex, Cursor (.cursorrules), Aider, and OpenHands.

## Code search and exploration

Use `semble_rs` instead of `ls -R`, `grep`, `cat`:

​```bash
semble_rs tree . --symbols                         # codebase map (cheap)
semble_rs search "<feature or symbol>" . --outline # pass 1
semble_rs search "<feature or symbol>" . --compact # pass 2
semble_rs deps   <file> .                          # what file imports / defines
semble_rs impact <file> .                          # files affected by changes
​```

Compress noisy command output before reading it:

​```bash
cargo build 2>&1 | semble_rs digest
pnpm install 2>&1 | semble_rs digest
gh run view <id> --log-failed | semble_rs digest
​```

semble_rs savings shows estimated tokens saved across past searches.

How it works

semble_rs chunks every file with tree-sitter at function / class / module boundaries (line-based fallback for unsupported languages), then scores every query with two complementary retrievers: static Model2Vec embeddings (default minishlab/potion-code-16M) for semantic similarity, and BM25 for lexical matches on identifiers and API names. Score lists are fused with Reciprocal Rank Fusion.

After fusion, results are reranked with code-aware signals:

The embedder is fully static (vocab embedding lookup → mean pool → SIF weighting → L2 normalize). All of this runs in milliseconds on CPU.

Benchmarks

Retrieval quality — 100-query benchmark (this repo)

100 hand-labelled queries across 5 categories: exact symbol names, natural-language feature descriptions, scenarios, acronyms, and Korean queries. Default model minishlab/potion-code-16M.

Query set: docs/eval_set_100.json · per-miss analysis: docs/benchmark_100.md.

Indexing and query latency by repo size

The index is rebuilt every run (no persistent cache).

digest is independent of repo size: 3.3 MB CI log → 35 KB in ~20 ms.

Token efficiency vs native shell tools

Measured on real projects:

Agents using grep + cat + ls -R spend most of their context window on irrelevant code and noise. semble_rs returns only what matters and compresses the rest.

Supported languages

License

MIT

Acknowledgements

• MinishLab/semble — original Python implementation by Stéphan Tulkens and Thomas van Dongen. semble_rs is a Rust port + superset of their work.
• Model2Vec and model2vec-rs — static distillation framework powering the embedder.
• Embedding model: minishlab/potion-code-16M.