## Introduction
When using AI programming assistants (e.g., Claude Code, DeepSeek Harness, Pi, etc.), long-running sessions are the biggest pain point. Tool outputs accumulate continuously, quickly filling up the context window—either getting truncated or being compressed in one go into a lossy summary, after which discarded content is irretrievable.
**billion-context** is an open-source context compression plugin. In 4.5 months of production use, it handled **18.76B cumulative input tokens** across 174,327 model calls (approximately 24.7B tokens across three hosts), achieving **zero overflow** on models with a 204,800-token context window; marathon sessions ran for 8,584–12,049 calls.
Its core insight is: **a 200K context window is sufficient**—no need to enlarge the window, add VRAM, or adopt RAG.
---
## Core Architecture: Proxy Middleware
billion-context is not a module embedded inside an agent but rather a **local proxy server**, intercepting all communication between the agent and the model API:
```
Agent (Claude Code / DSH / Pi / ...)
↓ Normal request sending
billion-context proxy (localhost:8787)
↓ Intercept, compress, forward
Model API (Anthropic / OpenAI / ...)
```
This means **any agent that supports setting a base URL can integrate seamlessly**, without modifying the agent’s source code.
---
## Compression Principle: The Fold
### 1. Context Layout
billion-context divides the context window into three zones:
```
┌─────────────────────────────────────┐
│ Distilled Prefix │ ← Compressed summary blocks (tier-1/2/3)
│ Occupies ~8% of context space │ Decompressible block-by-block
├─────────────────────────────────────┤
│ Raw Recent Zone │ ← Raw dialog of the most recent ~50K tokens
│ Most recent messages, uncompressed │ Full fidelity preserved
├─────────────────────────────────────┤
│ Protected Zone │ ← First user message (task intent)
│ System prompt, last user message │ Last user message
└─────────────────────────────────────┘
```
**Key design**: Compression applies only to the “consumed but no longer active” middle region—not to the entire history.
### 2. Model-Driven Compression (Not Engine Hard Truncation)
Traditional approaches trigger engine-level summarization once the context hits a threshold—then discard the original text. This is **irreversible, single-granularity, and task-agnostic**.
billion-context takes a radically different approach: **let the model itself decide what to compress, when to compress, and how much to compress**.
Specifically, it injects four tools into the conversation:
| Tool | Function |
|------|----------|
| `compress` | Fold a message range into a detailed summary |
| `decompress` | Restore compressed content on demand (reversible!) |
| `search_context` | Keyword search across compressed summaries and visible messages |
| `cp_status` | View context usage status, including which ranges are compressible |
As context grows, the system injects a **controlled nudge prompt**, presenting the model with a candidate list of compressible ranges (with actual sizes). The model then invokes the `compress` tool to execute compression. Once completed, the injected prompt is immediately removed from context—leaving no trace.
### 3. Multi-Generation Hierarchical Compression (Tier-1 → Tier-2 → Tier-3)
This is billion-context’s most sophisticated design. Compression does not produce one monolithic summary but instead generates multiple **CompressionBlocks**:
```
First compression: message range → tier-1 summary block
Second compression: multiple tier-1 blocks → tier-2 summary block (original message references preserved)
Third compression: multiple tier-2 blocks → tier-3 summary block
```
The paper dubs this **“an LSM tree over prose”**—analogous to database hierarchical compaction:
- Young generation absorbs new writes
- Old generation stores compressed history
- Compaction is incremental—not full reprocessing
Each block carries `effectiveMessageIds`, transitively covering every original message it summarizes. **Even after three layers of compression, you can still trace back to any original message**.
Production data:
- tier-1 occupies 92% of resident space, tier-2 7%, tier-3 <1%
- Yet tier-2+3 collectively represent 37% of historical content
- Median output per compression is only **185 tokens**, with 97.5% ≤ 2K tokens
### 4. Growth-Gated Nudge
Compression is not triggered by fixed thresholds but **only when the "consumed increment" reaches a certain volume**:
- Default growth step: ~50K tokens (tunable)
- Effective lower bound: ~22.5K tokens
- Only newly added content since the last compression is eligible
This implies:
- No repeated compression of the same content (traditional schemes re-summarize ~80% of history each round)
- Each piece of content is folded exactly once during its lifetime, then remains unchanged
- Information retention stabilizes at ~1/8, avoiding compound decay across rounds
### 5. KV-Cache Friendly
Because history is append-only and compression only deactivates suffix ranges, **the KV-cache prefix never invalidates due to compression**:
- Under active usage (call intervals < 2 minutes, accounting for 89% of traffic), cache read share reaches **94.2%**
- Traditional rewrite-style compression invalidates the entire prefix on every pass
---
## 9-Stage Processing Pipeline
Each turn passes through nine pure-function stages:
1. **assign-refs**: Assign stable `mNNNNN` IDs to new messages
2. **sync-blocks**: Project new content onto block boundaries
3. **merge-blocks**: Merge adjacent compressible ranges
4. **prune**: Enforce budget invariants
5. **filter**: Apply visibility rules by role/content type
6. **hide-compress-calls**: Remove noise from compression tool invocations in rendered view
7. **nudge-inject**: Inject compression guidance prompts
8. **emergency-truncate**: Final hard upper-bound safeguard
9. **render-refs**: Output final message list, annotated with per-message reference tags
Each stage is independently testable; the host never directly mutates history.
---
## Three Economic Advantages
### ① Long Sessions + Quality Assurance
- Heavy-load sessions average ~900K tokens of history; compressed down to just ~13K tokens of summary (~70× compression ratio)
- Recall probe scores remain flat; re-fetch calls drop by 63% (2.7× improvement)
### ② Resource Savings
- 94.2% cache read share
- Bounded context shrinks KV-cache resident memory
- Production environment saves 31–54% billed tokens
### ③ Attention Quality
- State-of-the-art models’ attention is most reliable below 200K tokens
- The Fold keeps total context within this range
---
## Per-Session Capacity Limit
For a 1M-token window, theoretical support includes:
- **~28.6B tokens** of sustainable work
- Sustained operation for **106 days**
- Without folding, the same session would hit capacity after ~104 calls (~53M tokens, less than one day)
---
## Comparison with Other Approaches
| Approach | Information Retention Rate | Resident Quality | Issues |
|----------|----------------------------|------------------|--------|
| Sliding-window truncation | R∞ = 0 | Linear | Content outside window permanently lost |
| Threshold-triggered auto-compression | ρ^k → 0 | Geometric decay | Re-summarizes each round; only 3–13% remains after 4 rounds |
| External memory (RAG) | Irreversible at write time | — | Preserves facts, but loses procedural context |
| LCM (lossless recursive) | R∞ = 100% | Never decays | But resident quality never shrinks |
| **billion-context** | **~1/8 (non-compounding)** | **Sublinear growth** | The only approach satisfying both asymptotic retention > 0 and sublinear resident growth |
---
## Paper Information
- **Title**: Model-Driven Incremental Hierarchical Compression: Training-Free Multi-Generational Context Management for Long-Lived Coding Agents
- **Author**: Xianglei Ran
- **Preprint**: v0.2, 2026-09-07
- **License**: MIT License (the paper itself is also open-sourced as part of the codebase)
- **GitHub**: https://github.com/ranxianglei/billion-context
---
## Summary
billion-context’s core innovation is not any single mechanism, but rather a **specific architectural combination**:
1. Proxy middleware (zero-intrusion integration)
2. The Fold layout (distilled prefix + raw recent zone + protected zone)
3. Model-driven compression (not engine hard truncation)
4. Multi-generation hierarchy (tier-1/2/3, analogous to LSM tree)
5. Growth-gated triggering (compress only consumed increments)
6. Reversible compression (any block can be `decompress`ed for recovery)
7. KV-cache friendly (append-only; prefix never invalidated)
This combination enables a 200K context window to sustain multi-month, multi-billion-token long-running sessions—without quality degradation.