fix(ctx_expand): fall back to reference_store for ref_-prefixed IDs by omar-khallaf · Pull Request #498 · yvgude/lean-ctx

omar-khallaf · 2026-06-22T06:30:17Z

Summary

ctx_expand only queried the on-disk archive store, so any output stored by the reference_results feature (ref_-prefixed IDs, in-memory) returned 'not found or expired.'

Fix: route by ID prefix — ref_ → reference_store, hex → archive. Extract shared format_range/format_search/format_json_keys into archive.rs so both paths use the same gutter/JSON formatting.

What does this PR change and why?

Test plan

cd rust && cargo test
cd rust && cargo clippy --all-targets --all-features -- -D warnings
cd rust && cargo fmt --check

ctx_expand only queried the on-disk archive store, so any output stored by the reference_results feature (ref_-prefixed IDs, in-memory) returned 'not found or expired'. Fix: route by ID prefix — ref_ → reference_store, hex → archive. Extract shared format_range/format_search/format_json_keys into archive.rs so both paths use the same gutter/JSON formatting. Signed-off-by: Omar Mohamed <mohamed.omar67492@gmail.com>

Hindsight-inspired epistemic structure for the knowledge store: - Epistemic typing: remember() now infers KnowledgeArchetype from the fact category (evidence vs. inference) instead of default(); infer_from_category gains provider categories. Feeds salience ranking immediately. (GL #803) - Deterministic observation synthesis: new background cognition step clusters raw facts per entity and writes condensed Observation facts via remember() (source = cognition-synthesis), so idempotency + versioning come for free and the hot-path recall stays byte-stable (#498). Optional LLM refinement via llm_enhance, deterministic by default. Wired as step 9 + introspect subsystem observation_synthesis. (GL #804) - Recall priority: synthesized observations get a balanced tier boost in recall_for_output / recall_by_category_for_output (not an absolute override). (GL #805) - Config archetype_aware_decay (off by default) gives evidence a conservative Ebbinghaus stability multiplier; cognition_synthesis_min_cluster gates the cluster threshold. Schema + generated config-keys regenerated. (GL #806) - Tests: archetype inference, deterministic synthesis + versioning, recall order, archetype-aware decay. Also serialized three flaky proxy tests on test_env_lock so the tee path no longer races on LEAN_CTX_DATA_DIR. Co-authored-by: Cursor <cursoragent@cursor.com>

…sm tests (yvgude#361) Foreign harnesses namespace their tools (forge_read, pi.shell, fs:grep), which the substring classifier missed — so a foreign source read risked compression and a foreign shell log went untouched. classify_tool_name now also matches the name's path-like segments as whole words (FileRead first, so a read is never misclassified), with negative tests guarding against substring false positives (thread/spread/readme). Adds end-to-end tests: a forge_* shell log compresses, a forge_read source body survives intact, and identical request bodies compress byte-for-byte (keeps the provider prompt-cache prefix stable, yvgude#498). Co-authored-by: Cursor <cursoragent@cursor.com>

The v3.8.3 fix added `type_ref` edges so C#/Java types consumed without a `using`/import (same-namespace/package visibility) are no longer false-negative leaves. Those edges only exist in a freshly built graph, but `ctx_impact` rebuilt the property graph only when it was completely empty — so after upgrading, an existing graph (built before the edges existed) was served unchanged and the consumer still reported "no impact". Stamp the graph with the engine generation that produced it (`engine_version` + `built_with` in `graph.meta.json`) and add `engine_outdated()`. The query paths (`analyze`/`diff`/`chain`) now open via `open_graph_fresh`, which rebuilds once when the graph is empty OR predates the current engine, then serves it. A fresh build re-stamps the version, so steady state never rebuilds. Tests: unit guard for `engine_outdated` + an upgrade-simulation E2E that stamps a built C# graph back to engine 0 and asserts the next `analyze` self-heals and surfaces the DI consumer. Determinism (yvgude#498) preserved: the stamp lives only in the meta sidecar, not in tool output bodies. Co-authored-by: Cursor <cursoragent@cursor.com>

… methods Close the two deferred yvgude#398 follow-ups (GL yvgude#640-yvgude#643) so ctx_impact builds the real C# blast radius in both builder paths. Namespace-aware resolution (yvgude#641): - TypeDef now carries its C# namespace (block/file-scoped/nested) via deep_queries::type_defs::csharp_namespace_for. - type_ref_targets resolves hybridly: a definer in the consumer's visible namespace (own ns + enclosing prefixes + usings, csharp_visible_namespaces) always links and discards out-of-namespace homonyms; with no visible match the global fallback links, with the too-generic cap raised 3 -> 5. - Java (no namespaces) keeps the fallback path unchanged. Extension methods (yvgude#642): - New deep_queries::ext_methods extracts C# `this`-parameter methods into an ExtMethodIndex; insert_ext_method_edges links `value.Foo()` calls to the defining file (file + symbol TypeRef edge), self-filtered and capped. Both wired into index_graph_file_embeddings and index_graph_file_minimal. New unit + e2e regressions are gated on tree-sitter so they exercise both builder paths. Outputs stay deterministic (sorted/deduped, bounded; yvgude#498). Co-authored-by: Cursor <cursoragent@cursor.com>

…onses & Gemini Make "own the whole context window" host-independent — the proxy is the universal engine that reaches every tool_result, not just the ~5% the ctx_* tool layer can wrap. - OpenAI Responses API: add cache-aware history pruning (was compression-only) while preserving function_call / function_call_output pairing. - Gemini: add history pruning + real tool-id routing into the shared compressor (was compress_tool_result(.., None), no pruning). - Determinism (yvgude#498) + cache-prefix byte-stability tests for both new rails. - OpenCode shadow_mode: guard that the bundled plugin actually routes bash/read/grep/glob/edit through the compressed ctx_* tools (R1 regression "102 native bash / 0 ctx_shell"). Co-authored-by: Cursor <cursoragent@cursor.com>

Let users delegate edits (and colorized diffs) to their host editor's native UI. With prefer_native_editor = true (or LEAN_CTX_PREFER_NATIVE_EDITOR=1) the edit tool (ctx_edit) is removed from disabled_tools_effective() so it is not advertised in list_tools, and dispatch_inner() refuses it (direct or via ctx_call) with a re-enable hint. Read/search/shell/memory tools are unaffected. Wires the config field + schema entry + generated config-keys doc; adds prefer_native_editor tests. Colorized diffs stay a host concern (byte-stable tool output, yvgude#498). Co-authored-by: Cursor <cursoragent@cursor.com>

…ude#498, yvgude#361) Make ctx_read output a pure function of (content, mode, crp_mode, task) so provider prompt caching applies: drop the time-relative scent foreign-claim hint and the read-count/proof lines from the response body, and remove the now-unused cache_hit_proof_line helper. Apply cap_to_raw to every lossy whole-file summary mode (auto and explicit) before caching via mode_allows_raw_cap, so a framed read never costs more than the raw file and re-read hits stay byte-stable.

…+ two-sided overhead Two parts, both about telling the truth in the savings/ROI numbers. Part A — model-correct ledger token counting The verified savings ledger now denominates tokens in the *active model's* tokenizer family instead of always assuming o200k. `ledger_family()` resolves the family once per process from the same model key `model_and_price()` uses; `count_for_ledger()` re-tokenizes in that family. For the default O200kBase model (OpenAI/Cursor/unknown) this is byte-identical to the old path at zero extra cost — only a resolved Claude/Gemini/Llama run triggers re-tokenizing. The ledger event label now reflects the family actually measured. The ledger record call is decoupled from the heatmap (which keeps its o200k counts) and made explicit at the three read chokepoints (tool_lifecycle, ctx_multi_read, ctx_read bg thread). The read/multi-read paths re-tokenize raw source (from cache) + sent output for non-o200k families, falling back to o200k on a cache miss (conservative). Tool-output framing/footers stay on count_tokens (o200k) to keep outputs byte-stable for provider prompt caching (yvgude#498) — count_for_ledger is internal-only. Fixes two ledger-arg bugs found while wiring: ctx_search passed `saved` instead of the sent-token count as actual_tokens (inverted savings), and server_metrics double-recorded ctx_shell with a delta where the output count belonged. Part B — two-sided overhead in the reported NET net_of_injection() and observed_turns() move to core::context_overhead so the gain path AND the verified ledger/ROI reconcile against the same math. RoiReport carries the per-turn injection, observed turns, total tax, and signed net-after-overhead (runtime annotations, NOT part of the signed batch — the batch stays byte-reproducible from the event chain). `lean-ctx roi` and `lean-ctx savings` now show the overhead-adjusted net, but only when the proxy actually observed turns — we never guess unseen turns, so without the proxy in path net == gross. Tests: tokenizer family detection for ledger model keys, real-BPE ledger counts, label/family agreement, RoiReport overhead population + headline, and roi/savings display of the injection line. cargo fmt + clippy -D warnings clean; full lib suite green (5392 passed). Co-authored-by: Cursor <cursoragent@cursor.com>

…vgude#683) The `auto` mode cascade mixed deterministic capability guards with adaptive learning signals (predictor, bandit, heatmap, adaptive policy, bounce/path memory) that read several on-disk stores per read and made the chosen mode a function of accumulated state — non-reproducible and, worse, a defeater of provider prompt caching (yvgude#498), which only discounts byte-stable inputs. Split the two: - The default cascade is now a pure function of (file, task): capability guards (instruction/binary/cache/small/config/diagnostic/suspect) + the deterministic intent and size heuristics. No per-read learning-store I/O. - The adaptive block moves behind the new `auto_mode_learning` flag (default off; env `LEAN_CTX_AUTO_MODE_LEARNING`), extracted into `resolve_adaptive`. Opt-in restores the previous behavior exactly — no code was deleted, so it is fully reversible. Net effect by default: reproducible mode selection, less disk I/O per read, and byte-stable output that maximizes prompt-cache hit rate. One test (`cached_compressed_only_file_does_not_escalate_to_full`) encoded the old learning-on default (predictor compressing a medium file); resized it so the deterministic heuristic itself compresses (large code -> map), keeping the real invariant under test — the cache_hit shortcut never fires for an entry whose full body was not delivered. All 5423 lib tests pass; clippy clean. Co-authored-by: Cursor <cursoragent@cursor.com>

…nsity (yvgude#714-yvgude#718) A single `aggressiveness` knob (0.0 lossless – 1.0 max) mapped onto the existing read modes, mirroring The Token Company's intensity control while staying local and deterministic. - yvgude#714 core/aggressiveness.rs: AggressivenessProfile::from_level (density_target / bpe_entropy / ib_budget_ratio), effective() resolution (arg > env > config), cache_fragment() bucketing. - yvgude#715 config field `compression_aggressiveness` + schema (LEAN_CTX_AGGRESSIVENESS) + get_f64 arg helper + local-config merge. - yvgude#716 ctx_read `aggressiveness` arg + ReadTuning threaded render → engine → handler; `auto` + aggressiveness routes through the proven density path. - yvgude#717 density / task / entropy renderers honour the knob (density_target, ib_budget_ratio, bpe_entropy override). - yvgude#718 compressed_cache_key carries the aggressiveness bucket; determinism + monotonicity tests. Determinism (yvgude#498): token_entropy_from_ids summed Shannon terms in HashMap order (f64 non-associative) → non-deterministic density/entropy output across processes. Switched to canonical sorted-count summation; output is byte-stable, preserving prompt-cache hits. Tests: clippy clean; full lib suite 5451 passed / 0 failed. Docs: vs-token-company comparison + competitive-improvements design spec. Co-authored-by: Cursor <cursoragent@cursor.com>

…afe markers + protect param (yvgude#719-yvgude#722) New core/protect.rs module gives agents two deterministic ways to force-keep content through the compression pipeline: - protect: ["token", …] on ctx_read force-keeps every line containing a needle across entropy filtering, dedup, and the information-bottleneck selector. - <lc_safe>…</lc_safe> universal markers preserve spans verbatim in the shell and proxy compression paths (single savings footer over the combined result). Threaded force_keep through entropy.rs and task_relevance.rs; protected lines score f64::INFINITY in IB and expand the budget by their count so they never displace other content. compressed_cache_key now folds in a canonical, order/dup-independent protect fragment so cache stays deterministic (yvgude#498). Tests: force-keep in entropy + IB, empty-protect byte-identity regressions, cache-key distinguishes protect. Docs regenerated for the new ctx_read param. Co-authored-by: Cursor <cursoragent@cursor.com>

…m/user compression (yvgude#723-yvgude#726) Sharpen the gateway/proxy by compressing natural-language prose in the frozen, cache-aware request region across all three rails (Anthropic, OpenAI, Gemini). Opt-in per role, off by default, byte-stable. yvgude#723 core/config: RoleAggressiveness {system,user} + ProseRole + resolved_role_aggressiveness() (env > config, clamped to [0,1]), schema section [proxy.role_aggressiveness], generated docs. yvgude#724 proxy/prose.rs: deterministic, anti-inflation prose squeeze with a conservative prose gate (sentence + symbol density). Wired into Anthropic system/user, OpenAI system/developer/user, Gemini systemInstruction/user — only inside [cached_prefix_len, boundary). yvgude#725 Assistant/model turns pass through verbatim on every rail (tested). yvgude#726 proxy/cache_safety.rs: cache_safe_ratio telemetry surfaced in /status; every frozen-region rewrite recorded as cache-safe. Cache safety (yvgude#448/yvgude#498): never rewrites behind a client cache_control breakpoint, never touches the live tail, and is a pure function of (text, aggressiveness) so the provider prompt-cache prefix stays valid. clippy --all-features -D warnings: clean. cargo test --lib: 5486 green. Co-authored-by: Cursor <cursoragent@cursor.com>

…(NO-GO for now) Epic D (yvgude#711) reality check: query-conditioned IB prose already ships in the entropy read mode (yvgude#542), so yvgude#728 is satisfied. The only model question left is 4B — a local delete-only prose classifier (TTC Bear-2 parity). This spike evaluates it against lean-ctx's invariants (determinism yvgude#498, 100% local, binary size, latency, license) with real candidates (LLMLingua-2: XLM-RoBERTa-large/mBERT, token classification, 512-tok, MIT, ONNX exports). Recommendation: NO-GO for now. Measure first (Accuracy@Rate vs the deterministic IB prose), gate any build on Epic 5a evidence + a cross-platform determinism conformance test. No model code until the gate is cleared. Adds docs/superpowers/specs/2026-06-19-prose-model-spike.md and a pointer from the parent design (Epic 4.3). Co-authored-by: Cursor <cursoragent@cursor.com>

…suite + cache-preservation ratio (yvgude#730-yvgude#732) yvgude#730: rust/eval/accuracy-suite.ndjson (needle-in-haystack, long-context QA, code-edit) over rust/eval/accuracy/ corpora. Every gold answer is a real, verifiable lean-ctx fact (no mock data). A model-free in-process test (eval_ab::accuracy_suite_tests) proves the deterministic floor of the "compressed >= raw" claim: lean-ctx's own assemble(LeanCtx) context must still contain a gold answer (SQuAD containment over the context itself), and the code task is provably solvable (reference solution passes the committed unit test, the stub fails). yvgude#731: --gate and --margin were already wired (ReportConfig.noninferiority_margin -> verdict_for over the bootstrap CI lower bound; --gate exits non-zero on Regressed) and even CI-enforced via the committed recording. Only the suite was missing — closed with evidence; design annotated. yvgude#732: cache_preservation_ratio added to DualArmScorecard (cache_read / billed), where the cold/warm token totals live. Derived from values already in the determinism_digest, so it is additive and digest-stable (yvgude#498); surfaced in to_human + JSON. Co-authored-by: Cursor <cursoragent@cursor.com>

… is enabled (yvgude#463) When an agent re-requests `full` or `lines:N-M` for a file it already read this session and the file changed on disk, return `mode=diff` instead of re-emitting content the model already holds; when unchanged, a `lines:` re-read of a fully-delivered file collapses to the full-mode `[unchanged]` stub. This enforces, on explicit re-reads, the RE-READ discipline lean-ctx already recommends. Opt-in and default OFF: config field `delta_explicit` plus the `LCTX_DELTA_EXPLICIT=1/0` env override (env wins in both directions, mirroring `no_degrade_effective`). First reads are unaffected and `fresh=true` always bypasses. - Staleness uses the verified mtime+md5 `is_cache_entry_stale_verified`, not mtime alone, so a same-second write on a coarse-granularity filesystem cannot be mistaken for "unchanged" and yield a misleading empty diff. - The diff base is the full cached source (via `handle_diff`), never a compressed map/signatures view. - The `[delta-explicit]` advisory and the diff carry no timestamps/counters, so identical inputs stay byte-stable (yvgude#498). The decision is factored into the pure, testable `ctx_read::resolve_explicit_delta_mode`; the MCP dispatcher calls it under a cache read-lock before the `lines:`->fresh guard so a changed-file `lines:` re-read can still be diverted to a diff. Tests: config round-trip + env-override (both directions); ctx_read behavior matrix (changed->diff, unchanged `lines:`->stub, OFF preserves current behavior, fresh bypasses, first read unaffected, auto-mode never diverted, diff base is full source, decision byte-stable). Schema + feature-catalog get a concise opt-in note. Co-authored-by: Parker Brown <267364116+parker-brown-family@users.noreply.github.com>

…vgude#739) Drop-in compress(messages, model) parity: rewrites every text payload in a chat-style messages array through the shared compression funnel and returns the rewritten messages plus a structured token-savings summary. - New proxy::compress_api module (messages-in/out; OpenAI + Anthropic shapes) - Footer-free bodies via protocol::strip_trailing_savings_footer; savings live in stats only -> byte-stable output for prompt caching (yvgude#498) - ctx_* tool results pass through verbatim (yvgude#479); images/tool_use untouched - 9 unit tests: determinism, footer-free, text/tool_result block handling Co-authored-by: Cursor <cursoragent@cursor.com>

…de#744) - determinism: yvgude#498 regression for /v1/compress (mixed conversation, byte-identical across N runs, footer-free) plus a daemon-free, #[ignore]d o200k_base benchmark over a real corpus (proxy::compress_api::tests). - benchmark: bench/compress/benchmark.py runs lean-ctx vs Headroom over the same real corpus with the same tokenizer, emitting JSON (ratio + latency). Measured-only — a missing tool/daemon is reported available:false, never estimated. Real lean-ctx point: 27 docs files, 69594->57615 tok (17.2%). - docs: compress() SDK cookbook (Py+TS) + docs/comparisons/vs-headroom.md, linked from both package READMEs and the comparisons index. Co-authored-by: Cursor <cursoragent@cursor.com>

Wire ten neuroscience/physics-motivated mechanisms to real hot-path call sites, deterministic by default (Rule yvgude#498 / prompt-cache intact), and make them provable via a new `lean-ctx introspect cognition` activity registry. - Phase 0: introspect registry + CLI + doctor; is_stochastic_enabled() gate - yvgude#2 time-variant Phi (EMA re-read); yvgude#1 Ebbinghaus decay + spacing effect - yvgude#3 Hebbian eviction + CLS consolidation (previously dead code, now wired) - yvgude#5 integration-aware Phi (greedy MMR) + content-based dedup fix - yvgude#6 global-workspace ignition; yvgude#4 learned field weights (bandit, argmax default) - yvgude#7 idle replay (sharp-wave-ripple); yvgude#9 FEP prefetch (active inference) - yvgude#8 immune detector vs context poisoning (coupled to workspace trust) - yvgude#10 QUBO selection spike behind LEAN_CTX_EXPERIMENTAL_QUBO (greedy stays default) Stochastic exploration gated behind LEAN_CTX_STOCHASTIC. Docs + CHANGELOG added. GitLab: epic root/lean-ctx#771 + foundation yvgude#772 + features yvgude#773-yvgude#782. 5653 lib tests green, zero clippy warnings. Co-authored-by: Cursor <cursoragent@cursor.com>

yvgude merged commit 79dc05e into yvgude:main Jun 22, 2026
22 of 24 checks passed

github-actions Bot locked and limited conversation to collaborators Jun 22, 2026

omar-khallaf deleted the fix/ctx-expand-reference-store-fallback branch June 23, 2026 22:54

Navigation Menu

Uh oh!

fix(ctx_expand): fall back to reference_store for ref_-prefixed IDs - #498

fix(ctx_expand): fall back to reference_store for ref_-prefixed IDs#498
yvgude merged 1 commit into
yvgude:mainfrom
omar-khallaf:fix/ctx-expand-reference-store-fallback

omar-khallaf commented Jun 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

omar-khallaf commented Jun 22, 2026

Summary

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants