Wayfinder: upstream session-lineage retrospective and evidence package

[Skywind5487]

Status — collector/index only; do not post upstream yet

This ticket owns the eventual single large upstream report for the session-search compression-lineage work completed in this fork.

The goal is not another internal decision ticket. It is to produce one coherent upstream-facing issue after the remaining narrow measurements and production implementation settle, so upstream maintainers can see:

• what problem we observed;
• what code/history we traced;
• what alternatives we tried;
• what failed and why;
• what benchmark/data evidence we collected;
• what production-corpus topology looks like;
• what implementation we ultimately selected;
• which semantics/tests appear worth upstreaming independently of our exact implementation.

Do not open the upstream issue yet. Keep this as a wayfinder/assembly ticket until #68 and the final measurement receipts are stable.

Primary internal sources

Current consolidated source of truth:

• [ARCHIVED] Session compression lineage + import provenance — consolidated source of truth #66 — compression lineage + import provenance consolidation.

Implementation/final gate:

• Spec: production compression-lineage resolver — always memo, B=2000, fail-closed safety #68 — selected production implementation contract;
• Session-lineage resolver research lane: validate always-memo + B/cycle receipts #54 / PR research(session-lineage): persist VM decision — Python memo + B=1500 (#54) #55 — benchmark evolution, VM/WSL measurements, real-corpus topology, counterbalanced runner;
• Production integration (BLOCKED on #54): session-search compression-lineage resolver #62 / PR feat(search): use compression-lineage memo resolver (#62) #63 — premature integration prototype; useful implementation evidence, not selected architecture.

Code/provenance/prior art:

• Research #45 execution map: session_search lineage flow, semantics, and provenance #46 / PR docs(research): map session-search lineage execution (#46) #49 — exact execution/code/provenance map;
• Research #45 prior art: lineage, session recall, bounded traversal, and analogous search work #47 / PR docs(research): map session-lineage prior art for #45 #48 — upstream and analogous prior art;
• Research #45 depth-64 provenance: recover the failure intent behind the lineage fuse #51 / PR docs(research): recover lineage depth-64 provenance #52 — depth-64 fuse provenance and bounded-work intent;
• [research] Separate ChatGPT-import lineage provenance from Hermes-native compaction lifecycle #58 — Hermes compression lifecycle, legacy rotation -> in-place transition, tail/drip explanation;
• Memoize session lineage resolution in search/list hot paths #29 / Session search lineage reuse: choose the simplest query-local traversal strategy #45 — historical design-space exploration and algorithm taxonomy.

Corpus/import context where relevant:

• [research] Reconstruct ChatGPT→Hermes DB import/merge from archived chats + fields #60 / PR research: reconstruct ChatGPT import provenance for #60 #61 — ChatGPT import provenance; use only to explain corpus composition and why imported rows must not distort lineage conclusions;
• Production recovery: local state.db rescue, #12 FTS migration, and pinned-runtime cutover #20/Recovery rollout: build and freeze the production-ready database #22 — canonical recovered DB identity and read-only safety contract;
• Wayfinder [DEFERRED]: post-recovery storage hygiene for proven ChatGPT importer replay #67 — separate post-recovery replay/storage cleanup; not part of resolver design except as corpus-quality context.

Required upstream narrative

The final upstream issue should read as a retrospective, not a dump of our internal issue history.

1. Problem in one screen

Explain the production seam:

ranked message candidates
  -> distinct owning sessions
  -> resolve compression lineage
  -> dedupe roots
  -> stop at K

State the repeated-work problem precisely: current seed-retaining recursive SQL repeats shared ancestors, and current fork code also groups generic parent_session_id ancestry more broadly than positive compression-continuation semantics justify.

2. Semantics discovered

Give the compact truth table for:

• legacy compression continuation;
• in-place compaction;
• branch;
• delegation;
• tool child;
• foreign inherited _branched_from / _delegate_from markers;
• missing parent;
• cycle;
• safety-bound exhaustion.

Distinguish upstream-accepted lifecycle/visibility semantics from fork-local winner SQL.

3. Why the real corpus has a tail

Document the lifecycle timeline and production-data regime change:

• legacy rotation created real parent chains;
• observed Hermes-native max depth 14 / lineage size 15;
• May-heavy lineage creation followed by a sharp June/July drop;
• anti-fork / trigger-rate changes plus June 20 in-place introduction and June 25 default-on;
• no fabricated single causal SHA for the exact breakpoint.

Include source/month topology and explicitly separate chatgpt-export imported history from Hermes-native runtime rows.

4. Production data characteristics

Use privacy-safe aggregates only. Include at minimum:

• total session count at the frozen receipt;
• source-stratified counts;
• positive compression-edge counts under accepted semantics;
• depth distribution and explicit tail;
• multi-session lineage counts/sizes;
• mean/p50/p90/p95/p99/max where useful;
• common shallow regime versus rare compatibility tail;
• candidate-prefix/ranked replay evidence if eventually collected.

Do not publish private conversation text, titles, IDs that reveal private content, raw evidence archives, or identifying message contents. Hashes/aggregate locators only where necessary.

5. Algorithms tried

Preserve the useful graveyard, but compress each family to:

approach	shape	why attractive	evidence/result	why rejected/retained

Include at least:

• current recursive SQL baseline;
• per-seed/no-reuse sequential point traversal;
• Python memo/path compression;
• lazy/no-memo-first hybrid;
• Pure TEMP memo;
• Fixed/shared recursive CTE variants;
• rank-priority/shared traversal experiments;
• candidate contraction / staged path reuse where they materially informed the design.

Do not make upstream maintainers reconstruct the taxonomy from old issue comments.

6. Benchmark methodology and mistakes

This is important evidence, not embarrassment. Include:

• synthetic correctness vs workload performance separation;
• e2-micro shared-core burst/sustained attribution and why 100–500 ms cliffs were not resolver/disk cliffs;
• fixed-order same-connection timing contamination and the counterbalanced correction;
• why aggregate averages hid the real tail;
• why fake/random candidate ranking was not treated as real search distribution;
• why B/work safety must be selected from logical work plus pathological bracketing, not one noisy wall-time cliff.

7. Final implementation and trade-off

After #68 lands, summarize:

• selected resolver shape;
• early-K behavior;
• query-local memo semantics;
• bounded-work/fail-closed behavior;
• final constants and their evidence;
• read snapshot/lock strategy;
• what modern candidate/ranking/hydration behavior was preserved;
• before/after benchmark and work counts.

If the final implementation differs from the current #68 hypothesis, update this section from the landed code rather than preserving stale planning text.

8. What upstream may want independently

Separate potentially upstreamable pieces:

• semantic regression tests for branch/delegate/tool vs compression continuation;
• shared root semantics between DB winner selection and current/title exclusion;
• bounded-work behavior that does not turn a fuse into fake lineage identity;
• query-local path compression / simpler resolver if evidence supports it;
• observability/benchmark fixtures where generally useful.

Do not require upstream to accept our entire fork architecture as one bundle.

Evidence quality rules

Classify statements as:

• UPSTREAM-MERGED / accepted;
• FORK-DEV / observed current fork behavior;
• PRODUCTION-CORPUS OBSERVATION;
• BENCHMARK RESULT;
• INFERENCE / hypothesis.

Pin immutable SHAs and exact artifact paths for every load-bearing code/history claim.

Never present open/unmerged upstream PRs as accepted behavior.

Final deliverable

Prepare one upstream-ready Markdown body in this ticket (or a durable repo artifact linked here) containing:

1. executive summary;
2. production/code problem;
3. lifecycle + semantic findings;
4. corpus characteristics;
5. attempted approaches / graveyard;
6. benchmark methodology + results;
7. final implementation;
8. proposed upstream actions/tests;
9. reproducibility / SHA receipt;
10. privacy-safe appendix with internal references.

Then review it against current upstream main immediately before posting so we do not report already-fixed behavior as current.

Exit criteria

• Spec: production compression-lineage resolver — always memo, B=2000, fail-closed safety #68 implementation direction is landed or otherwise final.
• Counterbalanced/crossover measurements used for final constants are recorded.
• Source-stratified/native corpus profile is final enough for public aggregate reporting.
• Every historical/current code claim has an immutable upstream/fork SHA.
• Rejected algorithm families have short, fair evidence summaries.
• Private corpus information is removed or aggregated.
• Current upstream main is rechecked for overlapping fixes immediately before publication.
• One coherent upstream issue body is ready; no upstream issue-spam split across many internal research tickets.

Refs #66 #68 #54 #46 #47 #51 #58 #60 #67 PR #55 PR #61 PR #63.