research: reconstruct ChatGPT import provenance for #60

[Skywind5487]

Summary

Research-only evidence for #60: reconstruct how the ChatGPT export/history was merged into an already-existing Hermes state.db, including earlier partial/test import generations, final-pass behavior, message replay side effects, native-row preservation, and final fidelity limits.

Ground truth: Hermes runtime began 2026-05-29; 2026-06-16 is the ChatGPT history import/merge date, not Hermes adoption.

Refs #60 #58 #54.

Added

Read-only, fail-closed evidence tools:

• research/chatgpt-import/extract_issue60_evidence.py
• research/chatgpt-import/extract_issue60_stage2.py
• research/chatgpt-import/extract_issue60_stage3.py
• research/chatgpt-import/extract_issue60_stage4.py
• research/chatgpt-import/extract_issue60_stage5.py

Durable public artifacts include:

• research/chatgpt-import/issue60/README.md
• research/chatgpt-import/issue60/evidence-locators.md
• research/chatgpt-import/issue60/evidence-manifest.json
• research/chatgpt-import/issue60/recovered-import_final.py
• research/chatgpt-import/issue60/import-final-run-output.txt
• research/chatgpt-import/issue60/stage4-findings.md
• research/chatgpt-import/issue60/stage5-run.md

Raw historical transcripts/evidence bundles remain private/local because this repo is public; exact locators and SHA identities are retained instead.

PROVEN so far

Existing Hermes DB

Historical representative-import evidence records discord=620 / cron=252 / cli=15 / chatgpt-export=57, proving the migration was into a populated Hermes DB. Stage 3 independently places the first surviving ChatGPT message at messages.id=92691, after messages from 883 distinct native sessions.

Exact final importer

Historical message 188425 contains the exact /tmp/chatgpt-export/import_final.py; 188428 contains its exact final run output.

The final importer linearizes only the active ChatGPT branch:

current_node
→ mapping.parent backward
→ skip null/non-message nodes
→ reverse
→ one Hermes session

Final recorded run:

loaded conversations        5447
existing conversation IDs   4654
considered missing           793
new sessions                 792
new messages                3322
final chatgpt-export        5447
final imported messages    89434
duplicate conversation IDs     0

The one remaining v1 orphan 20231013_125540_0041efa3 is subsequently repaired from the export.

Earlier importer generations survive final dedup

Stage 4 proves all 5,447 final imported sessions have a conversation ID and all satisfy session.id == UTC(started_at) + conversation_id[:8].

Surviving field shape splits exactly:

user_id == conversation_id      792
user_id != conversation_id     4655

That matches the historical final-pass count of 792 newly inserted sessions versus 4,654 existing conversation IDs + the subsequently repaired orphan. Treat this as strong importer-generation provenance; the final script's field mapping is not the field mapping of every final row.

Legacy pre-final metadata survives widely, including:

chatgpt_memory_scope        4254
chatgpt_is_study_mode       4254
chatgpt_is_do_not_remember  3143
chatgpt_voice                 17
chatgpt_pinned_time             2

Native-row collision/relabel path: not observed

Stage 3 finds zero chatgpt-export rows carrying audited native Hermes fingerprints (lineage/end/cwd/routing/tool/Discord-platform shapes). Stage 4's apparent 4,655 residuals are all only user_id_mismatch; there are zero generated-ID, missing-conversation-ID, native-field, or tool residuals.

There is therefore no surviving evidence that the final importer's IntegrityError -> UPDATE source='chatgpt-export' path relabelled a normal populated Hermes-native session. This is strong negative evidence, not a claim of logical impossibility.

Message replay side effect — PROVEN

1,174 imported sessions have stale stored message_count. Every affected session is an exact replay multiple:

3x   32 sessions
4x  618 sessions
5x  524 sessions

Physical excess over stored counts = 14,367 message rows.

Duplicate ChatGPT UUID accounting explains it exactly:

same-session extra UUID copies = 14,365
orphan empty-platform-ID rows  =      2
                                 -------
                                 14,367

So the final DB has 89,434 physical imported message rows versus 75,067 summed stored session counts because some import passes replayed entire message sequences.

The only empty/non-UUID imported platform IDs are two rows in the repaired v1 orphan with placeholder timestamps 1.0 and 2.0.

Final DB-only closure pass

Stage 5 is intentionally narrow. It only:

1. classifies the 953 duplicate ChatGPT UUID groups that span multiple imported sessions as exact clones vs conflicting data;
2. inspects the repaired orphan using hashes/metadata rather than publishing raw content;
3. records surviving imported message-field presence.

It does not restart broad provenance archaeology.

Remaining source-side fidelity question

The exact importer structurally drops or transforms source data it does not map, including alternate branches, non-string content.parts, content references/attachments/tool structures, and null/non-message mapping nodes. Quantifying data that never entered Hermes requires the original ChatGPT export or equally strong historical source evidence.

Canonical DB safety

/home/skywind/hermes-recovery/runs/20260807-081043/state.recovered.patched.db
SHA-256 23cfa3c8adb94ed403058329ae7e252e1d4c4bc01ead76e22ac7d0ff99948104

Expected counts: 7,268 sessions / 231,513 messages / 78 gateway-routing rows.

All extractors use immutable/query-only access and verify source identity/hash before and after.

Merge status

Still draft until Stage 5 resolves the last DB-only duplicate/orphan questions and #60 records the remaining source-side fidelity limits. No production code is changed.

[Skywind5487]

Stage 3 provenance-shape explorer added in 6eb6d36c67d0632cc77df7650cb38b0f4103347d.

Purpose: test the remaining pre-2026-06-16 lineage anomaly with independent fingerprints rather than more keyword archaeology:

• messages.id distribution as an insertion-order proxy versus the first final chatgpt-export message row;
• pre-cutoff tool-call names / earliest occurrences;
• platform-message-ID shape only (raw IDs are not emitted);
• cross-session session_search edges;
• pre-cutoff source / model / end_reason / parent-lineage distributions.

This was smoke-tested against a synthetic DB before commit. It retains the same pinned-SHA, mode=ro&immutable=1, PRAGMA query_only=ON, before/after identity/hash checks.

Important evidence motivating this pass: stage 2 already exposes pre-cutoff sessions such as 20260531_122400_196f4321 containing Hermes-specific tool activity (skill_view, session_search, etc.). Stage 3 is meant to determine whether the combined DB shape supports those rows having existed before the June 16 ChatGPT import, rather than being a later relabel/reinsert artifact.