-
Notifications
You must be signed in to change notification settings - Fork 1
docs(lost-substrate): full Day-1/2/3 corruption-triage substrate continuation (off-trunk; PR #756 squashed Day-0 only) #757
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
AceHack
merged 9 commits into
main
from
corruption-triage/full-substrate-onto-main-2026-04-29
Apr 29, 2026
Merged
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
64bc60f
docs(lost-substrate): peer-verification corrections per Aaron + Amara…
AceHack c968a59
docs(lost-substrate): Day-2 inventory results — 1,109 dangling object…
AceHack e4896e8
docs+memory: corruption triage results + memory file (Aaron + Amara —…
AceHack c4a6aed
docs(lost-substrate): 8d5e67fd corrupt blob is referenced only from d…
AceHack 5bca4a8
docs(lost-substrate): three-bucket reachability + content-equivalence…
AceHack a77b047
docs(soulfile): replace 1.2 MB raw diagnostic dumps with 28 KB extrac…
AceHack 9d3cd58
docs(lost-substrate): resolve PR #757 review threads — read-only fsck…
AceHack 756e8ed
docs(triage-report): MD032 fix — blank line before list (CI markdownl…
AceHack d3ecbf1
docs(lost-substrate): resolve 8 new PR #757 review threads — filename…
AceHack File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
620 changes: 620 additions & 0 deletions
620
docs/lost-substrate/artifacts/2026-04-29-corruption/fsck-connectivity.txt
Large diffs are not rendered by default.
Oops, something went wrong.
1,116 changes: 1,116 additions & 0 deletions
1,116
docs/lost-substrate/artifacts/2026-04-29-corruption/fsck-full-no-reflogs.txt
Large diffs are not rendered by default.
Oops, something went wrong.
627 changes: 627 additions & 0 deletions
627
docs/lost-substrate/artifacts/2026-04-29-corruption/fsck-full.txt
Large diffs are not rendered by default.
Oops, something went wrong.
9 changes: 9 additions & 0 deletions
9
docs/lost-substrate/artifacts/2026-04-29-corruption/gc-config-snapshot.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # git gc config snapshot — 2026-04-29 corruption triage | ||
| # (read-only inspection; no mutation) | ||
|
|
||
| gc.auto = (unset; default ~6700) | ||
| gc.autoPackLimit = (unset; default 50) | ||
| gc.pruneExpire = (unset; default 2.weeks.ago) | ||
| gc.reflogExpire = (unset; default 90.days.ago) | ||
| gc.reflogExpireUnreachable = (unset; default 30.days.ago) | ||
| gc.worktreePruneExpire = (unset; default 3.months.ago) |
Empty file.
2 changes: 2 additions & 0 deletions
2
docs/lost-substrate/artifacts/2026-04-29-corruption/hour-05Z-show-ref.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| d9feb3f08603e37118491e26527c03e6e5760d5b refs/heads/chore/heartbeat-batch-2026-04-26-hour-05Z | ||
| d9feb3f08603e37118491e26527c03e6e5760d5b refs/remotes/origin/chore/heartbeat-batch-2026-04-26-hour-05Z |
Empty file.
18,150 changes: 18,150 additions & 0 deletions
18,150
docs/lost-substrate/artifacts/2026-04-29-corruption/rev-list-all-objects.txt
Large diffs are not rendered by default.
Oops, something went wrong.
191 changes: 191 additions & 0 deletions
191
docs/lost-substrate/artifacts/2026-04-29-corruption/triage-report.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| # Corruption triage — 2026-04-29 — empirical artifact | ||
|
|
||
| This file captures the exact commands run, the raw outputs (in | ||
| sibling files), and the conclusion drawn. It exists so a future | ||
| reader can verify the prose conclusions in the ledger against the | ||
| artifacts, rather than trusting the summary. | ||
|
|
||
| ## Sibling artifact files | ||
|
|
||
| | File | Source command | | ||
| |---|---| | ||
| | `fsck-full.txt` | `git fsck --full --no-progress` (reflog-inclusive) | | ||
| | `fsck-full-no-reflogs.txt` | `git fsck --full --no-reflogs --no-progress` | | ||
| | `fsck-connectivity.txt` | `git fsck --connectivity-only --no-progress` | | ||
| | `rev-list-all-objects.txt` | `git rev-list --objects --all` | | ||
| | `rev-list-all-objects.err` | stderr from the above (corruption errors) | | ||
| | `gc-config-snapshot.txt` | per-key `git config --get gc.*` snapshot | | ||
|
|
||
| All files are point-in-time evidence; do not edit. | ||
|
|
||
| ## Two corrupt objects under triage | ||
|
|
||
| | SHA | Type | Size | Source path | | ||
| |---|---|---|---| | ||
| | `9bf2daee3ce53c88633824f9532a0158aaa92ed9` | blob (recovered from origin) | 16,455,417 bytes | unknown — large blob | | ||
| | `8d5e67fd313573855848705e4af114f3ff0eecbc` | blob (corrupt loose, locally) | unrecoverable from origin | `docs/hygiene-history/loop-tick-history.md` (intermediate version) | | ||
|
|
||
| ## Three-bucket reachability — `8d5e67fd` | ||
|
|
||
| Per Amara 2026-04-29: `git fsck` reachability is mode-dependent | ||
| (reflog-inclusive vs `--no-reflogs`). A correct classification | ||
| distinguishes three buckets: | ||
|
|
||
| | Bucket | Definition | Verdict for `8d5e67fd` | | ||
| |---|---|---| | ||
| | A | Live branch / tag / ref reachable | **YES** — `refs/heads/chore/heartbeat-batch-2026-04-26-hour-05Z` reaches it (local-only; the same-named remote-tracking ref is stale — origin no longer has the branch) | | ||
| | B | Reflog / stash / local-recovery reachable | Indeterminate from this scan; reflog scan was reflog-inclusive fsck (showed dangling adjacency); `refs/stash` is empty | | ||
| | C | Dangling / unreachable only | **NO** — bucket A applies, so C does not | | ||
|
|
||
| ### Verifying bucket A | ||
|
|
||
| ```text | ||
| $ git rev-list --objects --all | grep 8d5e67fd | ||
| 8d5e67fd313573855848705e4af114f3ff0eecbc docs/hygiene-history/loop-tick-history.md | ||
|
|
||
| $ git for-each-ref ... | per-ref rev-list grep | ||
| ref=refs/heads/chore/heartbeat-batch-2026-04-26-hour-05Z → reaches it | ||
| ref=refs/remotes/origin/chore/heartbeat-batch-2026-04-26-hour-05Z → reaches it (stale) | ||
| ``` | ||
|
|
||
| ### Verifying the remote-tracking ref is stale | ||
|
|
||
| ```text | ||
| $ git ls-remote origin refs/heads/chore/heartbeat-batch-2026-04-26-hour-05Z | ||
| (empty — origin no longer has this branch) | ||
|
|
||
| $ fresh-clone --branch chore/heartbeat-batch-2026-04-26-hour-05Z | ||
|
AceHack marked this conversation as resolved.
Outdated
|
||
| fatal: Remote branch chore/heartbeat-batch-2026-04-26-hour-05Z not found in upstream origin | ||
| ``` | ||
|
|
||
| ### Content-equivalence verification — `8d5e67fd` | ||
|
|
||
| Per Amara's correction: verify squash-preservation by **content**, | ||
| not by ancestry vibe. | ||
|
|
||
| ```text | ||
| $ BR=refs/heads/chore/heartbeat-batch-2026-04-26-hour-05Z | ||
| $ OBJ=8d5e67fd313573855848705e4af114f3ff0eecbc | ||
|
|
||
| $ git ls-tree -r "$BR" | grep "$OBJ" | ||
| (no path-match — corrupt blob is NOT at the branch tip) | ||
|
|
||
| $ git show "$BR":docs/hygiene-history/loop-tick-history.md | head -1 | ||
| # Loop-tick history ← branch tip's tick-history.md is readable | ||
|
|
||
| $ git diff --name-status origin/main..."$BR" \ | ||
| -- docs/hygiene-history/loop-tick-history.md | ||
| M docs/hygiene-history/loop-tick-history.md ← diff succeeds | ||
|
|
||
| $ git log --oneline -1 "$BR" | ||
| d9feb3f chore(loop-tick-history): hour-05Z bundle close 2026-04-26T05:59:36Z ... | ||
| ``` | ||
|
|
||
| **Branch tip is clean.** The corrupt blob `8d5e67fd` is from | ||
| the branch's **intermediate history** (an earlier commit on the | ||
| same branch path), NOT the current tip. The branch is also NOT | ||
| an ancestor of `main` (its tip is post-squash-merge and | ||
| abandoned), and `main` carries the equivalent content under a | ||
| different blob (`de670f72d6fd34208d04863818288d764150a151`). | ||
|
|
||
| ```text | ||
| main's tick-history blob: de670f72d6fd34208d04863818288d764150a151 | ||
| branch-tip's blob: (different, clean — readable via git show) | ||
| corrupt blob: 8d5e67fd313573855848705e4af114f3ff0eecbc | ||
| ← appears only in intermediate history | ||
| ``` | ||
|
|
||
| ## Final classification — `8d5e67fd` | ||
|
|
||
| ```text | ||
| CORRUPT_BLOB_REFERENCED_BY_LIVE_LOCAL_BRANCH_AND_STALE_REMOTE_TRACKING_REF | ||
| ``` | ||
|
|
||
| (sub-bucket of bucket A under the three-bucket schema; refined | ||
| sub-condition: branch TIP is clean, corrupt blob is in | ||
| intermediate history of the branch only.) | ||
|
|
||
| Substrate-loss assessment: | ||
| - **Current-state use** (checkout / hard-reset-to-tip / diff vs | ||
| main): zero impact. Branch tip and main are both readable. | ||
| - **Bisect-through-pre-merge-history**: impacted. Bisecting | ||
| `loop-tick-history.md` through this branch's intermediate | ||
| revisions would surface the corrupt blob; that pre-merge | ||
| bisect-step granularity is the only thing actually lost, | ||
| and it's a niche use case for a stale post-merge local | ||
| branch. | ||
|
|
||
| ## Hard-reset readiness — qualified statement | ||
|
|
||
| ```text | ||
| This corruption does not appear to affect current main, but it | ||
| DOES affect at least one live local branch (specifically, the | ||
| intermediate history of refs/heads/chore/heartbeat-batch-2026- | ||
| 04-26-hour-05Z). Hard reset / branch cleanup remains blocked | ||
| until the branch's content is classified as preserved, | ||
| obsolete, or explicitly abandoned. | ||
| ``` | ||
|
|
||
| Branch-tip checkout / hard-reset-to-tip is fine. Reading | ||
| intermediate-history blobs would surface the corruption. | ||
|
|
||
| ## Stale remote-tracking ref — evidence, not origin recovery | ||
|
|
||
| ```text | ||
| $ git show-ref | grep 'chore/heartbeat-batch-2026-04-26-hour-05Z' | ||
| d9feb3f08603e37118491e26527c03e6e5760d5b refs/heads/chore/heartbeat-batch-2026-04-26-hour-05Z | ||
| d9feb3f08603e37118491e26527c03e6e5760d5b refs/remotes/origin/chore/heartbeat-batch-2026-04-26-hour-05Z | ||
| ↑ same SHA → ref is stale, not authoritative | ||
|
|
||
| $ git ls-remote --heads origin chore/heartbeat-batch-2026-04-26-hour-05Z | ||
| (empty — origin no longer has this branch) | ||
| ``` | ||
|
|
||
| The remote-tracking ref is **stale** — origin deleted the | ||
| branch (presumably after squash-merge), but the local repo | ||
| still carries the same-named ref under | ||
| `refs/remotes/origin/...`. This stale ref is evidence about | ||
| the prior state of origin; it is NOT proof origin currently | ||
| has the object. Do not prune it during triage — `git fetch | ||
| --prune origin` and `git remote prune` would remove it, and | ||
| that's a destructive cleanup decision under the same rule as | ||
| GC. | ||
|
|
||
| ```text | ||
| Stale remote-tracking ref is evidence, not origin recovery. | ||
| Do not prune it while investigating lost substrate. | ||
| ``` | ||
|
|
||
| ## Cleanup posture | ||
|
|
||
| ```text | ||
| Cleanup is a destructive decision, not a repair step. | ||
| ``` | ||
|
|
||
| `git gc` / `git prune` / `git repack -Ad` / `git fsck --lost-found` | ||
| remain forbidden during triage. Auto-gc thresholds are at default | ||
| (per `gc-config-snapshot.txt`); below-threshold loose-object count | ||
|
AceHack marked this conversation as resolved.
AceHack marked this conversation as resolved.
|
||
| means auto-gc is not imminent, but the discipline is "no implicit | ||
| cleanup decision" not "we have time." | ||
|
|
||
| A future cleanup may remove this unreachable/corrupt evidence, so | ||
| cleanup must wait until related dangling/reflog/stash surfaces | ||
| are classified and either preserved or explicitly abandoned. | ||
|
|
||
| ## Composes with | ||
|
|
||
| - `docs/lost-substrate/inventory-ledger-2026-04-29.md` — the | ||
| inventory ledger that surfaced these findings. | ||
| - `memory/feedback_corruption_triage_discipline_object_health_incident_aaron_amara_2026_04_29.md` | ||
| — the durable discipline rule. | ||
|
|
||
| ## Distilled keepers (Amara 2026-04-29) | ||
|
|
||
| ```text | ||
| A corrupt object is not a backlog item — it is a substrate health incident. | ||
| Do not prune the evidence while investigating lost evidence. | ||
| Recoverable-from-origin requires fresh-clone cat-file type + size verification. | ||
| Cleanup is a destructive decision, not a repair step. | ||
| Corruption lane means exclusive lane. | ||
| If corruption is the incident, everything else is noise. | ||
| ``` | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.