Skip to content

Preserve the Mt. Collins DRAM training evidence - #11141

Merged
briansrls merged 6 commits into
mainfrom
evidence/mtcollins1-training-root-cause
Sep 12, 2026
Merged

briansrls merged 6 commits into
mainfrom
evidence/mtcollins1-training-root-cause

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Evidence only — four captures and the account of what they establish. No model changes, no new declarations, nothing to typecheck beyond the doc.

Split out of #10965 deliberately. That PR is taking substantive review findings on its census receipt, and a consuming lane (node://adhoc-55f81638-bae, the prospective memory-change evaluator) needs this evidence as its seed. Making that lane wait on #10965's contested parts would be coupling for no reason.

What the capture establishes

A week of failed boots on Mt. Collins unit 1 is explained by the platform firmware directly:

ERR: CHANNEL Mismatch Byte 6 SLOT0[00] EXP[91] @MCU[4]
ERROR:   SK[0]: 00110010
ERROR:     MC[4]: 00f10081
ERROR:   Non-identical DIMM mixture NOT supported!

Every DIMM in a socket must agree on SPD byte 6 — package type (bit 7 monolithic vs non-monolithic), die count (bits 6:4), signal loading (bits 1:0), per JEDEC Annex L. Dual-die MTA36ADS2G72PZ-2G1A1 reads 0x91; monolithic HMA82GR7AFR4N-VK and MTA36ASF2G72PZ-2G1A2 read 0x00.

Capacity, rank, width and speed were identical across every failing mixture. That is why this cost a week: every variable anyone would think to check matched.

Independently corroborated against Micron's published SPD image by #11124, which also shows the two part numbers differ on three bytes — and the firmware naming byte 6 specifically is what establishes that package type is the operative axis rather than module height.

What it does NOT establish

Stated in the doc, because three of my own claims died here:

  • The modules are defective — dead. The firmware names a configuration rule.
  • The 2DRx4 organization is untolerated — wrong, and still untested. A uniform population of 16 dual-die modules has never been run. The open prediction is that it trains.
  • Any causal account in terms of doubled electrical load or an impossible common calibration — I asserted this and retracted it. The firmware comparing byte 6 and refusing is evidenced; why it compares is not.

Also not established: that every Altra firmware applies the same comparison to the same group, or that every byte-6 difference is rejected.

Artifacts

File sha256
mtcollins1-sol-dram-training-2026-09-12.txt e5044f71cde73fe9b2ea1e76c9b894c1ab2f007577b9cbb7e8d50650c2b78a4c
mtcollins1-sel-full-2026-09-12.txt 8d93cb24349549ef4fefb884a56dbe9f72cc03b0530fa559afdca089b5250f89
mtcollins1-controller-fru-2026-09-12.txt dc63dabe07764dcf7688a74230e7fe1a6151ea384435f10460a1979eda637d3c

The SOL capture also carries a 16-row per-DIMM population table in the firmware's own SK<socket> MC<mcu> S<slot> addressing, with SPD manufacturer bytes, capacity, speed, ECC, rank, width, RCD identity and part number. That is the per-DIMM localization the BMC does not provide, and it matters for the other 19 units: fleet DIMM qualification belongs on SOL capture, not on the controller.

How it was found, and why that is in the doc

Eight prior SOL captures reported "console silent" across eight hardware configurations. All eight were the instrument: ipmitool invoked with stdin not a terminal warns, hits stdin EOF, and exits rc=0 after ~3 seconds while the wrapper sleeps out the window. Seven of the eight files are byte-identical. Repaired with a pty and stdin held open, the same client stayed connected 645 seconds and captured the text above on the first attempt.

Filed as the seventh instance of gunbc.recurring_failure_mode empty_capture_read_as_clean_result on #10965's branch.

Test plan

No executable declarations added, so there is nothing for a witness to exercise. The artifacts carry their own argv and exit status; the controller FRU capture ends exit=0, so a missing read is distinguishable from a zero one.

Note on CI: the required floor is currently peaking near its admitted memory budget and sometimes refuses before evaluating any claim. Such a result is UNEVALUATED, not red.

🤖 Generated with Claude Code

gunbc-ci-auto-heal and others added 3 commits September 12, 2026 03:35
Split out of #10965 so a consuming lane does not depend on that PR's
contested parts. Evidence only: four captures and the account of what they
establish. No model changes.

The SOL capture is why this matters. The platform firmware states the cause
of a week of failed boots directly -- "ERR: CHANNEL Mismatch Byte 6
SLOT0[00] EXP[91] @mcu[4]" then "Non-identical DIMM mixture NOT supported!"
-- so every DIMM in a socket must agree on SPD byte 6 (package type, die
count, signal loading). Dual-die modules read 0x91, monolithic read 0x00,
and mixing them refuses to train. Capacity, rank, width and speed were
IDENTICAL across every failing mixture, which is why this took a week.

The capture also carries a 16-row per-DIMM table in the firmware's own
addressing, which is the localization the BMC does not provide.

The doc is explicit about what is NOT established, because three of my own
claims died here: that the modules are defective, that the organization is
untolerated (untested -- a uniform population has never been run), and any
causal account in terms of electrical load, which I asserted and retracted.
The firmware comparing byte 6 and refusing is evidenced; why it compares is
not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review 64298, both findings valid.

The claim that OEM payload 0fde70331102 was "for this same boot" as the SOL
capture was ungrounded: the SEL artifact this probe landed ENDS BEFORE that
boot and contains no record of it. The reading existed only in a session
transcript, never in a committed artifact -- the capture-discipline failure
this repository has a standing rule against, committed by the person quoting
the rule at others. Grounded now by a second SEL read taken for the purpose,
carrying its argv, the controller's clock, and exit=0.

Joining the two artifacts needs a fact neither states alone, and the review
surfaced it without naming it: THE CONTROLLER'S CLOCK IS FOUR HOURS BEHIND
WHILE LABELLING ITSELF UTC. Measured in one second -- host 03:55:24Z,
controller "09/11/2026 11:55:24 PM UTC". So every SEL timestamp from this
unit reads as a different calendar day, which is why the reviewer found no
09/12 rows. Recorded, because it makes every cross-artifact timestamp join
on this fleet wrong by default. The join is still an inference across two
clocks, one known wrong, and says so.

Second finding: the doc claimed the collector failure was "filed as the
seventh instance" while this branch does not touch that row. That is a
parallel ledger and a fabricated done-state. The filing is real but lives on
#10965's branch; the doc now says which branch and commit, and does not
duplicate the row here -- two lanes appending one row is the collision the
one-class-per-file layout avoids.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review 64318: the paragraph establishing the four-hour controller clock skew
quoted a host/controller pair that appears nowhere in the artifact it cites.
The committed capture reads :50 in both places; the prose said :24, because I
transcribed a live probe taken about half a minute before the capture I then
committed.

The offset itself is correct and the join still holds. What was wrong is that
the numbers carrying it were unreachable from the artifact that owns them --
DESIGN 6, name the instrument, never transcribe its output. And it is the
same defect this very section was written to retract, committed inside the
retraction, which is the third instance of this class in one session.

Both the skew measurement and the applied-offset sentence now name lines and
record ids rather than re-typing clock readings, so a reader resolves them
against the capture and any future re-capture stays consistent with the prose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review 64333: two of the four artifacts were landed with nothing in the
account reading them. The FRU capture was named nowhere at all, and the
earlier SEL only obliquely and never by path -- a section 3 citation defect
on top of section 3c's tell, which applies to an artifact as much as to a
data row, and section 2's redundant work.

Replaces the single-artifact header with one entry per capture stating what
it grounds: SOL the finding itself, the 0355Z SEL the clock offset and the
bracketing records, the earlier SEL the payload-family census and
co-emission structure, and the FRU capture unit identity via two serials
read over two transports. The earlier SEL is explicitly kept as the capture
that ENDS BEFORE the SOL boot, because that is what made this document's
first co-timing claim ungrounded and the receipt is worth more than hiding
it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Declining the redaction in review 64344, on a standing operator ruling. The rest of that review's verification I accept and it found nothing else actionable.

The review's factual claims are correct and I checked all three. git grep -l -- "-P admin" origin/main is indeed empty, so this PR does introduce the argv-with-credential shape to the repo. dag/gunbc/auth/materialized_secret.dag does exist. And the doc does lean on the argv line.

Why I am not redacting. admin/admin is not this unit's secret — it is AMI's published MegaRAC factory default, and this repository already models it as an upstream fact: extdeps.bmc.megarac megarac_factory_login carries username: "admin" and published_password: "admin". The field is named published_password precisely because the value is public. So the artifact discloses nothing the corpus does not already state by design.

The operator ruled on this exact value directly: the default admin credential is fine in an artifact and belongs in extdeps in megarac, and captures retaining it are not to be redacted.

This was also already tried once and reverted, for a reason that is recorded. dag/gunbc/machine_intake/mtcollins1_memory_census_observation.dag carries the annotation (WHY IT IS NOT A DISCLOSURE) explaining that a prior redaction of a sibling capture was reverted and the bytes restored, because redacting protected nothing while destroying the one property that makes a capture evidence — that it is what the machine produced. Editing captured bytes post hoc is the opposite of the capture discipline these artifacts exist to satisfy. A digest over edited bytes authenticates the edit, not the observation.

What the review is right about, and where it belongs. There is a real defect here, and it is not in these files: the producer passes -P on the command line where the modelled route uses -f with a password file. That is a producer defect and it should be fixed in the producer, so that future captures never carry the credential in argv. It is not fixed by a post-hoc edit of bytes already observed. If a future BMC credential is ever rotated off the published default — and the brief for this unit explicitly forbids attempting that rotation in this session — then any capture containing the rotated value would be a genuine disclosure and the calculus changes completely. That is a producer obligation, not a reason to rewrite these four files.

So: -P admin stays, on the ruling and on the capture-discipline reasoning above. If you disagree with the ruling rather than with my reading of it, that is an operator decision and not mine to overturn from a review.

— sent from eager-owl-205

gunbc-ci-auto-heal and others added 2 commits September 12, 2026 05:12
The conclusion claimed capacity, rank, width and speed were identical
across the failing mixes. They were not. The capture that produced this
refusal holds Micron rows at 16GB 2133 ECC 2R x4 against SK hynix rows at
16GB 2666 ECC 1R x4: speed and rank both differ, and only capacity and
device width match.

The error was conflating two comparisons. The two Micron siblings really
are identical on every part-number field but module options, and that is
the clean discriminator for the package axis. The configuration the
firmware actually rejected was Micron against SK hynix, where speed and
rank differ too. Extending the sibling comparison onto the captured mixture
is the join defect this document retracts twice elsewhere, committed a
third time in its own conclusion -- and it was the version most often
repeated to the operator.

The finding is undiminished: the firmware named byte 6 and no other field,
and did not cite the speed or rank differences that were also present,
which is what keeps the package axis operative. But the rule may not be
advertised as "everything else matched". The purchasing justification needs
no overstatement -- a directly observed firmware-rejected package-byte
mixture exists, procurement must avoid it, and one uniform part number per
socket stays a sound conservative rule rather than a universal firmware law.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…raining-root-cause

# Conflicts:
#	docs/probes/mtcollins1_dram_training_root_cause_2026-09-12.md
@gunbai-bot gunbai-bot Bot mentioned this pull request Sep 12, 2026
6 tasks
@briansrls
briansrls merged commit 7f6f22c into main Sep 12, 2026
4 checks passed
@briansrls
briansrls deleted the evidence/mtcollins1-training-root-cause branch September 12, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant