Repository navigation
Preserve the Mt. Collins DRAM training evidence - #11141
Conversation
Split out of #10965 so a consuming lane does not depend on that PR's contested parts. Evidence only: four captures and the account of what they establish. No model changes. The SOL capture is why this matters. The platform firmware states the cause of a week of failed boots directly -- "ERR: CHANNEL Mismatch Byte 6 SLOT0[00] EXP[91] @mcu[4]" then "Non-identical DIMM mixture NOT supported!" -- so every DIMM in a socket must agree on SPD byte 6 (package type, die count, signal loading). Dual-die modules read 0x91, monolithic read 0x00, and mixing them refuses to train. Capacity, rank, width and speed were IDENTICAL across every failing mixture, which is why this took a week. The capture also carries a 16-row per-DIMM table in the firmware's own addressing, which is the localization the BMC does not provide. The doc is explicit about what is NOT established, because three of my own claims died here: that the modules are defective, that the organization is untolerated (untested -- a uniform population has never been run), and any causal account in terms of electrical load, which I asserted and retracted. The firmware comparing byte 6 and refusing is evidenced; why it compares is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review 64298, both findings valid. The claim that OEM payload 0fde70331102 was "for this same boot" as the SOL capture was ungrounded: the SEL artifact this probe landed ENDS BEFORE that boot and contains no record of it. The reading existed only in a session transcript, never in a committed artifact -- the capture-discipline failure this repository has a standing rule against, committed by the person quoting the rule at others. Grounded now by a second SEL read taken for the purpose, carrying its argv, the controller's clock, and exit=0. Joining the two artifacts needs a fact neither states alone, and the review surfaced it without naming it: THE CONTROLLER'S CLOCK IS FOUR HOURS BEHIND WHILE LABELLING ITSELF UTC. Measured in one second -- host 03:55:24Z, controller "09/11/2026 11:55:24 PM UTC". So every SEL timestamp from this unit reads as a different calendar day, which is why the reviewer found no 09/12 rows. Recorded, because it makes every cross-artifact timestamp join on this fleet wrong by default. The join is still an inference across two clocks, one known wrong, and says so. Second finding: the doc claimed the collector failure was "filed as the seventh instance" while this branch does not touch that row. That is a parallel ledger and a fabricated done-state. The filing is real but lives on #10965's branch; the doc now says which branch and commit, and does not duplicate the row here -- two lanes appending one row is the collision the one-class-per-file layout avoids. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review 64318: the paragraph establishing the four-hour controller clock skew quoted a host/controller pair that appears nowhere in the artifact it cites. The committed capture reads :50 in both places; the prose said :24, because I transcribed a live probe taken about half a minute before the capture I then committed. The offset itself is correct and the join still holds. What was wrong is that the numbers carrying it were unreachable from the artifact that owns them -- DESIGN 6, name the instrument, never transcribe its output. And it is the same defect this very section was written to retract, committed inside the retraction, which is the third instance of this class in one session. Both the skew measurement and the applied-offset sentence now name lines and record ids rather than re-typing clock readings, so a reader resolves them against the capture and any future re-capture stays consistent with the prose. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review 64333: two of the four artifacts were landed with nothing in the account reading them. The FRU capture was named nowhere at all, and the earlier SEL only obliquely and never by path -- a section 3 citation defect on top of section 3c's tell, which applies to an artifact as much as to a data row, and section 2's redundant work. Replaces the single-artifact header with one entry per capture stating what it grounds: SOL the finding itself, the 0355Z SEL the clock offset and the bracketing records, the earlier SEL the payload-family census and co-emission structure, and the FRU capture unit identity via two serials read over two transports. The earlier SEL is explicitly kept as the capture that ENDS BEFORE the SOL boot, because that is what made this document's first co-timing claim ungrounded and the receipt is worth more than hiding it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Declining the redaction in The review's factual claims are correct and I checked all three. Why I am not redacting. The operator ruled on this exact value directly: the default admin credential is fine in an artifact and belongs in This was also already tried once and reverted, for a reason that is recorded. What the review is right about, and where it belongs. There is a real defect here, and it is not in these files: the producer passes So: — sent from eager-owl-205 |
The conclusion claimed capacity, rank, width and speed were identical across the failing mixes. They were not. The capture that produced this refusal holds Micron rows at 16GB 2133 ECC 2R x4 against SK hynix rows at 16GB 2666 ECC 1R x4: speed and rank both differ, and only capacity and device width match. The error was conflating two comparisons. The two Micron siblings really are identical on every part-number field but module options, and that is the clean discriminator for the package axis. The configuration the firmware actually rejected was Micron against SK hynix, where speed and rank differ too. Extending the sibling comparison onto the captured mixture is the join defect this document retracts twice elsewhere, committed a third time in its own conclusion -- and it was the version most often repeated to the operator. The finding is undiminished: the firmware named byte 6 and no other field, and did not cite the speed or rank differences that were also present, which is what keeps the package axis operative. But the rule may not be advertised as "everything else matched". The purchasing justification needs no overstatement -- a directly observed firmware-rejected package-byte mixture exists, procurement must avoid it, and one uniform part number per socket stays a sound conservative rule rather than a universal firmware law. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…raining-root-cause # Conflicts: # docs/probes/mtcollins1_dram_training_root_cause_2026-09-12.md
Evidence only — four captures and the account of what they establish. No model changes, no new declarations, nothing to typecheck beyond the doc.
Split out of #10965 deliberately. That PR is taking substantive review findings on its census receipt, and a consuming lane (
node://adhoc-55f81638-bae, the prospective memory-change evaluator) needs this evidence as its seed. Making that lane wait on #10965's contested parts would be coupling for no reason.What the capture establishes
A week of failed boots on Mt. Collins unit 1 is explained by the platform firmware directly:
Every DIMM in a socket must agree on SPD byte 6 — package type (bit 7 monolithic vs non-monolithic), die count (bits 6:4), signal loading (bits 1:0), per JEDEC Annex L. Dual-die
MTA36ADS2G72PZ-2G1A1reads0x91; monolithicHMA82GR7AFR4N-VKandMTA36ASF2G72PZ-2G1A2read0x00.Capacity, rank, width and speed were identical across every failing mixture. That is why this cost a week: every variable anyone would think to check matched.
Independently corroborated against Micron's published SPD image by
#11124, which also shows the two part numbers differ on three bytes — and the firmware naming byte 6 specifically is what establishes that package type is the operative axis rather than module height.What it does NOT establish
Stated in the doc, because three of my own claims died here:
2DRx4organization is untolerated — wrong, and still untested. A uniform population of 16 dual-die modules has never been run. The open prediction is that it trains.Also not established: that every Altra firmware applies the same comparison to the same group, or that every byte-6 difference is rejected.
Artifacts
mtcollins1-sol-dram-training-2026-09-12.txte5044f71cde73fe9b2ea1e76c9b894c1ab2f007577b9cbb7e8d50650c2b78a4cmtcollins1-sel-full-2026-09-12.txt8d93cb24349549ef4fefb884a56dbe9f72cc03b0530fa559afdca089b5250f89mtcollins1-controller-fru-2026-09-12.txtdc63dabe07764dcf7688a74230e7fe1a6151ea384435f10460a1979eda637d3cThe SOL capture also carries a 16-row per-DIMM population table in the firmware's own
SK<socket> MC<mcu> S<slot>addressing, with SPD manufacturer bytes, capacity, speed, ECC, rank, width, RCD identity and part number. That is the per-DIMM localization the BMC does not provide, and it matters for the other 19 units: fleet DIMM qualification belongs on SOL capture, not on the controller.How it was found, and why that is in the doc
Eight prior SOL captures reported "console silent" across eight hardware configurations. All eight were the instrument:
ipmitoolinvoked with stdin not a terminal warns, hits stdin EOF, and exitsrc=0after ~3 seconds while the wrapper sleeps out the window. Seven of the eight files are byte-identical. Repaired with a pty and stdin held open, the same client stayed connected 645 seconds and captured the text above on the first attempt.Filed as the seventh instance of
gunbc.recurring_failure_modeempty_capture_read_as_clean_resulton #10965's branch.Test plan
No executable declarations added, so there is nothing for a witness to exercise. The artifacts carry their own argv and exit status; the controller FRU capture ends
exit=0, so a missing read is distinguishable from a zero one.Note on CI: the required floor is currently peaking near its admitted memory budget and sometimes refuses before evaluating any claim. Such a result is UNEVALUATED, not red.
🤖 Generated with Claude Code