Repository navigation
perf(pipeline): dedupe the TemplatesToMiGroups bridge and stop cloning records - #431
Conversation
…actory `add_simplex`, `add_duplex`, and `add_codec` each inlined a ~70-line copy of the `TemplatesToMiGroups` bridge step (the fused group→consensus path that splices each template's MI into the `MI` tag and runs records into per-MI groups). The three copies were identical except for one flag — `duplex_strip_strand` (true for duplex, false for simplex/codec). Hoist the closure into a single `templates_to_mi_step(limit_bytes, duplex_strip_strand)` factory and call it from all three sites. No behavior change; this is a pure deduplication so the upcoming clone-removal lands once.
…loning The `TemplatesToMiGroups` bridge tags each record's MI in a reused `scratch` buffer, then did `RawRecord::from(state.scratch.clone())` — a full per-record heap copy on top of the `extend_from_slice`. This runs once per read on the primary runall workload (fused sort→group→consensus / group→consensus). Move the buffer into the record with `std::mem::replace`, reinstalling a fresh `Vec` pre-sized to the previous capacity so the next record reuses it without a realloc (the per-record `clear()` keeps it warm). Same allocation count as the clone, minus the record-body memcpy. Output is unchanged.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feat-runall #431 +/- ##
==============================================
Coverage ? 93.40%
==============================================
Files ? 139
Lines ? 54817
Branches ? 0
==============================================
Hits ? 51200
Misses ? 3617
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughIn 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
…g records (#431) * refactor(pipeline): extract the TemplatesToMiGroups bridge into one factory `add_simplex`, `add_duplex`, and `add_codec` each inlined a ~70-line copy of the `TemplatesToMiGroups` bridge step (the fused group→consensus path that splices each template's MI into the `MI` tag and runs records into per-MI groups). The three copies were identical except for one flag — `duplex_strip_strand` (true for duplex, false for simplex/codec). Hoist the closure into a single `templates_to_mi_step(limit_bytes, duplex_strip_strand)` factory and call it from all three sites. No behavior change; this is a pure deduplication so the upcoming clone-removal lands once. * perf(pipeline): move tagged bytes into MI-bridge records instead of cloning The `TemplatesToMiGroups` bridge tags each record's MI in a reused `scratch` buffer, then did `RawRecord::from(state.scratch.clone())` — a full per-record heap copy on top of the `extend_from_slice`. This runs once per read on the primary runall workload (fused sort→group→consensus / group→consensus). Move the buffer into the record with `std::mem::replace`, reinstalling a fresh `Vec` pre-sized to the previous capacity so the next record reuses it without a realloc (the per-record `clear()` keeps it warm). Same allocation count as the clone, minus the record-body memcpy. Output is unchanged.
Wave 2 perf cleanup (#8 of the feat-runall review series) — the highest-value item: it hits the primary runall workload (fused sort→group→consensus / group→consensus). Two commits, read in order.
1.
refactor(pipeline): extract the bridge into one factoryadd_simplex,add_duplex, andadd_codeceach inlined a ~70-line copy of theTemplatesToMiGroupsbridge (the fused group→consensus path that splices each template's MI into theMItag and runs records into per-MI groups). The three copies were identical except for one flag —duplex_strip_strand(true for duplex, false for simplex/codec). Hoisted into a singletemplates_to_mi_step(limit_bytes, duplex_strip_strand)factory called from all three sites. Pure dedup, no behavior change.2.
perf(pipeline): move tagged bytes into records instead of cloningThe bridge tags each record's MI in a reused
scratchbuffer, then didRawRecord::from(state.scratch.clone())— a full per-record heap copy on top of theextend_from_slice, once per read on the main runall workload. Now moves the buffer into the record withstd::mem::replace, reinstalling a freshVecpre-sized to the previous capacity so the next record reuses it without a realloc (the per-recordclear()keeps it warm). Same allocation count as the clone, minus the record-body memcpy. Output unchanged.Verification
cargo ci-fmt/cargo ci-lint(clippy pedantic) clean.cargo nextest -E 'test(runall) or test(group_determinism)'— 149 tests pass, covering all three fused group→{simplex,duplex,codec} shapes and MI determinism.--agent) and a CodeRabbit-style self-review: 0 findings.