Skip to content

Conversation

@n8mgr
Copy link
Member

@n8mgr n8mgr commented Nov 27, 2025

Refactors sector proofs to use cached subtree roots to reduce disk IO when building a proof for part of a sector. (SiaFoundation/hostd#876, SiaFoundation/coreutils#364)

(Yes, I copied sectorAccumulator in but it's probably time we remove the RHP2 and RHP3 packages in a follow up)

@lukechampine feel free to take this over if you are unhappy with it.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors sector proof generation to use cached subtree roots at a fixed 64-leaf level, reducing disk I/O operations by allowing hosts to build proofs without reading the entire sector data. This optimization is particularly beneficial for reducing the amount of data read from disk during proof generation.

Key Changes:

  • Introduces a specialized sectorAccumulator for efficient Merkle tree computation
  • Adds CachedSectorSubtrees to compute and cache 64-leaf subtree roots (4 KiB chunks)
  • Refactors BuildSectorProof to construct proofs using cached subtrees and minimal segment data

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File Description
rhp/v4/merkle.go Adds caching infrastructure with sectorAccumulator, CachedSectorSubtrees, and refactored BuildSectorProof to use cached 64-leaf subtree roots
rhp/v4/merkle_test.go Adds comprehensive test coverage for BuildSectorProof with various range scenarios including edge cases and randomized inputs

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@lukechampine
Copy link
Member

LGTM. I agree that it's time to absorb the RHP2 and RHP3 packages into RHPv4.

var wg sync.WaitGroup
for i := range roots {
wg.Add(1)
go func(i int) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This'll spin up 1024 goroutines per sector right? Isn't that an excessive amount? Obviously it would come at the cost of a little more complexity but perhaps we could batch these and use runtime.GOMAXPROCS or use a pool or something?

@n8mgr n8mgr merged commit 1d703a0 into master Dec 1, 2025
13 checks passed
@n8mgr n8mgr deleted the nate/cached-roots branch December 1, 2025 14:30
@github-project-automation github-project-automation bot moved this from In Progress to Done in Sia Dec 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

5 participants