feat:splitstore:Configure max space used by hotstore and GC makes best effort to respect

[ZenGround0]

Related Issues

#10388

Proposed Changes

• Add size reporting to chain walking subfunctions
• Record sizes of the different DAGs walked during compaction
• New config value for targeting max total space used by hotstore
• GC logic to trigger moving GC adaptively based on hotstore size
• Most importantly: splitstore refuses to do moving GC if disk usage would overflow the targetted max

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
  • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green

[vyzo]

hmmmm, ok, looks sane.

Potential concern: can we end up continuously moving gc in every compaction?

[magik6k]

General idea makes a ton of sense. Couple of comments

Would be great to hook-up those sizes / size estimates to the metrics endpoint somehow - https://github.com/filecoin-project/lotus/blob/master/metrics/metrics.go#L160

[magik6k]

Probably worth using the alert system to raise a (generally much more visible / loud) alert.

See e.g. https://github.com/filecoin-project/lotus/blob/master/node/modules/alerts.go

(alerts are for things which tell the user that something either is, or is about to be severely broken, and they need to take action)

[ZenGround0]

Potential concern: can we end up continuously moving gc in every compaction?

Reasonable concern, this is definitely possible. In my opinion this is more of a feature than a bug however. Misconfigurations are certainly possible but I would say we don't expose any worse footgun than allowing FullGCFreq = 1 with current config.

What I've come to believe is there are only two real problems users face with splitstore: disk overflow and chain out of sync. The only risk with too much moving GC when it is done without overflowing disk is that it gets the chain out of sync. But I think we can address chain out of sync additionally in #10392 or an extension after fixing a solution to disk overflow.

[ribasushi]

FullGCFreq = 1 with current config.

This is how I configure my experimental splitstore nodes, anything besides "do moving GC every time" makes no sense from an operational perspective: you want your dataset to always be as small as possible.

[ZenGround0]

anything besides "do moving GC every time" makes no sense from an operational perspective

While I agree that "do moving GC every time" often makes sense from an operational perspective I think this is overstated. Moving GC is objectively more expensive than online in terms of memory, compute and disk io. Paying the cost on every compaction to keep datastore down is likely often a good idea but keeping more disk around and then doing this in batches is a reasonable point in the tradeoff space.

[ZenGround0]

@magik6k ready for another round. I ran out of time so filing this follow up to get alerting and metrics in place after the merge: #10428.

[magik6k]

One question, 2 small nits, but other than that looks good.

If it's not perfect bugs can be fixed in the RC process

[ZenGround0]

Thanks for the make gen assist @magik6k !!