Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: sealing: Use bitfields to manage sector numbers #9183

Merged
merged 10 commits into from
Aug 26, 2022

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Aug 18, 2022

Related Issues

Needed for SaaS

Proposed Changes

  • Move sector number assigning logic to use stored bitfields instead of stored counters
  • Make it possible to reserve ranges of sector numbers
    • Using lotus-miner sectors numbers reserve/free cli or SectorNumReserve/Free API.
  • Add a command which makes it possible to see the sector assigner state - lotus-miner sectors numbers info
  • Add a command which makes it possible to view sector number reservations - lotus-miner sectors numbers reservations

Some cli examples:

$ lotus-miner sectors numbers info
Next free: 4
Allocated: 0-3,11-12
Reserved: 0

$ lotus-miner sectors pledge
Created CC sector:  4

$ lotus-miner sectors numbers info
Next free: 5
Allocated: 0-4,11-12
Reserved: 0

$ lotus-miner sectors numbers reserve test-res 5-9

$ lotus-miner sectors numbers reservations
reservations: 1
test-res: count=5 5-9

$ lotus-miner sectors numbers info
Next free: 10
Allocated: 0-4,11-12
Reserved: 5-9

$ lotus-miner sectors pledge
Created CC sector:  10

## sectors 11/12 were already allocated for whatever reason, so the assigner will assign 13, which is the next available number

$ lotus-miner sectors numbers info
Next free: 13
Allocated: 0-4,10-12
Reserved: 5-9

$ lotus-miner sectors pledge
Created CC sector:  13

## if we drop the reservation without any sector numbers in that being used, the assigner will now try to reuse those numbers to reduce number gaps

$ lotus-miner sectors numbers free test-res
$ lotus-miner sectors numbers info
Next free: 5
Allocated: 0-4,10-13
Reserved: 0

$ lotus-miner sectors pledge
Created CC sector:  5

TODO

  • Make sealing pipeline use bitfields instad of a stored sector counter for tracking sector number assignments
    • Backwards compatible with the old stored counter so upgrading/downgrading lotus-miner is free of manual migrations
  • Add commands/utils to allocate / reserve blocks of sector numbers
    • Makes it possible for SaaS to make sure lotus won't use specified ranges of sector numbers
  • Make sector number allocation logic look at miner-actor state to not create sectors with numbers which are marked as allocated
  • Describe how to use here
  • Test all this
    • Itests
    • Manual testing

Checklist

Before you mark the PR ready for review, please make sure that:

  • All commits have a clear commit message.
  • The PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, INTERFACE BREAKING CHANGE, CONSENSUS BREAKING, build, chore, ci, docs,perf, refactor, revert, style, test
    • area: api, chain, state, vm, data transfer, market, mempool, message, block production, multisig, networking, paychan, proving, sealing, wallet, deps
  • This PR has tests for new functionality or change in behaviour
  • If new user-facing features are introduced, clear usage guidelines and / or documentation updates should be included in https://lotus.filecoin.io or Discussion Tutorials.
  • CI is green

@magik6k magik6k force-pushed the feat/sectornum-mgmt branch from f085428 to e371ea0 Compare August 22, 2022 20:55
@magik6k magik6k force-pushed the feat/sectornum-mgmt branch from e371ea0 to 129eba2 Compare August 22, 2022 20:58
@magik6k magik6k force-pushed the feat/sectornum-mgmt branch from b9a6e36 to 64290b6 Compare August 22, 2022 21:41
@magik6k magik6k marked this pull request as ready for review August 23, 2022 18:21
@magik6k magik6k requested a review from a team as a code owner August 23, 2022 18:21
storage/pipeline/numassign.go Outdated Show resolved Hide resolved
storage/pipeline/numassign.go Outdated Show resolved Hide resolved
cmd/lotus-miner/sectors.go Show resolved Hide resolved
lib/strle/human.go Outdated Show resolved Hide resolved
@magik6k magik6k merged commit 45d1bd6 into master Aug 26, 2022
@magik6k magik6k deleted the feat/sectornum-mgmt branch August 26, 2022 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants