Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dagstore lotus mount Implementation with tests #564

Merged
merged 4 commits into from
Jul 11, 2021

Conversation

aarshkshah1992
Copy link
Collaborator

@aarshkshah1992 aarshkshah1992 commented Jul 6, 2021

TODO

  • Unit Tests for the Loutus Mount API impl.
  • Replace mocked out DAG store with actual DAG store Integrate dag store into mount implementation #565
  • Register shard after handoff deal completes (when deal has been successfully sent for sealing)
  • Destroy shard when storage deal expires / is slashed
    • LotusMount.Stat.Exists should be false when storage deal expires.
  • Shard failure and recovery

Copy link
Contributor

@dirkmc dirkmc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of nits but LGTM 👍

The mount implementation isn't really specific to lotus, should we call it something other than LotusMount?

dagstore/mount.go Outdated Show resolved Hide resolved
dagstore/mount.go Outdated Show resolved Hide resolved
@aarshkshah1992
Copy link
Collaborator Author

@dirkmc Notes on integrating the Sharded DAG Store into Lotus and Markets:

  1. Instantiate the Sharded DAG Store with the correct Config in Lotus. It's likely some of that Config will flow from the Lotus Config. The instantiation will take care of "rehydrating" the DAG Store so we don't need to worry about that. We'll instantiate and inject the Lotus Mount/Mount factory/Type here so they make their way into the Mount Registry of the DAG Store.

  2. Register Shards in the storage market after deals have been successfully sent for sealing (i.e. storage deal enters the StorageDealAwaitingPrecommit state on the Miner). If this fails, keep retrying till this succeeds with the "Sharded DAG Store failure management and Recovery" mechanism. Note that we can pass in the CARv2 file that Graphsync wrote out for us as part of the "RegisterShard" call here in the RegisterOpts option there to ask the Sharded DAG Store to simply scrap off the Index from the CARv2 file we already have rather than fetching the CARv1 payload from the Unsealer and indexing it all over again.

  3. In the retrieval Market, we'd call "AcquireShard" to get a ShardAccessor /ReadOnlyBlockstore that we can serve the retrieval from. It's important that we call Close on this accessor when we no longer need it (successful or failed retrieval) so the DAG Store can clean up transient artefacts it creates for this retrieval. If the call "AcquireShard" fails, we'd have to use the same failure management/recovery mechanism mentioned above.

  4. Also note that we might eventually decide that we need lazy registration of Shards (after having this discussion with @raulk in today's huddle) for all pre-existing deals to avoid the tricky migration effort there and in that case, the lazy shard registration i.e. calling ""RegisterShard" would also happen in the retrieval market.

@aarshkshah1992 aarshkshah1992 merged commit 92e827d into feat/dagstore-retrieval Jul 11, 2021
@aarshkshah1992
Copy link
Collaborator Author

@dirkmc Shifting review and TODOs to the feat/dagstore-retrieval here so I can review all the changes together.

dirkmc added a commit that referenced this pull request Jul 13, 2021
* refactor: integrate dag store into retrieval market

* fix all tests and the padding issue

* refactor: move mount from shared testutil to dagstore dir

* refactor: add tests for lazy blockstore

* refactor: code cleanup

* feat: update go-car to latest

* Dagstore lotus mount Implementation with tests (#564)

* dagstore lotus mount impl

* refactor: nicer error messages

* mount api tests

* refactor: integrate dag store (#565)

Co-authored-by: Dirk McCormick <[email protected]>

* some storage market fixes by aarsh

* fix: better error messages in DAG store wrapper

* refactor: simplify mock dag store wrapper

* fix: TestBounceConnectionDealTransferOngoing

* refactor: remove some commented out code

* refactor: closable blockstore interface to use full blockstore

* fix: TestBounceConnectionDealTransferUnsealing

* refactor: add comment explaining lotus mount template

* test: verify that the lazy blockstore is only initialized once

* fix: comment

* fix: always finalize blockstore before reaching complete state (#567)

Co-authored-by: aarshkshah1992 <[email protected]>
aarshkshah1992 added a commit that referenced this pull request Jul 13, 2021
* dag store lotus mount

* storage client code complete-tests remain

* storage miner first draft

* second draft

* provider state tests are now working

* most unit tests working and more unit tests

* refactor: car store trackers (#559)

* Apply suggestions from code review

Co-authored-by: dirkmc <[email protected]>

* nits and review

* integration testing changes

* storage test harness changes

* added more TODOs

* fix itests for storage

* get offline deal tests working

* tests work

* more tests

* integration tests

* fix blockstore finalize

* Integrate CARv2 blockstore in the retrieval market (#560)

* refactor: integrate dag store into retrieval market

* fix all tests and the padding issue

* refactor: move mount from shared testutil to dagstore dir

* refactor: add tests for lazy blockstore

* refactor: code cleanup

* feat: update go-car to latest

* Dagstore lotus mount Implementation with tests (#564)

* dagstore lotus mount impl

* refactor: nicer error messages

* mount api tests

* refactor: integrate dag store (#565)

Co-authored-by: Dirk McCormick <[email protected]>

* some storage market fixes by aarsh

* fix: better error messages in DAG store wrapper

* refactor: simplify mock dag store wrapper

* fix: TestBounceConnectionDealTransferOngoing

* refactor: remove some commented out code

* refactor: closable blockstore interface to use full blockstore

* fix: TestBounceConnectionDealTransferUnsealing

* refactor: add comment explaining lotus mount template

* test: verify that the lazy blockstore is only initialized once

* fix: comment

* fix: always finalize blockstore before reaching complete state (#567)

Co-authored-by: aarshkshah1992 <[email protected]>

Co-authored-by: dirkmc <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants