M1 Testing With MinerX.2

[jennijuju]

Motivation & Goals

With the goal of supporting and enabling much greater online deal-making experience on Filecoin, lotus teams started to work on milestone M1 a few weeks ago. M1 also aimed to reduce the performance impact of the very large and continuously growing Filecoin datastore.

Today, we are ready to share the progress with Filecoin MinerX.2 fellows and start to test the new features, bug fixes, and enhancements. The goal of MinerX.2 M1 testing is to:

• collect feedback on the new features and gain confidence in their performance
• work closely with lotus teams to identify and troubleshooting the potential bugs
• provide feedback on documentation and come up with step-by-step tutorials for the broader community later
• define the gaps for future improvements

By the end of M1, we are expecting a much higher online storage and retrieval deal success rate from Filecoin using lotus.

What’s being built

To support the above goal, the team focused on implementing the following:

• Miner segregation with lotus markets process, you can read more about the evolution of a lotus node here.
• CARv2 and DAG store
• SplitStore v1

To better test and analyze storage and retrieval deal capability on the Filecoin network, the team also build a DealBot for sending out deals and collect success rate and failure reasons.

What’s next

The first iteration of M1 has been tagged as m1.0 this Wednesday. While a tremendous amount of testing has been done internally with the dev teams, potential bugs and further improvements are expected to be discovered throughout the testing. We will split MinerX.2 fellows into different groups to test different components in different stages. Each testing plan and instruction will be shared in its own post on this discussion page. MinerX.2 fellows that are selected for the current stage are expected to upgrade your lotus to a specific tag within 48 hours.

This discussion page will be used to communicate the testing plans and instructions. We will use this discussion for questions(❓) and feedbacks(🤩 ). A slack channel will be created for coordination and sync discussions. Issues should be reported by filing tickets in the lotus repo using the corresponding issue templates: M1 Bug Report For Deal Making and M1 Bug Report For Non Deal Making Issue

To the Filecoin Community

Most of the M1 work that has been well tested is merged into the master(dev) branch, and any future improvements will be merged ASAP as well. You can also expect multiple feature releases over the next few weeks, the first one is coming on July 27th and details will be available in the release note. As always, we would love you to try out the new release candidates, enjoy the new features and improvements, and let us know any issue and feedback you have so we can make lotus better together ❤️ .

---

Huge thanks to @magik6k, @raulk, @vyzo , @dirkmc , @aarshkshah1992 , @nonsense , @hannahhoward , @willscott , @Stebalien , @arajasek, @ZenGround0, @coryschwartz , @ognots , @travisperson, @iand , @masih , @mvdan , @jacobheun , @gammazero, @dkkapur and so on for making this happen 💙 .

[jennijuju]

Logging

Miner

Run
lotus-miner log set-level --system dt-impl --system dt-chanmon --system dt_graphsync --system markets --system data_transfer_network --system retrievaladapter --system retrieval --system stores --system markets-rtvl-reval debug

Client

lotus log set-level --system dt-impl --system dt-chanmon --system dt_graphsync --system markets --system data_transfer_network --system retrieval debug

[jennijuju]

Releases

2021-07-21

m1.0 for testing Stage I

2021-07-28

m1.2 for testing Stage II

2021-07-29

m1.3 for testing Stage III

m1.3.1

• fix a stall issue when restarting the miner

m1.3.2

• add ability to tune the migration concurrency factor through environment variable LOTUS_DAGSTORE_COPY_CONCURRENCY (default: 2); increase this factor for migration to proceed quicker, if your storage system can handle the IO load

2021-08-05

m1.3.4 for testing Stage IV

Release Note

2021-08-12

m1.3.5 for testing Stage V

Release Note

[jennijuju]

2021.07.24 - Stage I

Test Users

10 fellows

Assignment

Split lotus main miner and markets service processes, then accept online storage and retrieval deals.

Steps

1. Checkout and update your lotus and lotus miner/worker to m1.0
2. Whitelist the M1 client for your deal filters.
3. Read the instructions here thoroughly to set up the lotus market process.
4. Start your lotus daemon and miner process with the markets subsystem.
5. Run the command here to set the logging levels.
6. Set the ask price of storage deals from M1 client for:

• --verified-price to be 0.
• --min-piece-size to be no larger than 4GB.
• --max-piece-size to be no less than 32GB.

7. Start to accept deals, remember to save the logs!

Disclaimer

You don't need to stop your usual dealmaking activities for the testing, however, we do expect you to whitelist and accept deals from the M1 testing clients. If a trade-off is required, we do request that you prioritize the testing deals.

What to expect

You can expect storage deals from the M1 client starting today. Deals will be verified deals and the size range is 4GB-32GB. If you don't get any deals, please ping @jennijuju or @dkkapur in the MinerX.2 Slack channel.
In a couple of days, you can also expect retrieval deals on the data you stored with the storage deal made earlier.

Risks and Mitigations

If deal-making is not working well with the market service process, you can follow the instruction here to roll back to lotus-miner monolith process.

Known Issues With the Rollback

If a storage deal was made before splitting the market process and a retrieval deal on the data stored after the split - everything works.
However, if a storage deal is made with the split market service process, and the miner was rolled back to the monolith process, then to retrieve the data we did from AFTER the split - the retrieval deal will fail.

A lotus-shed tool to resolve this issue is being worked on here.

Happy testing!

[jennijuju]

2021.07.27 - Stage II

Recap From Stage I

Huge thanks to @TippyFlitsUK, @boqian-nbai, @Meatball13, and @WillTechMusing for participating in Stage I testing actively, provided valuable feedback on doc and bug reports! With that being said, we are now ready to move to the next stage!

Test Users

34 fellows

Assignment

Split lotus main miner and markets service processes, then accept online storage and retrieval deals.

Steps

1. Checkout and update your lotus and lotus miner/worker to m1.2
2. Whitelist the M1 client for your deal filters.
3. Read the instructions here thoroughly to set up the lotus market process. (If you participated in Stage I, please revisit the docs as well as we have updated it quite a lot based on the feedbacks!)
4. Start your lotus daemon and miner process with the markets subsystem.
5. Run the command here to set the logging levels.
6. Set the ask price of storage deals from M1 client for:

• -- --price no higher 0.00000002 FIL per GB / epoch
• --verified-price to be 0.
• --min-piece-size to be no larger than 4GB.
• --max-piece-size to be no less than your miner sector size(32/64 GiB).

7. Start to accept deals, remember to save the logs!
8. Once you get your daemon and miner running, you can use lotus -vv or lotus-miner -vv to trace which API endpoints are being used and cross-checking with your configuratoin.

Note: You might encounter some issues for setting up the lotus-miner markets subsystem for the first time and get it running, if you have any questions, leave a comment in this discussion.

Disclaimer

You don't need to stop your usual dealmaking activities for the testing, however, we do expect you to whitelist and accept deals from the M1 testing clients. If a trade-off is required, we do request that you prioritize the testing deals.

What to expect

You can expect storage deals from the M1 client starting today. Deals will include both regular and verified deals and the size range is 4GB-32/64GB. If you don't get any deals, please ping @jennijuju or @dkkapur in the MinerX.2 Slack channel.
In a couple of days, you can also expect retrieval deals on the data you stored with the storage deal made earlier.

Risks and Mitigations

If deal-making is not working well with the market service process, you can follow the instruction here to roll back to lotus-miner monolith process.

Happy deal making!

[raulk]

Hello! An update from the Lotus team on upcoming m1.3, across several posts.

DAG store migration

Several early testers noticed that the $LOTUS_MARKETS_HOME/dagStore/transients directory was filling up quickly, while IO utilisation was spiking. Some miners reported disruption of some proving activities like fault declaration (thanks @TippyFlitsUK for the collaboration!).

In m1.3 we have introduced various throttling directives:

1. a maximum of 2 unsealed deals will be fetched at any time from the storage cluster.
2. a maximum of 100 concurrent API calls to the storage miner for getting deal metadata, and checking if sectors are unsealed. These calls are quick API calls, but if you want to further limit the concurrency you can use env variable LOTUS_DAGSTORE_MOUNT_CONCURRENCY, e.g. LOTUS_DAGSTORE_MOUNT_CONCURRENCY=20.

It is worth noting that DAG store migration is mostly asynchronous: we register one shard per deal in the DAG store, and this is quick operation (but it will be limited by throttling directive 2 above).

Your markets node will not expose any APIs until the registrations are complete. This is normal.

Shards are initialized in the background; it does not block the markets process startup. This is how initialization works:

1. A .partial file is created under $LOTUS_MARKETS_HOME/dagStore/transients. You should expect as many partial files as active storage deals you have being created quickly.
2. Initially they are all zero-sized.
3. The DAG store then fetches the unsealed copy for each deal, subject to the throttling directive 1 above into the partial file. You should see at most 2 files being filled up at a time.
4. Once the download completes, the file will be renamed to .complete.
5. The deal is then indexed (tends to be a quick operation too), and the index is written to the $LOTUS_MARKETS_HOME/dagStore/index directory.
6. The .complete file is then marked for deletion.

Garbage collection

Every minute, the DAG store's garbage collection will reclaim (= delete) unused transients. That is, transients for shards whose initialization has completed as per above.

Deletions will be logged with message: deleted existing transient.

Monitoring the migration

m1.3 ships with much improved migration logging. Please:

1. tail + grep for lines containing migrator in the markets output to watch progress.
2. monitor the transients directory by running: watch ls -lt. This is a normal, healthy pattern of file usage:
   • at most 2 .partial files whose size is increasing (downloading).
   • initially, as many zero-sized .partial files as unsealed storage deals.
   • once downloaded, the .partial extension is replaced with .complete.
   • every minute, a bunch of .complete files are deleted (GC).

[nonsense]

Here is a video, which shows the migration in practice: https://asciinema.org/a/KJOIinWWQ7hCXIzxnuF0niYGK

[raulk]

Second update on m1.3.

Introducing markets environment variables and flags

We have made it a lot easier to manage a split miner/market environment. Before, you needed to override the LOTUS_MINER_PATH or MINER_API_INFO env variables to point to the markets node when running markets commands.

The Lotus CLI now supports the following, analogous to the existing miner and daemon vars and flags:

• MARKETS_API_INFO env variable
• LOTUS_MARKETS_PATH env variable
• --markets-api-url flag
• --markets-repo flag

When splitting miner and markets, we recommend adding .bashrc (or equivalent) with the API address / path for your markets node. You can mix and match, for example, the client configuration inside the machine running the markets node:

FULLNODE_API_INFO=...
MINER_API_INFO=...
LOTUS_MARKETS_PATH=...

A client configuration that speaks to all nodes remotely:

FULLNODE_API_INFO=...
MINER_API_INFO=...
MARKETS_API_INFO=...

Furthermore, we have made lotus-miner CLI commands more intelligent now, as depending on what they do, they will know whether to hit the markets node or the miner node (and they support the monolith deployment too).

[jennijuju]

2021.07.29 - Stage III

Recap From Stage II

Thanks to @TippyFlitsUK, @Meatball13 and @BenjenminH, a few dagstore enhancements were identified and are now implemented in m1.3! Learn more about the previous issue and the implemented solution in DAG store migration.

Another feedback we got is using the same LOTUS_MINER env var for both miner and market subsystem is confusing anderror-prone , therefore, we are also introducing dedicated markets environment variables and flags in the newer releases!

Stage III Overview

Given the feedbacks we received from the fellows from the previous stages, we decided to break the MinerX.2 group into even more smaller cohorts and test things gradually. So for this stage, we are testing with two cohorts:

• Bleeding Edge to test DAG+MRA with m1.3
• Shipperz to run v1.11.1-rc2

Cohort Bleeding-Edge

Test Users

5 fellows

Assignment

• Upgrade and restart lotus node and the monolith miner with m1.3 to perform a DAG migration
• Split main miner and main miner process

Steps

1. Checkout and update your lotus miner/worker and market(if applicable) to m1.3

2. Optional: If you have run m1.2 and have started a DAG store migration, do

   rm -rf ~/.lotusminer/dagStore to start fresh or you can keep it and lotus will pick it up.

3. Restart miners and workers.

4. Monitor the DAG migration as described in the Monitoring the migration section here.

5. If the DAG migration is ☠️, save the logs and gorountines, report the issue and follow the instructions in DAG store troubleshooting.

6. Once the DAG migration is ✅, read the instructions here thoroughly to set up the lotus market process. (If you participated in Stage I or II, please revisit the docs as well as we have updated it quite a lot based on the feedbacks!)

7. Restart your lotus daemon and miner process with the markets subsystem.

8. Run the command here to set the logging levels.

9. Set the ask price of storage deals from M1 client for:

• - --price no higher 0.00000002 FIL per GB / epoch
• -verified-price to be 0.
• -min-piece-size to be no larger than 4GB.
• -max-piece-size to be no less than your miner sector size(32/64 GiB).

1. Start to accept deals, remember to save the logs!
2. Once you get your daemon and miner running, you can use lotus -vv or lotus-miner -vv to trace which API endpoints are being used and cross-checking with your configuratoin.

Note: You might encounter some issues for setting up the lotus-miner markets subsystem for the first time and get it running, if you have any questions, leave a comment in this discussion.

Disclaimer

You don't need to stop your usual dealmaking activities for the testing, however, we do expect you to whitelist and accept deals from the M1 testing clients. If a trade-off is required, we do request that you prioritize the testing deals.

What to expect

You can expect storage deals from the M1 client starting today. Deals will include both regular and verified deals and the size range is 4GB-32/64GB. If you don't get any deals, please ping @jennijuju or @dkkapur in the MinerX.2 Slack channel.
In a couple of days, you can also expect retrieval deals on the data you stored with the storage deal made earlier.

Risks and Mitigations

If deal-making is not working well with the market service process, you can follow the instruction here to roll back to lotus-miner monolith process.

Cohort Shipperz

coming soon

[raulk]

m1.3.4 release notes

Tag: https://github.com/filecoin-project/lotus/releases/tag/m1.3.4.

For miners already running m1 releases

This release introduces two changes that require action on your side.

Prior to upgrading:

1. Rename your dagStore directory under $LOTUS_MARKETS_PATH or $LOTUS_MINER_PATH directory to dagstore (remove the capital S).
2. Cancel all transfers for inbound storage deals that are still transferring, using the lotus-miner data-transfers cancel command.
   • Reason: this release changes attributes in the internal deal state objects, and the paths to the staging CARs were the deal data was being placed will be lost.

After upgrading:

1. Restart the transfer for inbound storage deals that hadn't finished transferring, using the lotus-miner data-transfers restart command.

Changelog

• bugfix: CARv2 panic on read after closure.
• bugfix: races in Graphsync transfer led to unreliable cancellation.
• Redesigned dagstore migration logic. See below for more information.
• Introduced five dagstore JSON-RPC and CLI commands (see screenshots):
  • lotus-miner dagstore list-shards
  • lotus-miner dagstore initialize-shard <key>
  • lotus-miner dagstore initialize-all --concurrency=10
  • lotus-miner dagstore recover <key>
  • lotus-miner dagstore gc
• Dagstore configuration now performed through config.toml. Refer to documentation for further information. Default:

[DAGStore]
RootDir = "$LOTUS_HOME/dagstore"
MaxConcurrentIndex = 5
MaxConcurrentReadyFetches = 0
MaxConcurrencyStorageCalls = 100
GCInterval = "1m"

• The dagstore default fetch throtting value of 2 concurrent fetches has been removed. The default value is now unlimited, and migration concurrency can be controlled through the --concurrency flag of the lotus-miner dagstore initialize-all command.
• Quality of life improvement: fstmp files that were formerly created directly under the Lotus repo are now placed under a tmp subdirectory.
  -lotus-miner commands have been audited and fixed so they work properly in split markets/miner deployments.

Redesigned dagstore migration logic

m1.3.4 replaces the staging Badger store with a new component called dagstore. It is designed to provide higher efficiency and throughput, and reduce the system resource footprint of deal-making processes.

• On storage deals, miners no longer stage the data into a Badger store prior to forming the unsealed CAR. Instead, the unsealed CAR is built directly as the data is being transferred through graphsync.
• On retrieval deals, miners no longer load the unsealed CAR into a Badger blockstore first. Retrievals are served directly from the data at rest (the unsealed CAR).

Refer to the dagstore's technical design for more information.

Previous m1.x releases used a migration process to register all deals in the dagstore that proved problematic for miners with many deals.

Starting with m1.3.4, the migration will register all shards in lazy initialization mode. As deals come in, those shards are fetched and initialized just in time.

You can also force initialization using the initialize-all command, and you can set a concurrency level through a flag. You can stop/start this command at your wish/convenience as proving deadlines approach and elapse. Forcing initialization will be something we ask you to do to populate your local indices, so you can serve/export them to the network in the future.

In our test environments, we found the new migration to progress at a rate of 400-500 shards/deals per second.

[jennijuju]

2021.08.09 - Stage IV

Recap From Stage III

From Stage III, most of the fellows have upgraded to v1.11.1-rc2 and up and we are seeing a huge improvements in deal making:

Miners who have split the market process also have very positive feedback, for example:

Since then, we have implemented a new DAG shards registration process which gives the user more control according to their setup and avoids missing Window PoSt due to the CPU usage, most of the work is done and has been introduced in m1.3.4. The highlight being starting with m1.3.4, the migration will register all shards only in lazy initialization mode. Shards are fetched and initialized just in time as the deals come in. For existing storage deal data, shards will be initialized when the first retrieval deal arrives that access that data. After #7010 is resolved, likely in m1.3.5, we will ask miners to try the very user-controllable force initializations to register all shards for existing data.

Stage IV Overview

For this testing stage, we will be focusing on splitting the market system.

Test Users

Cohort Bleeding-Edge and f099

Assignment

• Before you start, make sure you have read:
  • MRA docs, including subsystem concepts, split the market process and how to interact across the nodes via CLI.
  • CARv2 + DAG store implementation notes
• Upgrade and restart lotus node and the monolith miner with m1.3.4 to split the market process first
  • Provide feedback on the [latest MRA guides]( read the instructions here thoroughly) and let us know what's missing or still need clarification.
• Dag Migration (this step is done as you start the node with m1.3.4)

Note: Register shards for existing deals in dagstore is out of the scope of this testing stage. Please do not run lotus-miner dagstore initialize-all until furthur notice before m1.3.5.

Using M1 discussion for questions and disucssion, create an M1 issue in lotus for bug report and create an issue in filecoin docs for feedbacks on the guides.

Steps

1. Optional*: If you have started a DAG store migration previously, you can either start fresh by running rm -rf ~/.lotusminer/dagStore or rename your existing dagStore directory under $LOTUS_MARKETS_PATH or $LOTUS_MINER_PATH directory to dagstore (remove the capital S).
2. Cancel all transfers for inbound storage deals that are still transferring, using the lotus-miner data-transfers cancel command.
   Reason: this release change attributes in the internal deal state objects, and the paths to the staging CARs were the deal data was being placed will be lost.
3. Checkout and update your lotus miner/worker and market(if applicable) to m1.3.4
4. Restart miners and workers.
5. If you have not split your market yet, read the instructions here thoroughly to set up the lotus market process. Follow it step by step, don't skip or jump any lines! (If you participated in Stage I, II or III, please revisit the docs as well as we have updated it quite a lot based on the feedbacks!)
6. You can also play around with the operations following CLI Interaction Across Subsystems Process.
7. Run the command here to set the logging levels.
8. Set the ask price of storage deals from M1 client for:

• - --price no higher 0.00000002 FIL per GB / epoch
• -verified-price to be 0.
• -min-piece-size to be no larger than 4GB.
• -max-piece-size to be no less than your miner sector size(32/64 GiB).

9. Start to accept deals, remember to save the logs!
10. If you have completed dag migration before, you should check the state of your shards to confirm that the dagstore recognised them. They should all be in ShardStateAvailable (or ShardStateServing if a shard is serving a retrieval). Any shards in ShardStateErrored should be recovered with lotus-miner dagstore recover-shard <key>.

Disclaimer

You don't need to stop your usual dealmaking activities for the testing, however, we do expect you to whitelist and accept deals from the M1 testing clients. If a trade-off is required, we do request that you prioritize the testing deals.

What to expect

You can expect storage deals from the M1 client starting today. Deals will include both regular and verified deals and the size range is 4GB-32/64GB. If you don't get any deals, please ping @jennijuju or @dkkapur in the MinerX.2 Slack channel.
In a couple of days, you can also expect retrieval deals on the data you stored with the storage deal made earlier.

Risks and Mitigations

If deal-making is not working well with the market service process, you can follow the instruction here to roll back to lotus-miner monolith process.

[raulk]

m1.3.5 release notes

This is a minor bugfix release.

Tag: https://github.com/filecoin-project/lotus/releases/tag/m1.3.5.

For miners already running m1 releases (e.g. m1.3.4)

This release fixes an issue that led to some deals not being registered in the dagstore during migration.

It is safe to trigger migration again with m1.3.5 -- this will not lead to any IO. Any shards that were already registered the first time will simply lead to an inoffensive warning in the log files. Any shards that were not registered the first time will be registered this time.

To force the migration with the new version:

1. Stop your markets node.
2. Delete your $LOTUS_MARKETS_PATH/dagstore/.shard-registration-complete marker file. Note that this is a hidden file.
3. Start your m1.3.5 markets node.

Changelog

• initialize-all: only initialize unsealed deals by default: M1: lotus-miner dagstore initialize-all --sealed #7010
• not all deals were migrated by default: M1: miner with 2k deals but 0 migrated to dagstore #7017
• fix in go-car for dagstore: mmap-related panic when serving retrieval #7005 for extra reliability

[jennijuju]

2021.08.11 - Stage V

Stage V Overview

For this testing stage, we will invite more fellows to split their market process out of the main miner and work two cohorts to further test the new dagstore migration and initialization tool.

Test Users

Cohort Bleeding: Edges 6 fellows
Cohort f099: 9 fellows
Cohort ShipperZ: 11 fellows

Readings

Before you proceed, make sure you have read:

• MRA - split your markets node docs, including subsystem concepts, split the market process and how to interact across the nodes via CLI.
• 🆕 🌟[DAGStore docs] (https://docs.filecoin.io/mine/lotus/dagstore/#frontmatter-title) Learn more about what dagstore is and how to interact with it! This is the first iteration of the dagstore doc, feedback is welcomed!
  • If you want to go deeper: CARv2 + DAG store implementation notes
• ‼️Especially for those who were running m1.3.4, make sure you have readm1.3.5 release note

Note: Please find the Assignments based on your cohort below.

Communication Platform

If you hit any blockers or have any questions, leave a comment:

• Bleeding Edges: in cohort channel or M1 Testing discussion
• f099 and ShipperZ: in minerx2 channel or M1 Testing discussion

Disclaimer

You don't need to stop your usual dealmaking activities for the testing, however, we do expect you to whitelist and accept deals from the M1 testing clients. If a trade-off is required, we do request that you prioritize the testing deals.

What to expect

You can expect storage deals from the M1 client starting today. Deals will include both regular and verified deals and the size range is 4GB-32/64GB. If you don't get any deals, please ping @jennijuju or @dkkapur in the MinerX.2 Slack channel.
In a couple of days, you can also expect retrieval deals on the data you stored with the storage deal made earlier.

Risks and Mitigations

If deal-making is not working well with the market service process, you can follow the instruction here to roll back to lotus-miner monolith process.

Cohort Bleeding Edge and Cohort f099

Task Assignments

• You should have split your market process by now, if not, it's not too late but please do it ASAP!
• DAG migration and Initizations.
  • Provide feedback on the dagstore documentation

Steps

1. If you were runing any previous M1 releases, stop your market node(If you have split the market process, only stop your market node. Otherwise, stop your miner node.) Delete your $LOTUS_MARKETS_PATH/dagstore/.shard-registration-complete marker file. Note that this is a hidden file.
2. Checkout and update your market node to m1.3.5 if you have split the market process already. If not, update your miner/worker node.
3. Start your markets node or miners/workers.
4. Monitor the DAG store first-time migration.
5. If you have not split your marekt yet, read the instructions here thoroughly to set up the lotus market process. Follow it step by step, don't skip or jump any lines! (If you participated in Stage I, II or III, please revisit the docs as well as we have updated it quite a lot based on the feedbacks!)
6. Make sure you have read CLI Interaction Across Subsystems Process.
7. Run the command here to set the logging levels.
8. Set the ask price of storage deals from M1 client for:

• -price no higher 0.00000002 FIL per GB / epoch
• -verified-price to be 0.
• -min-piece-size to be no larger than 4GB.
• -max-piece-size to be no less than your miner sector size(32/64 GiB).

9. Start to accept deals, remember to save the logs for troubleshooting potential issues!
10. If you have completed dag migration before, you should check the state of your shards to confirm that the dagstore recognised them. They should all be in ShardStateAvailable (or ShardStateServing if a shard is serving a retrieval). Any shards in ShardStateErrored should be recovered with lotus-miner dagstore recover-shard <key>.
11. If you haven't finished one complete dag migration and initialization, perform a bulk initialization and monitor the process. Note: if any error occured, please see how they will recover here. Let us know if you run into any issue in discussion or slack!

Cohort ShipperZ

Task Assignments

• Upgrade and restart lotus node and the monolith miner with m1.3.5 then split the market process
  • Provide feedback on the [latest MRA guides]( read the instructions here thoroughly) and let us know what's missing or still need clarification.
• Dag Migration (this process starts as you start the node with m1.3.5)
  • Provide feedback on the dagstore documentation

Steps

1. Cancel all transfers for inbound storage deals that are still transferring, using the lotus-miner data-transfers cancel command.
   Reason: this release changes attributes in the internal deal state objects, and the paths to the staging CARs were the deal data was being placed will be lost.
2. Checkout and update your lotus miner/worker to m1.3.5
3. Restart miners and workers.
4. Assuming you have not split your market node yet, read the instructions here thoroughly to set up the lotus market process. Follow it step by step, don't skip or jump any lines!
5. Make sure you have read CLI Interaction Across Subsystems Process.
6. Run the command here to set the logging levels.
7. Set the ask price of storage deals from M1 client for:

• --price no higher 0.00000002 FIL per GB / epoch
• --verified-price to be 0.
• --min-piece-size to be no larger than 4GB.
• --max-piece-size to be no less than your miner sector size(32/64 GiB).

9. Start to accept deals, remember to save the logs for troubleshooting potential issues.

[aarshkshah1992]

Logging

Miner process

Run

lotus-miner log set-level --system stores debug

Markets process

Run

lotus-miner --call-on-markets log set-level --system dt-impl --system dt-chanmon --system dt_graphsync --system markets --system data_transfer_network --system retrievaladapter --system retrieval --system markets-rtvl-reval --system dagstore debug

Client

lotus log set-level --system dt-impl --system dt-chanmon --system dt_graphsync --system markets --system data_transfer_network --system retrieval debug