Documentation change for new retrieval functionality (#187)

README.md

@@ -92,11 +92,10 @@ This is useful if you only need deal preparation but not deal making.
 You can still have deal making enabled, but disabling it will use slightly less system resources.  
 In [default.toml](./config/default.toml) from your repo
 
-1. change `index_service.enabled` to false
-2. change `ipfs.enabled` to false
-3. change `deal_tracking_service.enabled` to false
-4. change `deal_replication_service.enabled` to false
-5. change `deal_replication_worker.enabled` to false
+1. change `ipfs.enabled` to false
+2. change `deal_tracking_service.enabled` to false
+3. change `deal_replication_service.enabled` to false
+4. change `deal_replication_worker.enabled` to false
 
 ### Use External MongoDb database
 
@@ -131,7 +130,6 @@ Commands:
   init              Initialize the configuration directory in SINGULARITY_PATH
                     If unset, it will be initialized at HOME_DIR/.singularity
   daemon            Start a daemon process for deal preparation and deal making
-  index             Manage the dataset index which will help map the dataset path to actual piece
   preparation|prep  Manage deal preparation
   help [command]    display help for command
 ```
@@ -453,6 +451,20 @@ location /propose {
 }
 ```
 
+## Retrieval
+
+The recommended way for Retrieval is via bitswap protocol.
+You need the storage provider to run [booster-bitswap](https://boost.filecoin.io/bitswap-retrieval).
+
+Then you may use `ipfs get <RootCid>/sub/path/to/file` to retrieve the file or folder. The `ipfs` version needs to be 0.18.0+.
+
+The `RootCid` can be found in `singularity prep list` and will be automatically generated when the dataset is fully prepared.
+
+If you find `RootCid` missing, or you're using an older version of Singularity (before 3.0.0),
+you can regenerate the `RootCid` by running `singularity prep dag <dataset>`.
+This will generate another CAR file that encapsulates the IPLD DAG of the whole dataset.
+You will need to get that new CAR file sealed before you can perform bitswap retrieval.
+
 ## Configuration
 
 Look for `default.toml` in the initialized repo.
@@ -587,8 +599,7 @@ however this currently only works when the tmpDir is used.
 
 ### Does it work on Windows
 
-Only Deal Preparation works and Indexing works on Windows.
-Deal Replication and Retrieval only works in Linux/Mac due to dependency restrictions.
+This software is not extensively tested on Windows.
 
 ### Error - too many open files
 

config/default.toml

@@ -22,9 +22,6 @@ deal_preparation_service = 'http://127.0.0.1:7001'
 deal_replication_service = 'http://127.0.0.1:7004'
 deal_self_service = 'http://127.0.0.1:7005'
 
-# Connection string to index service, used by Singularity CLI
-index_service = 'http://127.0.0.1:7003'
-
 [deal_preparation_service]
 enabled = true
 # API to serve singularity prep commands
@@ -52,18 +49,6 @@ num_workers = 4
 # Buffer for child spawn
 max_buffer = 1073741824
 
-[index_service]
-enabled = true
-# API to serve singularity index commands
-bind = "0.0.0.0"
-port = 7003
-# If not using the built-in IPFS daemon, use below URI to connect to IPFS API
-ipfs_http = "http://localhost:5001"
-
-[ipfs]
-# Whether to start a built-in IPFS daemon - experimental
-enabled = false
-
 [http_hosting_service]
 enabled = true
 bind = "0.0.0.0"

getting-started.md

@@ -165,56 +165,16 @@ singularity repl start -m 10 -c '0 * * * *' -x 1000 -xp 100 MyData f01111 f1...
 
 This command creates a schedule to send 10 deals each hour for up to 1000 deals in total with up to 100 pending deals.
 
-## Indexing and Retrieval
-
-Each file and folder in the dataset has its own unique CID.
-Usually, when you retrieve the file or folder from storage provider, you need to specify the CID of the file or folder.
-
-However, for most people coming from web2.0, we are used to retrieving resources using URI.
-That's where the index and retrieval module shines.
-
-The index module publishes the mapping from URI path to the CID,
-so end users can retrieve content using URI instead of CID.
-
-### Prerequisites
-
-Install [IPFS](https://docs.ipfs.tech/install/) and start the daemon service
-
-```shell
-ipfs init
-ipfs daemon
-```
-
-The index will be published on IPFS which is required to be run by client and retrievers.
-
-### Create Index
-
-Index creation is as simple as running a single command for a specific dataset.
-
-```shell
-singularity index create MyData
-```
-
-It will return the IPFS path that contains the index,
-and you can choose to publish it with DNSLink to make it more end-user friendly.
-
-```text
-Add or update the TXT record for _dnslink.mydata.net
-  _dnslink.mydata.net  34  IN  TXT "dnslink=/ipfs/bafy..."
-```
+You can also setup a Deal Making API to grant full control of deal making speed to storage providers.
+See [Deal Making API](./README.md#deal-making-self-service) for more details.
 
 ### Retrieval
 
-With index published on IPFS, the end user can now browse the dataset from Filecoin network
+Retrieval is as simple as `ipfs get <RootCid>/path/to/file/or/folder`.
+You can check the `RootCid` from the result of `singularity prep list`.
 
-```shell
-# ls command tells what file or folder is inside the sub path of the dataset
-singularity-retrieve ls -v singualrity://ipns/mydata.net/sub/path
-# show command tells the CID of the file or folder and how to assemble them back 
-singularity-retrieve show singualrity://ipns/mydata.net/sub/path
-# cp command retrieves the data from a specific storage provider to a local path and reassemble them if needed
-singularity-retrieve cp -p f01111 singualrity://ipns/mydata.net/sub/path ./local/path
-```
+Note that the storage providers need to setup [booster-bitswap](https://boost.filecoin.io/bitswap-retrieval) and you'll need
+to use ipfs version 0.18.0 or above to make this to work.
 
 ## Next Steps