Early blocks not found with REST api with archive_mode = true

[leonbell0101]

Describe the bug
After full sync (from completely empty chain) with archive_mode = true REST API returns 404 for all blocks earlier than certain height. It still returns block headers for that height.

Full block not found:

curl 127.0.0.1:3413/v1/blocks/38000 --verbose
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3413 (#0)
> GET /v1/blocks/38000 HTTP/1.1
> Host: 127.0.0.1:3413
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< content-length: 0
< date: Thu, 21 Feb 2019 10:42:19 GMT
< 
* Connection #0 to host 127.0.0.1 left intact

However header exists:

curl 127.0.0.1:3413/v1/headers/38000 --verbose
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 3413 (#0)
> GET /v1/headers/38000 HTTP/1.1
> Host: 127.0.0.1:3413
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< content-length: 1443
< date: Thu, 21 Feb 2019 10:42:13 GMT
< 
{
  "hash": "0000024dc21c58783cf729f170325870f655b0bf336d8054795a347b12efbdf4",
...omitted

To Reproduce
Steps to reproduce the behavior:

1. build grin from tag v1.0.1
2. launch daemon to create ~/.grin/ directory and populate default config
3. stop daemon
4. change ~/.grin/main/grin-server.toml: change archive_mode to from false to true, remove line with api_secret_path
5. remove existing ~/.grin/main/chain_data
6. launch daemon and wait for a full sync
7. try to request 127.0.0.1:3413/v1/blocks/1 (first block that's not a genesis) and see 404 (expected full block)
8. try to request 127.0.0.1:3413/v1/headers/1 and see that it exists (as expected)

Relevant Information
Nothing relevant in logs, no errors, warnings or anything.

Desktop:

• OS: Ubuntu 16.04
• Version: grin v1.0.1

Additional context
I tried (completely different tries, unrelated to steps to reproduce):

1. waiting for a full sync in fast mode, then changing to archive_mode
2. starting from scratch in archive_mode
3. changing chain_data directory so it will resync it in archive_mode
4. rebuilding grin from tag 1.0.1 with a patch to default settings, so it will by-default start in archive_mode

and archived nothing.

[antiochp]

This is expected behavior.

archive_mode=true will retain historical blocks (will not delete them from the db), but it will not necessarily have these blocks to begin with after a fast sync.

Your "archive" node will only archive blocks received since it started processing blocks.
i.e. If you fast sync at height 50,000 you will only have blocks in the db from 50,000 onwards.

[leonbell0101]

@antiochp So how can I download entire historical blockchain (non-pruned)? I need to find node with full chain or what? It's entirely unclear from comments or from documentation.

[antiochp]

There is no "official" way of doing this. To support this we would need to commit to supporting archive nodes on the network that actually had all this data.
We made the decision not to support this as this data is not necessary to validate chain state (but we do accept this data is valuable for the wider ecosystem).

You will need to find someone who is willing to provide a db export.
Best bet is to ask here - https://gitter.im/grin_community/support

[leonbell0101]

@antiochp If I find node with full-chain, will I receive entirety of chain_data (via peers_allow in grin-server.toml)? Or do I need to copy files directly and find people who are willing to transfer it?

[antiochp]

You will need to copy the files. There is no support in the grin networking layer to request or provide historical blocks.