Blockchain tip not set in REST context

[thevictor13]

Context

Unless I'm missing some configuration changes, it appears that the latest two cardano-wallet versions don't work paired with jormungandr. Either with existing or wiped database (chain folder).

Information	-
Version	v2020.5.6 (git revision: cf3d9c9) with jormungandr v0.8.18 AND v2020-04-28 with jormungandr v0.8.15 or v0.8.18
Platform	Linux the least
Installation	From Github Release

Steps to Reproduce

$ cardano-wallet launch --node-port 3100 --genesis-block-hash $(cat genesis-hash.txt) -- --config ./stakepool-config.yaml

Above referenced files exist and are valid, as previous versions of cardano-wallet and jormungandr v0.8.15 and v0.8.18 on their own run well with them.

Expected behavior

Jormungandr bootstraps successfully and cardano-wallet starts communicating with it.

Actual behavior

Fails as follows:

[cardano-wallet.main:Info:4] [2020-05-08 23:01:04.58 UTC] Running as v2020.5.6 (git revision: cf3d9c91e895d1a01f90e4ab3d10b81e260b3510)
[cardano-wallet.main:Info:4] [2020-05-08 23:01:04.58 UTC] Command line: /home/thevictor/.local/bin/cardano-wallet launch --node-port 3100 --genesis-block-hash 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676 -- --config ./stakepool-config.yaml
[cardano-wallet.main:Info:4] [2020-05-08 23:01:04.58 UTC] LaunchArgs {_hostPreference = Host "127.0.0.1", _listen = ListenOnPort 8090, _nodePort = Just 3100, _stateDir = Nothing, _syncTolerance = SyncTolerance 300s, _logging = LoggingOptions {loggingMinSeverity = Debug, loggingTracers = Tracers {applicationTracer = Const (Just Info), apiServerTracer = Const (Just Info), walletEngineTracer = Const (Just Info), walletDbTracer = Const (Just Info), stakePoolEngineTracer = Const (Just Info), stakePoolDbTracer = Const (Just Info), networkTracer = Const (Just Info), ntpClientTracer = Const (Just Info)}, loggingTracersDoc = Nothing}, _jormungandrArgs = JormungandrArgs {genesisBlock = Left (Hash {getHash = "\142M*4?=\207\147\&0\173\144\&5\179\232\209h\230r\137\EOT&/,CJO\143\147N\199\182v"}), extraJormungandrArgs = ["--config","./stakepool-config.yaml"]}}
[cardano-wallet.main:Info:4] [2020-05-08 23:01:04.58 UTC] Wallet state: Using directory: /home/thevictor/.local/share/cardano-wallet/jormungandr/testnet
[cardano-wallet.main:Info:4] [2020-05-08 23:01:04.58 UTC] Wallet databases: Using directory: /home/thevictor/.local/share/cardano-wallet/jormungandr/testnet/wallets
[cardano-wallet.application:Info:4] [2020-05-08 23:01:04.58 UTC] Wallet backend server starting. Launching Jörmungandr...
[cardano-wallet.application:Info:4] [2020-05-08 23:01:04.58 UTC] Node is Jörmungandr on testnet (0)
[cardano-wallet.network:Notice:4] [2020-05-08 23:01:04.58 UTC] Starting process jormungandr
     --genesis-block-hash 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676
     --rest-listen 127.0.0.1:3100
     --storage /home/thevictor/.local/share/cardano-wallet/jormungandr/testnet/chain
     --config ./stakepool-config.yaml

[cardano-wallet.network:Info:4] [2020-05-08 23:01:04.58 UTC] [jormungandr.8657] Process started
May 08 23:01:04.594 INFO Starting jormungandr 0.8.18 (HEAD-690eb547, release, linux [x86_64]) - [rustc 1.42.0 (b8cedc004 2020-03-09)], task: init
May 08 23:01:04.594 WARN Node started without path to the stored secret keys (not a stake pool or a BFT leader), task: init
May 08 23:01:04.595 INFO storing blockchain in '"/home/thevictor/.local/share/cardano-wallet/jormungandr/testnet/chain/blocks.sqlite"', task: init
May 08 23:01:05.259 INFO loading from storage, currently at d79f2082-00000001-0.530 processing=0ns (0ns per block) ..., task: bootstrap
[cardano-wallet.network:Info:12] [2020-05-08 23:01:05.59 UTC] Waiting for Jörmungandr to be ready on tcp/3100
[cardano-wallet.network:Notice:12] [2020-05-08 23:01:05.59 UTC] [jormungandr.8657] Terminating child process
[cardano-wallet.main:Debug:4] [2020-05-08 23:01:05.59 UTC] Logging shutdown.
cardano-wallet: ErrUnexpectedNetworkFailure (Link {_segments = ["api","v0","tip"], _queryParams = []}) (FailureResponse (Request {requestPath = (BaseUrl {baseUrlScheme = Http, baseUrlHost = "127.0.0.1", baseUrlPort = 3100, baseUrlPath = ""},"/api/v0/tip"), requestQueryString = fromList [], requestBody = Nothing, requestAccept = fromList [text/plain;charset=utf-8], requestHeaders = fromList []), requestHttpVersion = HTTP/1.1, requestMethod = "GET"} (Response {responseStatusCode = Status {statusCode = 500, statusMessage = "Internal Server Error"}, responseHeaders = fromList [("content-length","38"),("content-type","text/plain; charset=utf-8"),("date","Fri, 08 May 2020 23:01:05 GMT")], responseHttpVersion = HTTP/1.1, responseBody = "Blockchain tip not set in REST context"}))

---

Resolution

---

QA

[rvl]

Thanks for the report.

I can reproduce this error locally with the itn_rewards_v1 testnet but not the jormungandr self node. So that would explain why the jormungandr integration tests are still passing.

[KtorZ]

I recall a configuration change in Jörmungandr regarding the API prefix. It used to be included in the configuration but was later removed and assumed to always be "/api" and implicitly required. Could it be linked?

[piotr-iohk]

If I am not mistaken the issue is that currently during Jormungandr bootstrap when you ask it for a network tip the response is 500.

Start Jorm:

jormungandr \
     --genesis-block-hash 8e4d2a343f3dcf9330ad9035b3e8d168e6728904262f2c434a4f8f934ec7b676 \
     --rest-listen 127.0.0.1:8080 \
     --storage ./data_incentivized/chain \
     --config node_config_incentivized.yaml

Check status:

GET http://127.0.0.1:8080/api/v0/node/stats

v

Response body:
{"version":"jormungandr 0.8.19-776347cf","state":"Bootstrapping"}

While it is boostrapping, try to get the tip:

GET http://127.0.0.1:8080/api/v0/tip"

v

failed to make a REST request
  |-> node internal error
  |-> HTTP status server error (500 Internal Server Error) for url (http://127.0.0.1:8080/api/v0/tip)

And I suppose this is what is happening, we are asking for tip while Jormungandr is still bootsrapping and it returns 500 then.

Probably here -> https://github.com/input-output-hk/cardano-wallet/blob/master/lib/jormungandr/src/Cardano/Wallet/Jormungandr/Network.hs#L277, instead of asking for tip make sure that Jormungandr is no longer in Boostrapping state.

[rvl]

Doh ... it keeps changing. The endpoint used to return HTTP status 503 during bootstrap. But before that I think it did return 500 - however now we consider 500 to be a fatal error.
I don't think it's right to check the /api/v0/node/stats endpoint. The thing we need to succeed is /api/v0/tip. And there shouldn't be a reason why this endpoint can't work during bootstrap.

[piotr-iohk]

/api/v0/node/stats is a bit misleading name I guess... But it actually shows the state of Jorm and, as far as I see, currently on state:"Boostrappting" pretty much all other endpoints return 500 (except for /api/v0/node/stats)... so, currently it seems to be the only valid endpoint to query for the state.

Well, the alternative is to raise error in Jormungandr and desrcibe what would we expect. 🤷‍♂️