[CBR-495] Fix inconsistent metadata store after deletion

[KtorZ]

Description

We store some information related to transaction in the metadata store (sqlite). However, when looking up transaction with metadata referring to wallet we don't know about, we fail with a not so friendly error "WalletNotFound" despite no wallet being given as part of the query.

This fix is actually two folds:

• It discards incoherent transactions fetched from the DB, if any,
  and shout a warning in the log. This is in order to make the system
  more resilient to conconcurrent calls while a wallet or account is
  being deleted (since metadata and accounts / wallets are stored in
  separated databases, we can't easily run both delete in a single
  transaction).

• It also deletes corresponding metadata when an account or a wallet
  is removed. This may cause extra damage? What if there are pending
  transactions when we delete the account or wallet.

Linked issue

CBR-495

Type of change

• 🐞 Bug fix (non-breaking change which fixes an issue)
• 🛠 New feature (non-breaking change which adds functionality)
• ⚠️ Breaking change (fix or feature that would cause existing functionality to change)
• 🏭 Refactoring that does not change existing functionality but does improve things like code readability, structure etc
• 🔨 New or improved tests for existing code
• ⛑ git-flow chore (backport, hotfix, etc)

Developer checklist

• I have read the style guide document, and my code follows the code style of this project.
• If my code deals with exceptions, it follows the guidelines.
• I have updated any documentation accordingly, if needed. Documentation changes can be reflected in opening a PR on cardanodocs.com, amending the inline Haddock comments, any relevant README file or one of the document listed in the docs directory.
• CHANGELOG entry has been added and is linked to the correct PR on GitHub.

Testing checklist

• I have added tests to cover my changes.
• All new and existing tests passed.

QA Steps

On input-output-hk/cardano-sl I've added the following integration tests:

    scenario "Reg#141: metadata become inconsistent after a wallet is deleted" $ do
        fixture <- setup $ defaultSetup
            & initialCoins .~ [14]

        response <- request $ Client.getTransactionIndexFilterSorts
            $- Nothing
            $- Nothing
            $- Nothing
            $- defaultPage
            $- defaultPerPage
            $- NoFilters
            $- NoSorts

        verify response
            [ expectSuccess
            ]

    scenario "Reg#141: metadata become inconsistent after an account is deleted" $ do
        fixture <- setup $ defaultSetup
            & initialCoins .~ [500000]

        request_ $ Client.postAccount
            $- fixture ^. wallet . walletId
            $- NewAccount
                noSpendingPassword
                "My Second Account"

        request_ $ Client.deleteAccount
            $- fixture ^. wallet . walletId
            $- defaultAccountId

        response <- request $ Client.getTransactionIndexFilterSorts
            $- Just (fixture ^. wallet . walletId)
            $- Nothing
            $- Nothing
            $- defaultPage
            $- defaultPerPage
            $- NoFilters
            $- NoSorts

        verify response
            [ expectSuccess
            ]

Both now passes. Beside, in case of future inconsistency like this in the DB, the whole endpoint won't fail but instead, inconsistent transactions will be discarded and a warning printed in the logs:

[cardano-sl.walletapi:Info:ThreadId 1201] [2018-12-08 15:05:49.40 UTC] 
GET Request #28
    :> api
    :> v1
    :> transactions
    :> 'wallet_id' field: -
    :> 'account_index' field: -
    :> 'address' field: -
    :> pagination: page=1, per_page=10
    :> filter_by: -
    :> sort_by: -
[cardano-sl.edge:Warning:ThreadId 1201] [2018-12-08 15:05:49.40 UTC] Inconsistent entry in the metadata store: UnknownHdAccountRoot HdRootId Ae2tdPwUPEZKUpTf4jDurpvX7fdUrmZvFPM56TAwihnBbAuyvMZ47NeHHdY
[cardano-sl.walletapi:Info:ThreadId 1201] [2018-12-08 15:05:49.40 UTC] 
    Response #28 OK 0.002410603s

Screenshots (if available)

How to merge

Send the message bors r+ to merge this PR. For more information, see
docs/how-to/bors.md.

[KtorZ]

Added an extra integration test to illustrate last comment and @kderme addition.

    scenario "Reg#141: metadata becomes inconsistent after a wallet is deleted and restored" $ do
        fixture <- setup $ defaultSetup
            & initialCoins .~ [500000]

        [verifyTxInHistory, verifyWalletIsRestored] <- sequence
            [ do -- // 1 Make sure there's a transaction in the history
                response <- request $ Client.postTransaction $- Payment
                    (defaultSource fixture)
                    (defaultDistribution 42 fixture)
                    defaultGroupingPolicy
                    noSpendingPassword
                verify response
                    [ expectTxStatusEventually (fixture ^. wallet) [InNewestBlocks, Persisted]
                    ]
                -- // 3 Later, check that we can recover that transaction
                return $ verify response
                    [ expectTxInHistoryOf  (fixture ^. wallet)
                    ]
            , do -- // 2 Remove and then, restore the wallet
                request_ $ Client.deleteWallet $- fixture ^. wallet . walletId
                response <- request $ Client.postWallet $- NewWallet
                    (fixture ^. backupPhrase)
                    noSpendingPassword
                    defaultAssuranceLevel
                    defaultWalletName
                    RestoreWallet
                return $ verify response
                    [ expectWalletEventuallyRestored
                    ]
            ]
        verifyWalletIsRestored >> verifyTxInHistory

Succeeds with the last addition, but otherwise fails with:

  test/integration/Test/Integration/Framework/DSL.hs:131:23: 
  1) Transactions Reg#141: metadata becomes inconsistent after a wallet is deleted and restored
       uncaught exception: IOException of type UserError
       user error (expectWalletEventuallyRestored: waited too long for restoration.)

[KtorZ]

Another integration test added to cover these bits 👍 , cf comments.

[KtorZ]

bors r+

[iohk-bors]

Timed out (retrying...)

[iohk-bors]

Timed out

[jmitchell]

bors r+

[iohk-bors]

Build succeeded

• ci/hydra:serokell:cardano-sl:required
• buildkite/cardano-sl