Use SlotNo and TimeInterpreter

[Anviking]

Issue Number

ADP-356 #1868 #1869

Overview

• Add TimeInterpreter, necessary queries, and a way to run them
  Add tests comparing them with old implementation

• Fundamentally change wallet to use SlotNo instead of SlotId

  • Patch together cardano-wallet-jormungandr to still work, despite the
    binary using SlotId.

• Remove redundant byron code

• Sync progress calculation compares times now instead of slots

• Rename the slot tables to slot_no to force automatic migration, preventing catastrophic re-interpretation of old data

Comments

• Depends on Move slotting to separate module #1890

[Anviking]

☝️ We may now have a TimeInterpreter IO that sometimes takes a long time to run a conversion query (when it needs to refetch the summary over LSQ).

Not sure if we should change…

The re-fetching should happen on very rare occasions though…

[rvl]

I think it would be better to have the network layer query the Interpreter on a schedule, and stash it in a mutable variable. So that all queries can be pure.

[Anviking]

So that all queries can be pure

I presume it would still be in IO (to access the var), so no change to the interface.

In that case I'd be concerned with what callers would see if there is nothing in the var. There are many call-sites, and I think blocking until it is available is the simplest thing to. Which is somewhat like this.

But preemptively refreshing before a conversion fails, sounds sane to do… optimally.

[rvl]

Let's see. I was thinking that the initial Interpreter will be one constructed from genesis parameters.
It will get updated once the node has synced far enough. (Question - how far is enough?)
This would prevent the wallet from being unresponsive while the node is syncing.

[Anviking]

I suspect we may need manual migration

[rvl]

Don't fret over DB migration if it's too hard. We can start from a fresh state directory if necessary.

[Anviking]

On second thought, I fear/wonder if users would silently get corrupted data, as the meaning of the slot table has changed, but not the name.

Is there a good way to test / detect this?

Or I guess testing manually using CLI works. I tried using the Daedalus mainnet installation, and replacing the wallet and node manually, but I think too many other things have changed for it to work.

[rvl]

If the type of the column changes, I think persistent auto migrations will either throw an exception or helpfully delete the table.

Let's decide next week whether to automatically migrate or make a fresh state directory.

[rvl]

What's the benefit of wrapping the provided Interpreter in our own type? Can we use the bare Interpreter?

[Anviking]

What's the benefit of wrapping the provided Interpreter in our own type?

The original interpreter only works with RelativeTime, which is relative to genesis.

The wrapper carries the genesis start date along with it.

[Anviking]

bors try

[iohk-bors]

try

Build failed

• ci/hydra-build:required

[Anviking]

bors try

[Anviking]

I am seeing rare failures of this sort. I am investigating

  test/unit/Cardano/Pool/DB/Properties.hs:348:17:
  1) Cardano.Pool.DB.Sqlite.Sqlite, Stake Pool properties, readPoolProduction for a given epoch should always give slots from given epoch
       Falsifiable (after 10 tests and 1 shrink):
         StakePoolsFixture {poolSlots = [(PoolId {getPoolId = "w\EOTe\227Aa\169\r=\fK%v>C\178\r/\ETX\v\FS\131>\ACK\225\aG$u\a\v>"},BlockHeader {slotNo = SlotNo {unSlotNo = 10}, blockHeight = Quantity {getQuantity = 0}, headerHash = Hash {getHash = "00000000000000000000000000000001"}, parentHeaderHash = Hash {getHash = "00000000000000000000000000000000"}}),(PoolId {getPoolId = "\138P0PcK\141\161.\a\DC1\DC3L\ESC\a\SOT\FS\US\245[\209&Q\CAN\DLEWU\249\DC1\165?"},BlockHeader {slotNo = SlotNo {unSlotNo = 0}, blockHeight = Quantity {getQuantity = 0}, headerHash = Hash {getHash = "00000000000000000000000000000001"}, parentHeaderHash = Hash {getHash = "00000000000000000000000000000000"}}),(PoolId {getPoolId = "w\EOTe\227Aa\169\r=\fK%v>C\178\r/\ETX\v\FS\131>\ACK\225\aG$u\a\v>"},BlockHeader {slotNo = SlotNo {unSlotNo = 8}, blockHeight = Quantity {getQuantity = 0}, headerHash = Hash {getHash = "00000000000000000000000000000001"}, parentHeaderHash = Hash {getHash = "00000000000000000000000000000000"}})], rollbackSlots = [SlotNo {unSlotNo = 10},SlotNo {unSlotNo = 0},SlotNo {unSlotNo = 8}]}
       expected: fromList [SlotNo {unSlotNo = 0},SlotNo {unSlotNo = 8}]
        but got: fromList [SlotNo {unSlotNo = 10}]

[Anviking]

bors try

[iohk-bors]

try

Build succeeded

• buildkite/cardano-wallet
• ci/hydra-eval
• ci/hydra-build:required

[rvl]

Nice work - this is a really big PR.
I have checked Cardano.Wallet.Primitive and Cardano.Wallet.Shelley.
Is there anything else I should pay attention to?

[rvl]

Will these all be removed in a future PR, or moved to a tests-only module?

[Anviking]

I suppose we should remove them completely, and re-write the tests to use the new ones.

Now we are testing the old ones, and that the new ones are equivalent to the old ones (Generators aren't perfect though).

[rvl]

I like this. :-)

[rvl]

With the database migration issue, we can't re-use the slot columns if the meaning of their values has changed. So we have to do something about this situation.

We could either:

1. implement some "manual migration" code to restore all wallets from genesis if loading a DB from before this change.
2. say that the DB is incompatible with previous versions, and a fresh state directory must be used.

In the case of 2, the DB migration test can be disabled, because there is no migration to test. I like option 2, because there are other user stories such as transaction TTL that also need major DB schema changes.

[Anviking]

Migrating from 2020-07-06 works. Migrating from the current flight candidate to this PR doesn't:

But in the real world, users would be upgrading from an older Daedalus Mainnet release, which might trigger automatic migration.

In the case of 2, the DB migration test can be disabled, because there is no migration to test. I like option 2, because there are other user stories such as transaction TTL that also need major DB schema changes.

✔️

Regardless, I think thinking about this later is fine, as you suggested in another comment.

[Anviking]

bors r+

[iohk-bors]

Canceled

[Anviking]

bors r+

[iohk-bors]

Build failed

• ci/hydra-eval

[Anviking]

bors r+

[iohk-bors]

Build failed

• ci/hydra-build:required

  test/unit/Cardano/Wallet/Primitive/SlottingSpec.hs:69:13: 
  1) Cardano.Wallet.Primitive.Slotting.slotting, runQuery NEW singleEraInterpreter == OLD . fromFlatSlot, slotRangeFromTimeRange and slotRangeFromTimeRange'
       uncaught exception: ErrorCall
       fromFlatSlot: The specified flat slot number (7469933546) is higher than the maximum flat slot number (6442450943) for the specified epoch length (3).
       CallStack (from HasCallStack):
         error, called at src/Cardano/Wallet/Primitive/Slotting.hs:331:9 in cardano-wallet-core-2020.7.6-FPg0XPMd6uj7UsprjiHPOH:Cardano.Wallet.Primitive.Slotting
       (after 912301 tests and 3 shrinks)
         GenesisParameters {getGenesisBlockHash = Hash {getHash = "Genesis Hash"}, getGenesisBlockDate = StartTime 1865-01-03 04:58:42 UTC, getSlotLength = SlotLength {unSlotLength = 0.145323864972s}, getEpochLength = EpochLength {unEpochLength = 3}, getEpochStability = Quantity {getQuantity = 0}, getActiveSlotCoefficient = ActiveSlotCoefficient {unActiveSlotCoefficient = 1.0}}
         Range {inclusiveLowerBound = Nothing, inclusiveUpperBound = Just 1899-05-29 13:18:56 UTC}
         Exception thrown while showing test case:
           fromFlatSlot: The specified flat slot number (7469933546) is higher than the maximum flat slot number (6442450943) for the specified epoch length (3).
           CallStack (from HasCallStack):
             error, called at src/Cardano/Wallet/Primitive/Slotting.hs:331:9 in cardano-wallet-core-2020.7.6-FPg0XPMd6uj7UsprjiHPOH:Cardano.Wallet.Primitive.Slotting
         

  To rerun use: --match "/Cardano.Wallet.Primitive.Slotting/slotting/runQuery NEW singleEraInterpreter == OLD . fromFlatSlot/slotRangeFromTimeRange and slotRangeFromTimeRange'/"

Randomized with seed 212063873

[Anviking]

bors r+

[iohk-bors]

Build failed

• ci/hydra-eval

error: unable to fork: Cannot allocate memory

[Anviking]

bors r+

[iohk-bors]

Build succeeded

• ci/hydra-build:required
• buildkite/cardano-wallet
• ci/hydra-eval