Testing pool fetching runtime selection #2225

piotr-iohk · 2020-10-08T12:11:01Z

Issue Number

Overview

8c0c580
More detailed description of poolMetadataSource in swagger.yaml
439f8f0
Integration test checking metadata is re-synced after settings update
1d13fd5
Fix existing stake pool test checking metadata exists

Comments

hasufell · 2020-10-08T12:35:11Z

specifications/api/swagger.yaml

@@ -303,7 +303,13 @@ x-settings: &settings
  properties:
    pool_metadata_source:
      <<: *poolMetadataSource
-      description: How to fetch pool metadata
+      description: |
+        Select stake pool metadata fetching strategy:


The values are already explained here, no?

x-poolMetadataSource: &poolMetadataSource description: | Pool metadata source. This sets the metadata fetching strategy. Possible values are * null -> no fetching * direct -> direct fetching * uri -> use SMASH server oneOf: - type: string enum: - none - direct - title: URI type: string format: uri example: https://smash.cardano-mainnet.iohk.io/

Yes, but I figured it renders nicely in the API doc. :) (see screenshot in Comment in the pr)

Hmm, how do we make sure the doc doesn't go async with the other place?

There's this -> https://github.com/input-output-hk/cardano-wallet/blob/master/lib/core/test/unit/Cardano/Wallet/Api/TypesSpec.hs#L411-L414.

Descriptions are manual maintenance, but having stuff clearly explained improves user experience.

hasufell · 2020-10-08T12:35:39Z

specifications/api/swagger.yaml

+          - `direct` - metadata is fetched directly from chain,
+          - `uri` - metadata is fetched from the external Stake-Pool Metadata Aggregation Server (SMASH)
+
+        After update existing metadata will be dropped forcing it to re-sync automatically with the new setting.


This sentence is useful indeed!

hasufell · 2020-10-08T12:41:25Z

lib/core-integration/src/Test/Integration/Scenario/API/Shelley/Settings.hs

+        eventually "1. There is no metadata" $ do
+            metadatas <- getMetadata
+            metadatas `shouldSatisfy` (all isNothing)


Haven't checked if it compiles, but a little golfing so we could have it on one line maybe:

Suggested change

eventually "1. There is no metadata" $ do

metadatas <- getMetadata

metadatas `shouldSatisfy` (all isNothing)

eventually "1. There is no metadata" $ getMetadata >>= (`shouldSatisfy` all isNothing)

hasufell · 2020-10-08T12:43:42Z

lib/core-integration/src/Test/Integration/Scenario/API/Shelley/StakePools.hs

@@ -735,6 +737,7 @@ spec = describe "SHELLEY_STAKE_POOLS" $ do
                saturation `shouldSatisfy` (any (> 0))

        it "contains pool metadata" $ \ctx -> do
+            updateMetadataSource ctx "direct"


Hmm, what's the default of the tests? Why did it work before? 😄

It used to check on-chain by default before, but now its none... that's what it is also now in the integration test cluster.
Yes, I was surprised it worked, with this check passing:

metadataActual `shouldSatisfy` (`Set.isSubsetOf` metadataPossible)

so it turns out that:

> (fromList []) `isSubsetOf` (fromList [1]) True

hasufell · 2020-10-08T12:44:52Z

lib/core-integration/src/Test/Integration/Scenario/API/Shelley/Settings.hs

+    it "SETTINGS_02 - Changing pool_metadata_source re-syncs metadata" $ \ctx -> do
+        let toNone = "none"
+            toDirect = "direct"
+            getMetadata = map (view #metadata) . snd <$> unsafeRequest


I personally prefer fmap over map, because even if we change from list to set or nonempty list, it still compiles... but ymmv

piotr-iohk · 2020-10-08T13:10:50Z

lib/core-integration/src/Test/Integration/Scenario/API/Shelley/StakePools.hs

@@ -771,6 +774,8 @@ spec = describe "SHELLEY_STAKE_POOLS" $ do
                                mapMaybe (fmap getApiT . view #metadata) pools
                        metadataActual
                            `shouldSatisfy` (`Set.isSubsetOf` metadataPossible)
+                        metadataActual
+                            `shouldSatisfy` (not . Set.null)


Therefore I added this...

piotr-iohk · 2020-10-08T14:55:57Z

bors r+

2225: Testing pool fetching runtime selection r=piotr-iohk a=piotr-iohk # Issue Number #2163 / [ADP-427](https://jira.iohk.io/browse/ADP-427) # Overview - 8c0c580 More detailed description of poolMetadataSource in swagger.yaml - 439f8f0 Integration test checking metadata is re-synced after settings update - 1d13fd5 Fix existing stake pool test checking metadata exists # Comments ![Screenshot from 2020-10-08 14-06-59](https://user-images.githubusercontent.com/42900201/95456553-ca995b80-096f-11eb-89ee-b1b42ae8d40f.png) Co-authored-by: Piotr Stachyra <[email protected]>

iohk-bors · 2020-10-08T17:19:18Z

Timed out.

Failures:

  src/Test/Integration/Scenario/API/Shelley/Settings.hs:86:5: 
  1) API Specifications, SHELLEY_SETTINGS, SETTINGS_02 - Changing pool_metadata_source re-syncs metadata
       uncaught exception: IOException of type UserError
       user error (Waited longer than 90s (more than 2 epochs) for an action to resolve. Action: "2. There is metadata". Error condition: Just (HUnitFailure (Just (SrcLoc {srcLocPackage = "cardano-wallet-core-integration-2020.9.30-6PfBDqTjuY813ZWL2vU0Do", srcLocModule = "Test.Integration.Scenario.API.Shelley.Settings", srcLocFile = "src/Test/Integration/Scenario/API/Shelley/Settings.hs", srcLocStartLine = 98, srcLocStartCol = 30, srcLocEndLine = 98, srcLocEndCol = 56})) (Reason "predicate failed on: [Nothing,Nothing,Nothing]")))

  To rerun use: --match "/API Specifications/SHELLEY_SETTINGS/SETTINGS_02 - Changing pool_metadata_source re-syncs metadata/"

#expected

piotr-iohk · 2020-10-08T17:20:00Z

bors r+

2225: Testing pool fetching runtime selection r=piotr-iohk a=piotr-iohk # Issue Number #2163 / [ADP-427](https://jira.iohk.io/browse/ADP-427) # Overview - 8c0c580 More detailed description of poolMetadataSource in swagger.yaml - 439f8f0 Integration test checking metadata is re-synced after settings update - 1d13fd5 Fix existing stake pool test checking metadata exists # Comments ![Screenshot from 2020-10-08 14-06-59](https://user-images.githubusercontent.com/42900201/95456553-ca995b80-096f-11eb-89ee-b1b42ae8d40f.png) Co-authored-by: Piotr Stachyra <[email protected]>

iohk-bors · 2020-10-08T18:39:09Z

Build failed:

ci/hydra-build:required


  src/Test/Integration/Scenario/API/Shelley/Settings.hs:86:5: 
  1) API Specifications, SHELLEY_SETTINGS, SETTINGS_02 - Changing pool_metadata_source re-syncs metadata
       uncaught exception: IOException of type UserError
       user error (Waited longer than 90s (more than 2 epochs) for an action to resolve. Action: "2. There is metadata". Error condition: Just (HUnitFailure (Just (SrcLoc {srcLocPackage = "cardano-wallet-core-integration-2020.9.30-6PfBDqTjuY813ZWL2vU0Do", srcLocModule = "Test.Integration.Scenario.API.Shelley.Settings", srcLocFile = "src/Test/Integration/Scenario/API/Shelley/Settings.hs", srcLocStartLine = 98, srcLocStartCol = 30, srcLocEndLine = 98, srcLocEndCol = 56})) (Reason "predicate failed on: [Nothing,Nothing,Nothing]")))

  To rerun use: --match "/API Specifications/SHELLEY_SETTINGS/SETTINGS_02 - Changing pool_metadata_source re-syncs metadata/"

#expected

hasufell · 2020-10-08T18:59:48Z

The setting here is less related to epoch or the chain. The resync can take an arbitrary amount of time, depending on the amount of pools metadata that needs fetching. So we probably want a custom timeout?

Anviking · 2020-10-09T09:03:10Z

I'm guessing the failure above is expected? I'm marking it so, but please change it to a reference to a flaky test ticket if that's the reason.

piotr-iohk · 2020-10-09T09:26:02Z

The setting here is less related to epoch or the chain. The resync can take an arbitrary amount of time, depending on the amount of pools metadata that needs fetching. So we probably want a custom timeout?

Well, in integration tests there are only 3 pools, so I'd expect it to resolve rather fast 🤔 ... Anyway, I've changed timeout for those tests to 120s. I've actually also risen global timeout for eventually to 100s...

piotr-iohk · 2020-10-09T09:26:09Z

bors try

iohk-bors · 2020-10-09T10:24:46Z

try

Build failed:

ci/hydra-build:required

Failures:

  src/Test/Integration/Scenario/API/Shelley/Settings.hs:87:5: 
  1) API Specifications, SHELLEY_SETTINGS, SETTINGS_02 - Changing pool_metadata_source re-syncs metadata
       uncaught exception: IOException of type UserError
       user error (Waited longer than 120s for an action to resolve. Action: "2. There is metadata". Error condition: Just (HUnitFailure (Just (SrcLoc {srcLocPackage = "cardano-wallet-core-integration-2020.9.30-6PfBDqTjuY813ZWL2vU0Do", srcLocModule = "Test.Integration.Scenario.API.Shelley.Settings", srcLocFile = "src/Test/Integration/Scenario/API/Shelley/Settings.hs", srcLocStartLine = 101, srcLocStartCol = 30, srcLocEndLine = 101, srcLocEndCol = 56})) (Reason "predicate failed on: [Nothing,Nothing,Nothing]")))

  To rerun use: --match "/API Specifications/SHELLEY_SETTINGS/SETTINGS_02 - Changing pool_metadata_source re-syncs metadata/"

#expected
(or at least not due to on-master-related flakiness)

piotr-iohk · 2020-10-09T11:55:05Z

Ok, so it failed on hydra again with bigger timeout of 120s. I thought that maybe that's MacOS related, cause hydra is on Mac, but checked on MacOS VM (manually) and it works just fine. Changes are applied instantaneously on updating settings and metadata is fetched again (or dropped when setting to none).

I added additional check to verify if metadata setting actually changed after updating... 🤷‍♂️

piotr-iohk · 2020-10-09T11:55:16Z

bors try

hasufell · 2020-10-09T12:10:21Z

Does the integration test run with an actual SQlite db or inside memory?

piotr-iohk · 2020-10-09T12:58:34Z

Does the integration test run with an actual SQlite db or inside memory?

Actual SQlite.

iohk-bors · 2020-10-09T13:14:38Z

try

Build failed:

ci/hydra-build:required

Failures:

  src/Test/Integration/Scenario/API/Shelley/Settings.hs:87:5: 
  1) API Specifications, SHELLEY_SETTINGS, SETTINGS_02 - Changing pool_metadata_source re-syncs metadata
       uncaught exception: IOException of type UserError
       user error (Waited longer than 120s for an action to resolve. Action: "2. There is metadata". Error condition: Just (HUnitFailure (Just (SrcLoc {srcLocPackage = "cardano-wallet-core-integration-2020.9.30-6PfBDqTjuY813ZWL2vU0Do", srcLocModule = "Test.Integration.Scenario.API.Shelley.Settings", srcLocFile = "src/Test/Integration/Scenario/API/Shelley/Settings.hs", srcLocStartLine = 103, srcLocStartCol = 30, srcLocEndLine = 103, srcLocEndCol = 56})) (Reason "predicate failed on: [Nothing,Nothing,Nothing]")))

  To rerun use: --match "/API Specifications/SHELLEY_SETTINGS/SETTINGS_02 - Changing pool_metadata_source re-syncs metadata/"

Same thing...

#expected
(or at least not due to on-master-related flakiness)

piotr-iohk · 2020-10-12T07:02:23Z

bors try

iohk-bors · 2020-10-12T08:16:33Z

try

Build failed:

ci/hydra-build:required

Failures:

  src/Test/Integration/Scenario/API/Shelley/Transactions.hs:949:5:
  1) API Specifications, SHELLEY_TRANSACTIONS, TRANS_EXTERNAL_02 - Multiple Outputs Transaction - Shelley witnesses
       uncaught exception: IOException of type UserError
       user error (Waited longer than 100s for an action to resolve. Action: "wFaucet and wSrc balances are as expected". Error condition: Just (HUnitFailure (Just (SrcLoc {srcLocPackage = "cardano-wallet-core-integration-2020.9.30-8S5x2S8YoF48Q0OmGSe0jU", srcLocModule = "Test.Integration.Scenario.API.Shelley.Transactions", srcLocFile = "src/Test/Integration/Scenario/API/Shelley/Transactions.hs", srcLocStartLine = 1024, srcLocStartCol = 18, srcLocEndLine = 1024, srcLocEndCol = 67})) (ExpectedButGot Nothing "Quantity {getQuantity = 999989843200}" "Quantity {getQuantity = 999989843100}")))

  To rerun use: --match "/API Specifications/SHELLEY_TRANSACTIONS/TRANS_EXTERNAL_02 - Multiple Outputs Transaction - Shelley witnesses/"

Ah, different one...

#2233

piotr-iohk · 2020-11-03T10:24:34Z

bors try

iohk-bors · 2020-11-03T11:27:14Z

try

Build failed:

ci/hydra-build:required

Failures:

  src/Test/Integration/Framework/DSL.hs:618:11: 
  1) API Specifications, SHELLEY_TRANSACTIONS, TRANS_EXTERNAL_02 - Multiple Outputs Transaction - Shelley witnesses
       Quantity {getQuantity = 999989843200} ∉ [Quantity {getQuantity = 999989843000}, Quantity {getQuantity = 999989843100}
       Waited longer than 120s to resolve action: "wFaucet and wSrc balances are as expected".

  To rerun use: --match "/API Specifications/SHELLEY_TRANSACTIONS/TRANS_EXTERNAL_02 - Multiple Outputs Transaction - Shelley witnesses/"

Randomized with seed 1791888633

Finished in 2161.2486 seconds
688 examples, 1 failure, 7 pending

#2287

piotr-iohk · 2020-11-03T14:31:22Z

bors try

iohk-bors · 2020-11-03T17:36:49Z

try

Build succeeded:

piotr-iohk · 2020-11-03T17:53:43Z

bors try

iohk-bors · 2020-11-03T21:01:29Z

try

Build succeeded:

piotr-iohk · 2020-11-04T08:02:54Z

bors r+

2225: Testing pool fetching runtime selection r=piotr-iohk a=piotr-iohk # Issue Number #2163 / [ADP-427](https://jira.iohk.io/browse/ADP-427) # Overview - 8c0c580 More detailed description of poolMetadataSource in swagger.yaml - 439f8f0 Integration test checking metadata is re-synced after settings update - 1d13fd5 Fix existing stake pool test checking metadata exists # Comments ![Screenshot from 2020-10-08 14-06-59](https://user-images.githubusercontent.com/42900201/95456553-ca995b80-096f-11eb-89ee-b1b42ae8d40f.png) Co-authored-by: Piotr Stachyra <[email protected]>

iohk-bors · 2020-11-04T09:52:47Z

Build failed:

buildkite/cardano-wallet

cardano-wallet-2020.11.3: Test suite unit passed
--
  | Progress 6/7: cardano-wallet-core-2020.11.3
  | Timed out after 60 minutes.

Seems like a cardano-wallet-core:unit timeout in buildkite.

And it seems stuck here:

Cardano.Wallet.DB.Sqlite.Types
  Values can be persisted and unpersisted successfully
    can persist and unpersist values of type 'SlotNo'
      +++ OK, passed 100 tests.
Cardano.Wallet.DB.Sqlite
  Sqlite
    Extra Properties about DB initialization
      createWallet . listWallets yields expected results
        +++ OK, passed 100 tests.
      creating same wallet twice yields an error
        +++ OK, passed 100 tests.
      removing

#2292

… timeout for eventually to 100s

piotr-iohk · 2020-11-04T11:14:40Z

bors try

iohk-bors · 2020-11-04T12:29:37Z

try

Build succeeded:

piotr-iohk · 2020-11-04T12:34:57Z

bors r+

iohk-bors · 2020-11-04T13:32:10Z

Build succeeded:

piotr-iohk added the IMPROVEMENT Mark a PR as an improvement, for auto-generated CHANGELOG label Oct 8, 2020

piotr-iohk requested a review from hasufell October 8, 2020 12:11

piotr-iohk self-assigned this Oct 8, 2020

piotr-iohk force-pushed the piotr/smash-runtime-settings-testing branch from f020bc3 to 439f8f0 Compare October 8, 2020 12:14

hasufell reviewed Oct 8, 2020

View reviewed changes

piotr-iohk commented Oct 8, 2020

View reviewed changes

piotr-iohk force-pushed the piotr/smash-runtime-settings-testing branch from 1d13fd5 to c109bd8 Compare October 8, 2020 13:19

hasufell approved these changes Oct 8, 2020

View reviewed changes

iohk-bors bot added a commit that referenced this pull request Oct 9, 2020

Try #2225:

2858c14

piotr-iohk force-pushed the piotr/smash-runtime-settings-testing branch from 1e42823 to a84c137 Compare October 9, 2020 11:54

iohk-bors bot added a commit that referenced this pull request Oct 9, 2020

Try #2225:

a917f2b

iohk-bors bot added a commit that referenced this pull request Oct 12, 2020

Try #2225:

c312358

iohk-bors bot added a commit that referenced this pull request Nov 3, 2020

Try #2225:

0d6d70c

iohk-bors bot added a commit that referenced this pull request Nov 3, 2020

Try #2225:

01b7763

iohk-bors bot added a commit that referenced this pull request Nov 3, 2020

Try #2225:

9a2058c

Anviking mentioned this pull request Nov 4, 2020

Flaky test - various DB properties causing timeout #2292

Closed

Piotr Stachyra added 9 commits November 4, 2020 12:11

More detailed description of poolMetadataSource in swagger.yaml

e191241

Integration test checking metadata is re-synced after settings update

246417e

Fix existing stake pool test checking metadata exists

6881e4e

Use custom timeout of 120s for SETTINGS_02 test; slightly rise global…

801498a

… timeout for eventually to 100s

Additional check verifyMetadataSource in SETTINGS_02

378e364

180s timeout for SETTINGS_02

33f5cd0

set global timeout for eventually to 120s

0b14e13

Reword poolMetadataSource in swagger

6f85868

Adjust timeouts

f38e3a6

piotr-iohk force-pushed the piotr/smash-runtime-settings-testing branch from 8d82bc7 to f38e3a6 Compare November 4, 2020 11:13

iohk-bors bot added a commit that referenced this pull request Nov 4, 2020

Try #2225:

d83745c

iohk-bors bot merged commit 74ebfca into master Nov 4, 2020

iohk-bors bot deleted the piotr/smash-runtime-settings-testing branch November 4, 2020 13:32

Testing pool fetching runtime selection #2225

Testing pool fetching runtime selection #2225

Conversation

piotr-iohk commented Oct 8, 2020 • edited Loading

Issue Number

Overview

Comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hasufell Oct 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

piotr-iohk commented Oct 8, 2020

iohk-bors bot commented Oct 8, 2020 • edited by Anviking Loading

piotr-iohk commented Oct 8, 2020

iohk-bors bot commented Oct 8, 2020 • edited by Anviking Loading

hasufell commented Oct 8, 2020

Anviking commented Oct 9, 2020

piotr-iohk commented Oct 9, 2020

piotr-iohk commented Oct 9, 2020

iohk-bors bot commented Oct 9, 2020 • edited by Anviking Loading

try

piotr-iohk commented Oct 9, 2020

piotr-iohk commented Oct 9, 2020

hasufell commented Oct 9, 2020

piotr-iohk commented Oct 9, 2020

iohk-bors bot commented Oct 9, 2020 • edited by Anviking Loading

try

piotr-iohk commented Oct 12, 2020

iohk-bors bot commented Oct 12, 2020 • edited by piotr-iohk Loading

try

piotr-iohk commented Nov 3, 2020

iohk-bors bot commented Nov 3, 2020 • edited by piotr-iohk Loading

try

piotr-iohk commented Nov 3, 2020

iohk-bors bot commented Nov 3, 2020

try

piotr-iohk commented Nov 3, 2020

iohk-bors bot commented Nov 3, 2020

try

piotr-iohk commented Nov 4, 2020

iohk-bors bot commented Nov 4, 2020 • edited by Anviking Loading

piotr-iohk commented Nov 4, 2020

iohk-bors bot commented Nov 4, 2020

try

piotr-iohk commented Nov 4, 2020

iohk-bors bot commented Nov 4, 2020

piotr-iohk commented Oct 8, 2020 •

edited

Loading

hasufell Oct 8, 2020 •

edited

Loading

iohk-bors bot commented Oct 8, 2020 •

edited by Anviking

Loading

iohk-bors bot commented Oct 8, 2020 •

edited by Anviking

Loading

iohk-bors bot commented Oct 9, 2020 •

edited by Anviking

Loading

iohk-bors bot commented Oct 9, 2020 •

edited by Anviking

Loading

iohk-bors bot commented Oct 12, 2020 •

edited by piotr-iohk

Loading

iohk-bors bot commented Nov 3, 2020 •

edited by piotr-iohk

Loading

iohk-bors bot commented Nov 4, 2020 •

edited by Anviking

Loading