Cluster of stake pools in integration tests #1734

KtorZ · 2020-06-09T15:39:23Z

Issue Number

N/A

Overview

de6fec0
📍 add command-line helpers for generating stake key and certificates
50ae861
📍 improve error handling and reporting when member of the cluster fail to start
It's actually nice to be able to tell what's going on. The command-line will likely undergo changes so it's really likely
that a command that used to work will fail. The previous message was sort of opaque / cryptic. Now we get a better error.
772c813
📍 fixup add new gen methods
bb2b82e
📍 register stake pool and delegate funds to it using genesis faucets.
This makes sure that a stake pool joins the cluster only after it has been registered and is ready to process block.
e08d5cd
📍 Drop use of 'withAsync' so pools boot in parallel
The issue with 'withAsync' is that it's actually synchronous.
While the underlying action is executed in a thread, 'withAsync' will
only resolve when the async action is done. Yet, in our case, we want
all async actions to run in parallel and to be stopped when the test
suite is completed.
de7559f
📍 make the node a little less verbose regarding forging
With a slot length of 0.2, blocks are produced at a much faster pace.
b2b3c0b
📍 tweak genesis parameters with sensible values
A value of '1' for the active slot coefficient makes actually no sense. Only small values of it are
considered sensible. Tweaking also the decentralization level so that some blocks are deferred to
registered stake pools
adf9dda
📍 start a cluster of 3 pools when running integration tests
aaf7d6d
📍 show little disclaimer when starting integration tests
2d7ece5
📍 do not restart the cluster between CLI and API spec
We have a limited number of faucets, and it takes quite some time now

Comments

With this, we should be able to test all sort of things regarding pools o/

…to start It's actually nice to be able to tell what's going on. The command-line will likely undergo changes so it's really likely that a command that used to work will fail. The previous message was sort of opaque / cryptic. Now we get a better error.

This makes sure that a stake pool joins the cluster only after it has been registered and is ready to process block.

The issue with 'withAsync' is that it's actually synchronous. While the underlying action is executed in a thread, 'withAsync' will only resolve when the async action is done. Yet, in our case, we want all async actions to run in parallel and to be stopped when the test suite is completed.

With a slot length of 0.2, blocks are produced at a much faster pace.

rvl · 2020-06-10T05:29:03Z

lib/core-integration/src/Test/Integration/Scenario/API/Network.hs

@@ -104,14 +104,13 @@ spec = do
        let networkParams = getFromResponse id r
        networkParams `shouldBe`
            toApiNetworkParameters (ctx ^. #_networkParameters)
-        let Right zeroPercent = Quantity <$> mkPercentage 0
+        let Right zeroPercent = Quantity <$> mkPercentage 75


yeah, why 75?

Because the d-parameter is now set to 0.25 (was 1) to allow both the BFT node and pools to produce blocks. We actually report (1 - d) in the API, hence 75% here. We are 75% decentralized in the integration test.

paweljakubas · 2020-06-10T06:58:38Z

lib/shelley/test/data/cardano-node-shelley/genesis.yaml

@@ -32,7 +31,8 @@ updateQuorum: 5
 maxMajorPV: 25446
 initialFunds: {}
 maxLovelaceSupply: 45000000000000000
-networkMagic: 1
+protocolMagicId: 1
+networkMagic: 764824073


why we need this?

While new Shelley addresses uses the new protocolMagic (1), the network still uses the old magic (764824073). This is therefore needed to match the right network configuration of the cluster.

paweljakubas · 2020-06-10T06:59:25Z

lib/shelley/src/Cardano/Wallet/Shelley/Launch.hs

+        ]
+    pure file
+
+-- | Generate a raw transaction. We kill two birds one stone here by also


KtorZ · 2020-06-10T08:33:47Z

bors r+

1734: Cluster of stake pools in integration tests r=KtorZ a=KtorZ # Issue Number  N/A # Overview  - de6fec0 📍 **add command-line helpers for generating stake key and certificates** - 50ae861 📍 **improve error handling and reporting when member of the cluster fail to start** It's actually nice to be able to tell what's going on. The command-line will likely undergo changes so it's really likely that a command that used to work will fail. The previous message was sort of opaque / cryptic. Now we get a better error. - 772c813 📍 **fixup add new gen methods** - bb2b82e 📍 **register stake pool and delegate funds to it using genesis faucets.** This makes sure that a stake pool joins the cluster only after it has been registered and is ready to process block. - e08d5cd 📍 **Drop use of 'withAsync' so pools boot in parallel** The issue with 'withAsync' is that it's actually synchronous. While the underlying action is executed in a thread, 'withAsync' will only resolve when the async action is done. Yet, in our case, we want all async actions to run in parallel and to be stopped when the test suite is completed. - de7559f 📍 **make the node a little less verbose regarding forging** With a slot length of 0.2, blocks are produced at a much faster pace. - b2b3c0b 📍 **tweak genesis parameters with sensible values** A value of '1' for the active slot coefficient makes actually no sense. Only small values of it are considered sensible. Tweaking also the decentralization level so that some blocks are deferred to registered stake pools - adf9dda 📍 **start a cluster of 3 pools when running integration tests** - aaf7d6d 📍 **show little disclaimer when starting integration tests** - 2d7ece5 📍 **do not restart the cluster between CLI and API spec** We have a limited number of faucets, and it takes quite some time now # Comments  With this, we should be able to test all sort of things regarding pools o/  Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-06-10T08:59:01Z

Build failed

buildkite/cardano-wallet

A value of '1' for the active slot coefficient makes actually no sense. Only small values of it are considered sensible. Tweaking also the decentralization level so that some blocks are deferred to registered stake pools

We have a limited number of faucets, and it takes quite some time now

KtorZ · 2020-06-10T12:31:32Z

bors r+

1734: Cluster of stake pools in integration tests r=KtorZ a=KtorZ # Issue Number  N/A # Overview  - de6fec0 📍 **add command-line helpers for generating stake key and certificates** - 50ae861 📍 **improve error handling and reporting when member of the cluster fail to start** It's actually nice to be able to tell what's going on. The command-line will likely undergo changes so it's really likely that a command that used to work will fail. The previous message was sort of opaque / cryptic. Now we get a better error. - 772c813 📍 **fixup add new gen methods** - bb2b82e 📍 **register stake pool and delegate funds to it using genesis faucets.** This makes sure that a stake pool joins the cluster only after it has been registered and is ready to process block. - e08d5cd 📍 **Drop use of 'withAsync' so pools boot in parallel** The issue with 'withAsync' is that it's actually synchronous. While the underlying action is executed in a thread, 'withAsync' will only resolve when the async action is done. Yet, in our case, we want all async actions to run in parallel and to be stopped when the test suite is completed. - de7559f 📍 **make the node a little less verbose regarding forging** With a slot length of 0.2, blocks are produced at a much faster pace. - b2b3c0b 📍 **tweak genesis parameters with sensible values** A value of '1' for the active slot coefficient makes actually no sense. Only small values of it are considered sensible. Tweaking also the decentralization level so that some blocks are deferred to registered stake pools - adf9dda 📍 **start a cluster of 3 pools when running integration tests** - aaf7d6d 📍 **show little disclaimer when starting integration tests** - 2d7ece5 📍 **do not restart the cluster between CLI and API spec** We have a limited number of faucets, and it takes quite some time now # Comments  With this, we should be able to test all sort of things regarding pools o/  Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-06-10T14:58:42Z

Build failed

ci/hydra-build:required

KtorZ · 2020-06-10T15:24:23Z

bors r+

1734: Cluster of stake pools in integration tests r=KtorZ a=KtorZ # Issue Number  N/A # Overview  - de6fec0 📍 **add command-line helpers for generating stake key and certificates** - 50ae861 📍 **improve error handling and reporting when member of the cluster fail to start** It's actually nice to be able to tell what's going on. The command-line will likely undergo changes so it's really likely that a command that used to work will fail. The previous message was sort of opaque / cryptic. Now we get a better error. - 772c813 📍 **fixup add new gen methods** - bb2b82e 📍 **register stake pool and delegate funds to it using genesis faucets.** This makes sure that a stake pool joins the cluster only after it has been registered and is ready to process block. - e08d5cd 📍 **Drop use of 'withAsync' so pools boot in parallel** The issue with 'withAsync' is that it's actually synchronous. While the underlying action is executed in a thread, 'withAsync' will only resolve when the async action is done. Yet, in our case, we want all async actions to run in parallel and to be stopped when the test suite is completed. - de7559f 📍 **make the node a little less verbose regarding forging** With a slot length of 0.2, blocks are produced at a much faster pace. - b2b3c0b 📍 **tweak genesis parameters with sensible values** A value of '1' for the active slot coefficient makes actually no sense. Only small values of it are considered sensible. Tweaking also the decentralization level so that some blocks are deferred to registered stake pools - adf9dda 📍 **start a cluster of 3 pools when running integration tests** - aaf7d6d 📍 **show little disclaimer when starting integration tests** - 2d7ece5 📍 **do not restart the cluster between CLI and API spec** We have a limited number of faucets, and it takes quite some time now # Comments  With this, we should be able to test all sort of things regarding pools o/  Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-06-10T15:47:51Z

Build failed

ci/hydra-build:required

This prevents some race-condition where pools would start making queries but being not fully started _yet_.

KtorZ · 2020-06-10T15:52:53Z

bors r+

1734: Cluster of stake pools in integration tests r=KtorZ a=KtorZ # Issue Number  N/A # Overview  - de6fec0 📍 **add command-line helpers for generating stake key and certificates** - 50ae861 📍 **improve error handling and reporting when member of the cluster fail to start** It's actually nice to be able to tell what's going on. The command-line will likely undergo changes so it's really likely that a command that used to work will fail. The previous message was sort of opaque / cryptic. Now we get a better error. - 772c813 📍 **fixup add new gen methods** - bb2b82e 📍 **register stake pool and delegate funds to it using genesis faucets.** This makes sure that a stake pool joins the cluster only after it has been registered and is ready to process block. - e08d5cd 📍 **Drop use of 'withAsync' so pools boot in parallel** The issue with 'withAsync' is that it's actually synchronous. While the underlying action is executed in a thread, 'withAsync' will only resolve when the async action is done. Yet, in our case, we want all async actions to run in parallel and to be stopped when the test suite is completed. - de7559f 📍 **make the node a little less verbose regarding forging** With a slot length of 0.2, blocks are produced at a much faster pace. - b2b3c0b 📍 **tweak genesis parameters with sensible values** A value of '1' for the active slot coefficient makes actually no sense. Only small values of it are considered sensible. Tweaking also the decentralization level so that some blocks are deferred to registered stake pools - adf9dda 📍 **start a cluster of 3 pools when running integration tests** - aaf7d6d 📍 **show little disclaimer when starting integration tests** - 2d7ece5 📍 **do not restart the cluster between CLI and API spec** We have a limited number of faucets, and it takes quite some time now # Comments  With this, we should be able to test all sort of things regarding pools o/  Co-authored-by: KtorZ <[email protected]>

iohk-bors · 2020-06-10T16:19:08Z

Build failed

ci/hydra-build:required

KtorZ · 2020-06-10T16:53:05Z

bors try

iohk-bors · 2020-06-10T17:32:02Z

try

Build failed

ci/hydra-build:required

the initial submitTx was also failing sometimes

rvl · 2020-06-11T09:34:11Z

bors try

iohk-bors · 2020-06-11T10:05:09Z

try

Build failed

ci/hydra-build:required

This reverts commit 8b89aad. It didn't work anyway.

It now makes a single payment with many outputs, instead of many payments with a single output. This is to cope with the absence of transaction scheduler in the wallet at the moment where pending outgoing transaction aren't automatically re-submitted after rollback.

KtorZ · 2020-06-11T12:44:58Z

bors r+

iohk-bors · 2020-06-11T13:18:40Z

Build succeeded

KtorZ added 6 commits June 9, 2020 17:01

add command-line helpers for generating stake key and certificates

de6fec0

fixup add new gen methods

772c813

register stake pool and delegate funds to it using genesis faucets.

bb2b82e

This makes sure that a stake pool joins the cluster only after it has been registered and is ready to process block.

make the node a little less verbose regarding forging

de7559f

With a slot length of 0.2, blocks are produced at a much faster pace.

KtorZ requested review from Anviking and paweljakubas June 9, 2020 15:39

KtorZ self-assigned this Jun 9, 2020

rvl approved these changes Jun 10, 2020

View reviewed changes

paweljakubas reviewed Jun 10, 2020

View reviewed changes

paweljakubas approved these changes Jun 10, 2020

View reviewed changes

KtorZ added 6 commits June 10, 2020 13:32

tweak genesis parameters with sensible values

a4a1982

A value of '1' for the active slot coefficient makes actually no sense. Only small values of it are considered sensible. Tweaking also the decentralization level so that some blocks are deferred to registered stake pools

start a cluster of 3 pools when running integration tests

87aebee

show little disclaimer when starting integration tests

43dddd9

do not restart the cluster between CLI and API spec

a17f3fa

We have a limited number of faucets, and it takes quite some time now

provide cardano-cli to integration tests via Nix

63d80f1

move Shelley-specific network integration test into dedicated package

2f94a92

KtorZ force-pushed the KtorZ/shelley-stake-pool-setup-part-2 branch from 8d615d4 to 2f94a92 Compare June 10, 2020 12:04

pass the parent socket to pools when querying node state

2f1c24e

This prevents some race-condition where pools would start making queries but being not fully started _yet_.

KtorZ force-pushed the KtorZ/shelley-stake-pool-setup-part-2 branch from 064e6ac to 2f1c24e Compare June 10, 2020 15:52

temp debugging

44f3870

iohk-bors bot added a commit that referenced this pull request Jun 10, 2020

Try #1734:

461926d

rvl added 10 commits June 11, 2020 15:29

hydra: run macos shelley integration tests on every commit

8b89aad

Adjust when the CARDANO_NODE_SOCKET_PATH variable is set

17e466e

enhance test data script

8499a11

number stake pool state directories

2038a66

retry cardano-cli shelley query stake-distribution if it fails

0a40d51

add NO_CLEANUP variable for debugging purposes

6aa3002

more elaborate method of waiting for socket

07b8382

the initial submitTx was also failing sometimes

fixup! more elaborate method of waiting for socket

9e4a7f2

fixup! more elaborate method of waiting for socket

ec855df

fixup! more elaborate method of waiting for socket

e34ac7d

iohk-bors bot added a commit that referenced this pull request Jun 11, 2020

Try #1734:

053e4e4

rvl and others added 3 commits June 11, 2020 20:10

Revert "hydra: run macos shelley integration tests on every commit"

0e2b6f6

This reverts commit 8b89aad. It didn't work anyway.

remove temporary debug line regarding pool socket

378b929

iohk-bors bot merged commit 4cd2ce6 into master Jun 11, 2020

iohk-bors bot deleted the KtorZ/shelley-stake-pool-setup-part-2 branch June 11, 2020 13:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cluster of stake pools in integration tests #1734

Cluster of stake pools in integration tests #1734

KtorZ commented Jun 9, 2020

rvl Jun 10, 2020

paweljakubas Jun 10, 2020

KtorZ Jun 10, 2020

paweljakubas Jun 10, 2020

KtorZ Jun 10, 2020

paweljakubas Jun 10, 2020

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

rvl commented Jun 11, 2020

iohk-bors bot commented Jun 11, 2020

KtorZ commented Jun 11, 2020

iohk-bors bot commented Jun 11, 2020

Cluster of stake pools in integration tests #1734

Cluster of stake pools in integration tests #1734

Conversation

KtorZ commented Jun 9, 2020

Issue Number

Overview

Comments

rvl Jun 10, 2020

Choose a reason for hiding this comment

paweljakubas Jun 10, 2020

Choose a reason for hiding this comment

KtorZ Jun 10, 2020

Choose a reason for hiding this comment

paweljakubas Jun 10, 2020

Choose a reason for hiding this comment

KtorZ Jun 10, 2020

Choose a reason for hiding this comment

paweljakubas Jun 10, 2020

Choose a reason for hiding this comment

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

Build failed

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

Build failed

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

Build failed

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

Build failed

KtorZ commented Jun 10, 2020

iohk-bors bot commented Jun 10, 2020

try

Build failed

rvl commented Jun 11, 2020

iohk-bors bot commented Jun 11, 2020

try

Build failed

KtorZ commented Jun 11, 2020

iohk-bors bot commented Jun 11, 2020

Build succeeded