Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restoration bench: increase heap and only bench {0%,0.2%} x {seq,rnd} #2688

Merged
merged 3 commits into from
Jun 7, 2021

Conversation

Anviking
Copy link
Member

@Anviking Anviking commented Jun 3, 2021

Issue Number

ADP-846

Overview

Let the nightly benchmark finish successfully within the alotted time.

  • Increase max heap from 8GB to 16GB
  • Replace 0.1% and 0.4% rnd and seq benchmarks with 0.2% to prevent timeout.

Comments

Build status

Anviking added 2 commits June 2, 2021 10:27
It seems the final listTransactions benchmark at the end surpasses the
8GB of memory.

A quick workaround could be to just bump the limit. We can investigate
the memory usage of listing transactions separately if we deem it
worthfile.
@Anviking Anviking self-assigned this Jun 3, 2021
@@ -313,7 +313,8 @@ cardanoRestoreBench tr c socketFile = do
, benchRestoreSeqWithOwnership (Proxy @4)

, benchRestoreRndWithOwnership (Proxy @0)
, benchRestoreRndWithOwnership (Proxy @1)
-- benchRestoreRndWithOwnership (Proxy @1)
-- Removed to prevent timeout in CI.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping it commented to prevent someone from thinking the absence of it is a mistake.

And I kept 0.4% rnd, because I recall it being similar to some exchanges (but I don't really know).

@Anviking Anviking requested review from rvl and piotr-iohk June 3, 2021 13:56
@Anviking
Copy link
Member Author

Anviking commented Jun 3, 2021

Looking at the current progress, removing 0.1%-rnd will most certainly not be enough.

Before removing anything, it started 0.4%-rnd 30 minutes before the timeout. So even just removing 0.4%-rnd will be tight.

I suspect we might need to remove both 0.4%-seq and 0.4%-rnd…

@Anviking Anviking removed request for rvl and piotr-iohk June 3, 2021 18:46
@Anviking
Copy link
Member Author

Anviking commented Jun 3, 2021

list transactions: 2946 s

From 0.4%-seq src

😳

@Anviking Anviking marked this pull request as draft June 3, 2021 18:47
Copy link
Contributor

@rvl rvl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing it - looks good to me.


, benchRestoreRndWithOwnership (Proxy @0)
, benchRestoreRndWithOwnership (Proxy @1)
, benchRestoreRndWithOwnership (Proxy @4)
, benchRestoreRndWithOwnership (Proxy @2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little bit confusing to have the address discovery ratio as a unitless type parameter, but now I see this means 0% and 0.2%.

Probably just one or two percentage benchmarks is enough anyway.

Next thing we should do is improve the performance so we can increase these numbers. 😄.

And also to benchmark the sequential address pool code in isolation.

@rvl rvl marked this pull request as ready for review June 6, 2021 03:26
@rvl
Copy link
Contributor

rvl commented Jun 6, 2021

bors r+

iohk-bors bot added a commit that referenced this pull request Jun 6, 2021
2688: Restoration bench: increase heap and remove 0.1%-rnd r=rvl a=Anviking

# Issue Number

ADP-846

# Overview

Let the nightly benchmark finish successfully within the alotted time.

- [x] Increase max heap from 8GB to 16GB
- [x] Replace 0.1% and 0.4% rnd and seq benchmarks with 0.2% to prevent timeout.

# Comments

[![Build status](https://badge.buildkite.com/59ea9363b8526e867005ca8839db47715bc5f661f36e490143.svg?branch=anviking%2FADP-846%2Fmore-heap)](https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds?branch=anviking%2FADP-846%2Fmore-heap)


Co-authored-by: Johannes Lund <[email protected]>
@iohk-bors
Copy link
Contributor

iohk-bors bot commented Jun 6, 2021

Build failed:

@Anviking
Copy link
Member Author

Anviking commented Jun 6, 2021

bors r+

iohk-bors bot added a commit that referenced this pull request Jun 6, 2021
2688: Restoration bench: increase heap and remove 0.1%-rnd r=Anviking a=Anviking

# Issue Number

ADP-846

# Overview

Let the nightly benchmark finish successfully within the alotted time.

- [x] Increase max heap from 8GB to 16GB
- [x] Replace 0.1% and 0.4% rnd and seq benchmarks with 0.2% to prevent timeout.

# Comments

[![Build status](https://badge.buildkite.com/59ea9363b8526e867005ca8839db47715bc5f661f36e490143.svg?branch=anviking%2FADP-846%2Fmore-heap)](https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds?branch=anviking%2FADP-846%2Fmore-heap)


Co-authored-by: Johannes Lund <[email protected]>
@iohk-bors
Copy link
Contributor

iohk-bors bot commented Jun 6, 2021

Build failed:

weeder
--
&nbsp; | error: Command exited with code -9!

But there's no more details 🤔

@Anviking Anviking changed the title Restoration bench: increase heap and remove 0.1%-rnd Restoration bench: increase heap and only bench {0%,0.2%} x {seq,rnd} Jun 6, 2021
@Anviking
Copy link
Member Author

Anviking commented Jun 6, 2021

bors r+

iohk-bors bot added a commit that referenced this pull request Jun 6, 2021
2688: Restoration bench: increase heap and only bench {0%,0.2%} x {seq,rnd} r=Anviking a=Anviking

# Issue Number

ADP-846

# Overview

Let the nightly benchmark finish successfully within the alotted time.

- [x] Increase max heap from 8GB to 16GB
- [x] Replace 0.1% and 0.4% rnd and seq benchmarks with 0.2% to prevent timeout.

# Comments

[![Build status](https://badge.buildkite.com/59ea9363b8526e867005ca8839db47715bc5f661f36e490143.svg?branch=anviking%2FADP-846%2Fmore-heap)](https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds?branch=anviking%2FADP-846%2Fmore-heap)


Co-authored-by: Johannes Lund <[email protected]>
@iohk-bors
Copy link
Contributor

iohk-bors bot commented Jun 6, 2021

Build failed:

@rvl
Copy link
Contributor

rvl commented Jun 7, 2021

bors r+

iohk-bors bot added a commit that referenced this pull request Jun 7, 2021
2686: cabal: Add version constraint on random r=rvl a=rvl

# Issue Number

ADP-961

# Overview

In `random>=1.2.0`, `StdGen` no longer has a `Read` instance.

- Add version constraint so Cabal uses previous version.
- Clean up `cabal.project`, with the assumption of Cabal >= 3.4.0.0.
- Add a nightly CI step to run the cabal build.
- Use cabal-cache to make the nightly build faster.
- Make sure every Haskell tool built for the nix-shell has a materialization, for faster evaluation without warnings.

# Comments

Resolves #2685.

We may fix the usage of `instance Read StdGen` later.


2688: Restoration bench: increase heap and only bench {0%,0.2%} x {seq,rnd} r=rvl a=Anviking

# Issue Number

ADP-846

# Overview

Let the nightly benchmark finish successfully within the alotted time.

- [x] Increase max heap from 8GB to 16GB
- [x] Replace 0.1% and 0.4% rnd and seq benchmarks with 0.2% to prevent timeout.

# Comments

[![Build status](https://badge.buildkite.com/59ea9363b8526e867005ca8839db47715bc5f661f36e490143.svg?branch=anviking%2FADP-846%2Fmore-heap)](https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds?branch=anviking%2FADP-846%2Fmore-heap)


Co-authored-by: Rodney Lorrimar <[email protected]>
Co-authored-by: Johannes Lund <[email protected]>
@iohk-bors
Copy link
Contributor

iohk-bors bot commented Jun 7, 2021

Build failed (retrying...):

  src/Test/Integration/Framework/DSL.hs:971:11: 
  1) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_JOIN_01rewards - Can join a pool, earn rewards and collect them
       Quantity {getQuantity = 0} does not satisfy (> Quantity {getQuantity = 0})
       While verifying (Status {statusCode = 200, statusMessage = "OK"},Right (ApiWallet {id = ApiT {getApiT = WalletId {getWalletId = 20a555fd33e3c42a79e87d1b8aab256804dc7433}}, addressPoolGap = ApiT {getApiT = AddressPoolGap {getAddressPoolGap = 20}}, balance = ApiWalletBalance {available = Quantity {getQuantity = 999998855100}, total = Quantity {getQuantity = 999998855100}, reward = Quantity {getQuantity = 0}}, assets = ApiWalletAssetsBalance {available = ApiT {getApiT = TokenMap (fromList [])}, total = ApiT {getApiT = TokenMap (fromList [])}}, delegation = ApiWalletDelegation {active = ApiWalletDelegationNext {status = Delegating, target = Just (ApiT {getApiT = PoolId {getPoolId = "\236(\243=\203\230\214@\n\RS^3\155\208d|\ts\202l\f\249\194\187\230\131\141\198"}}), changesAt = Nothing}, next = []}, name = ApiT {getApiT = WalletName {getWalletName = "Faucet Wallet"}}, passphrase = Just (ApiWalletPassphraseInfo {lastUpdatedAt = 2021-06-07 04:39:13.855782412 UTC}), state = ApiT {getApiT = Ready}, tip = ApiBlockReference {absoluteSlotNumber = ApiT {getApiT = SlotNo 5452}, slotId = ApiSlotId {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 109}}, slotNumber = ApiT {getApiT = SlotInEpoch {unSlotInEpoch = 2}}}, time = 2021-06-07 04:40:59.4 UTC, block = ApiBlockInfo {height = Quantity {getQuantity = 1978}}}}))
       Waited longer than 90s to resolve action: "Wallet gets rewards".

  To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_JOIN_01rewards - Can join a pool, earn rewards and collect them/"

  src/Test/Integration/Framework/DSL.hs:971:11: 
  2) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_JOIN_04 - Rewards accumulate
       Quantity {getQuantity = 0} does not satisfy (> Quantity {getQuantity = 0})
       While verifying (Status {statusCode = 200, statusMessage = "OK"},Right (ApiWallet {id = ApiT {getApiT = WalletId {getWalletId = 08e85bba85772dae58b18193182614952e478cec}}, addressPoolGap = ApiT {getApiT = AddressPoolGap {getAddressPoolGap = 20}}, balance = ApiWalletBalance {available = Quantity {getQuantity = 999998855100}, total = Quantity {getQuantity = 999998855100}, reward = Quantity {getQuantity = 0}}, assets = ApiWalletAssetsBalance {available = ApiT {getApiT = TokenMap (fromList [])}, total = ApiT {getApiT = TokenMap (fromList [])}}, delegation = ApiWalletDelegation {active = ApiWalletDelegationNext {status = Delegating, target = Just (ApiT {getApiT = PoolId {getPoolId = "\236(\243=\203\230\214@\n\RS^3\155\208d|\ts\202l\f\249\194\187\230\131\141\198"}}), changesAt = Nothing}, next = []}, name = ApiT {getApiT = WalletName {getWalletName = "Faucet Wallet"}}, passphrase = Just (ApiWalletPassphraseInfo {lastUpdatedAt = 2021-06-07 04:39:32.891149242 UTC}), state = ApiT {getApiT = Ready}, tip = ApiBlockReference {absoluteSlotNumber = ApiT {getApiT = SlotNo 5549}, slotId = ApiSlotId {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 110}}, slotNumber = ApiT {getApiT = SlotInEpoch {unSlotInEpoch = 49}}}, time = 2021-06-07 04:41:18.8 UTC, block = ApiBlockInfo {height = Quantity {getQuantity = 2013}}}}))
       Waited longer than 90s to resolve action: "Wallet gets rewards".

  To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_JOIN_04 - Rewards accumulate/"

  src/Test/Integration/Framework/DSL.hs:971:11: 
  3) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_LIST_01 - List stake pools, has non-zero saturation & stake
       Quantity {getQuantity = Percentage {getPercentage = 0 % 1}} does not satisfy (> Quantity {getQuantity = Percentage {getPercentage = 0 % 1}})
       While verifying (Status {statusCode = 200, statusMessage = "OK"},Right [ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\236(\243=\203\230\214@\n\RS^3\155\208d|\ts\202l\f\249\194\187\230\131\141\198"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 262204104674}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 0 % 1}}, saturation = 8.101452618265432e-6, producedBlocks = Quantity {getQuantity = 0}}, metadata = Nothing, cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 2000000000000}, retirement = Nothing, flags = []},ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\187\DC1L\179}u\250\ENQ&\ETX(\194\&5\163\218\226\149\163=\v\166t\165\235\RS>V\142"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 253315753931}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 3151 % 10000}}, saturation = 0.9453324099475117, producedBlocks = Quantity {getQuantity = 713}}, metadata = Nothing, cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 1000000000000}, retirement = Just (ApiEpochInfo {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 100000}}, epochStartTime = 2021-06-18 18:09:29 UTC}), flags = []},ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\180Wh\193\162\218K\209>\188\170\RS\165\DC4\b\237\163\GS\204!v\\\203\212\a\205\169\242"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 161034834888}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 2817 % 5000}}, saturation = 1.690229863617087, producedBlocks = Quantity {getQuantity = 1624}}, metadata = Nothing, cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 1000000000000}, retirement = Just (ApiEpochInfo {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 1000000}}, epochStartTime = 2021-09-30 22:09:29 UTC}), flags = []}])
       Waited longer than 90s to resolve action: "list pools returns non-empty list".

  To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_LIST_01 - List stake pools/has non-zero saturation & stake/"

Randomized with seed 2057366705
retry: Quantity {getQuantity = Percentage {getPercentage = 0 % 1}} does not satisfy (> Quantity {getQuantity = Percentage {getPercentage = 0 % 1}})
retry: While verifying (Status {statusCode = 200, statusMessage = "OK"},Right [ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\236(\243=\203\230\214@\n\RS^3\155\208d|\ts\202l\f\249\194\187\230\131\141\198"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 87926288846}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 0 % 1}}, saturation = 8.100573047141676e-6, producedBlocks = Quantity {getQuantity = 0}}, metadata = Nothing, cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 2000000000000}, retirement = Nothing, flags = []},ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\187\DC1L\179}u\250\ENQ&\ETX(\194\&5\163\218\226\149\163=\v\166t\165\235\RS>V\142"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 83952194106}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 3151 % 10000}}, saturation = 0.9453522135694138, producedBlocks = Quantity {getQuantity = 767}}, metadata = Nothing, cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 1000000000000}, retirement = Just (ApiEpochInfo {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 100000}}, epochStartTime = 2021-06-18 18:09:29 UTC}), flags = []},ApiStakePool {id = ApiT {getApiT = PoolId {getPoolId = "\180Wh\193\162\218K\209>\188\170\RS\165\DC4\b\237\163\GS\204!v\\\203\212\a\205\169\242"}}, metrics = ApiStakePoolMetrics {nonMyopicMemberRewards = Quantity {getQuantity = 54000144542}, relativeStake = Quantity {getQuantity = Percentage {getPercentage = 2817 % 5000}}, saturation = 1.6902496268433929, producedBlocks = Quantity {getQuantity = 1740}}, metadata = Nothing, cost = Quantity {getQuantity = 0}, margin = Quantity {getQuantity = Percentage {getPercentage = 1 % 10}}, pledge = Quantity {getQuantity = 1000000000000}, retirement = Just (ApiEpochInfo {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 1000000}}, epochStartTime = 2021-09-30 22:09:29 UTC}), flags = []}])

2 of the pools have stake, but the third one doesn't. So a problem in the test setup.

#2720

iohk-bors bot added a commit that referenced this pull request Jun 7, 2021
2688: Restoration bench: increase heap and only bench {0%,0.2%} x {seq,rnd} r=rvl a=Anviking

# Issue Number

ADP-846

# Overview

Let the nightly benchmark finish successfully within the alotted time.

- [x] Increase max heap from 8GB to 16GB
- [x] Replace 0.1% and 0.4% rnd and seq benchmarks with 0.2% to prevent timeout.

# Comments

[![Build status](https://badge.buildkite.com/59ea9363b8526e867005ca8839db47715bc5f661f36e490143.svg?branch=anviking%2FADP-846%2Fmore-heap)](https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds?branch=anviking%2FADP-846%2Fmore-heap)


Co-authored-by: Johannes Lund <[email protected]>
@iohk-bors
Copy link
Contributor

iohk-bors bot commented Jun 7, 2021

Build failed:

Failures:

  test/unit/Network/Wai/Middleware/LoggingSpec.hs:217:5: 
  1) Network.Wai.Middleware.Logging, Logging Middleware, different request ids
       uncaught exception: HttpException
       HttpExceptionRequest Request {
         host                 = "localhost"
         port                 = 36113
         secure               = False
         requestHeaders       = []
         path                 = "/get"
         queryString          = ""
         method               = "GET"
         proxy                = Nothing
         rawBody              = False
         redirectCount        = 10
         responseTimeout      = ResponseTimeoutDefault
         requestVersion       = HTTP/1.1
       }
        (ConnectionFailure Network.Socket.connect: <socket: 220>: failed (Socket is already connected (WSAEISCONN)))
       (after 64 tests)
         NumberOfRequests 91

  To rerun use: --match "/Network.Wai.Middleware.Logging/Logging Middleware/different request ids/"

Randomized with seed 1360170012

#2368

@Anviking
Copy link
Member Author

Anviking commented Jun 7, 2021

bors r+

iohk-bors bot added a commit that referenced this pull request Jun 7, 2021
2688: Restoration bench: increase heap and only bench {0%,0.2%} x {seq,rnd} r=Anviking a=Anviking

# Issue Number

ADP-846

# Overview

Let the nightly benchmark finish successfully within the alotted time.

- [x] Increase max heap from 8GB to 16GB
- [x] Replace 0.1% and 0.4% rnd and seq benchmarks with 0.2% to prevent timeout.

# Comments

[![Build status](https://badge.buildkite.com/59ea9363b8526e867005ca8839db47715bc5f661f36e490143.svg?branch=anviking%2FADP-846%2Fmore-heap)](https://buildkite.com/input-output-hk/cardano-wallet-nightly/builds?branch=anviking%2FADP-846%2Fmore-heap)


Co-authored-by: Johannes Lund <[email protected]>
@iohk-bors
Copy link
Contributor

iohk-bors bot commented Jun 7, 2021

Build failed:

src/Test/Integration/Scenario/API/Shelley/StakePools.hs:523:5: 
  1) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_JOIN_01 - Can rejoin another stakepool
       uncaught exception: IOException of type UserError
       user error (next delegation should contain exactly one element)

  To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_JOIN_01 - Can rejoin another stakepool/"

Randomized with seed 1300255809

#2497

@Anviking
Copy link
Member Author

Anviking commented Jun 7, 2021

bors r+

@iohk-bors
Copy link
Contributor

iohk-bors bot commented Jun 7, 2021

Build succeeded:

@iohk-bors iohk-bors bot merged commit e426116 into master Jun 7, 2021
@iohk-bors iohk-bors bot deleted the anviking/ADP-846/more-heap branch June 7, 2021 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants