Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Retry failed PVF prepare jobs #6213

Merged
merged 5 commits into from
Nov 8, 2022
Merged

Retry failed PVF prepare jobs #6213

merged 5 commits into from
Nov 8, 2022

Conversation

mrcnski
Copy link
Contributor

@mrcnski mrcnski commented Oct 30, 2022

PULL REQUEST

Overview

  • Keep track of the time of failure and the number of failures.
  • If we get a request to prepare a PVF, retry if:
    • Enough time has elapsed since a failure, and
    • The number of failures is not greater than NUM_PREPARE_RETRIES

TODO

  • How to test this?

Issues closed

Closes #4288

- Keep track of the time of failure and the number of failures.
- If we get a request to prepare a PVF, retry if:
  - Enough time has elapsed since a failure, and
  - The number of failures is not greater than `NUM_PREPARE_RETRIES`

Closes #4288
@mrcnski mrcnski added A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit. labels Oct 30, 2022
// The artifact is already being prepared, so we don't need to do anything.
},
ArtifactState::FailedToProcess(_) => {},
ArtifactState::FailedToProcess { last_time_failed, num_failures, error: _ } => {
// TODO: Do we want to retry for heads-up requests?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Figured we wanted the same logic for heads-up requests, but wanted to double-check.

@mrcnski
Copy link
Contributor Author

mrcnski commented Oct 30, 2022

I figured out how to test this, but I will wait until the general structure of the change is approved.

@mrcnski
Copy link
Contributor Author

mrcnski commented Oct 30, 2022

I had a conversation with @ordian about this change. He was wondering if we should retry for every PrepareError or if we could skip on e.g. Prevalidation errors, as a retry probably wouldn't fix that.

I am inclined to keep it simple and not add extra hacks. I think @pepyakin would say something similar, that this is cold code and so the performance for this case doesn't matter too much.

@slumber
Copy link
Contributor

slumber commented Oct 30, 2022

I had a conversation with @ordian about this change. He was wondering if we should retry for every PrepareError or if we could skip on e.g. Prevalidation errors, as a retry probably wouldn't fix that.

I am inclined to keep it simple and not add extra hacks. I think @pepyakin would say something similar, that this is cold code and so the performance for this case doesn't matter too much.

I wouldn't expect validators to retry compiling bad code, and I don't see a reason to call it a hack, it is a simple match on an error.

However, if the code is indeed malicious, then we should detect it during prechecking, submit invalid vote and reject it, thus no retries.

If it was prechecked before, no Prevalidation errors should happen. Which is again an external guarantee.

All in all, not so much difference

@mrcnski
Copy link
Contributor Author

mrcnski commented Oct 30, 2022

I wouldn't expect validators to retry compiling bad code, and I don't see a reason to call it a hack, it is a simple match on an error.

@slumber Fair enough, to me it just felt like a lot of logic for an edge case. But I can add it, with a note explaining that we don't expect to hit that match arm.

However, if the code is indeed malicious, then we should detect it during prechecking, submit invalid vote and reject it, thus no retries.

Gotcha. Can you please point me to where this happens in the code?

@slumber
Copy link
Contributor

slumber commented Oct 30, 2022

Gotcha. Can you please point me to where this happens in the code?

When the validation is requested, validator always fetches the code from the Runtime, either by hash or relay parent.
For example,

let validation_code = runtime_api_request(
sender,
descriptor.relay_parent,
RuntimeApiRequest::ValidationCode(descriptor.para_id, assumption, code_tx),
code_rx,
)
.await;

In order for the code to get stored in the Runtime storage, it should undergo prechecking process (it's not yet enabled but soon will be), this happens in the paras pallet (on upgrade or initialisation):

fn kick_off_pvf_check(
cause: PvfCheckCause<T::BlockNumber>,
code_hash: ValidationCodeHash,
code: ValidationCode,
cfg: &configuration::HostConfiguration<T::BlockNumber>,
) -> Weight {

@@ -48,6 +48,13 @@ pub const PRECHECK_COMPILATION_TIMEOUT: Duration = Duration::from_secs(60);
// NOTE: If you change this make sure to fix the buckets of `pvf_preparation_time` metric.
pub const EXECUTE_COMPILATION_TIMEOUT: Duration = Duration::from_secs(180);

/// The time period after which a failed preparation artifact is considered ready to be retried. Note that we will only
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: reaches soft wrap limit and no good reason to not wrap the line.

last_time_failed: SystemTime,
/// The number of times this artifact has failed to prepare.
num_failures: u32,
/// The prepare error.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: tautology. What is this error exactly? The last one I presume?

};

if let ArtifactState::Preparing { waiting_for_response } = state {
let num_failures = if let ArtifactState::Preparing { waiting_for_response, num_failures } =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I wonder if it's time to move it under the match above? Then, let state would become let (state, num_failures) and/or waiting_for_response.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave it a try, but I wasn't able to work around the compiler errors when I do that.

@pepyakin
Copy link
Contributor

Yeah, @ordian has a good point. Whenever the preparation worker returns an error gracefully, I expect it will be reproducible.

that this is cold code and so the performance for this case doesn't matter too much.

Actually, the code that does the bookkeeping of the PVF host is cold. The code that does prevalidation, compilation, etc is quite heavy (up to 60s) and can consume quite some memory. Moreover, it can be malicious.

Copy link
Member

@eskimor eskimor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, great work @m-cat !

let _ = result_sender.send(PrepareResult::Err(result.clone()));
},
ArtifactState::FailedToProcess { last_time_failed, num_failures, error } =>
if can_retry_prepare_after_failure(*last_time_failed, *num_failures) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: For preparation we actually agreed that retrying is not required/desired here. The pre-checking logic will not actually retry anyway, so this code will never get executed - so it is fine, but given that not doing it actually simplifies things, instead of complicating things (due to this complete separate implementation) - I would suggest to just not do it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would also remove some code duplication.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, removed.

@eskimor
Copy link
Member

eskimor commented Oct 31, 2022

@slumber has a good point, but as he also argued: We do not retry during prechecking and after prechecking, malicious code can be ruled out. So I am fine with retrying in any case, especially given that we will only do so every 15 minutes.

Copy link
Contributor

@slumber slumber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to add tests, please

@mrcnski mrcnski self-assigned this Nov 7, 2022
Copy link
Member

@eskimor eskimor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @m-cat !

@mrcnski mrcnski merged commit b19973c into master Nov 8, 2022
@mrcnski mrcnski deleted the m-cat/pvf-preparation-retry branch November 8, 2022 13:30
ordian added a commit that referenced this pull request Nov 9, 2022
* master:
  Companion for substrate#12530: Consolidate and deduplicate MMR API methods (#6167)
  Companion for substrate#12560 (#6226)
  [Companion] Bound Election and Staking by MaxActiveValidators (#6157)
  Retry failed PVF execution (AmbiguousWorkerDeath) (#6235)
  Make rolling session more resilient in case of long finality stalls (#6106)
  companion for fast unstake batching (#6253)
  staking miner: remove needless queue check (#6221)
  Retry failed PVF prepare jobs (#6213)
  Add a `last change` footer to the implementers guide (#6216)
  Update polkadot inflation to take into account auctions (#5872)
  Increase max rewardable nominators (#6230)
  BlockId removal: &Hash to Hash (#6246)
devdanco added a commit to gasp-xyz/polkadot that referenced this pull request Feb 14, 2023
* Companion for EPM duplicate submissions (paritytech#6115)

* make it work

* add migration

* fix

* Update utils/staking-miner/src/opts.rs

Co-authored-by: Niklas Adolfsson <[email protected]>

* Update utils/staking-miner/src/monitor.rs

* small tweaks

* Update utils/staking-miner/src/opts.rs

Co-authored-by: Bastian Köcher <[email protected]>

* fmt

* fix print'

* fmt

* update lockfile for {"substrate"}

Co-authored-by: Niklas Adolfsson <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: parity-processbot <>

* try and fix build (paritytech#6170)

* sync versions with current release (0.9.31) (paritytech#6176)

* Bump spec_version to 9310

* bump transaction_version (0.9.31) (paritytech#6171)

* Bump transaction_version for polkadot

* Bump transaction_version for kusama

* Bump transaction_version for rococo

* Bump transaction_version for westend

* Bump transaction_version for polkadot

* Bump transaction_version for kusama

* Bump transaction_version for rococo

* Bump transaction_version for westend

* Bump crate versions (0.9.31)

* Use a more typesafe approach for managing indexed data (paritytech#6150)

* Fix for issue #2403

* Nightly fmt

* Quick documentation fixes

* Default Implementation

* iter() function integrated

* Implemented iter functionalities

* Fmt

* small change

* updates node-network

* updates in dispute-coordinator

* Updates

* benchmarking fix

* minor fix

* test fixes in runtime api

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <[email protected]>

* Removal of [index], shorting of FromIterator, Renaming of GroupValidators to ValidatorGroups

* Removal of ops import

* documentation fixes for spell check

* implementation of generic type

* Refactoring

* Test and documentation fixes

* minor test fix

* minor test fix

* minor test fix

* Update node/network/statement-distribution/src/lib.rs

Co-authored-by: Andronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <[email protected]>

* removed IterMut

* Update node/core/dispute-coordinator/src/import.rs

Co-authored-by: Andronik <[email protected]>

* Update node/core/dispute-coordinator/src/initialized.rs

Co-authored-by: Andronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: Andronik <[email protected]>

* fmt

* IterMut

* documentation update

Co-authored-by: Andronik <[email protected]>

* minor adjustments and new TypeIndex trait

* spelling fix

* TypeIndex fix

Co-authored-by: Andronik <[email protected]>

* Add missing prerequisite in README for implementers guide (paritytech#6181)

* Companion for #12457 (Bounded Multisig) (paritytech#6172)

* u16 -> u32

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Make some fixes to logging in PVF subsystem (paritytech#6180)

* Log exit status code for workers

* Make log for execute job conclusion match prepare job conclusion

Trace log for conclusion of prepare job:

```rs
gum::debug!(
	target: LOG_TARGET,
	validation_code_hash = ?artifact_id.code_hash,
	?worker,
	?rip,
	"prepare worker concluded",
);
```

Co-authored-by: parity-processbot <>

* Co #12558: Update `pallet-multisig` benches (paritytech#6188)

* Typo

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add multisig weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update multisig weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* update lockfile for {"substrate"}

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: parity-processbot <>

* Update impl guide README (paritytech#6197)

* update impl guide readme

* Update README.md

* [Companion] StakingInterface adjustments (paritytech#6199)

* [Companion] StakingInterface adjustments

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Companion for update `wasm-opt` (paritytech#6209)

* Update cc

* Update regex

* Update thiserror

* Update anyhow

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Make ValidateUnsigned available on all chains for paras. (paritytech#6214)

Co-authored-by: eskimor <[email protected]>

* fix upgrade node test, use latest as base image (paritytech#6215)

* Check if approval voting db is empty on startup (paritytech#6219)

* Add approval voting db sanity check

Signed-off-by: Andrei Sandu <[email protected]>

* add newline back

Signed-off-by: Andrei Sandu <[email protected]>

* no need for overlay to read

Signed-off-by: Andrei Sandu <[email protected]>

Signed-off-by: Andrei Sandu <[email protected]>

* PVF timeouts follow-up (paritytech#6151)

* Rename timeout consts and timeout parameter; bump leniency

* Update implementor's guide with info about PVFs

* Make glossary a bit easier to read

* Add a note to LENIENT_PREPARATION_TIMEOUT

* Remove PVF-specific section from glossary

* Fix some typos

* BlockId removal: refactor: BlockBackend::block_body (paritytech#6223)

* BlockId removal: refactor: BlockBackend::block_body

It changes the arguments of `BlockBackend::block_body` method from: `BlockId<Block>` to: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Replace parachain/parathread boolean by enum (paritytech#6198)

* Replace parachain/parathread boolean by enum

* Address PR comments

* Update dependencies

* ParaType -> ParaKind

* Swap enum field order to avoid migration

* Rename paratype field to parakind

* Manual en-/decocing of Parakind

* Manual TypeInfo for ParaKind

* rename field back to parachain

* minor

* Update runtime/parachains/src/paras/mod.rs

Co-authored-by: Andrei Sandu <[email protected]>

* Manual serde Serialize and Deserialize for ParaKind

* cargo fmt

* Update runtime/parachains/src/paras/mod.rs

Co-authored-by: Andronik <[email protected]>

* Add test for serde_json encoding/decoding

* Move serde_json dep to dev-deps

Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Andronik <[email protected]>

* BlockId removal: refactor: Backend::justifications (paritytech#6229)

* BlockId removal: refactor: Backend::justifications

It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* formatting

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* BlockId removal: refactor: Backend::block_indexed_body (paritytech#6233)

* BlockId removal: refactor: Backend::block_indexed_body

It changes the arguments of `Backend::block_indexed_body` method from: `BlockId<Block>` to: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Xcm-Simulator Docs (paritytech#6178)

* Xcm-Simulator Docs

* spelling

* examples

* better docs

Co-authored-by: parity-processbot <>

* Add stake.plus bootnodes for westend, kusama, polkadot (paritytech#6224)

* add stake.plus bootnodes

* add stake.plus bootnodes for westend, kusama and polkadot

Co-authored-by: senseless <[email protected]>
Co-authored-by: parity-processbot <>

* clean up executed runtime migrations (paritytech#6206)

* kusama: clean up executed migrations

* polkadot: clean up executed migrations

* westend: clean up executed migrations

* Co #12085: Update `k256` (paritytech#6238)

* Typo

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update Cargo.lock

* update lockfile for {"substrate"}

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: parity-processbot <>

* cleanup deps (paritytech#6242)

* Pipeline with ci image with rust 1.65 (paritytech#6243)

* Pipeline with ci image with rust 1.65

* fix a warning

* return production image

Co-authored-by: Andronik <[email protected]>

* BlockId removal: &Hash to Hash (paritytech#6246)

* BlockId removal: &Hash to Hash

It changes &Block::Hash argument to Block::Hash.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* missing file corrected

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Increase max rewardable nominators (paritytech#6230)

* Increase max rewardable nominators

* update kusama as well

* Update polkadot inflation to take into account auctions (paritytech#5872)

* Update polkadot inflation to take into account auctions

* a possible solution -- but needs a rather untrivial data seeding

* some additional comments

* Use LOWEST_PUBLIC_ID as a guide to filter out system/common good para ids

* Fixes

* move tests

* fix

Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>

* Add a `last change` footer to the implementers guide (paritytech#6216)

* Add a `last change` footer to the implementers guide

Some of the newcomers were noticing outdated pages in the implementer's guide.
This idea came up as a heuristic for how up-to-date an individual page is.

* Update `build-implementers-guide` CI job

* Retry failed PVF prepare jobs (paritytech#6213)

* staking miner: remove needless queue check (paritytech#6221)

* staking miner: remove needless queue check

If the queue is full and the "mined solution" is better than solutions in the queue according to the "strategy"
then the solution with worse score will be kicked out from the queue.

Thus, the check can be removed completly.

* fix compile warns

* companion for fast unstake batching (paritytech#6253)

* update

* add migrations

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Make rolling session more resilient in case of long finality stalls (paritytech#6106)

* Impl dynamic window size. Keep sessions for unfinalized chain

Signed-off-by: Andrei Sandu <[email protected]>

* feedback

Signed-off-by: Andrei Sandu <[email protected]>

* Stretch also in contructor plus  tests

Signed-off-by: Andrei Sandu <[email protected]>

* review feedback

Signed-off-by: Andrei Sandu <[email protected]>

* fix approval-voting tests

Signed-off-by: Andrei Sandu <[email protected]>

* grunting: dispute coordinator tests

Signed-off-by: Andrei Sandu <[email protected]>

* add session window column

Signed-off-by: Andrei Sandu <[email protected]>

* integrate approval vote and fix tests

Signed-off-by: Andrei Sandu <[email protected]>

* fix rolling session tests

Signed-off-by: Andrei Sandu <[email protected]>

* Small refactor

Signed-off-by: Andrei Sandu <[email protected]>

* WIP, tests failing

Signed-off-by: Andrei Sandu <[email protected]>

* Fix approval voting tests

Signed-off-by: Andrei Sandu <[email protected]>

* fix dispute-coordinator tests

Signed-off-by: Andrei Sandu <[email protected]>

* remove uneeded param

Signed-off-by: Andrei Sandu <[email protected]>

* fmt

Signed-off-by: Andrei Sandu <[email protected]>

* fix loose ends

Signed-off-by: Andrei Sandu <[email protected]>

* allow failure and tests for it

Signed-off-by: Andrei Sandu <[email protected]>

* fix comment

Signed-off-by: Andrei Sandu <[email protected]>

* comment fix

Signed-off-by: Andrei Sandu <[email protected]>

* style fix

Signed-off-by: Andrei Sandu <[email protected]>

* new col doesn't need to be ordered

Signed-off-by: Andrei Sandu <[email protected]>

* fmt and spellcheck

Signed-off-by: Andrei Sandu <[email protected]>

* db persist tests

Signed-off-by: Andrei Sandu <[email protected]>

* Add v2 config and cols

Signed-off-by: Andrei Sandu <[email protected]>

* DB upgrade WIP

Signed-off-by: Andrei Sandu <[email protected]>

* Fix comments

Signed-off-by: Andrei Sandu <[email protected]>

* add todo

Signed-off-by: Andrei Sandu <[email protected]>

* update to parity-db to "0.4.2"

Signed-off-by: Andrei Sandu <[email protected]>

* migration complete

Signed-off-by: Andrei Sandu <[email protected]>

* One session window size

Signed-off-by: Andrei Sandu <[email protected]>

* fix merge damage

Signed-off-by: Andrei Sandu <[email protected]>

* fix build errors

Signed-off-by: Andrei Sandu <[email protected]>

* fmt

Signed-off-by: Andrei Sandu <[email protected]>

* comment fix

Signed-off-by: Andrei Sandu <[email protected]>

* fix build

Signed-off-by: Andrei Sandu <[email protected]>

* make error more explicit

Signed-off-by: Andrei Sandu <[email protected]>

* add comment

Signed-off-by: Andrei Sandu <[email protected]>

* refactor conflict merge

Signed-off-by: Andrei Sandu <[email protected]>

* rename col_data

Signed-off-by: Andrei Sandu <[email protected]>

* add doc comment

Signed-off-by: Andrei Sandu <[email protected]>

* fix build

Signed-off-by: Andrei Sandu <[email protected]>

* migration: move all cols to v2

Signed-off-by: Andrei Sandu <[email protected]>

Signed-off-by: Andrei Sandu <[email protected]>

* Retry failed PVF execution (AmbiguousWorkerDeath) (paritytech#6235)

* Fix a couple of typos

* Retry failed PVF execution

PVF execution that fails due to AmbiguousWorkerDeath should be retried once.
This should reduce the occurrence of failures due to transient conditions.

Closes paritytech#6195

* Address a couple of nits

* Write tests; refactor (add `validate_candidate_with_retry`)

* Update node/core/candidate-validation/src/lib.rs

Co-authored-by: Andronik <[email protected]>

Co-authored-by: eskimor <[email protected]>
Co-authored-by: Andronik <[email protected]>

* [Companion] Bound Election and Staking by MaxActiveValidators (paritytech#6157)

* add maximum winners to multi phase election provider

* fallback to noelection

* fmt

* missing values

* convert boundedvec to inner before sort

* dont clone

* pr feedback

* update lockfile for {"substrate"}

* run onchain election on westend benchmark

Co-authored-by: parity-processbot <>

* Companion for substrate#12560 (paritytech#6226)

* Companion for substrate#12560

Signed-off-by: koushiro <[email protected]>

* update num-format v0.4.0 ==> v0.4.3

* Fix

* update lockfile for {"substrate"}

Signed-off-by: koushiro <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: parity-processbot <>

* Companion for substrate#12530: Consolidate and deduplicate MMR API methods (paritytech#6167)

* histor. batch proof: make best block arg optional

* make generate_batch_proof stub for historical

* merge generate_{historical_}batch_proof functions

* merge generate_{batch_}proof functions

* merge verify_{batch_}proof functions

* merge verify_{batch_}proof_stateless functions

* rename BatchProof->Proof

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Add a few staking params to fast-runtime build (paritytech#5424)

Co-authored-by: Squirrel <[email protected]>

* State trie migration rococo runtime changes. (paritytech#6127)

* add state-trie-migration (warn key need to be changed)

* rococo root

* restore master benchs (weights from substrate are used).

* use ord_parameter macro.

* do not upgrade runtime version yet

* apply review changes

* to test ci

* Revert "to test ci"

This reverts commit 5df6c5c.

* test ci

* Revert "test ci"

This reverts commit 0747761.

Co-authored-by: parity-processbot <>

* Brad implementers guide revisions 2 (paritytech#6239)

* Add disputes subsystems fix

* Updated dispute approval vote import reasoning

* Improved wording of my changes

* Resolving issues brought up in comments

* Update disputes prioritisation in `dispute-coordinator` (paritytech#6130)

* Scraper processes CandidateBacked events

* Change definition of best-effort

* Fix `dispute-coordinator` tests

* Unit test for dispute filtering

* Clarification comment

* Add tests

* Fix logic

If a dispute is not backed, not included and not confirmed we
don't participate but we do import votes.

* Add metrics for refrained participations

* Revert "Add tests"

This reverts commit 7b8391a.

* Revert "Unit test for dispute filtering"

This reverts commit 92ba5fe.

* fix dispute-coordinator tests

* Fix scraping

* new tests

* Small fixes in guide

* Apply suggestions from code review

Co-authored-by: Andrei Sandu <[email protected]>

* Fix some comments and remove a pointless test

* Code review feedback

* Clarification comment in tests

* Some tests

* Reference counted `CandidateHash` in scraper

* Proper handling for Backed and Included candidates in scraper

Backed candidates which are not included should be kept for a
predetermined window of finalized blocks. E.g. if a candidate is backed
but not included in block 2, and the window size is 2, the same
candidate should be cleaned after block 4 is finalized.

Add reference counting for candidates in scraper. A candidate can be
added on multiple block heights so we have to make sure we don't clean
it prematurely from the scraper.

Add tests.

* Update comments in tests

* Guide update

* Fix cleanup logic for `backed_candidates_by_block_number`

* Simplify cleanup

* Make spellcheck happy

* Update tests

* Extract candidate backing logic in separate struct

* Code review feedback

* Treat  backed and included candidates in the same fashion

* Update some comments

* Small improvements in test

* spell check

* Fix some more comments

* clean -> prune

* Code review feedback

* Reword comment

* spelling

Co-authored-by: Andrei Sandu <[email protected]>

* approval-voting: remove redundant validation check (paritytech#6266)

* approval-voting: remove a redundant check

* candidate-validation: remove unreachable check

* remove fill_block (paritytech#6200)

Co-authored-by: parity-processbot <>

* fix a compilation warning (paritytech#6279)

Fixes paritytech#6277.

* Only report concluded if there is an actual dispute. (paritytech#6270)

* Only report concluded if there is an actual dispute.

Hence no "non"-disputes will be added to disputes anymore.

* Fix redundant check.

* Test for no onesided disputes.

Co-authored-by: eskimor <[email protected]>

* [ci] fix buildah image (paritytech#6281)

* Revert special casing of Kusama for grandpa rounds. (paritytech#6217)

Co-authored-by: eskimor <[email protected]>

* Fixes "for loop over an `Option`" warnings (paritytech#6291)

Was seeing these warnings when running `cargo check --all`:

```
warning: for loop over an `Option`. This is more readably written as an `if let` statement
    --> node/core/approval-voting/src/lib.rs:1147:21
     |
1147 |             for activated in update.activated {
     |                              ^^^^^^^^^^^^^^^^
     |
     = note: `#[warn(for_loops_over_fallibles)]` on by default
help: to check pattern in a loop use `while let`
     |
1147 |             while let Some(activated) = update.activated {
     |             ~~~~~~~~~~~~~~~         ~~~
help: consider using `if let` to clear intent
     |
1147 |             if let Some(activated) = update.activated {
     |             ~~~~~~~~~~~~         ~~~
```

My guess is that `activated` used to be a SmallVec or similar, as is
`deactivated`. It was changed to an `Option`, the `for` still compiled (it's
technically correct, just weird), and the compiler didn't catch it until now.

* companion for #12599 (paritytech#6290)

* companion for #12599

* update Cargo.lock

* use cargo path instead of diener

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* [ci] Improve pipeline stopper (paritytech#6300)

* [ci] Improve pipeline stopper

* break test-linux-stable

* fix test-linux-stable

* Provisioner should ignore unconfirmed disputes (paritytech#6294)

* Fix typos

* Filter unconfirmed disputes in provisioner -  random_selection

* Rework dispute coordinator to return `DisputeStatus` with
`ActiveDisputes` message.
* Rework the random_selection implementation of `select_disptues` in
  `provisioner` to return only confirmed disputes.

* Filter unconfirmed disputes in provisioner - prioritized_selection

* Add test for unconfirmed disputes handling

* Fix `dispute-distribution` tests

* Add Helikon boot nodes for Polkadot, Kusama and Westend. (paritytech#6240)

* Dedup subsystem name (paritytech#6305)

Signed-off-by: Andrei Sandu <[email protected]>

Signed-off-by: Andrei Sandu <[email protected]>

* Change best effort queue behaviour in `dispute-coordinator` (paritytech#6275)

* Change best effort queue behaviour in `dispute-coordinator`

Use the same type of queue (`BTreeMap<CandidateComparator,
ParticipationRequest>`) for best effort and priority in
`dispute-coordinator`.

Rework `CandidateComparator` to handle unavailable parent
block numbers.

Best effort queue will order disputes the same way as priority does - by
parent's block height. Disputes on candidates for which the parent's
block number can't be obtained will be treated with the lowest priority.

* Fix tests: Handle `ChainApiMessage::BlockNumber` in `handle_sync_queries`

* Some tests are deadlocking on sending messages via overseer so change `SingleItemSink`to `mpsc::Sender` with a buffer of 1

* Fix a race in test after adding a buffered queue for overseer messages

* Fix the rest of the tests

* Guide update - best-effort queue

* Guide update: clarification about spam votes

* Fix tests in `availability-distribution`

* Update comments

* Add `make_buffered_subsystem_context` in `subsystem-test-helpers`

* Code review feedback

* Code review feedback

* Code review feedback

* Don't add best effort candidate if it is already in priority queue

* Remove an old comment

* Fix insert in best_effort

* [ci] fix build implementers guide (paritytech#6306)

* [ci] fix build implementers guide

* add comment

* rm git fetch from publish-docs

* Remove the `wasmtime` feature flag (companion for substrate#12684) (paritytech#6268)

* Remove the `wasmtime` feature flag

* Update `substrate` to the newest `master`

* Update `substrate` to the newest `master`

* Add `starts_with` to v0 and v1 MultiLocation (paritytech#6311)

* add `starts_with` to v0 and v1 MultiLocation

* add tests

* fmt

* Extend lower bound of `manage_lease_period_start` from `runtime_common::slots` (paritytech#6318)

* Update async-trait version to v0.1.58 (paritytech#6319)

* Update async-trait version

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Add PVF module documentation (paritytech#6293)

* Add PVF module documentation

TODO (once the PRs land):

- [ ] Document executor parametrization.

- [ ] Document CPU time measurement of timeouts.

* Update node/core/pvf/src/lib.rs

Co-authored-by: Andrei Sandu <[email protected]>

* Clarify meaning of PVF acronym

* Move PVF doc to implementer's guide

* Clean up implementer's guide a bit

* Add page for PVF types

* pvf: Better separation between crate docs and implementer's guide

* ci: Add "prevalidating" to the dictionary

* ig: Remove types/chain.md

The types contained therein did not exist and the file was not referenced
anywhere.

Co-authored-by: Andrei Sandu <[email protected]>

* Rate limit improvements (paritytech#6315)

* We actually don't need to rate limit redundant requests.

Those redundant requests should not actually happen, but still.

* Add some logging.

* Also log message when the receiving side hit the rate limit.

* Update node/network/dispute-distribution/src/sender/mod.rs

Co-authored-by: Alexandru Vasile <[email protected]>

Co-authored-by: eskimor <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>

* [ci] fix build-implementers-guide (paritytech#6335)

* [ci] fix build-implementers-guide

* fix user

* Added Amforc bootnodes for Polkadot and Kusama (paritytech#6077)

* add wss and update dns

* fix wrong node id

* ig: Fix description of execution retry delay (paritytech#6342)

The delay is 3s and not 1s. I removed the reference to a specific number of
seconds as it may be too specific for a high-level description.

* Add support for outbound only configs on request/response protocols (paritytech#6343)

* Add option ot add outbound_only configurations

* Improve comment

* cargo update -p sp-io (paritytech#6351)

* Add more granularity to prometheus histogram buckets (paritytech#6348)

* Add buckets below 5ms

* Add more specific histogram buckets

* Add more specific buckets

* cargo fmt

* Provide some more granular metrics for polkadot_pvf_execution_time (paritytech#6346)

* [ci] fix implementer guide job (paritytech#6357)

* [DNM] debug implementer guide job

* remove git depth

* change git strategy

* add git depth

* try k8s runner

* fix k8s template

* test job

* fix test

* fix

* return pipeline back

* enable disabled deploy-parity-testnet

* dispute-coordinator: fix earliest session checks for pruning and import (paritytech#6358)

* RollingSession: add fn contains

Signed-off-by: Andrei Sandu <[email protected]>

* handle_import_statements fix ancient dispute check

Signed-off-by: Andrei Sandu <[email protected]>

* work with earliest session instead of latest

Signed-off-by: Andrei Sandu <[email protected]>

* update comment

Signed-off-by: Andrei Sandu <[email protected]>

Signed-off-by: Andrei Sandu <[email protected]>

* remove executed migrations (0.9.33) (paritytech#6364)

* Companion for: pallet-mmr: move offchain logic to client-side gadget (paritytech#6321)

* Spawn MMR gadget when offchain indexing is enabled

* companion PR code review changes: 1st iteration

* Code review changes: 2nd iteration

* update lockfile for {"substrate"}

Co-authored-by: acatangiu <[email protected]>
Co-authored-by: parity-processbot <>

* Remove `parity-util-mem` from `runtime-api` cache (paritytech#6367)

* Add Collectives as Trusted Teleporter (paritytech#6326)

* add collectives as trusted teleporter

* fix statemint-specific doc

* sync versions with current release (0.9.33) (paritytech#6363)

* westend: update transaction version

* polkadot: update transaction version

* kusama: update transaction version

* Bump spec_version to 9330

* bump versions to 0.9.33

* Clippyfy (paritytech#6341)

* Add clippy config and remove .cargo from gitignore

* first fixes

* Clippyfied

* Add clippy CI job

* comment out rusty-cachier

* minor

* fix ci

* remove DAG from check-dependent-project

* add DAG to clippy

Co-authored-by: alvicsam <[email protected]>

* Companion for: MMR: move RPC code from frame/ to client/ (paritytech#6369)

* rpc: mmr rpc crate name change

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* support opengov calls in proxy definitions (paritytech#6366)

* Use CPU clock timeout for PVF jobs (paritytech#6282)

* Put in skeleton logic for CPU-time-preparation

Still needed:
- Flesh out logic
- Refactor some spots
- Tests

* Continue filling in logic for prepare worker CPU time changes

* Fix compiler errors

* Update lenience factor

* Fix some clippy lints for PVF module

* Fix compilation errors

* Address some review comments

* Add logging

* Add another log

* Address some review comments; change Mutex to AtomicBool

* Refactor handling response bytes

* Add CPU clock timeout logic for execute jobs

* Properly handle AtomicBool flag

* Use `Ordering::Relaxed`

* Refactor thread coordination logic

* Fix bug

* Add some timing information to execute tests

* Add section about the mitigation to the IG

* minor: Change more `Ordering`s to `Relaxed`

* candidate-validation: Fix build errors

* guide: remove refences to outdated secondary checkers (paritytech#6309)

* guide: remove refences to outdated secondary checkers

* Update roadmap/implementers-guide/src/glossary.md

* guide: remove refences to Fisherman

* revert changes to roadmap/parachains.md

* Kusama: approve/reject treasury prop by treasurer (paritytech#6354)

* Add buckets on lower end of distribution to network bridge latency metrics (paritytech#6359)

* Add two more buckets on lower end of distribution

* add even smaller buckets

* cargo fmt

* Reduce provisioner work (paritytech#6328)

* Store values needed to create inherent data when needed instead of creating them early on

* Point deps to substrate branch

* Arc the client

* Cargo update

* Fix main cargo files

* Undo cargo file changes

* Add overseer dep to inherents

* Update deps

* Simplify code

* Update benchmark

* Update node/client/src/benchmarking.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/parachains-inherent/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/parachains-inherent/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Revert "Update node/core/parachains-inherent/src/lib.rs"

This reverts commit 8b9555d.

* Revert "Update node/core/parachains-inherent/src/lib.rs"

This reverts commit 816c92d.

* cargo update -p sp-io

* fmt

Co-authored-by: Bastian Köcher <[email protected]>

* update deprecated alias `--all` (paritytech#6383)

`--all` is a deprecated alias for `--workspace` (https://doc.rust-lang.org/cargo/commands/cargo-test.html)

* Upgrade tokio to 1.22.0 (paritytech#6262)

Co-authored-by: Sebastian Kunert <[email protected]>

* Set polkadot version in one place (paritytech#6095)

* rust 1.64 enables workspace properties

* add edition, repository and authors.

* of course, update the version in one place.

Co-authored-by: Andronik <[email protected]>

* Introduce NIS functionality into Kusama/Rococo (paritytech#6352)

* Integrate NIS into Kusama/Rococo

* Missing files

* Fix

* Fix

* Formatting

* Add Kusama weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add Rococo weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Bump

* Bump

* ".git/.scripts/bench-bot.sh" runtime kusama-dev pallet_nis

* ".git/.scripts/bench-bot.sh" runtime rococo-dev pallet_nis

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: command-bot <>

* OpenGov improvements for Kusama (paritytech#6372)

* Tweaks to optimise gov2

* Use new inactive funds

* Introduce migrations

* Fixes

* Fixes

* Fixes

* Some constant updates for Fellowship

* Further tweaks

* Lower floor for whitelisted

* Tweak some NIS params (paritytech#6393)

* Reduce base period, increase queue count

* Reduce base period, increase queue count

* allow root with gov2 origins (paritytech#6390)

* Bump Substrate (paritytech#6394)

* crowdloan: Fix migration. (paritytech#6397)

The migration would not have been run because of the `current_version ==
1` check.

* Companion of Substrate PR 12837 (paritytech#6385)

* rename some crates for publishing to crates.io

* s/remote-ext/frame-remote-externalities

* cargo update

* Make submission deposit reasonable (paritytech#6402)

* kusama whitelist pallet preimage dep upgrade (paritytech#6392)

* Bump (paritytech#6404)

* Companion for paritytech/substrate#12795 (paritytech#6374)

* Begin removing `parity-util-mem`; remove `collect_memory_stats`

* Update some dependencies that were using `parity-util-mem`

* Remove `trie-memory-tracker` feature

* Update Cargo.lock

* Update `kvdb-shared-tests`

* Add back jemalloc

* Add missing license header

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
Co-authored-by: Andronik <[email protected]>

* Let the PVF host kill the worker on timeout (paritytech#6381)

* Let the PVF host kill the worker on timeout

* Fix comment

* Fix inaccurate comments; add missing return statement

* Fix a comment

* Fix comment

* Companion for paritytech/substrate#12788 (paritytech#6360)

* Companion for paritytech/substrate#12788

* migrations

* rustfmt

* update lockfile for {"substrate"}

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Make sure to preserve backing votes (paritytech#6382)

* Guide updates

* Consider more dead forks.

* Ensure backing votes don't get overridden.

* Fix spelling.

* Fix comments.

* Update node/primitives/src/lib.rs

Co-authored-by: Tsvetomir Dimitrov <[email protected]>

Co-authored-by: eskimor <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>

* Refactoring to condense disputes tests (paritytech#6395)

* Refactoring to condense disputes tests

* Removing unhelpful comment lines

* Addressing Tsveto's suggestions

* Fixing formatting nit

* [ci] Adjust check-runtime-migration job (paritytech#6107)

* [WIP][ci] Add node to check-runtime-migration job

* add image

* remove sscache from before_script

* add nodes

* restart pipeline

* restart pipeline2

* disable other jobs

* debug

* remove debug

* add ports

* restart pipeline

* restart pipeline

* restart pipeline

* try kusama first

* run polkadot 1st

* disable some jobs

* try command from command bot

* cargo run

* test passing variables

* run without condition

* adjust kusama and westend

* fix

* return jobs

* fix small nits

* split check-runtime-migration

Co-authored-by: parity-processbot <>

* update weights (sync with 0.9.33) (paritytech#6362)

* update weights (0.9.33) (paritytech#6299)

* kusama: update weights

* polkadot: update weights

* rococo: update weights

* westend: update weights

* fix deps

* Resolve merge

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Reset Kusama whitelist weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* Feature gate test `benchmark_storage_works` (paritytech#6376)

* Feature gate storage bench test

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* typo

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: parity-processbot <>

* Companion for paritytech/substrate#12868 (paritytech#6406)

* Replace WEIGHT_PER_* with WEIGHT_REF_TIME_PER_*

* cargo fmt

* Update substrate

* OpenGov: Tweak parameters further (paritytech#6416)

* Tweak parameters further

* Further tweaks to deposits

* companion for #12663 jsonrpsee v0.16 (paritytech#6339)

* companion for #12663 jsonrpsee v0.16.1

* update substrate

* merge master

* Update rpc/Cargo.toml

Co-authored-by: Alexandru Vasile <[email protected]>

* Update utils/staking-miner/Cargo.toml

Co-authored-by: Alexandru Vasile <[email protected]>

* update lockfile for {"substrate"}

Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: parity-processbot <>

* CI: Remove the excessive variable declaration (paritytech#6426)

* companion slash chilling update (paritytech#6424)

* update weights

* goddamit

* update weights

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* swap error responses (paritytech#6421)

* [ci] fix check-transaction-versions (paritytech#6425)

* [ci] fix check-transaction-versions

* allow fail the job

* approval-distribution: batched approval/assignment sending (paritytech#6401)

* Imple batched send

Signed-off-by: Andrei Sandu <[email protected]>

* Add batch tests

Signed-off-by: Andrei Sandu <[email protected]>

* pub MAX_NOTIFICATION_SIZE

Signed-off-by: Andrei Sandu <[email protected]>

* spell check

Signed-off-by: Andrei Sandu <[email protected]>

* spellcheck ...

Signed-off-by: Andrei Sandu <[email protected]>

* Fix comment

Signed-off-by: Andrei Sandu <[email protected]>

* o.O

Signed-off-by: Andrei Sandu <[email protected]>

* 2 constants

Signed-off-by: Andrei Sandu <[email protected]>

* Ensure batch size is at least 1 element

Signed-off-by: Andrei Sandu <[email protected]>

* feedback impl

Signed-off-by: Andrei Sandu <[email protected]>

Signed-off-by: Andrei Sandu <[email protected]>

* add westend bootnode (paritytech#6434)

* [ci] add job switcher (paritytech#6433)

* [ci] add job switcher

* add before_script to docker and k8s runners

* upd runners

* sccache :(

* companion for try-runtime revamp (paritytech#6187)

* update to reflect latest try-runtime stuff

* update to latest version

* fix

* fix miner

* update

* update

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* Fix wrong rate limit + add a few logs. (paritytech#6440)

* Add trace log

* More tracing.

* Fix redundant rate limit.

* Add trace log.

* Improve logging.

Co-authored-by: eskimor <[email protected]>

* Bump workspace unified version

* Bump spec_version to 9360

* fix diener output

* bump substrate

* clean up migrations

* bump version (0.9.36)

* sync transaction_version with 0.9.35 (compatible)

* sync transaction_version with 0.9.35 (compatible)

* update weights (0.9.36) (paritytech#6450)

* polkadot: update weights (0.9.36)

* kusama: update weights (0.9.36)

* rococo: update weights (0.9.36)

* westend: update weights (0.9.36)

* delete unnecessary files

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Andrei Sandu <[email protected]>
Signed-off-by: koushiro <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Niklas Adolfsson <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Mara Robin B <[email protected]>
Co-authored-by: Boluwatife Bakre <[email protected]>
Co-authored-by: Andronik <[email protected]>
Co-authored-by: Marcin S <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Dan Shields <[email protected]>
Co-authored-by: Roman Useinov <[email protected]>
Co-authored-by: Alexander Theißen <[email protected]>
Co-authored-by: eskimor <[email protected]>
Co-authored-by: eskimor <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Michal Kucharczyk <[email protected]>
Co-authored-by: alexgparity <[email protected]>
Co-authored-by: Sergej Sakac <[email protected]>
Co-authored-by: Tom <[email protected]>
Co-authored-by: senseless <[email protected]>
Co-authored-by: Xiliang Chen <[email protected]>
Co-authored-by: Alexander Samusev <[email protected]>
Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: Ankan <[email protected]>
Co-authored-by: Qinxuan Chen <[email protected]>
Co-authored-by: Robert Hambrock <[email protected]>
Co-authored-by: moh-eulith <[email protected]>
Co-authored-by: Squirrel <[email protected]>
Co-authored-by: cheme <[email protected]>
Co-authored-by: Bradley Olson <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>
Co-authored-by: Kutsal Kaan Bilgin <[email protected]>
Co-authored-by: Koute <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Aaro Altonen <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: tugy <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Mattia L.V. Bradascio <[email protected]>
Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: acatangiu <[email protected]>
Co-authored-by: alvicsam <[email protected]>
Co-authored-by: Muharem Ismailov <[email protected]>
Co-authored-by: amab8901 <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>
Co-authored-by: João Paulo Silva de Souza <[email protected]>
Co-authored-by: Vlad <[email protected]>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow retrying for rejected PVFs
4 participants