Fix treasury benchmarks when no SpendOrigin #3049

bgallois · 2024-01-24T15:47:18Z

Issue

It was impossible to benchmark the pallet_treasury when SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>; was specified.

Done

Use weight = 0 for all extrinsics that are un-callable with no SpendOrigin.
Fix benchmarks for extrinsics requiring a Spend even if SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>;

cla-bot-2021 · 2024-01-24T15:47:26Z

User @bgallois, please sign the CLA here.

bgallois · 2024-01-27T09:24:22Z

I'm unable to sign the CLA: Content Encoding Error. An error occurred during a connection to cla.parity.io.

ggwpez

Thanks!

substrate/frame/treasury/src/benchmarking.rs

ggwpez · 2024-01-27T13:07:33Z

I'm unable to sign the CLA: Content Encoding Error. An error occurred during a connection to cla.parity.io.

Hm, can you try again?
Not sure whom to ping here @mutantcornholio maybe?

bgallois · 2024-01-27T14:55:38Z

I'm unable to sign the CLA: Content Encoding Error. An error occurred during a connection to cla.parity.io.

Hm, can you try again? Not sure whom to ping here @mutantcornholio maybe?

Still no luck.

bkchr

This is actually not correct. You are right that the benchmarks are not correct the way they are written, but these calls clearly don't have zero weight when SpendOrigin fails. If you look at the functions of the dispatchables, you will see that we access Spends aka reading from the DB which is one of the most expensive operations we have.

bgallois · 2024-01-28T05:44:02Z

This is actually not correct. You are right that the benchmarks are not correct the way they are written, but these calls clearly don't have zero weight when SpendOrigin fails. If you look at the functions of the dispatchables, you will see that we access Spends aka reading from the DB which is one of the most expensive operations we have.

In fact, these extrinsics should not be callable when SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>;. In this case, we can authorize weight=0 so that the benchmark can pass for this pallet (or that is the expected behavior I understand from the benchmarks already written).

Another solution is to benchmark the path until failure occurs and use this weight as the default weight, and these extrinsics could be callable even with SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>;.

ggwpez · 2024-01-28T13:38:06Z

You are right that the benchmarks are not correct the way they are written, but these calls clearly don't have zero weight when SpendOrigin fails.

Yes true. But he is fixing the case that SpendOrigin is configured to be impossible to pass. There should be no DB reads in that case, right?
Otherwise the try_success_origin for benchmarks is advised to return something that always works by using a feature flag. Although its better to go through the normal setup to also benchmark the effort of the try_success_origin.

bkchr · 2024-01-28T14:24:47Z

polkadot-sdk/substrate/frame/treasury/src/lib.rs

Lines 812 to 815 in acd043b

    
           pub fn payout(origin: OriginFor<T>, index: SpendIndex) -> DispatchResult { 
        
           	ensure_signed(origin)?; 
        
           	let mut spend = Spends::<T, I>::get(index).ok_or(Error::<T, I>::InvalidIndex)?; 
        
           	let now = frame_system::Pallet::<T>::block_number();

polkadot-sdk/substrate/frame/treasury/src/lib.rs

Lines 855 to 861 in acd043b

    
           pub fn check_status(origin: OriginFor<T>, index: SpendIndex) -> DispatchResultWithPostInfo { 
        
           	use PaymentState as State; 
        
           	use PaymentStatus as Status; 
        
           	ensure_signed(origin)?; 
        
           	let mut spend = Spends::<T, I>::get(index).ok_or(Error::<T, I>::InvalidIndex)?; 
        
           	let now = frame_system::Pallet::<T>::block_number();

polkadot-sdk/substrate/frame/treasury/src/lib.rs

Lines 909 to 911 in acd043b

    
           pub fn void_spend(origin: OriginFor<T>, index: SpendIndex) -> DispatchResult { 
        
           	T::RejectOrigin::ensure_origin(origin)?; 
        
           	let spend = Spends::<T, I>::get(index).ok_or(Error::<T, I>::InvalidIndex)?;

As we can see, there are reads. If SpendOrigin isn't set, this prevents that any value is stored in Spends. However, it doesn't prevent that these other functions are doing a read.

bgallois · 2024-01-28T14:26:01Z

You are right that the benchmarks are not correct the way they are written, but these calls clearly don't have zero weight when SpendOrigin fails.

Yes true. But he is fixing the case that SpendOrigin is configured to be impossible to pass. There should be no DB reads in that case, right? Otherwise the try_success_origin for benchmarks is advised to return something that always works by using a feature flag. Although its better to go through the normal setup to also benchmark the effort of the try_success_origin.

Alright, but in this case, we need to adapt the three benchmarks(check_status, void_spend and payout) that always failed not because of a SpendOrigin check, but because the extrinsic will fail when reading the Spend storage, which will always be empty. For these, the weight will be one signed origin check and then one read to the database.

bkchr · 2024-01-28T19:16:30Z

Alright, but in this case, we need to adapt the three benchmarks(check_status, void_spend and payout) that always failed not because of a SpendOrigin check,

Yes, I actually wanted this. I should have been more clear :D

mutantcornholio · 2024-01-29T08:45:22Z

I'm unable to sign the CLA: Content Encoding Error. An error occurred during a connection to cla.parity.io.

Hm, can you try again? Not sure whom to ping here @mutantcornholio maybe?

Thanks for report, I'm on it

bkchr · 2024-01-29T14:25:06Z

substrate/frame/treasury/src/benchmarking.rs

+fn force_add_spend<T: Config<I>, I: 'static>(
+	asset_kind: Box<T::AssetKind>,
+	amount: AssetBalanceOf<T, I>,
+	beneficiary: Box<BeneficiaryLookupOf<T, I>>,
+) -> Result<(), &'static str> {


This is not correct. What you should do is to skip adding something to Spend when T::SpendOrigin::try_successful_origin() fails.

So,

if let Ok(origin) = T::SpendOrigin::try_successful_origin() { Treasury::<T, _>::spend( origin, force_add_spend::<T, _>( Box::new(asset_kind.clone()), amount, Box::new(beneficiary_lookup), None, )?; }

is the logic I would expect for these benchmarks below.

If we omit adding something to the spend, the benchmark will fail during execution with the 'InvalidIndex' error. While it is probably possible to bypass this error using a solution like this, it appears error-prone with an extrinsic that will sometime complete and sometime fail.

As extrinsic benchmark cover worst case scenario, It seems not far-fetched that even with SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64> we take this impossible case.

I mean we should be able to use the same condition as above to check either on success or error?

CC @ggwpez

I know that the #[extrinsic_call] cannot be included inside a conditional block.

If there is a way to perform a benchmark with a condition and two expected results, I have never seen it.

You can use #[block] and put the if inside:

#[block] { if condition { ... } else { ... } }

Condition should be a bool already to not mess up the timing. Otherwise you can create a new benchmark just for the failing case and then use the maximum of the two as weight.

The more general case (forcing a spend then processing it) has a weight that is superior to the edge case (one read then an error), thus posing no risk of spamming without overly complicating the benchmark.

Let take the case of:

polkadot-sdk/substrate/frame/treasury/src/lib.rs

Lines 812 to 831 in 8a8f6f9

pub fn payout(origin: OriginFor<T>, index: SpendIndex) -> DispatchResult {

ensure_signed(origin)?;

let mut spend = Spends::<T, I>::get(index).ok_or(Error::<T, I>::InvalidIndex)?;

let now = frame_system::Pallet::<T>::block_number();

ensure!(now >= spend.valid_from, Error::<T, I>::EarlyPayout);

ensure!(spend.expire_at > now, Error::<T, I>::SpendExpired);

ensure!(

matches!(spend.status, PaymentState::Pending | PaymentState::Failed),

Error::<T, I>::AlreadyAttempted

);

let id = T::Paymaster::pay(&spend.beneficiary, spend.asset_kind.clone(), spend.amount)

.map_err(|_| Error::<T, I>::PayoutError)?;

spend.status = PaymentState::Attempted { id };

Spends::<T, I>::insert(index, spend);

Self::deposit_event(Event::<T, I>::Paid { index, payment_id: id });

Ok(())

The worst-case scenario occurs when a valid spend is found and paid. Therefore, even if SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>, and in reality, no spend can be added, we can still benchmark the worst case by forcibly adding a spend (what I have done in the last commit).

It is analog to an extrinsic than necessitate a root origin, we write a benchmark using the root origin and always charge the weight of this path. We did not split the benchmark to one with success when root and one with a fail when not root. The only time we split the benchmark is when the extrinsic will always fail and when there is no valid path, that is not the case with payout, check_status, and void_spend, but is it the case for spend, spend_local.

The worst-case scenario occurs when a valid spend is found and paid.

But this isn't the worst-case when you have SpendOrigin = NeverEnsureOrigin<u64>;. Then the worst-case is basically the best case as well and just means that there is only one read of the storage.

The worst-case scenario occurs when a valid spend is found and paid.

But this isn't the worst-case when you have SpendOrigin = NeverEnsureOrigin<u64>;. Then the worst-case is basically the best case as well and just means that there is only one read of the storage.

From the extrinsic point of view, it is the worst path because there is no mention of SpendOrigin in its implementation. In all the benchmarks that I found in Substrate, the worst path is determined from the extrinsic perspective, rather than considering all the (sometimes complicated) interference of all the parameters that can play a role outside the extrinsic implementation (and that can be quite complicated to anticipate when writing the benchmark, see this error for example).

I think that the implementation you're asking is suboptimal by adding unnecessary complexity to the benchmark for something that doesn't pose any threat to the chain and just add a slight overweight. However, you seem very committed to this specific approach, so I will move this pull request to draft status and I will implement it according to your suggestions when I have some time.

Not sure your arguments are correct here. I mean the problem here is that the benchmark is failing because it is written in an incorrect way. The benchmark is "abusing" SpendOrigin to setup the state and the benchmark is failing because SpendOrigin is set to NeverEnsureOrigin. Now if we fix the benchmark, we should do it properly that it generates the correct weights. The thing I'm proposing here is also not really that complicated and can be implemented in some minutes.

Yes, I agree that the benchmark is failing because it was written incorrectly. In my opinion, this failure directly stems from not approaching it from the extrinsic point of view, as I explained before.

No, it's not complicated. However, since I'm paid to work on another Substrate chain, this fix isn't a priority in my work schedule that is quite full at the moment. Feel free to complete it if you want it done rapidly (i.e. before the end of a week).

mutantcornholio · 2024-01-29T15:59:00Z

@bgallois could you try signing the CLA again please?

bgallois · 2024-01-29T16:01:57Z

@bgallois could you try signing the CLA again please?

Thanks it is working :)

Hugo-Trentesaux · 2024-09-13T11:05:10Z

The thing I'm proposing here is also not really that complicated and can be implemented in some minutes.

@bkchr, those minutes transformed into months and the benchmarks are still broken.

If you are still interested by a fix, we have it on our fork: https://github.com/duniter/duniter-polkadot-sdk/commits/duniter-substrate-v1.14.0/ (latest commit here).

bkchr · 2024-09-13T13:34:57Z

@Hugo-Trentesaux I have done now the changes as I had requested.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

It was impossible to benchmark the pallet_treasury when `SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>;` was specified. - [x] Use `weight = 0` for all extrinsics that are un-callable with no `SpendOrigin`. - [x] Fix benchmarks for extrinsics requiring a Spend even if `SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>;` --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: Oliver Tale-Yazdi <[email protected]>

bgallois · 2024-09-13T18:28:03Z

Thank you for the fix.

There is still a small attack vector if a chain is improbably (mis)configured with SpendOrigin set to NeverOrigin, but RejectOrigin can be successful. In this case, the code from remove_approval could be executed without any weight at all.

bkchr · 2024-09-13T21:58:56Z

Ty! #5713

paritytech-cmd-bot-polkadot-sdk · 2024-09-16T16:06:42Z

Git push to origin failed for stable2407 with exitcode 1

paritytech-cmd-bot-polkadot-sdk · 2024-09-16T16:06:46Z

Successfully created backport PR for stable2409:

[stable2409] Backport #3049 #5730

### Issue It was impossible to benchmark the pallet_treasury when `SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>;` was specified. ### Done - [x] Use `weight = 0` for all extrinsics that are un-callable with no `SpendOrigin`. - [x] Fix benchmarks for extrinsics requiring a Spend even if `SpendOrigin = frame_support::traits::NeverEnsureOrigin<u64>;` --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: Oliver Tale-Yazdi <[email protected]> (cherry picked from commit 51f3367)

paritytech-cmd-bot-polkadot-sdk · 2024-09-16T16:11:53Z

Created backport PR for stable2407:

[stable2407] Backport #3049 #5734 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-3049-to-stable2407
git worktree add --checkout .worktree/backport-3049-to-stable2407 backport-3049-to-stable2407
cd .worktree/backport-3049-to-stable2407
git reset --hard HEAD^
git cherry-pick -x 51f336711a0391987db69d6281c9b57bfe49d925
git push --force-with-lease

paritytech-cmd-bot-polkadot-sdk · 2024-09-16T16:11:57Z

Git push to origin failed for stable2409 with exitcode 1

Backport #3049 into `stable2409` from bgallois. See the [documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md) on how to use this bot.  Co-authored-by: Benjamin Gallois <[email protected]>

Backport #3049 into `stable2407` from bgallois. See the [documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md) on how to use this bot.  --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Benjamin Gallois <[email protected]> Co-authored-by: Oliver Tale-Yazdi <[email protected]>

bgallois requested review from a team January 24, 2024 15:47

bgallois marked this pull request as draft January 24, 2024 16:05

bgallois force-pushed the fix_treasury_benchmark branch from 20c2e85 to 67691bf Compare January 24, 2024 16:29

fix treasury benchmarks when no SpendOrigin

b8509ed

bgallois force-pushed the fix_treasury_benchmark branch from 67691bf to b8509ed Compare January 24, 2024 16:38

bgallois marked this pull request as ready for review January 24, 2024 16:42

ggwpez approved these changes Jan 27, 2024

View reviewed changes

substrate/frame/treasury/src/benchmarking.rs Outdated Show resolved Hide resolved

fix comments

f587bf1

bkchr approved these changes Jan 27, 2024

View reviewed changes

bkchr requested changes Jan 28, 2024

View reviewed changes

bgallois marked this pull request as draft January 28, 2024 14:37

fix benchmarks

08e6bd7

bgallois marked this pull request as ready for review January 29, 2024 10:59

bkchr reviewed Jan 29, 2024

View reviewed changes

bgallois marked this pull request as draft February 3, 2024 11:42

Fix benchmarks

042159a

bkchr added the T2-pallets This PR/Issue is related to a particular pallet. label Sep 13, 2024

PRDOC

6d28fc9

bkchr added the A4-needs-backport Pull request must be backported to all maintained releases. label Sep 13, 2024

ggwpez added 2 commits September 13, 2024 17:36

Fix on_init bench

44fbb39

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Merge remote-tracking branch 'origin/master' into fix_treasury_benchmark

f8ba289

ggwpez approved these changes Sep 13, 2024

View reviewed changes

ggwpez enabled auto-merge September 13, 2024 15:37

ggwpez added this pull request to the merge queue Sep 13, 2024

Merged via the queue into paritytech:master with commit 51f3367 Sep 13, 2024
204 of 206 checks passed

bgallois deleted the fix_treasury_benchmark branch September 13, 2024 18:31

bkchr mentioned this pull request Sep 14, 2024

NoOp Impl Polling Trait #5311

Merged

ggwpez added A4-needs-backport Pull request must be backported to all maintained releases. and removed A4-needs-backport Pull request must be backported to all maintained releases. labels Sep 16, 2024

paritytech-cmd-bot-polkadot-sdk bot mentioned this pull request Sep 16, 2024

[stable2409] Backport #3049 #5730

Merged

ggwpez added A4-needs-backport Pull request must be backported to all maintained releases. and removed A4-needs-backport Pull request must be backported to all maintained releases. labels Sep 16, 2024

paritytech-cmd-bot-polkadot-sdk bot mentioned this pull request Sep 16, 2024

[stable2407] Backport #3049 #5734

Merged

ggwpez removed the A4-needs-backport Pull request must be backported to all maintained releases. label Sep 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix treasury benchmarks when no SpendOrigin #3049

Fix treasury benchmarks when no SpendOrigin #3049

bgallois commented Jan 24, 2024 •

edited

Loading

cla-bot-2021 bot commented Jan 24, 2024

bgallois commented Jan 27, 2024

ggwpez left a comment

ggwpez commented Jan 27, 2024

bgallois commented Jan 27, 2024

bkchr left a comment

bgallois commented Jan 28, 2024 •

edited

Loading

ggwpez commented Jan 28, 2024 •

edited

Loading

bkchr commented Jan 28, 2024 •

edited

Loading

bgallois commented Jan 28, 2024

bkchr commented Jan 28, 2024

mutantcornholio commented Jan 29, 2024

bkchr Jan 29, 2024

bgallois Jan 29, 2024 •

edited

Loading

bkchr Jan 29, 2024

bgallois Jan 30, 2024 •

edited

Loading

ggwpez Jan 30, 2024

bgallois Jan 31, 2024 •

edited

Loading

bkchr Feb 2, 2024

bgallois Feb 3, 2024 •

edited

Loading

bkchr Feb 5, 2024

bgallois Feb 6, 2024

mutantcornholio commented Jan 29, 2024

bgallois commented Jan 29, 2024

Hugo-Trentesaux commented Sep 13, 2024

bkchr commented Sep 13, 2024

bgallois commented Sep 13, 2024

bkchr commented Sep 13, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 16, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 16, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 16, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 16, 2024

	pub fn payout(origin: OriginFor<T>, index: SpendIndex) -> DispatchResult {
	ensure_signed(origin)?;
	let mut spend = Spends::<T, I>::get(index).ok_or(Error::<T, I>::InvalidIndex)?;
	let now = frame_system::Pallet::<T>::block_number();
	ensure!(now >= spend.valid_from, Error::<T, I>::EarlyPayout);
	ensure!(spend.expire_at > now, Error::<T, I>::SpendExpired);
	ensure!(
	matches!(spend.status, PaymentState::Pending \| PaymentState::Failed),
	Error::<T, I>::AlreadyAttempted
	);

	let id = T::Paymaster::pay(&spend.beneficiary, spend.asset_kind.clone(), spend.amount)
	.map_err(\|_\| Error::<T, I>::PayoutError)?;

	spend.status = PaymentState::Attempted { id };
	Spends::<T, I>::insert(index, spend);

	Self::deposit_event(Event::<T, I>::Paid { index, payment_id: id });

	Ok(())

Fix treasury benchmarks when no SpendOrigin #3049

Fix treasury benchmarks when no SpendOrigin #3049

Conversation

bgallois commented Jan 24, 2024 • edited Loading

Issue

Done

cla-bot-2021 bot commented Jan 24, 2024

bgallois commented Jan 27, 2024

ggwpez left a comment

Choose a reason for hiding this comment

ggwpez commented Jan 27, 2024

bgallois commented Jan 27, 2024

bkchr left a comment

Choose a reason for hiding this comment

bgallois commented Jan 28, 2024 • edited Loading

ggwpez commented Jan 28, 2024 • edited Loading

bkchr commented Jan 28, 2024 • edited Loading

bgallois commented Jan 28, 2024

bkchr commented Jan 28, 2024

mutantcornholio commented Jan 29, 2024

bkchr Jan 29, 2024

Choose a reason for hiding this comment

bgallois Jan 29, 2024 • edited Loading

Choose a reason for hiding this comment

bkchr Jan 29, 2024

Choose a reason for hiding this comment

bgallois Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

ggwpez Jan 30, 2024

Choose a reason for hiding this comment

bgallois Jan 31, 2024 • edited Loading

Choose a reason for hiding this comment

bkchr Feb 2, 2024

Choose a reason for hiding this comment

bgallois Feb 3, 2024 • edited Loading

Choose a reason for hiding this comment

bkchr Feb 5, 2024

Choose a reason for hiding this comment

bgallois Feb 6, 2024

Choose a reason for hiding this comment

mutantcornholio commented Jan 29, 2024

bgallois commented Jan 29, 2024

Hugo-Trentesaux commented Sep 13, 2024

bkchr commented Sep 13, 2024

bgallois commented Sep 13, 2024

bkchr commented Sep 13, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 16, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 16, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 16, 2024

paritytech-cmd-bot-polkadot-sdk bot commented Sep 16, 2024

bgallois commented Jan 24, 2024 •

edited

Loading

bgallois commented Jan 28, 2024 •

edited

Loading

ggwpez commented Jan 28, 2024 •

edited

Loading

bkchr commented Jan 28, 2024 •

edited

Loading

bgallois Jan 29, 2024 •

edited

Loading

bgallois Jan 30, 2024 •

edited

Loading

bgallois Jan 31, 2024 •

edited

Loading

bgallois Feb 3, 2024 •

edited

Loading