Transaction Validation #736

SethDusek · 2023-12-28T23:58:40Z

Initial stateless validation work (things we can validate without needing blockchain context). Also includes a fix where we were allowing as many as u16::MAX inputs/outputs (65535) when ergo only allows i16::MAX (Short.MaxValue in scala) which is (32767). https://github.com/ergoplatform/ergo/blob/48239ef98ced06617dc21a0eee5670235e362933/ergo-core/src/main/scala/org/ergoplatform/modifiers/mempool/ErgoTransaction.scala#L93

coveralls · 2023-12-29T00:28:41Z

Pull Request Test Coverage Report for Build 8953175335

Details

165 of 228 (72.37%) changed or added relevant lines in 15 files are covered.
8 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.1%) to 80.473%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
ergo-lib/src/chain/transaction/ergo_transaction.rs	28	29	96.55%
ergotree-interpreter/src/eval/error.rs	0	2	0.0%
ergotree-interpreter/src/eval/scontext.rs	3	5	60.0%
ergotree-interpreter/src/eval/scoll.rs	8	11	72.73%
ergotree-ir/src/mir/value.rs	3	6	50.0%
ergo-lib/src/chain/transaction.rs	7	11	63.64%
ergo-lib/src/chain/transaction/storage_rent.rs	7	21	33.33%
ergo-lib/src/chain/parameters.rs	12	28	42.86%
ergo-lib/src/wallet/tx_context.rs	73	91	80.22%

Files with Coverage Reduction	New Missed Lines	%
ergotree-interpreter/src/eval/error.rs	2	21.88%
ergotree-ir/src/serialization/types.rs	6	83.28%

Totals
Change from base Build 8951357735:	0.1%
Covered Lines:	10723
Relevant Lines:	13325

💛 - Coveralls

SethDusek · 2024-03-18T11:20:58Z

Should be good to go now! Some notes:

This PR despite the usual unit tests was tested using ergovalidation which downloads transactions from node (requires extra indexing to be enabled) and validates them. After fixing some script bugs we're now down to 15 transactions on mainnet that sigma-rust rejects, all of which rely on buggy behavior from v4.0 that no longer holds in v5.0, including CONTEXT.selfBoxIndex being -1 and sigmaProp(sigmaProp(true)) being allowed. I believe fixing these (and adding version-specific actions to ergotree-interpreter) is out of scope for this PR. Gist of failing transactions
I had to bump rust version since sigma-rust no longer builds as some dependencies don't support rust 1.64 anymore. Clippy rejection seems to be related and should be ignored for now
There are some limitations still in interpreter. The biggest one is that atLeast(0, ...) isn't supported (I plan on addressing this separately in another PR, but I fixed atLeast(1, ...) not being supported in this PR).
There are some breaking changes in this PR, namely ErgoStateContext now also requiring Parameters. I believe this is an acceptable trade-off even if parameters aren't needed for signing. Later when sigma-rust adds support for costing we could use Parameters to calculate costs during signing
Also fixed quadratic scaling in TransactionContext::new
old:

new:

SethDusek · 2024-03-18T11:31:50Z

ergotree-interpreter/src/eval/scoll.rs

@@ -24,15 +24,20 @@ pub(crate) static INDEX_OF_EVAL_FN: EvalFn = |_env, _ctx, obj, args| {
            .get(0)
            .cloned()
            .ok_or_else(|| EvalError::NotFound("indexOf: missing first arg".to_string()))?;
-        let fallback_index = args


indexOf now has the same semantics as scala version, https://github.com/ScorexFoundation/sigmastate-interpreter/blob/efa68e7079393d70d25d43dd63dfba9a18d03415/data/shared/src/main/scala/sigma/ast/methods.scala#L880

ergo-lib/src/chain/transaction/ergo_transaction.rs

kushti · 2024-04-26T11:09:56Z

Please fix linter issues

…saction Scala's Short.MaxValue is equivalent to the maximum a 16 bit signed integer can hold

set_hook can only be called once, this means displaying an error multiple times causes a panic in sigma-rust. ignoring seems like the best solution

When converting a signed integer i8 like -1 (0xff) to a bigger signed integer like i64, the 2's complement representation will be padded with ff (0xffffffff...). Thus if input[7] for example is -1, when it is |'ed, all high bits will be set to 1. Converting u8 0xff to i64 0xff leads to correct conversion (0x000000...ff) This fixes validation for transactions like https://ergexplorer.com/transactions#7f838c5060185125cb554323401d87c0fc135a0c5fc545f9e315ffc38b97b06e

The current indexOf implementation doesn't match the behavior of the one in sigmastate-interpreter. The previous version wrongly treated the second argument as a fallback index whereas it is actually the position in the array to start searching from

Some mainnet transactions like 9f4d40607a561c14a504c96919362f150fca3337ab43219d9a683f1617b74b0a have proofs that are smaller than usual but still valid. Relevant sigmastate-interpreter commit ergoplatform/sigmastate-interpreter@ef6ff08

In v5.0 onwards flatMap can have any arbitrary lambda in body as long as it returns a Coll[T]

This brings down TransactionContext creation for transactions with 32767 inputs down from ~1.2 seconds to ~160 milliseconds

This reduces validation time for some transactions like 34fecabe033ca5b516ddaf99a251baf1a580db5be1ccd7ca41bfcd953bb1923f from 11 seconds down to ~1.2

SethDusek force-pushed the txvalidation branch 2 times, most recently from 678472f to a4e5a92 Compare February 5, 2024 01:06

kushti added the bounty - 500 SigUSD label Feb 14, 2024

SethDusek force-pushed the txvalidation branch from 9f1d83e to 8614bab Compare February 23, 2024 11:29

SethDusek force-pushed the txvalidation branch 6 times, most recently from 8818810 to fdedb60 Compare March 16, 2024 04:46

SethDusek marked this pull request as ready for review March 18, 2024 11:14

SethDusek requested a review from ross-weir March 18, 2024 11:29

SethDusek commented Mar 18, 2024

View reviewed changes

SethDusek mentioned this pull request Apr 16, 2024

Criterion Benchmarks #739

Open

SethDusek assigned SethDusek and unassigned SethDusek Apr 25, 2024

kushti approved these changes Apr 26, 2024

View reviewed changes

ergo-lib/src/chain/transaction/ergo_transaction.rs Outdated Show resolved Hide resolved

SethDusek added 10 commits May 5, 2024 00:55

Use i16::MAX instead of u16::MAX for maximum inputs/outputs in a tran…

f224a72

…saction Scala's Short.MaxValue is equivalent to the maximum a 16 bit signed integer can hold

Add stateless validation for transactions

861b3e8

Add ArbBoxParameters for ErgoBox/ErgoBoxCandidate proptests

996d293

Add Parameters to ErgoStateContext

9c9d326

Stateful Transaction Validation

abac93a

Add documentation for Parameters/TxValidationError

17c6a1e

Add monotonic height validation

ece67fe

Add Parameters to WASM/C bindings

f345e7f

Ignore miette set_hook errors

2d06ca2

set_hook can only be called once, this means displaying an error multiple times causes a panic in sigma-rust. ignoring seems like the best solution

SethDusek added 13 commits May 5, 2024 00:55

Fix indexOf implementation

18c1021

The current indexOf implementation doesn't match the behavior of the one in sigmastate-interpreter. The previous version wrongly treated the second argument as a fallback index whereas it is actually the position in the array to start searching from

Add additional byteArrayToLong test

32f595e

Add CONTEXT.minerPubkey method

60fd0c7

Relax flatMap restrictions + fix inference bugs

ffb8989

In v5.0 onwards flatMap can have any arbitrary lambda in body as long as it returns a Coll[T]

Handle negative from/replaced in Coll.patch

fc88669

perf: Fix quadratic TransactionContext::new creation

b6d7dca

This brings down TransactionContext creation for transactions with 32767 inputs down from ~1.2 seconds to ~160 milliseconds

Add Storage rent support in transaction validation

4e5e393

perf: re-use bytes_to_sign() in verify_tx_input_proof

5b15f57

This reduces validation time for some transactions like 34fecabe033ca5b516ddaf99a251baf1a580db5be1ccd7ca41bfcd953bb1923f from 11 seconds down to ~1.2

Add TX validation bindings

392f2ac

support atLeast(1, ...)

66b15c1

Fix doc comments + merge TxValidationError/TxVerifyError

3f98116

Fix flakey transaction validation tests

517f683

SethDusek force-pushed the txvalidation branch from df822c6 to 517f683 Compare May 4, 2024 19:57

Clippy + update documentation for TxValidationError::BoxSizeExceeded

0fdf2d2

SethDusek force-pushed the txvalidation branch from 127133d to 0fdf2d2 Compare May 4, 2024 20:51

SethDusek linked an issue May 7, 2024 that may be closed by this pull request

Tx verification #703

Closed

SethDusek merged commit fe5de6d into ergoplatform:develop May 7, 2024
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transaction Validation #736

Transaction Validation #736

SethDusek commented Dec 28, 2023 •

edited

Loading

coveralls commented Dec 29, 2023 •

edited

Loading

SethDusek commented Mar 18, 2024 •

edited

Loading

SethDusek Mar 18, 2024

kushti commented Apr 26, 2024

Transaction Validation #736

Transaction Validation #736

Conversation

SethDusek commented Dec 28, 2023 • edited Loading

coveralls commented Dec 29, 2023 • edited Loading

Pull Request Test Coverage Report for Build 8953175335

Details

💛 - Coveralls

SethDusek commented Mar 18, 2024 • edited Loading

SethDusek Mar 18, 2024

Choose a reason for hiding this comment

kushti commented Apr 26, 2024

SethDusek commented Dec 28, 2023 •

edited

Loading

coveralls commented Dec 29, 2023 •

edited

Loading

SethDusek commented Mar 18, 2024 •

edited

Loading