Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic runtime metering #626

Merged
merged 5 commits into from
Jun 16, 2022
Merged

Basic runtime metering #626

merged 5 commits into from
Jun 16, 2022

Conversation

gumb0
Copy link
Collaborator

@gumb0 gumb0 commented Oct 27, 2020

Method 1 of #621

@gumb0 gumb0 force-pushed the runtime-metering branch 3 times, most recently from 4e0e6be to 340309f Compare October 27, 2020 14:17
@codecov
Copy link

codecov bot commented Oct 27, 2020

Codecov Report

Merging #626 (642a3d5) into master (91bab22) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #626   +/-   ##
=======================================
  Coverage   99.27%   99.27%           
=======================================
  Files          88       88           
  Lines       13158    13184   +26     
=======================================
+ Hits        13062    13088   +26     
  Misses         96       96           
Flag Coverage Δ
rust 98.47% <ø> (ø)
spectests 89.92% <82.35%> (-0.08%) ⬇️
unittests 99.22% <100.00%> (+<0.01%) ⬆️
unittests-32 99.31% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
lib/fizzy/execute.hpp 100.00% <ø> (ø)
lib/fizzy/execution_context.hpp 100.00% <ø> (ø)
lib/fizzy/execute.cpp 99.30% <100.00%> (+<0.01%) ⬆️
lib/fizzy/instructions.cpp 100.00% <100.00%> (ø)
test/unittests/execute_test.cpp 100.00% <100.00%> (ø)

lib/fizzy/execute.cpp Outdated Show resolved Hide resolved
lib/fizzy/execute.cpp Outdated Show resolved Hide resolved
@gumb0 gumb0 force-pushed the runtime-metering branch 7 times, most recently from 8480da1 to ba9f744 Compare December 9, 2020 13:10
@gumb0
Copy link
Collaborator Author

gumb0 commented Dec 15, 2020

GCC 10.2 with LTO

fizzy/execute/blake2b/512_bytes_rounds_1_mean                     +0.2008         +0.2007           164           196           164           196
fizzy/execute/blake2b/512_bytes_rounds_16_mean                    +0.2174         +0.2173          2440          2971          2441          2971
fizzy/execute/ecpairing/onepoint_mean                             +0.1772         +0.1770        828851        975707        828881        975626
fizzy/execute/keccak256/512_bytes_rounds_1_mean                   +0.3596         +0.3588           173           235           173           235
fizzy/execute/keccak256/512_bytes_rounds_16_mean                  +0.3778         +0.3771          2498          3442          2498          3440
fizzy/execute/memset/256_bytes_mean                               +0.2451         +0.2445            14            17            14            17
fizzy/execute/memset/60000_bytes_mean                             +0.2421         +0.2416          2977          3697          2977          3696
fizzy/execute/mul256_opt0/input1_mean                             +0.2435         +0.2429            50            62            50            62
fizzy/execute/ramanujan_pi/33_runs_mean                           +0.2260         +0.2255           239           293           239           292
fizzy/execute/sha1/512_bytes_rounds_1_mean                        +0.2171         +0.2167           178           217           178           217
fizzy/execute/sha1/512_bytes_rounds_16_mean                       +0.2168         +0.2164          2483          3022          2483          3021
fizzy/execute/sha256/512_bytes_rounds_1_mean                      +0.3375         +0.3371           167           224           167           224
fizzy/execute/sha256/512_bytes_rounds_16_mean                     +0.3479         +0.3474          2285          3080          2285          3079
fizzy/execute/taylor_pi/pi_1000000_runs_mean                      +0.1714         +0.1710         77758         91086         77758         91057
fizzy/execute/micro/eli_interpreter/exec105_mean                  +0.1075         +0.1072            10            11            10            11
fizzy/execute/micro/factorial/20_mean                             +0.0525         +0.0522             1             1             1             1
fizzy/execute/micro/fibonacci/24_mean                             +0.1432         +0.1429         10754         12294         10754         12291
fizzy/execute/micro/host_adler32/1_mean                           +0.1755         +0.1752             0             0             0             0
fizzy/execute/micro/host_adler32/1000_mean                        +0.2317         +0.2314            61            75            61            75
fizzy/execute/micro/icall_hash/1000_steps_mean                    +0.0674         +0.0671           144           153           144           153
fizzy/execute/micro/spinner/1_mean                                +0.1697         +0.1694             0             0             0             0
fizzy/execute/micro/spinner/1000_mean                             +0.1243         +0.1241            20            23            20            23

Clang 11.0.1 with LTO

fizzy/execute/blake2b/512_bytes_rounds_1_mean                     -0.0668         -0.0648           194           181           193           181
fizzy/execute/blake2b/512_bytes_rounds_16_mean                    -0.0555         -0.0537          2923          2761          2918          2761
fizzy/execute/ecpairing/onepoint_mean                             -0.0234         -0.0228        948203        926024        947633        926073
fizzy/execute/keccak256/512_bytes_rounds_1_mean                   -0.1415         -0.1413           259           222           259           222
fizzy/execute/keccak256/512_bytes_rounds_16_mean                  -0.1421         -0.1421          3814          3272          3814          3272
fizzy/execute/memset/256_bytes_mean                               -0.0568         -0.0568            16            15            16            15
fizzy/execute/memset/60000_bytes_mean                             -0.0626         -0.0626          3563          3340          3563          3340
fizzy/execute/mul256_opt0/input1_mean                             -0.1127         -0.1128            69            61            69            61
fizzy/execute/ramanujan_pi/33_runs_mean                           -0.1308         -0.1308           277           241           277           241
fizzy/execute/sha1/512_bytes_rounds_1_mean                        -0.0379         -0.0379           204           196           204           196
fizzy/execute/sha1/512_bytes_rounds_16_mean                       -0.0202         -0.0203          2838          2781          2838          2781
fizzy/execute/sha256/512_bytes_rounds_1_mean                      -0.0342         -0.0342           183           177           183           177
fizzy/execute/sha256/512_bytes_rounds_16_mean                     -0.0388         -0.0388          2515          2417          2515          2417
fizzy/execute/taylor_pi/pi_1000000_runs_mean                      +0.1043         +0.1043         87643         96786         87644         96786
fizzy/execute/micro/eli_interpreter/exec105_mean                  +0.0487         +0.0486            10            11            10            11
fizzy/execute/micro/factorial/20_mean                             +0.0146         +0.0146             1             1             1             1
fizzy/execute/micro/fibonacci/24_mean                             +0.0260         +0.0259         11354         11649         11354         11648
fizzy/execute/micro/host_adler32/1_mean                           -0.0290         -0.0290             0             0             0             0
fizzy/execute/micro/host_adler32/1000_mean                        -0.2443         -0.2443            95            72            95            72
fizzy/execute/micro/icall_hash/1000_steps_mean                    -0.0612         -0.0613           152           143           152           143
fizzy/execute/micro/spinner/1_mean                                -0.0599         -0.0600             0             0             0             0
fizzy/execute/micro/spinner/1000_mean                             -0.0232         -0.0233            20            19            20            19

@axic
Copy link
Member

axic commented Dec 15, 2020

So that is rather curious that on clang it is faster with metering on 😅

@gumb0 gumb0 force-pushed the runtime-metering branch 2 times, most recently from c7fd3f2 to c333e40 Compare April 7, 2021 13:20
@gumb0 gumb0 force-pushed the runtime-metering branch from c333e40 to d4e9e83 Compare April 15, 2021 10:48
@gumb0 gumb0 force-pushed the runtime-metering branch 2 times, most recently from 75d8408 to ac50361 Compare April 16, 2021 16:34
@gumb0 gumb0 changed the base branch from master to test-cleanup April 16, 2021 16:44
Base automatically changed from test-cleanup to master April 16, 2021 18:22
@gumb0 gumb0 force-pushed the runtime-metering branch from ac50361 to 21393a7 Compare April 19, 2021 10:37
@gumb0 gumb0 requested review from chfast and axic April 19, 2021 10:39
@axic
Copy link
Member

axic commented Apr 19, 2021

I think this looks good code wise and now the change is rather small, so it is easy to keep rebasing it. However we probably need to make our measurements without merging it, so we can easily see the overhead. (Unless it is possible to merge this with a compile-time option enabling/disabling it.)

lib/fizzy/execute.cpp Outdated Show resolved Hide resolved
lib/fizzy/instructions.cpp Show resolved Hide resolved
@gumb0 gumb0 force-pushed the runtime-metering branch from 21393a7 to f008f1f Compare April 20, 2021 09:32
@axic
Copy link
Member

axic commented May 23, 2022

Rebased.

@axic axic marked this pull request as ready for review May 23, 2022 17:40

ExecutionResult execute(
template <bool MeteringEnabled>
ExecutionResult execute_local_function(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe consider specialising invoke as well so that we generate one less branch?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which branch? I think invoke_function does not change with MeteringEnabled... It still needs ExecutionContext to count depth.

Copy link
Member

@axic axic May 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I actually meant execute and not invoke, because it is called by invoke_function.

@gumb0 gumb0 force-pushed the runtime-metering branch 2 times, most recently from c65cc7a to 7095691 Compare May 30, 2022 14:16
@gumb0 gumb0 force-pushed the runtime-metering branch from 7095691 to a26d244 Compare May 30, 2022 14:51
@@ -444,6 +444,222 @@ constexpr uint8_t instruction_max_align_table[256] = {
/* f32_reinterpret_i32 = 0xbe */ 0,
/* f64_reinterpret_i64 = 0xbf */ 0,
};

constexpr int16_t instruction_cost_table[256] = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the way to have a user supplied cost_table?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess probably passing const int16_t* to execute()

@axic axic changed the title Runtime metering prototype Basic runtime metering May 30, 2022
@axic axic requested a review from chfast May 30, 2022 16:51
Copy link
Member

@axic axic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good, benchmarking can decide whether the last commit makes it worse or better.

@chfast
Copy link
Collaborator

chfast commented Jun 14, 2022

fizzy/execute/blake2b/512_bytes_rounds_1_mean                     +0.0015         +0.0016            74            74            74            74
fizzy/execute/blake2b/512_bytes_rounds_16_mean                    -0.0021         -0.0021          1113          1110          1113          1110
fizzy/execute/ecpairing/onepoint_mean                             +0.0041         +0.0041        370547        372062        370547        372063
fizzy/execute/keccak256/512_bytes_rounds_1_mean                   +0.0870         +0.0870            90            98            90            98
fizzy/execute/keccak256/512_bytes_rounds_16_mean                  +0.0983         +0.0983          1312          1440          1312          1440
fizzy/execute/memset/256_bytes_mean                               -0.0036         -0.0036             6             6             6             6
fizzy/execute/memset/60000_bytes_mean                             -0.0038         -0.0038          1336          1331          1336          1331
fizzy/execute/mul256_opt0/input1_mean                             -0.0428         -0.0428            25            24            25            24
fizzy/execute/ramanujan_pi/33_runs_mean                           -0.0214         -0.0214           103           101           103           101
fizzy/execute/sha1/512_bytes_rounds_1_mean                        +0.0081         +0.0081            80            80            80            80
fizzy/execute/sha1/512_bytes_rounds_16_mean                       +0.0087         +0.0087          1109          1119          1109          1119
fizzy/execute/sha256/512_bytes_rounds_1_mean                      +0.0108         +0.0108            73            74            73            74
fizzy/execute/sha256/512_bytes_rounds_16_mean                     +0.0071         +0.0071          1007          1014          1007          1014
fizzy/execute/taylor_pi/pi_1000000_runs_mean                      -0.0022         -0.0022         37890         37807         37890         37807
fizzy/execute/micro/eli_interpreter/exec105_mean                  +0.0062         +0.0062             4             4             4             4
fizzy/execute/micro/factorial/20_mean                             -0.0043         -0.0043             1             1             1             1
fizzy/execute/micro/fibonacci/24_mean                             +0.0154         +0.0154          4980          5057          4980          5057
fizzy/execute/micro/host_adler32/1_mean                           +0.0409         +0.0409             0             0             0             0
fizzy/execute/micro/host_adler32/1000_mean                        -0.0086         -0.0086            28            28            28            28
fizzy/execute/micro/icall_hash/1000_steps_mean                    +0.0622         +0.0622            68            72            68            72
fizzy/execute/micro/spinner/1_mean                                +0.0840         +0.0840             0             0             0             0
fizzy/execute/micro/spinner/1000_mean                             +0.0004         +0.0003             8             8             8             8

Copy link
Member

@axic axic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, please rebase.

@gumb0 gumb0 force-pushed the runtime-metering branch from a26d244 to 642a3d5 Compare June 16, 2022 13:38
@axic axic merged commit 5c5fae0 into master Jun 16, 2022
@axic axic deleted the runtime-metering branch June 16, 2022 15:02
@axic axic mentioned this pull request Nov 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants