feat: truncate SHA hashes inside circuits

[MirandaWood]

Will close #2019

This PR converts SHA hashing inside noir circuits from outputting 2 128-bit fields to outputting 1 248-bit field. To fit inside the field, we truncate one byte.

---

Noir Changes

The constant NUM_FIELDS_PER_SHA256 is now 1, so any hardcoded test values and function returns have been changed to use an array of one. I've kept it as an array rather than a single Fr to minimise changes across the repo and ensure if we want to revert NUM_FIELDS_PER_SHA256 in future, it won't be so painful. However, we can also just use a single Fr if that's preferable.

TX_EFFECTS_HASH_LOG_FIELDS

Methods:

• field_from_bytes_32_trunc: Converts a 32 byte array to a 31 byte field element (useful for comparisons with new sha256_to_field), tests in types/src/utils/field.nr.
• sha256_to_field: Uses the same method as the previous version to convert the sha result (BE) bytes array to field, but leaves out the final byte.
• accumulate_sha256: Used almost exclusively for enc/unenc logs hashing - takes in 2 31 byte field elements, assumed to be outputs of a previous sha hash, pads to 32 bytes and hashes them with sha256_to_field as a 64 byte array. Note that as before, other circuits that use sha (like tx effects hash and messages hash) do not use this method and instead create a flat byte array, then call sha256_to_field.

---

L1 Contract Changes

To match the Noir method, the sha256ToField function now truncates a byte and prepends a blank byte. Not prepending the blank byte means changing many struct fields from bytes32 to bytes31. This (IIRC) is the same gas cost and creates more awkward encoding, so I kept the length with a blank byte. This also changes the slither file, as I removed some of the old encoding which flagged with new encoding... which also flags.

Only the 'leaves' used in computing the txsHash in TxsDecoder and logs hashes have been changed to 31 bytes to match the Noir SHA accumulation (since we are repeating hashes of hashes).

The TS code (see below) does pack the Header struct with 31 bytes per SHA, so we must shift the decoding in HeaderLib` by 3 bytes.

As of 21.3, there have been a lot of changes in master to the way the txs effect hash (formerly calldata hash/txs hash) is calculated. Plus, now we actually recalculate the in/outHash (i.e. the root of the sha tree of messages) in the contract, so I have reverted to using 32 bytes everywhere with a prepended blank byte.

---

TS Changes

All .hash() methods which are also computed in the circuit have been changed to match the Noir code. In most places this just means truncating a byte with .subarray(0, 31) on the buffer.
The ContentCommitment serialise/deserialise methods have been modified, as keeping NUM_BYTES_PER_SHA256 = 32 caused a lot of issues in the background. Changing it to 31 to match Noir does mean slightly different encoding, but many fewer changes across the repo (and hopefully less confusion).
As of 21.3, due to changes in master, it's now cleaner to keep NUM_BYTES_PER_SHA256 = 32 and be sure to truncate and pad all SHA hashes which touch the Noir circuits.
Since I've kept the hash output as an array of one in Noir, there are many tuples of one in ts (for the above reasoning) - this can be changed if preferable.

Methods:

• toTruncField: Mirrors Noir's field_from_bytes_32_trunc to convert to a field element - used in place of old method to2Fields (tested in free_funcs.test.ts).
• fromTruncField: Converts the above back to a 31 byte buffer (tested as above).

---

[github-actions]

Changes to circuit sizes

Generated at commit: 8841a4b1f2ec172931a26f13038dce6754fd35bd, compared to commit: 0f09b63dc40969e9c5ac810faad1422abb40f586

🧾 Summary (100% most significant diffs)

Program	ACIR opcodes (+/-)	%	Circuit size (+/-)	%
parity_base	-728 ✅	-60.26%	0 ➖	0.00%
rollup_merge	-160 ✅	-30.89%	0 ➖	0.00%
rollup_root	-160 ✅	-14.65%	0 ➖	0.00%
parity_root	-240 ✅	-33.33%	0 ➖	0.00%
private_kernel_tail	-5 ✅	-0.00%	-10 ✅	-0.00%
public_kernel_tail	-7 ✅	-0.01%	-12 ✅	-0.00%
rollup_base	-56 ✅	-0.03%	-1,112 ✅	-0.06%
private_kernel_init	-115 ✅	-0.24%	-389 ✅	-0.11%
private_kernel_inner	-167 ✅	-0.17%	-2,047 ✅	-0.39%
public_kernel_setup_simulated	-5 ✅	-0.55%	-10 ✅	-0.55%
public_kernel_tail_simulated	-5 ✅	-0.55%	-10 ✅	-0.55%
public_kernel_app_logic_simulated	-5 ✅	-0.55%	-10 ✅	-0.55%
public_kernel_teardown_simulated	-5 ✅	-0.55%	-10 ✅	-0.55%
private_kernel_inner_simulated	-5 ✅	-0.55%	-10 ✅	-0.55%
private_kernel_init_simulated	-5 ✅	-0.55%	-10 ✅	-0.55%
public_kernel_setup	-90 ✅	-0.31%	-1,541 ✅	-0.84%
public_kernel_teardown	-132 ✅	-0.45%	-1,583 ✅	-0.86%
public_kernel_app_logic	-166 ✅	-0.30%	-2,747 ✅	-0.87%
private_kernel_tail_simulated	-5 ✅	-1.10%	-10 ✅	-1.09%
rollup_base_simulated	0 ➖	0.00%	-2 ✅	-5.71%

---

Full diff report 👇

Program	ACIR opcodes (+/-)	%	Circuit size (+/-)	%
parity_base	480 (-728)	-60.26%	79,811 (0)	0.00%
rollup_merge	358 (-160)	-30.89%	45,970 (0)	0.00%
rollup_root	932 (-160)	-14.65%	77,898 (0)	0.00%
parity_root	480 (-240)	-33.33%	79,811 (0)	0.00%
private_kernel_tail	373,472 (-5)	-0.00%	839,573 (-10)	-0.00%
public_kernel_tail	124,962 (-7)	-0.01%	358,953 (-12)	-0.00%
rollup_base	171,459 (-56)	-0.03%	1,730,700 (-1,112)	-0.06%
private_kernel_init	48,592 (-115)	-0.24%	348,602 (-389)	-0.11%
private_kernel_inner	95,831 (-167)	-0.17%	519,398 (-2,047)	-0.39%
public_kernel_setup_simulated	903 (-5)	-0.55%	1,809 (-10)	-0.55%
public_kernel_tail_simulated	903 (-5)	-0.55%	1,809 (-10)	-0.55%
public_kernel_app_logic_simulated	903 (-5)	-0.55%	1,809 (-10)	-0.55%
public_kernel_teardown_simulated	903 (-5)	-0.55%	1,809 (-10)	-0.55%
private_kernel_inner_simulated	902 (-5)	-0.55%	1,807 (-10)	-0.55%
private_kernel_init_simulated	902 (-5)	-0.55%	1,807 (-10)	-0.55%
public_kernel_setup	29,330 (-90)	-0.31%	182,766 (-1,541)	-0.84%
public_kernel_teardown	29,397 (-132)	-0.45%	182,834 (-1,583)	-0.86%
public_kernel_app_logic	56,052 (-166)	-0.30%	314,134 (-2,747)	-0.87%
private_kernel_tail_simulated	451 (-5)	-1.10%	905 (-10)	-1.09%
rollup_base_simulated	1 (0)	0.00%	33 (-2)	-5.71%

[AztecBot]

Benchmark results

Metrics with a significant change:

• note_history_successful_decrypting_time_in_ms (10): 3,087 (+25%)

Detailed results

All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.

This benchmark source data is available in JSON format on S3 here.

Values are compared against data from master at commit 0f09b63d and shown if the difference exceeds 1%.

L2 block published to L1

Each column represents the number of txs on an L2 block published to L1.

Metric	8 txs	32 txs	64 txs
l1_rollup_calldata_size_in_bytes	676	676	676
l1_rollup_calldata_gas	6,400	6,400 (-1%)	6,424
l1_rollup_execution_gas	585,733 (+1%)	585,733 (+1%)	585,757 (+1%)
l2_block_processing_time_in_ms	1,444 (+10%)	4,697 (-3%)	8,912 (-2%)
note_successful_decrypting_time_in_ms	173 (-11%)	548 (-2%)	1,007 (-3%)
note_trial_decrypting_time_in_ms	78.6 (-24%)	65.2 (-18%)	37.7 (-32%)
l2_block_building_time_in_ms	17,790	65,352 (-3%)	128,552 (-4%)
l2_block_rollup_simulation_time_in_ms	8,486 (+5%)	28,467 (-1%)	55,439 (-2%)
l2_block_public_tx_process_time_in_ms	9,284 (-5%)	36,829 (-5%)	73,021 (-5%)

L2 chain processing

Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric	5 blocks	10 blocks
node_history_sync_time_in_ms	13,997 (-2%)	27,454 (+2%)
note_history_successful_decrypting_time_in_ms	1,280 (+2%)	⚠️ 3,087 (+25%)
note_history_trial_decrypting_time_in_ms	110 (+13%)	173 (-10%)
node_database_size_in_bytes	18,653,264	35,082,320
pxe_database_size_in_bytes	29,859	59,414

Circuits stats

Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit	circuit_simulation_time_in_ms	circuit_input_size_in_bytes	circuit_output_size_in_bytes
private-kernel-init	240 (-15%)	44,302	28,181
private-kernel-ordering	213 (-2%)	52,805	14,263
base-parity	1,852 (+1%)	128	311
base-rollup	712 (-1%)	165,724	861 (-7%)
root-parity	1,624 (+1%)	1,244	311
root-rollup	50.2 (-4%)	4,359 (-3%)	725
private-kernel-inner	593 (-10%)	73,644	28,181
public-kernel-app-logic	398 (-8%)	35,164	28,151
public-kernel-tail	172 (-2%)	40,862	28,151
merge-rollup	7.13 (-17%)	2,568 (-5%)	861 (-7%)

Tree insertion stats

The duration to insert a fixed batch of leaves into each tree type.

Metric	1 leaves	16 leaves	64 leaves	128 leaves	512 leaves	1024 leaves	2048 leaves	4096 leaves	32 leaves
batch_insert_into_append_only_tree_16_depth_ms	10.0 (-1%)	16.1	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_count	16.8	31.6	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_16_depth_hash_ms	0.584 (-1%)	0.496	N/A	N/A	N/A	N/A	N/A	N/A	N/A
batch_insert_into_append_only_tree_32_depth_ms	N/A	N/A	45.8 (-4%)	73.1 (+1%)	232	447 (-2%)	875 (+1%)	1,725	N/A
batch_insert_into_append_only_tree_32_depth_hash_count	N/A	N/A	96.0	159	543	1,055	2,079	4,127	N/A
batch_insert_into_append_only_tree_32_depth_hash_ms	N/A	N/A	0.470 (-4%)	0.451 (+1%)	0.422	0.418 (-2%)	0.415 (+1%)	0.413	N/A
batch_insert_into_indexed_tree_20_depth_ms	N/A	N/A	53.7 (-4%)	106 (-1%)	336 (-1%)	670	1,309 (-1%)	2,617	N/A
batch_insert_into_indexed_tree_20_depth_hash_count	N/A	N/A	104	207	691	1,363	2,707	5,395	N/A
batch_insert_into_indexed_tree_20_depth_hash_ms	N/A	N/A	0.478 (-4%)	0.479	0.457 (-1%)	0.460	0.454	0.453	N/A
batch_insert_into_indexed_tree_40_depth_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	61.2 (-3%)
batch_insert_into_indexed_tree_40_depth_hash_count	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	109
batch_insert_into_indexed_tree_40_depth_hash_ms	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	0.534 (-3%)

Miscellaneous

Transaction sizes based on how many contract classes are registered in the tx.

Metric	0 registered classes
tx_size_in_bytes	21,949

Transaction processing duration by data writes.

Metric	0 new note hashes	1 new note hashes
tx_pxe_processing_time_ms	3,087 (-3%)	1,648 (-7%)

Metric	0 public data writes	1 public data writes
tx_sequencer_processing_time_ms	11.1 (-17%)	1,144 (-7%)

[LeilaWang]

Just some small things and questions. Looks good overall 😃

[LeilaWang]

Sounds good!

[MirandaWood]

I had a go at this, but found the gas cost would increase quite a lot in replicating for calculating the inHash. Basically, l1toL2Msgs are stored in bytes32 in solidity so replicating the accumulate nr hash requires truncating each one manually over a loop, then concatenating and hashing.
Unfortunately, to avoid casting issues, I've already changed other hashes (like the txEffectHash) to be bytes31 so this creates a discrepancy. Perhaps in another PR I have two choices:

• force all sha hashes that are touched in nr to be bytes31 (requires a lot of changes in ts/sol and possibly higher gas)
• revert the hashes that I have cast to bytes31, and prepend/drop a byte where required in solidity (probably messier code)

I think both are valid choices, so would appreciate a pointer!

[MirandaWood]

We have decided to go for 32 bytes everywhere, however we would require two generics in accumulate_sha256 - one for the number of input fields and one for 32 * that number. Unfortunately Noir doesn't realise 32*N can be known at compile time, so I would need another generic or some constant, which wouldn't make sense.
I did notice though that this function (compute_messages_hash) now appears to be unused, so perhaps this is not a problem anymore?

[LeilaWang]

Can probably drop the as [Fr]s?

[MirandaWood]

Doable but as it will touch a lot of files, I will do this in a future PR if that's ok!

[MirandaWood]

In integrating the truncated SHA into this PR, I discovered a discrepancy in byte casting between Solidity and Noir/ts.
e.g. say normal sha256 gives a result of 0x2e7ff14389eef3dc51597529149e01b49cb33829f7089438c8c145c8f352c17b.
In Solidity the cleanest way it seems to drop a byte is bytes31(thing). Turning it back into a bytes32 for checks and structs gives:
0x2e7ff14389eef3dc51597529149e01b49cb33829f7089438c8c145c8f352c100.

However, both toBuffer() in ts and to_be_bytes() in Noir would give:
0x002e7ff14389eef3dc51597529149e01b49cb33829f7089438c8c145c8f352c1

The solidity version fails in-field checks since it still fills the most significant bytes, so I went for the Noir/TS version wherever checks are needed. That's why in some places sha256ToField32 is used, and in others sha256ToField (which returns bytes31).
I also changed logs hashes to be treated as bytes31 wherever possible to avoid any issues with casting. The above fixes are a bit janky so I'm happy for any feedback on them.