send duplicate shred proofs for conflicting shred scenarios by AshwinSekar · Pull Request #32965 · solana-labs/solana

AshwinSekar · 2023-08-23T23:18:17Z

These are multiple last_shred_in_slot shreds and
coding shreds with conflicting erasure metas.

Problem

These scenarios are detected and stored in blockstore, but never sent to gossip or the state machine.

Summary of Changes

Fix the plumbing so these are sent to gossip and the state machine.

Split from #32862

codecov · 2023-08-24T03:14:01Z

Codecov Report

Merging #32965 (3812210) into master (79c02c8) will increase coverage by 0.0%.
Report is 8 commits behind head on master.
The diff coverage is 89.9%.

❗ Current head 3812210 differs from pull request most recent head 5907204. Consider uploading reports for the commit 5907204 to get more accurate results

@@           Coverage Diff            @@
##           master   #32965    +/-   ##
========================================
  Coverage    81.9%    82.0%            
========================================
  Files         785      784     -1     
  Lines      213295   212931   -364     
========================================
- Hits       174794   174704    -90     
+ Misses      38501    38227   -274

bw-solana

LGTM

AshwinSekar · 2023-08-25T18:31:47Z

Will wait for behzad before merging.

behzadnouri · 2023-08-28T14:54:32Z

+    if shred1.index() != shred2.index() && shred1.shred_type() == ShredType::Code {
+        if shred1.fec_set_index() != shred2.fec_set_index() {
+            return Err(Error::InvalidErasureMetaConflict);
+        }
+        let erasure_meta =
+            ErasureMeta::from_coding_shred(shred1).expect("Shred1 should be a coding shred");
+        if erasure_meta.check_coding_shred(shred2) {
+            return Err(Error::InvalidErasureMetaConflict);


This is not technically exhaustive.
Any 2 shreds with different but overlapping erasure sets can be considered duplicate.
In particular doesn't even need to have the same fec_set_index.

Technically the overlapping erasure sets might result in non-duplicate shreds, but a leader shouldn't generate overlapping erasure sets unless malicious or buggy.

We don't currently identify those cases because we just lookup by the erasure set:
https://github.com/solana-labs/solana/blob/5de27d9a0/ledger/src/blockstore.rs#L1197-L1206
but I think that has to be fixed at some point.

Good to know, right now i've only replicated the cases that we catch in blockstore.
I created #33037, and will extend the checks to catch these types of conflicts in a follow up pr.

Can you put a comment reference to #33037 here?

behzadnouri · 2023-08-28T15:11:49Z

+    // Shred index of the first shred in the proof
+    pub(crate) shred_index: u32,


Is there a use to this any more?
Can we just change this to _unused?

behzadnouri · 2023-08-28T15:12:21Z

+    #[error("invalid last index conflict")]
+    InvalidLastIndexConflict,
+    #[error("invalid erasure meta conflict")]
+    InvalidErasureMetaConflict,


Can you keep this enum sorted?

behzadnouri · 2023-08-28T15:13:25Z

+            (ix1, true, ix2, false) if ix1 > ix2 => return Err(Error::InvalidLastIndexConflict),
+            (ix1, false, ix2, true) if ix1 < ix2 => return Err(Error::InvalidLastIndexConflict),


What if both are last_in_slot but ix1 != ix2?

It will get caught by the _ => () case as this is a valid last index conflict proof.

This part (and now the function as a whole) has become too confusing.
We want to check if the shreds indeed indicate a duplicate slot.
So we need to check that the shreds have the same slot and type and then either of these conditions hold:

same index, different payload.

one has last_in_slot but the other has higher index.

the erasure configs are conflicting.

It is more readable to check if these conditions hold than check if one of them does not hold.

behzadnouri · 2023-08-29T13:38:10Z

+
+    #[allow(dead_code)]
    shred_index: u32,
+


Just rename this to _unused.
You will need to update frozen-abi digest in a couple of places but that is fine.
Also please don't add unnecessary blank lines:
https://google.github.io/styleguide/cppguide.html#Vertical_Whitespace

behzadnouri · 2023-08-29T13:42:29Z

+        return Err(Error::ShredTypeMismatch);
+    }
+
+    if shred1.index() == shred2.index() && shred1.payload() == shred2.payload() {


If shred1.index() != shred2.index() then the payload cannot be the same.
The index is encoded in the payload.
So the first check is unnecessary.

behzadnouri · 2023-08-29T13:44:40Z

+    if shred1.index() != shred2.index() && shred1.shred_type() == ShredType::Code {
+        if shred1.fec_set_index() != shred2.fec_set_index() {
+            return Err(Error::InvalidErasureMetaConflict);
+        }
+        let erasure_meta =
+            ErasureMeta::from_coding_shred(shred1).expect("Shred1 should be a coding shred");
+        if erasure_meta.check_coding_shred(shred2) {
+            return Err(Error::InvalidErasureMetaConflict);


Can you put a comment reference to #33037 here?

behzadnouri · 2023-08-29T13:55:19Z

+            (ix1, true, ix2, false) if ix1 > ix2 => return Err(Error::InvalidLastIndexConflict),
+            (ix1, false, ix2, true) if ix1 < ix2 => return Err(Error::InvalidLastIndexConflict),


This part (and now the function as a whole) has become too confusing.
We want to check if the shreds indeed indicate a duplicate slot.
So we need to check that the shreds have the same slot and type and then either of these conditions hold:

same index, different payload.

one has last_in_slot but the other has higher index.

the erasure configs are conflicting.

It is more readable to check if these conditions hold than check if one of them does not hold.

behzadnouri · 2023-08-29T14:02:22Z

        ));
        assert!(blockstore.has_duplicate_shreds_in_slot(0));
+        assert_eq!(duplicate_shreds.len(), 1);
+        matches!(


assert_matches

matches by itself does not do any assertions here.

behzadnouri · 2023-08-29T14:02:54Z

        ));
+
+        assert_eq!(duplicate_shreds.len(), 1);
+        matches!(


behzadnouri · 2023-08-29T14:06:35Z

    // Returns true if the erasure fields on the shred
    // are consistent with the erasure-meta.
-    pub(crate) fn check_coding_shred(&self, shred: &Shred) -> bool {
+    pub fn check_coding_shred(&self, shred: &Shred) -> bool {


To minimize the public api surface, instead of making these 2 functions public, can you add a method which takes two shreds and returns false if their erasure configs mismatch.
That function can call these 2 functions here internally.

behzadnouri · 2023-08-29T14:10:57Z

+                if !blockstore.has_duplicate_shreds_in_slot(shred_slot) {
+                    if let Some(existing_shred_payload) = blockstore.is_shred_duplicate(&shred) {
+                        blockstore.store_duplicate_slot(
+                            shred_slot,
+                            existing_shred_payload.clone(),
+                            shred.clone().into_payload(),
+                        )?;
+                        (shred, existing_shred_payload)
+                    } else {
+                        // Shred is not duplicate
+                        return Ok(());
+                    }
+                } else {
+                    // Shred has already been handled
+                    return Ok(());
+                }


This would be more readable if you avoid nested branches:

if blockstore.has_duplicate_shreds_in_slot(shred_slot) { return Ok(()); // A duplicate is already recorded. } let Some(existing_shred_payload) = blockstore.is_shred_duplicate(&shred) else { return Ok(()); // Not a duplicate. } blockstore.store_duplicate_slot( shred_slot, existing_shred_payload.clone(), shred.clone().into_payload(), )?; (shred, existing_shred_payload)

The scenarios are multiple last_shred_in_slot shreds and coding shreds with conflicting erasure metas.

behzadnouri

lgtm, aside from that extraneous #[allow(dead_code)].

behzadnouri · 2023-08-31T12:36:25Z

    pub(crate) wallclock: u64,
    pub(crate) slot: Slot,
-    shred_index: u32,
+    #[allow(dead_code)]


When you start the field name with underscore, don't need #[allow(dead_code)] anymore.

thanks for the review! I've removed the #[allow(dead_code)]

mergify · 2023-08-31T18:22:53Z

automerge label removed due to a CI failure

AshwinSekar force-pushed the duplicate-shred-proof-last-slot-erasure-meta branch from 5c90b2c to 3812210 Compare August 24, 2023 00:33

AshwinSekar requested review from bw-solana and carllin August 24, 2023 04:01

bw-solana reviewed Aug 25, 2023

View reviewed changes

Comment thread gossip/src/duplicate_shred.rs

bw-solana reviewed Aug 25, 2023

View reviewed changes

Comment thread gossip/src/duplicate_shred_handler.rs Outdated

bw-solana reviewed Aug 25, 2023

View reviewed changes

Comment thread ledger/src/blockstore.rs

AshwinSekar requested a review from behzadnouri August 25, 2023 18:22

bw-solana previously approved these changes Aug 25, 2023

View reviewed changes

behzadnouri reviewed Aug 28, 2023

View reviewed changes

AshwinSekar dismissed bw-solana’s stale review via cfa2d69 August 28, 2023 21:18

AshwinSekar force-pushed the duplicate-shred-proof-last-slot-erasure-meta branch 2 times, most recently from cfa2d69 to f45864c Compare August 28, 2023 22:05

AshwinSekar requested a review from behzadnouri August 28, 2023 23:52

behzadnouri reviewed Aug 29, 2023

View reviewed changes

AshwinSekar added 3 commits August 30, 2023 18:20

send duplicate shred proofs for conflicting shred scenarios

1192973

The scenarios are multiple last_shred_in_slot shreds and coding shreds with conflicting erasure metas.

Pr feedback: deprecate shred_index for gossip duplicate shred proofs

4059787

Pr feedback

8ee8631

AshwinSekar force-pushed the duplicate-shred-proof-last-slot-erasure-meta branch from f45864c to 8ee8631 Compare August 30, 2023 18:54

AshwinSekar requested a review from behzadnouri August 30, 2023 20:04

behzadnouri previously approved these changes Aug 31, 2023

View reviewed changes

Remove extraneous dead code

5907204

AshwinSekar dismissed behzadnouri’s stale review via 5907204 August 31, 2023 17:46

behzadnouri approved these changes Aug 31, 2023

View reviewed changes

AshwinSekar added the automerge Merge this Pull Request automatically once CI passes label Aug 31, 2023

mergify Bot removed the automerge Merge this Pull Request automatically once CI passes label Aug 31, 2023

AshwinSekar merged commit 6d2dae6 into solana-labs:master Aug 31, 2023

AshwinSekar self-assigned this Nov 29, 2023

AshwinSekar mentioned this pull request Dec 7, 2023

Add feature flag for LastIndex and Erasure duplicate proofs #34360

Merged

		// Shred index of the first shred in the proof
		pub(crate) shred_index: u32,

		(ix1, true, ix2, false) if ix1 > ix2 => return Err(Error::InvalidLastIndexConflict),
		(ix1, false, ix2, true) if ix1 < ix2 => return Err(Error::InvalidLastIndexConflict),

Conversation

AshwinSekar commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Summary of Changes

Uh oh!

codecov Bot commented Aug 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bw-solana left a comment

Choose a reason for hiding this comment

Uh oh!

AshwinSekar commented Aug 25, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

behzadnouri Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

behzadnouri Aug 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

behzadnouri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AshwinSekar commented Aug 23, 2023 •

edited

Loading

codecov Bot commented Aug 24, 2023 •

edited

Loading

behzadnouri Aug 29, 2023 •

edited

Loading

behzadnouri Aug 29, 2023 •

edited

Loading