v3.1: bug: use try_send in handle_forwarded_packets (backport of #10498)

[mergify]

Problem

• FetchStage's handle_forwarded_packets calls send on a channel that is now bounded (streamer/sigverify: use bounded channels between streamers and sigver #9732)
• A call to send will block until there is free capacity

Summary of Changes

• Call try_send and drop packets if the channel is at capacity

Fixes #

---

This is an automatic backport of pull request #10498 done by [Mergify](https://mergify.com).

[codecov-commenter]

Codecov Report

❌ Patch coverage is 64.28571% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.2%. Comparing base (01a77c7) to head (b36d86d).

Additional details and impacted files

@@            Coverage Diff            @@
##             v3.1   #10552     +/-   ##
=========================================
- Coverage    83.2%    83.2%   -0.1%     
=========================================
  Files         865      865             
  Lines      376601   376612     +11     
=========================================
- Hits       313609   313587     -22     
- Misses      62992    63025     +33     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[t-nelson]

this pr needs to land before #10551 to avoid introducing a bug on channel full upon switching to bounded channel

[steviez]

this pr needs to land before #10551 to avoid introducing a bug on channel full upon switching to bounded channel

Agreed

[KirillLykov]

If we assume that users use released version of 3.1 and not raw branch v3.1 then why not it is enough to ensure that both of these changes land to the same release?

[steviez]

If we assume that users use released version of 3.1 and not raw branch v3.1 then why not it is enough to ensure that both of these changes land to the same release?

Generally, yes, I expect that most people will update to the v3.1.9 tag which means they'll pickup both commits. But, we have no way to guarantee this. Canaries pick up the latest commit from their respective branches, and someone running a git bisect could end up running on one of these intermediate commits.

The point is to be deliberate with how we release upgrades and to remove/minimize the chance of introducing a foot-gun. What is we merged 10551 first because "it should be fine, 10552 coming soon" but then 10552 gets held up because it breaks something else. There is nothing to be gained by assuming that people will pick up both commits, and there is no reason to merge them in the "wrong" order if we know that the "wrong" order creates a bug

[KirillLykov]

If we assume that users use released version of 3.1 and not raw branch v3.1 then why not it is enough to ensure that both of these changes land to the same release?

Generally, yes, I expect that most people will update to the v3.1.9 tag which means they'll pickup both commits. But, we have no way to guarantee this. Canaries pick up the latest commit from their respective branches, and someone running a git bisect could end up running on one of these intermediate commits.

The point is to be deliberate with how we release upgrades and to remove/minimize the chance of introducing a foot-gun. What is we merged 10551 first because "it should be fine, 10552 coming soon" but then 10552 gets held up because it breaks something else. There is nothing to be gained by assuming that people will pick up both commits, and there is no reason to merge them in the "wrong" order if we know that the "wrong" order creates a bug

So in theory, if we are again in such situation and we have commit introducing some bug and after that PR fixing it if nothing has been released and changes are minimal, is the best way to revert original PR and introduce a new one with bug fixed?

[steviez]

So in theory, if we are again in such situation and we have commit introducing some bug and after that PR fixing it if nothing has been released and changes are minimal, is the best way to revert original PR and introduce a new one with bug fixed?

Just making sure - you are asking about the master PR's, not the v3.1 PR's right ? If so, then yes, I think that it is typically preferred to back out the original and then reapply with fix. When we find that commits break something very obviously (CI, canaries, etc), we back it out as soon as the bug is discovered

If a couple weeks pass before the bug is discovered, things can get a little ambiguous if the commit doesn't revert cleanly.

[apfitzge]

^ To add onto @steviez's comment - I did this incorrectly. I think your PR could have been cleanly reverted and fixed. Then backported as a whole.