Skip to content

chore(sequencer-relayer): add timeout to gRPCs to Celestia app#1191

Merged
Fraser999 merged 3 commits intomainfrom
fraser/timeout-celestia-grpcs
Jun 18, 2024
Merged

chore(sequencer-relayer): add timeout to gRPCs to Celestia app#1191
Fraser999 merged 3 commits intomainfrom
fraser/timeout-celestia-grpcs

Conversation

@Fraser999
Copy link
Contributor

Summary

Added a five second timeout to all gRPCs targeting the Celestia app.

Background

Currently if the Celestia app doesn't respond to any gRPC in a timely manner, the relayer's submission flow is blocked by this. We want to avoid this by not waiting indefinitely for any gRPC.

Changes

  • Added timeout to the gRPC channel. This is applied to all requests sent via that channel. We can move to more fine-grained timeouts (different per request) in the future if required.

Testing

  • Added a unit test. I opted for this approach over black box testing for a couple of reasons: we can leverage tokio's time::advance to avoid the test running for the actual duration of the timeout, and we have a better chance to inspect the actual error returned by the RPC to ensure it's a timeout variant.

Related Issues

Closes #1187.

@Fraser999 Fraser999 requested a review from a team as a code owner June 18, 2024 09:51
@Fraser999 Fraser999 requested a review from noot June 18, 2024 09:51
@github-actions github-actions bot added the sequencer-relayer pertaining to the astria-sequencer-relayer crate label Jun 18, 2024
Copy link
Contributor

@SuperFluffy SuperFluffy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the patch, but I am not a fan of the test (described in a comment on the code).

What would make for a useful test is externally observable behavior, such as a request timing out and a second request being issued. This would test relayer's expected behavior in light of failure.

I don't see a reason to block this PR on these grounds (because adding this blackbox test requires more work on the mock grpc server), but I would be fine if the unit test was removed.

use super::*;

#[tokio::test(start_paused = true)]
async fn get_tx_should_timeout() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At its core, this tests the timeout inside Client::get_tx and thus if the underlying timeout logic of tonic's channel works. This limits the usefulness of this test (because we should always be able to rely on the correct behavior of our building blocks).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, I agree on removing this test, with a view to add black box tests in the near future for sad path cases like gRPC errors, including timeouts.

@Fraser999 Fraser999 enabled auto-merge June 18, 2024 12:23
@Fraser999 Fraser999 added this pull request to the merge queue Jun 18, 2024
Merged via the queue into main with commit f6171b1 Jun 18, 2024
@Fraser999 Fraser999 deleted the fraser/timeout-celestia-grpcs branch June 18, 2024 12:35
steezeburger added a commit that referenced this pull request Jun 19, 2024
* main:
  chore(bridge-withdrawer): add missing errors and clean up names (#1178)
  feat(sequencer): add ttl and invalid cache to app mempool (#1138)
  chore(astria-merkle): add benchmarks (#1179)
  chore(sequencer-relayer): add timeout to gRPCs to Celestia app (#1191)
  refactor(core): parse ics20 denoms as ibc or trace prefixed variants (#1181)
  Mycodecrafting/sequencer seed node (#1188)
  chore: register all metrics during startup (#1144)
  feat(charts): option to purge geth mempool (#1182)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sequencer-relayer pertaining to the astria-sequencer-relayer crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add timeouts to sequencer-relayer RPC calls

2 participants