Skip to content

Drain rabbitmq consumers slowly from Cannon#4342

Merged
elland merged 6 commits intodevelopfrom
WPB-11458/rabbit-drainage
Nov 19, 2024
Merged

Drain rabbitmq consumers slowly from Cannon#4342
elland merged 6 commits intodevelopfrom
WPB-11458/rabbit-drainage

Conversation

@elland
Copy link
Contributor

@elland elland commented Nov 18, 2024

https://wearezeta.atlassian.net/browse/WPB-11458

In case Cannon gets terminated, we want to drain the consumers, like we do with websocket queues.

Checklist

  • Add a new entry in an appropriate subdirectory of changelog.d
  • Read and follow the PR guidelines

@echoes-hq echoes-hq bot added the echoes/initiative: scale Enterprise Readiness Initiatives label Nov 18, 2024
@elland elland force-pushed the WPB-11458/rabbit-drainage branch from 2f95a06 to 9d7abe5 Compare November 18, 2024 15:02
@zebot zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Nov 18, 2024
Comment on lines +110 to +112
mkEnv external o cs l d conns p g t rabbitmqOpts =
Env o l d conns (RequestId defRequestId) $
WS.env external (o ^. cannon . port) (encodeUtf8 $ o ^. gundeck . host) (o ^. gundeck . port) l p d conns g t (o ^. drainOpts) rabbitmqOpts cs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's time to start using a record as a replacement to all these arguments to mkEnv. The same goes with arguments to WS.env.

Comment on lines +61 to +62
Q.closeConnection conn
void $ D.remove key e.rabbitConnections
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if Cannon crashes in between these two?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it completely crashes, then there's no issue, it's not like the in-memory data will be preserved. Since this is bookkeeping for the graceful termination. If it starts graceful termination between these two steps, the worst case is trying to close a connection twice, which should be a no-op.

@elland elland merged commit 12f3687 into develop Nov 19, 2024
@elland elland deleted the WPB-11458/rabbit-drainage branch November 19, 2024 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

echoes/initiative: scale Enterprise Readiness Initiatives ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants