[evanhutnik]

Summary

Adding separate retry queue for gmail inbox update pubsub processing.

Problem

When users perform bulk operations (e.g., archiving 10,000 emails), we have to make separate calls to the Gmail API for each message being updated, which results in us hitting the rate limit quota for the user, in turn causing webhook processing to repeatedly retry. This creates a backlog that delays inbox updates for all users, not just the one performing the bulk operation, as all of these messages were being processed by the same workers.

Solution

Implement a two-tier webhook processing system:

• Primary queue: Processes normal webhook operations. When rate-limited, offloads the operation to the retry queue and moves on.
• Retry queue: Handles rate-limited operations separately with its own worker pool, allowing the primary queue to maintain low latency.

This architecture prevents head-of-line blocking—the primary queue continues processing fresh updates for all users while rate-limited operations are handled independently.

Changes

• Rate-limited operations in the primary worker are enqueued to a new SQS retry queue and marked as processed
• A separate pool of retry workers consumes from this queue
• Retry workers treat rate limits as retryable errors (with high maxReceiveCount to improve chance it doesn't dlq)
• Added check_gmail_rate_limit_webhook() helper to deduplicate rate limit handling logic

Testing

Verified in dev environment:

• Primary queue continues processing during bulk operations
• Rate-limited operations successfully migrate to retry queue
• Retry workers properly handle backoff and eventual success

Screenshots, GIFs, and Videos

[linear]

EMA-42 Improve rate limit handling for webhook pubsub queue

[whutchinson98]

I think it's time to move out the workers into their own container that runs isolated from email service.

[whutchinson98]

please make a ticket for that and implement when you have the bandwidth, medium priority

[evanhutnik]

agreed, will do

[evanhutnik]

https://linear.app/macro-eng/issue/EMA-46/move-email-workers-out-of-service-pods

[whutchinson98]

Prefer anyhow::bail! here

[whutchinson98]

small nit but overall seems fine.

we really should be running these workers in a separate container to the main email service though.