Enable Cross-Worker Queue Testing Locally

[StefanStuehrmann]

Proposal: SQLite-based Queue Implementation for Local Development

Background

Currently, wrangler's local queue implementation uses an in-memory array for storing queue messages, which prevents queue usage across different workers during local development. This limitation makes it difficult to test distributed worker architectures locally.

Proposal

Extend the current queue implementation to optionally use SQLite for message storage (which is already used for D1).
This would enable:

• Shared queue access across different workers
• Persistent queue state between restarts
• Better debugging capabilities
• More production-like behavior for local development

Implementation Details

Configuration

# wrangler.toml
[queues]
storagetType = "sqlite"  # Optional, defaults to "memory" for backwards compatibility
persistTo = ".wrangler/queues.db" # needs to be introduced for queues but would be consistent to e.g. D1, R2
retentionPeriod = 86400  # Optional, defaults to 24 hours

You could think about if a queue state should always reside with the consumer worker as queues can only have one consumer but multiple producers.

Key Features

1. Backward Compatible: Maintains existing memory-based implementation as default
2. Concurrent Access: Uses SQLite transactions for safe concurrent queue operations
3. Message Lifecycle: Proper handling of message states, retries, and dead-letter queues
4. Performance: Indexed queries and batch operations for efficient message processing
5. Debugging: Messages persist between runs and can be inspected using SQLite tools

Benefits

• Test distributed worker architectures locally, which is the main purpose of introducing queues
• Debug queue-related issues more effectively
• More accurate representation of production behavior
• Simplified testing of retry and error handling logic

Implementation Impact

This change would be isolated to the local development environment and wouldn't affect production queues. The implementation would live entirely within the miniflare package and tests and wouldn't require changes to the core Workers runtime.

Looking forward to hearing the queue teams thoughts on this approach.
The effort estimation for the implementation seems to be relatively small (few days).
So in case it does not make it to the queues team backlog I would be willing to contribute once we agree on details and it will make it into the main branch.