First draft for Vitess Queues design document.

[alainjobart]

No description provided.

[harshit-gangal]

As per the doc, this seems to be an unordered event relaying.

[derekperkins]

This is our #1 priority for our Vitess migration (8 week timeline), so I'm highly motivated to get this in motion. We're replacing the Google App Engine task queue, so that informs most of my opinions. https://cloud.google.com/appengine/docs/go/taskqueue/ Here's a screenshot of the queue UI

Key Needs

• support both push/pull
• highly elastic write loads (create 1M jobs/events in < 1 min)
• scheduled events
• support queue limits
  • max QPS execution speeds
  • max concurrent tasks

Thoughts

Primary Key Proposal

Using a timestamp as the PK is likely going to cause serious write hotspotting when adding many tasks simultaneously and read hotspotting at execution time.

Add in an event_name column that will act as the primary / sharding key. This also allows individual tasks to be retrieved / canceled

1. This is optionally user supplied, and the user is responsible for hotspotting
2. Otherwise generated as a UUID or similar option

Configurable Cleanup

Keep task history for N days, but also allow for users to manually purge individual task names or entire queues.

Push / Pull Queues

The core Queue Manager functions as a pull queue with acks, so the push component could potentially be external to the system. The queue limits would need to be applied at the Queue Manager level, which may necessitate a limitation of only 1 listener per queue, if only for the ease of implementation. Google Tasks likely implements it this way, as they have a hardcoded 500/s limit. If faster execution is necessary, the user is responsible for sharding queues.

These are my initial thoughts, I'll keep mulling over the proposal. I'm super excited for this! What is your expected timeline / availability? Thanks for the super detailed proposal!

[derekperkins]

@rnavarro and @bbeaudreault both expressed some interest in Vitess queue capabilities.

[sougou]

Yes: https://github.com/youtube/vitess/blob/master/go/vt/tabletserver/table_info.go#L37

[sougou]

drop very here and below.

[sougou]

Note:, or Note that

[sougou]

Will we provide an ordering guarantee? @harshit-gangal was asking.

[derekperkins]

We don't have a need for guaranteed ordering, though if it were provided, I'm sure we could find a use case. :) That would be one of the first things I'd punt on. I think that could maybe be a queue type CREATE OrderedQueue that could be plugged in after the fact.

[harshit-gangal]

@derekperkins There are other parameters that needs to be considered to provide ordering guarantee.
As you might have an event which needs to be dispatched first but it is not yet committed whereas the event after that is committed.

[derekperkins]

@harshit-gangal I know that ordered queues is a non-trivial problem, which is why we don't rely on them currently. It feels like we'd have a better shot at doing it than a NoSQL store. All in all, I think the extra layers of complexity for it would introduce unwanted tradeoffs for systems that don't need ordering, which is why I think an OrderedQueue should probably be done as a separate queue type, if it is even tackled at all.

[derekperkins]

Task Metadata / Tagging

A pretty common use case we currently have that is difficult to manage is "when all X tasks complete, then do Y". This may just be covered by user field data, but may be globally useful. If you had a single Tag or JobID or TaskGroup field, then when I started a logical job, I could give it a unique ID. e.g.

1. Create 1M tasks for immediate execution with JobID=user-41-rebuild
2. Create 1 task to be executed in 5 minutes, that inside my application would fail and retry until QueueManager.CountTasksByJobID("user-41-rebuild") == 0