Skip to content
This repository has been archived by the owner on Mar 20, 2023. It is now read-only.

Scheduling FIFO queue #8

Open
rvolosatovs opened this issue Jun 21, 2022 · 0 comments
Open

Scheduling FIFO queue #8

rvolosatovs opened this issue Jun 21, 2022 · 0 comments

Comments

@rvolosatovs
Copy link
Member

rvolosatovs commented Jun 21, 2022

We want a few properties from the queue:

  1. FIFO
  2. Persistence and fail-safety, once a workload is scheduled by the user it does not get lost even on server crashes
  3. Efficiency and scalability, N workloads should be able to be safely scheduled concurrently and M workload executors should be able to pick elements from the queue concurrently
  4. At most one element per GitHub ID, that means that the data structure should do some bookkeeping

The actual queue items should be just Drawbridge slugs, which are resolved at execution time

I propose to use a Redis stream https://redis.io/docs/manual/data-types/streams/ for the workload queue and possibly pair it with a hash set for bookkeeping. This would provide us with a very robust solution, which we can trivially scale (by just starting more instances) and easily debug (by looking into the Redis queue). I have extensive experience working with Redis doing almost exactly this, so I'd be happy to pick this up

@dpal dpal moved this to New in Profian Board Nov 23, 2022
@dpal dpal moved this from New to Backlog in Profian Board Nov 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Status: Backlog
Development

No branches or pull requests

1 participant