About a new backend infrastructure...

[GuillaumeDerval]

Introduction

This issue is actually not one, it's rather a requirement list for a new "backend infrastructure".

The current infrastructure currently suffers from a few problems/limitations. The most problematic ones:

• The "backend" (inginious-backend, which is mostly a queue that can handle the various requirements and specifities of our agents) is a SPOF
• We cannot handle "long running jobs", where long-running is potentially several days. One example of this requirement is the long-awaited "peer-reviewing" feature, where students would be able to review exercises made by other students. In that case, we could consider the student as a very (very) slow agent.

Requirements of a new infra

Mandatory

• Ability to run multiple types of jobs (MCQ, docker, kata, ...) like currently. The general idea of environments (Grading Units) should not change.
• Ability to send message to and from running jobs. From example to kill them, or to create interactive tasks.
• No SPOFs (We can consider that MongoDB is never a SPOF, so we can use it at will)
• The threat model should ensure that agent can be compromised, and that should not break anything more than the jobs being run on that agent
• Long running jobs, with job in DB for multiple days
• Jobs should survive to restarts of any component
• If any component crash, the system should be able to detect it and rollback to a previous safe state. We should never lose information, apart from maybe information never comitted to DB.
• Performance. It should be able tens or hundreds of new submissions per second for short burst, and be able to cope for long time with very large queues.
• Jobs should be handled in a first-come-first-served way.

(what about fairness and starvation? from an agent perspective, that's not too worrying).

Nice to have

• Multiple-steps jobs: it would be nice to be able to run first a MCQ step then send the result to a docker container and then show it to students in a peer-review ;-)
• Ability to give an idea of the size of the queue, and position inside it
• Even better, to give an idea of how much time remains to be processed, approximately
• Ability to send message to and from running and waiting jobs
• It would be nice to prove that the system cannot fail (under the assumption that everything is implemented as designed), using FSMs or something like that. Time to reinstall LTSA ;-)
• Implementing the system AS an FSM would be ultra nice.
• Agent should be able to store additional state about themselves. For example, all agents using Docker could add an internal id to the running Docker container they launch, and when this ID is not present in DB, any other agent using Docker could close the container (as the original agent would have crashed).

An example of possible infrastructure

A proposal I explored 2 years is to remove the backend process and replace it by MongoDB, that would handle the queue. The frontend and the agent would connect to MongoDB and push/pull information about jobs to do/done directly.

The work I did is here: https://github.com/UCL-INGI/INGInious/tree/backend_free_arch/inginious/new_agent

The idea is a follows:

• Each agent proposes multiple Grading Units (that we currently call "environments")
• Agent interact directly with the DB
• A job is composed of multiples steps. A step is the name of a Grading Unit (for example, mcq or docker-cpp). The steps are stored as an array in the job in DB.
• The process to find and process a job is the following:
  1. The agents finds a job in DB which
     1. is not reserved (currently worked on by someone else)
     2. has as first step in its steps any value that it can handle (any of its Grading Units)
  2. The agent reserves the job. This is done in the same time as the first step, atomatically.
  3. The agent launch the Grading Unit, that does its specific job
  4. Once the Grading Unit is done and has returned the result
     1. The result is put in the job
     2. The first step is removed from the array of steps
     3. The job is released for another agent
• Agents have to update a row in a status collection in DB each 10 seconds.
• There is a process in each frontend whose job is to gather done jobs (jobs that are free and with no steps remaining) and to gather stalling jobs, i.e. jobs that are reserved and for which the agent did not update the timer for a too long time (let's say 30s). This ensures that if an agent crashes, the system recovers, by freeing the submissions of crashed agents.
• In order to ensure the agents do not spam the DB, agents listen to the events in DB rather than pulling every X seconds. If an event indicates that a new job was added with one of their grading unit as next step, they try to reserve it.
• There is a collection in DB that allows to send messages to a running job. For example to kill it. Once a job is started, the agent listen for update in the collection and read messages concerning the running job. Grading Units can send messages to frontends using that same collection.
• Everything is based on Motor, a MongoDB client using asyncio.

This infrastructure has multiple advantages. It is elegant: agents are highly decorelated from Grading Units, and thus the creation of specific new grading units is super easy (creating a peer-review grading unit would take something like 5 lines of code once the user-facing interface is done). It respects most of the requirements, but two:

1. We lose the ability to give time estimates. This is a nice to have, anyway...
2. but more importantly, agent have direct, full access to the DB and that is bad™. It would be nice if they could only update some specific fields in specific collections, to ensure that if agents are compromised then they are not able to break everything. Note that it's not THAT bad. In practice, the agent has only access to the submission queue and specific agent-related things so I can only break things that are "new jobs" and not old jobs, that are not stored there but in a collection specific to the frontend. Maybe we can live with this relaxed threat model.

Another possibility

Our friends at Domjudge actually have the agents (the "judges") pull periodically by HTTP(S) on the frontend to ask for work. It complicates a bit real-time communication between frontends and agents, but it may work well too.