Task / Course storage

[nrybowski]

Currently, tasks and courses are stored on a file system interfaced by the LocalFSProvider class.

This implies that this FS has to be mounted somehow in each environment running a frontend instance, e.g. https://github.com/UCL-INGI/INGInious/pull/882/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3R69-R70.
Even worse, this FS also has to be mounted in the Agents. See #352 for more details and https://github.com/UCL-INGI/INGInious/pull/882/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3R30-R32 for an example.

We could use the MongoDB instances backing INGInious to store the tasks and the courses. We could also take this opportunity to decouple tasks from courses.

This discussion proposes a new way to store tasks and courses within MongoDB.
Any inputs are welcome to validate/improve this new design.

New Task Model

• A task is a unique object, i.e. it does not belong per se to any course.
• A task is uniquely identified by its ObjectId.
• A task is a MongoDB document whose keys are the current file paths and the values are the serialized files. The current task.yaml file is directly embedded as a sub-document of the task. Rather than embedding directly serialized files in the task, one could embed only an ObjectId pointing to the serialized file. Images can be stored as base64 encoded strings.
• A task could be a derivative of another task. It should then mention the parent's ObjectId.
• A derivative task only contains the updated values against its parent task. For example, one could store git diff alike structures for the task's files.

New Course Model

• A course is a MongoDB document uniquely identified by its ObjectId.
• It embeds directly the current course.yaml as a sub-document.
• It contains a list of tasks being represented by their ObjectIds.
• The current '$common' directory is embedded as a task and referenced in the course by its ObjectId.

Additional notes

• This new way of structuring tasks allows an easy decoupling of the concepts of Tasks and Courses. Now, if a task is shared between multiple courses, no more duplications are required. On the other hand, if a course needs an updated version of a task, one could generate a derivative task that only stores the required diff, hence minimizing the additional storage amount needed.
• The interface for this new model MUST respect the FileSystemProvider interface to minimize the changes within the frontend code.
• The replication feature of MongoDB ensures that tasks and courses are always available. The Frontend environment is not a SPOF anymore.
• There is no need anymore to mount File Systems via NFS, sshfs, ... within the Frontend and Agents.

[anthonygego]

MongoDB can store binary files into its GridFS, this is currently used to store the submission payload. I was always skeptical about the overhead for exporting the whole task files to the container filesystem but it may be acceptable as the MongoDB cache may be more efficient than a classic filesystem cache, if there is no too many files.

Tasks should remain serialisable in a filesystem format as most users rely on modifying the files on their machines, and to allow sharing courses between instances

[GuillaumeDerval]

MongoDB cache may be more efficient than a classic filesystem cache, if there is no too many files.

On the database-server-side yes, but on the local side, there is no cache, and it's where we need one...

[GuillaumeDerval]

One the main question is the performance. MongoDB is fast, but will never be as fast as a local filesystem. Even if the filesystem is remote (nfs, sshfs, whatever) it's probable that the performance would be better than MongoDB, notably due to serialization and the way messages are passed by MongoDB on the wire.

There are some tasks with multi-GB folders, and clearly redownloading them from MongoDB each time a task starts is not possible. A kind of local caching will be needed, and we will not be able to use the caching system of the kernel (above NFS/sshfs), so it would be on our end to implement it.

As an alternative, maybe we can look at MongoDB-based FS compatible with FUSE?

[anthonygego]

FYI the largest task folder we have is 691MB large, 1.5% of our 6750 tasks are larger than 10MB, and 0.28% are larger than 100MB (duplicates included).

[anthonygego]

Also note that currently the whole task folder is copied before launching the container and this cannot be neglected in this discussion. The whole process has to be considered.

[nrybowski]

Is there any counter-indication to bind-mount the task folder in read-only in the grading containers?

[GuillaumeDerval]

Yes, it needs to be writable. But we could bind-mount it as the lower level of an overlayfs.

[GuillaumeDerval]

(I actually have a toy implementation of this somewhere)

[nrybowski]

#834 is another related feature we should consider in this discussion.

[nrybowski]

We should also take the opportunity to reconsider the syllabus integration with this discussion. It could play nicely with the new concept of Course.