Generic Non-Blocking Task Management ("Queue") for discovery and nodes domains

[emmacasolin]

Specification

Unattended discovery was added in #320, however, there is no concept of priority within the queue. There are three ways that a vertex (a node or identity) can be added to the discovery queue, and they should follow this order of priority:

1. Manually, via the discovery methods queueDiscoveryByNode() and queueDiscoveryByIdentity() (these are called in the commands identities discover (explicit discovery) and identities trust (explicitly setting a permission, so we want the Gestalt to be updated via discovery).
2. As a step in the discovery process whereby child vertices are added into the discovery queue in order to discover the entire connected gestalt.
3. Automatically by a process of rediscovery when we want to update existing Gestalts in the Gestalt Graph (to be addressed in Discovery - revisiting Gestalt Vertices and error handling #328).

Vertexes with a higher priority should be discovered first, either by being placed at the front of the queue or by modifying the traversal method of the queue. The priority queue could also be further optimised by grouping vertices from the same gestalt together when this is known (for example when adding child vertices).

Additional context

• Original PR for implementing Unattended Discovery: Growing the Gestalt Graph and Implementing Social Discovery #320
• Issue for re-adding discovered vertices into the queue to update the gestalt graph Discovery - revisiting Gestalt Vertices and error handling #328
• Indexing Domains and DB Levels for Faster & Flexible Lookup #188 - Indexing the LevelDB may be required to be able to implement a persistent priority queue
• Integrate Automatic Indexing into DB js-db#1 - general indexing into js-db
• https://nicolas-van.github.io/modern-async/modern-async/1.1.0/Queue.html

Tasks

1. Modify the existing Discovery Queue to be a Priority Queue
2. Ensure that when a user interactively wants to discover a gestalt vertex that it becomes the highest priority and gets executed first
3. Look into the potential for further optimising the priority queue, for example by having multiple points of comparison with varying levels of importance that can influence the priority of a particular vertex in the queue

[CMCDragonkai]

There's a go implementation of persistent priority queue backed by leveldb here: https://github.com/beeker1121/goque. It can be used as a reference for this. Also a JS implementation here: https://github.com/eugeneware/level-q (the goque is probably more comprehensive).

Our priority queue needs to by default maintain order, because we do want to know the sorted list of jobs. But also allow us to add a special priority number on top.

From my imagination:

This reminds me of the indexing problem, where you can ask for a list of rows sorted by several columns. The first column would dictate the base sort, then subsequent columns would sort any ambiguous sub-orders.

Imagine we had 2 indexes. The first being your priority index using an arbitrary number, the second being the monotonic time index using IdSortable. You could sort on the priority index first, then sort on IdSortable second.

Maybe this then has a relationship to #188.

However this would only be for if you are looking up items. If we are streaming data from the level db that may be more complicated.

[CMCDragonkai]

This can be done with a compound index. Prefix can be the priority number (lexinted), suffix can be IdSortable.

This means you can then stream results that are always ordered in terms of priority first then by time second.

Priority can start at 0 by default, and one can increment priorities depending on the origin of the task. Like tasks emitted by user wanting to lookup something can be set to a higher priority number.

We could do this directly by changing the queue domain key. But I'd suggest first solving the indexing issue in general first then building a compound index on top.

[CMCDragonkai]

We discovered that the priority queue can also benefit from a uniqueness index creating a uniqueness constraint: #311 (comment)

This means that duplicate tasks cannot go into the priority queue. Not entirely sure if this is required because a queue can still say they should process the same task over and over.