Move persistent client queue from slurm collector to clients

[stefan-k]

The Slurm collector currently has a "sending queue" where records to be sent are placed on. This queue is processed at regular intervals. If sending fails, the record is placed on the queue again and tried again on the next run. The queue is persisted on disk in a sqlite database, therefore the collector can resume after a crash.

This should be moved into the client instead, such that it becomes available in other collectors as well (also in python). This will require quite a bit of work. I suggest implementing this for the async client first. When this works, it can be tried to do the same for the blocking client. Either way this will require a background thread which processes the queue.

[rkleinem]

Hi,
I had a look and think it makes sense to do this before the Kubernetes collector.
My suggestion would be that I add a QueuedAuditorClient and keep the interface as close to the AuditorClient as possible. The new client would be created by AuditorClientBuilder as the others, put everything in the persistent storage and send the records to AUDITOR periodically.
Also, I would move the client to a separate package because sqlx doesn't like two different database drivers in one package.
Does that sound ok to you?

[QuantumDancer]

Sounds good (especially if Stefan agrees :-) )!

[QuantumDancer]

This issue has been split into two parts:

• Move queueing mechanism to client: Add QueuedAuditorClient #776
• Adaption of Slurm Collector: Slurm Collector: Use QueuedAuditorClient #798