Add offline caching to the agent

[ysmilda]

📃 Description

This pull request adds offline caching capabilities to the agent as discussed in #1262. The goal is to keep gathering statistics while the agent cannot reach the hub. Once the connection is restored the backlog is committed to the hub.

Currently there is a basic cache implemented for the agent. What still needs to be done before this can be merged:

• Add support to the hub for processing multiple statistics at once without breaking compatibility with older clients.
• Check if alerts need to be changed to timestamp as well.
• Add graph to the hub for showing connection status over time. To indicate which measurements were live and which were cached.
• Add tests.

Closes #1262

🪵 Changelog

➕ Added

• Offline cache to the agent
• Timestamp field to the CombinedData struct
• Timestamp field to the database instead of using the created field. This allows the data to be backfilled.

✏️ Changed

• AgentResponse SystemData field is now an array of []*system.CombinedData. This is a breaking change for older agents.

[henrygd]

Awesome, thanks very much!

It may be a week or two before I can get to this as I'm trying to finish up with S.M.A.R.T. support and have a couple other long-waiting PRs at the top of my queue.

Don't worry about any merge conflicts that may be introduced in the meantime 👍

[ysmilda]

The PR is now functional. It does introduce a breaking change within the AgentResponse which would require the update of the hub and agent simultaneously. There are some workarounds possible, but none particularly elegant.

While I made the argument for just an in memory store, I would like to revisit that. It is not guaranteed the agent will get a connection to the hub before a shutdown. So adding optional persistency seems to be a nice feature. Introducing an option to cache to sqlite wouldn't be too hard to do.

[henrygd]

Thanks! Should be able to get to this during this upcoming week.

[henrygd]

Apologies, I'm still wrapped up with #1153 and a couple small things. I will switch this to in-progress and try to get to it as soon as I can.

[ysmilda]

No worries, if you'd have the time to do a review or give some pointers I would be more than happy to continue the development and carry some of the load.

[ysmilda]

@henrygd Any chance we can get this moving again?

[henrygd]

Sorry for the wait and lack of updates!

I have a good idea of what I want to do with this, and made some precursor architectural tweaks. Then had to pause work on the whole project for a bit, and I'm still working through a pile of smaller things that accumulated.

My plan is to get #539 added for 0.19 (it's been waiting for nearly a year, and I finally have a Proxmox machine). Then this is the top priority for 0.20.

[ysmilda]

Great to hear it's on the roadmap. If at that time you could use some help I will be available!

[henrygd]

Thanks, I'll let you know!