Disk Queue GA meta-issue

[faec]

This is a meta-issue to track the remaining work on the disk queue, which went beta in 7.10. This is for planning / prioritization as we push it towards GA, so anything that is a candidate for going into the first full release should be listed here.

This is in some ways a successor to #6859, although the new transactionless design and related simplifications means that much of that list is no longer applicable (transaction errors, page management, etc) or already complete in the current base package (full error handling / blocking), or is more advanced than we plan to include in v1 (multiple queues, advanced upgrade options and recovery tools...).

Phase 1

• Finalize on-disk schema ([libbeat] Track frame counts in saved segments in the disk queue #22970)
• Unit testing
  • Core loop logic ([libbeat] Fix potential deadlock in the disk queue + add more unit tests #21930, [libbeat] Add more disk queue unit tests and fix a size-check bug #22107)
  • Consumer ACKs
  • Helper goroutines (reader / writer / deleter)
  • Shutdown
• End-to-end tests

Phase 2

• Metrics
  • Report full queue metrics (including events from previous session) in monitoring.metrics.beat.libbeat.pipeline.events.active. #29296
  • Add additional queue-specific metrics?

Documentation with help of the doc team?

• Document the implementation logic (a lot is included in the comments and tests but there should be a standalone design summary) Describe what are the Guarantee are for the user.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[EricDavisX]

@faec Hi, when you get into this one, please do reach out to me and I am happy to chat about testing in general, and especially the e2e-testing framework if you aren't familiar with it. cheers.

[simitt]

This is also very interesting for the APM Server, where event sizes can vary quite a bit, and spooling to disk could help reducing the variance in memory consumption.

[felixbarny]

To add to Silvia's point: As applications monitored by APM may have bursty traffic. The amount of data APM Server processes and sends to Elasticsearch proportional to the monitored application's traffic. A persistent queue could help to smooth out the traffic spikes. Currently, APM Server has to drop events if its in-memory queue fills up when Elasticsearch is busy indexing a burst of events.

[faec]

Edit: moved part of phase 2 (#22970, the schema change that uses absolute instead of relative file locations to track queue positions) to phase 1, since the remaining tests depend on its changes, and since the new schema greatly simplifies any future updates. The metrics reporting itself is still phase 2.