Skip to content

chore(database_observability): Use Fanout, Consume and Drain#5866

Merged
kalleep merged 3 commits into
mainfrom
kalleep/database-observabilit
Mar 24, 2026
Merged

chore(database_observability): Use Fanout, Consume and Drain#5866
kalleep merged 3 commits into
mainfrom
kalleep/database-observabilit

Conversation

@kalleep
Copy link
Copy Markdown
Contributor

@kalleep kalleep commented Mar 24, 2026

Pull Request Details

We have added some shared functionallity for working with loki pipelines and should use them in our database_observability. This will make it easier in the future to start to migrate to new architecture for loki pipelines and avoid common pitfalls.

I did not review all collectors but just did a sample and e.g. explain_plan can cause a deadlock when component is stopping and in turn preventing alloy from stopping because nothing is consuming from handler.

Changes:

  1. Use loki.Fanout - This is doing internal locking of logs receivers so components do not have to care about it. It will also stop forwarding when context is canceled.
  2. Use loki.Consume - Runs the consume loop from a LogsReceiver and will exit when context is canceled.
  3. Use loki.Drain - When components is stopping we need to still forward / drain to make sure collectors can stop. Since fix: Stop components in a deterministic order #5613 we have a deterministic order for stopping components so it is safe to forward to downstream components during stops but we still have a timeout after we would drain into nothing.
  4. Use loki.NewEntry - This makes sure we set created timestramp of entries so they can be properly tracked by loki_write_entry_propagation_latency_seconds.

Issue(s) fixed by this Pull Request

Part of: #5826

Notes to the Reviewer

PR Checklist

  • Documentation added
  • Tests updated
  • Config converters updated

@kalleep kalleep requested a review from a team as a code owner March 24, 2026 12:52
@kalleep kalleep requested a review from Copilot March 24, 2026 13:03
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates database_observability (MySQL + Postgres) components to use the shared Loki pipeline helpers (Fanout, Consume, Drain, NewEntry) to reduce shutdown deadlocks and align with the repo’s evolving Loki pipeline architecture.

Changes:

  • Replace manual receiver-forwarding loops with loki.Fanout + loki.Consume in MySQL/Postgres Run.
  • Add loki.Drain(..., loki.DefaultDrainTimeout, ...) to ensure logs are drained/forwarded while collectors stop during shutdown.
  • Switch BuildLokiEntryWithTimestamp to loki.NewEntry to ensure entry “created” timestamps are set; update affected tests for the new fanout field.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
internal/component/database_observability/postgres/component.go Use Fanout/Consume for forwarding and Drain during shutdown; add fanout field.
internal/component/database_observability/postgres/component_test.go Update component construction to initialize fanout.
internal/component/database_observability/mysql/component.go Use Fanout/Consume for forwarding and Drain during shutdown; add fanout field.
internal/component/database_observability/mysql/component_test.go Update component construction to initialize fanout.
internal/component/database_observability/loki_entry.go Build entries via loki.NewEntry to set internal created timestamp.

Comment thread internal/component/database_observability/postgres/component.go
Comment thread internal/component/database_observability/mysql/component.go
defer c.mut.Unlock()

c.args = args.(Arguments)
c.fanout.UpdateChildren(c.args.ForwardTo)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before both of these components stored both args and receivers but never actually updated the second one. So if a config change was made with new receivers that would not work and we would keep sending logs to the old receiver list causing all kinds of problems.

Copy link
Copy Markdown
Contributor

@matthewnolf matthewnolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@kalleep kalleep merged commit b2d980a into main Mar 24, 2026
48 of 49 checks passed
@kalleep kalleep deleted the kalleep/database-observabilit branch March 24, 2026 14:03
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Apr 8, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants