feat: Add fanout latency metric for loki pipelines#5656
Conversation
| wl := prometheus.NewHistogram(prometheus.HistogramOpts{ | ||
| Name: "loki_fanout_latency", | ||
| Help: "Write latency for sending to components", | ||
| Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10, 30, 60}, |
There was a problem hiding this comment.
Do we need all these buckets? I think we can do with fewer and still have a good idea of 'something is wrong' or 'everything looks fine'.
There was a problem hiding this comment.
Absolutely, I just used the same bucktes we have for prometheus and pyroscope pipelines
There was a problem hiding this comment.
I changed buckets in f62e04f.
These would be able to measure potential throughput at 5k/s to 1/s but I am happy to adjust these more if you think we should do it
| processIn: loki.NewLogsReceiver(), | ||
| receiver: loki.NewLogsReceiver(loki.WithComponentID(o.ID)), | ||
| fanout: loki.NewFanout(args.ForwardTo), | ||
| fanout: loki.NewFanout(args.ForwardTo, o.Registerer), |
There was a problem hiding this comment.
Is this registerer scoped to the component?
Well it's a draft, also need to update docs for each component using fanout to include this metric |
kalleep
changed the title
kalleep
force-pushed
the
kalleep/loki-pipeline-write-latency
branch
from
0ee3f56 to
b97898f
Compare
|
💻 Deploy preview deleted (feat: Add fanout latency metric for loki pipelines). |
|
Btw right now I just fixed existing tests that check metrics to ignore these, do you think we should have test for the metric itself? |
clayton-cornell
added
the
type/docs
Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>
|
Docs changes are OK for now. Are there more changes incoming for docs? |
|
Closing in favor of #5702 |
### Pull Request Details Alternative to #5656. In this pr I added `loki_write_entry_propagation_latency`. This is a histogram with the time it takes from from a source until it's written over the wire. Every source now uses `loki.NewEntry` or `loki.NewEntryWithCreated`, and multiline will propagate created from first entry. When the entry was either sent or dropped by loki write we record the time it took for all of them. I also added a new version for entries in WAL so that the created time is encoded. ### Issue(s) fixed by this Pull Request <!-- Uncomment the following line and fill in an issue number if you want a GitHub issue to be closed automatically when this PR gets merged. --> <!-- Fixes #issue_id --> ### Notes to the Reviewer <!-- Add any relevant notes for the reviewers and testers of this PR. --> ### PR Checklist <!-- Remove items that do not apply. For completed items, change [ ] to [x]. --> - [x] Documentation added - [x] Tests updated - [ ] Config converters updated --------- Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com>
3 participants
Pull Request Details
Issue(s) fixed by this Pull Request
Notes to the Reviewer
PR Checklist