[ENH]: add metric for total # of uncompacted log records #2696
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
codetheweb
commented
Aug 21, 2024
| @@ -85,9 +79,14 @@ k8s_yaml([ | |||
|
|
|||
| # Extra stuff to make debugging and testing easier | |||
| k8s_yaml([ | |||
| 'k8s/test/namespace.yaml', | |||
There was a problem hiding this comment.
resources in ks_yaml(...) take global precedence over anything else defined in the Tiltfile, so I moved namespace creation here so that it's available for these resources
(creation of the new configmaps was erroring in CI, but not locally for some reason)
codetheweb
force-pushed
the
feat-compaction-queue-depth
branch
from
992e68d
to
1850693
Compare
codetheweb
commented
Aug 21, 2024
HammadB
reviewed
Aug 22, 2024
go/database/log/queries/queries.sql
Outdated
| @@ -33,3 +33,6 @@ INSERT INTO collection (id, record_enumeration_offset_position, record_compactio | |||
|
|
|||
| -- name: PurgeRecords :exec | |||
| DELETE FROM record_log r using collection c where r.collection_id = c.id and r.offset <= c.record_compaction_offset_position; | |||
|
|
|||
| -- name: GetTotalUncompactedRecordsCount :one | |||
| SELECT SUM(record_enumeration_offset_position - record_compaction_offset_position) AS total_uncompacted_depth FROM collection; | |||
There was a problem hiding this comment.
Is there any concern of this query being slow / hogging too much of the node? Maybe we could consider this to move to a read replica if so (in future), since it would be OK with data log.
There was a problem hiding this comment.
I think it should be ok for now, takes 5-7ms to execute. But good to keep in mind.
HammadB
approved these changes
Aug 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Adds a new metric, emitted by the log service, that describes the total number of uncompacted log records. Here's what it looks like during
test_sanity.py:In service of this, I also added several resources to the local Tilt stack: the OpenTelemetry collector, Prometheus, and Grafana. These help debug and explore metrics as Jaeger unfortunately only does traces. All the containers are fairly lightweight and add about 200MB of memory consumption. I'm open to punting adding these local observability containers, but I think they will be useful when we start sampling traces and likely adding additional metrics to make up for the lost granularity.
Alternatives:
collection_id: would have liked to do this, but requires some careful consideration to avoid emitting 1 event per collection every 10s (or whatever interval we choose). OpenTelemetry supports instruments that only transmit a delta at each collection interval, so I think this should be possible.Test plan
How are these changes tested?
Confirmed new metric works with Grafana.
Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?
n/a