Initial metrics ingestion, processing, write to influxdb, and grafana dashboards. #1159
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Some interesting trends were certainly notable. See a snapshot of graphs generated using everything in this PR. |
jberry-suse
force-pushed
the
metrics-v1
branch
from
f45453d
to
a156b29
Compare
jberry-suse
force-pushed
the
metrics-v1
branch
from
a156b29
to
34aa7e9
Compare
jberry-suse
force-pushed
the
metrics-v1
branch
from
183f75b
to
fdb0e4b
Compare
jberry-suse
force-pushed
the
metrics-v1
branch
from
fdb0e4b
to
efe2969
Compare
|
If anyone plans to review and would like more time please comment, otherwise I'll merge this tomorrow. |
… dashboards. A lot of room for improvement and additional metrics that can be extracted. Including non-final state requests would allow for analyzing the current staging state instead of only historical state. Additionally, the current state can be used to present an activity log. Handling incremental updates is non-trivial given the deltas are evaluated and stored in sum state. A few possible approaches, but likely not worth the hassle given the relatively short processing time and infrequent desire to update data (daily at minimum).
Excludes large JSON files from main package as most users will not need.
jberry-suse
force-pushed
the
metrics-v1
branch
from
efe2969
to
298ca5e
Compare
|
rebased post other merges |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A lot of room for improvement and additional metrics that can be extracted. Including non-final state requests would allow for analyzing the current staging state instead of only historical state. Additionally, the current state can be used to present an activity log.
Handling incremental updates is non-trivial given the deltas are evaluated and stored in sum state. A few possible approaches, but likely not worth the hassle given the relatively short processing time and infrequent desire to update data (daily at minimum).
One of the biggest things missing is walking the history of the dashboard container to know the state of config options and psuedo-ignore list so those requests can be separated out from backlog.
For simplicity the non-final requests are not included at all which means that for
openSUSE:Factorythe totals for backlog and such will not reflect currently open request.Additionally, I have it set to infinitely cache request results until I decide how to handle only looking for new requests. Removing the cache directory (
~/.cache/osc-plugin-factory-metrics) or the last few pages will handle looking for new ones. Perhaps the easiest tweak would be sorting the search results on last update for requests to make it easy to append newly changed requests.The overall structure is to loop over all request creating measurement points, many of which are deltas (like add one to this bucket, or minus one from another). After all requests are process the points are walked to evaluate the deltas and re-create the state at that time.
Due to a number of issues with the data returned from OBS the Factory graphs will show negative values at a couple locations due to completely incorrect dates. The final numbers zero out correctly since the incorrect dates are still within dataset start and end dates. One caveat is that Grafana seems to not always grab the last record and thus shows a trailing one, but the data generated correctly ends at zero. The following are the OBS related issues:
I also have an experimental version of #730 being generated by Grafana. With a few tweaks and including open requests (at least for the log) I expect this to provide a rather nice activity log.
Lastly, a test could be added in a several ways, but this it seems like overkill to add to in-depth of a test for this. The simplest approach would be a test against a finished project using live data and confirm some points of interest and total records created and such. Alternatively, a hand-crafted set of requests could be returned using the current mock setup...or a full OBS setup and release requests setup to be queried. The live approach seems better since the data should not change, provides a large and unique enough data-set to be relevant, and does not requiring adding large XML dumps to this repository.
Finally, ingesting the annotation events from an external data source would definitely be preferable if one exists. Additionally, ingesting upstream release dates of interest or annotating version changes of certain packages signifying major updates may also be interesting.
Since open-build-service team did not indicate interest in hosting the tools I filed an IT ticket to have a publicly facing machine so the metrics can be available to anyone. May also encourage others to expand what is extracted.