Skip to content

Appends the saved objects documents count to the CoreUsageData service#124308

Merged
TinaHeiligers merged 6 commits intoelastic:mainfrom
TinaHeiligers:core-usage-data-use-count-api
Feb 3, 2022
Merged

Appends the saved objects documents count to the CoreUsageData service#124308
TinaHeiligers merged 6 commits intoelastic:mainfrom
TinaHeiligers:core-usage-data-use-count-api

Conversation

@TinaHeiligers
Copy link
Copy Markdown
Contributor

@TinaHeiligers TinaHeiligers commented Feb 2, 2022

Resolves #120485

This PR adds the actually saved objects document count to the CoreUsageDataService payload, retaining the original Lucene document count and other information about the store size from the GET _cat/indices/.kibana API.
In addition, this PR adds debug level logs for the Lucene document count and the saved objects documents count to make debugging easier.

Why not just change what we report?

I decided to keep the original payload and add the additional information rather than replace the docs count for 2 reasons:

  1. Reindexing the upstream data is a pain but would need to be done if we were to rename the fields or replace them.
  2. We already have data reporting the Lucene document count and other useful information (primary store size) from the _cat/indices/.kibana API. It would be confusing to simply replace the lucene docs count with that of the saved objects docs count because that's not what the _cat/indices/.kibana API API returns.

Changes:

In the example data payload (Advanced settings -> Usage Data -> cluster data:

Screen Shot 2022-02-02 at 11 56 28

In the logs

Logging configuration

// kibana.yml
appenders:
    console:
      type: console
      layout:
        type: pattern
        highlight: true
        pattern: "[%date][%level][%logger] [%meta]---%message"
root:
    appenders: [default, console]
    level: warn
  loggers:
    - name: core-usage-stats-service
      appenders: [console]
      level: debug

Logs

2022-02-02T11:56:08.425-07:00][DEBUG][core-usage-stats-service] []---Lucene documents count 267 from index .kibana
[2022-02-02T11:56:08.426-07:00][DEBUG][core-usage-stats-service] []---Saved objects documents count 148 from index .kibana
[2022-02-02T11:56:08.448-07:00][DEBUG][core-usage-stats-service] []---Lucene documents count 18 from index .kibana_task_manager
[2022-02-02T11:56:08.448-07:00][DEBUG][core-usage-stats-service] []---Saved objects documents count 18 from index .kibana_task_manager

How to test this:

  • Configure logging for the core-usage-stats-service in kibana.yml as described above
  • Start elasticsearch and kibana
  • Navigate to Advanced Settings -> Usage Data and click on cluster stats to see an example of the data Kibana collects
  • Search for savedObjects under services in the flyout and observe that we report the savedObjectsDocsCount as another entry per index.
  • Logs: search for the core-usage-stats-service logs and observe we have two entries for docs counts: one for the Lucene document count and one for the saved objects document counts.

Checklist

Risk Matrix

Risk Probability Severity Mitigation/Notes
Making an additional API call reduces telemetry collection performance Low Low Caching and the backoff mechanism will ensure that usage data collection does not impact cluster performance to any large degree.

For maintainers

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Deprecated - use backport:version if exact versions are needed bug Fixes for quality problems that affect the customer experience Feature:Saved Objects release_note:skip Skip the PR/issue when compiling release notes v7.17.1 v8.0.0 v8.1.0 v8.2.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CoreUsageDataService savedObjects.indices[0].docsCount should use _count API

5 participants