[Usage Counters] Enhancements to the APIs

[gsoldevila]

Summary

Part of #186530
Follow-up of #187064

The goal of this PR is to provide the necessary means to allow implementing the Counting views part of the Dashboards++ initiative.
We do this by extending the capabilities of the usage counters APIs:

• We support custom retention periods. Currently data is only kept in SO indices for 5 days. Having 90 days worth of counting was required for Dashboards++.
• We expose a Search API that will allow retrieving persisted counters.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

Why do we need this internal Core module import (@kbn/core-saved-objects-api-server-internal)? Is that only to stub getCurrentTime during the integration tests?

[gsoldevila]

Yes I'm afraid so, it bothers me too.
Perhaps I can try to find a cleaner way to insert old counters for testing purposes.

[gsoldevila]

In f90542c I used the standard incrementCounter to create the counters, and then I used esClient.updateByQuery() to modify their updated_at dates. Not ideal, but cleaner than the mock IMO, and addresses your feedback above.

[gsoldevila]

@afharo @Bamieh I have extracted all of the CollectorSet methods into a separate interface ICollectorSet, in an effort to make it cleaner / clearer that the usage-collection plugin offers 2 different things in the setup contract.

[gsoldevila]

I moved the rollups logic (aka delete counters older than 5 days) to the usage_collection plugin, IMO it makes more sense to have it there:

• It is specific of usage counters, kibana_usage_collection handles plenty of other collectors.
• If someone disables kibana_usage_collection, usage counters would be captured and persisted indefinitely.

WDYT?

[afharo]

Makes sense to me! Thanks!

[pgayvallet]

Implementation looks fine to me, but I'm not the one with the most knowledge of this area, so a second review would probably make sense

[pgayvallet]

NIT: we're not following our service pattern here, this method shouldn't be public / used directly. But it's not really that significant.

[gsoldevila]

++ ATM I am working on some enhancements that include making this private.
UPDATE: Addressed with fe9fb62

[afharo]

Overall LGTM. I added a few comments that I'd like to discuss before approving.

[afharo]

Makes sense to me! Thanks!

[afharo]

nit: how about returning an empty response instead? I wonder if throwing this error will create the need for plugins to check if it's enabled (and we don't provide an API to share if it's enabled or not).

Alternatively, we could allow searching, only that, when enabled: false, we don't store more data. WDYT?

[gsoldevila]

The problem is I only obtain the search method when calling the service.start(). Either:

• I call start (and we'll have a few RxJS timers running without buffering any events)
• Or I make the search method public (we break consistency with other services).
• Or I expose the search in the response of the stop() hook.

[gsoldevila]

I chose the 3rd option. Fixed in f58e23b

[afharo]

hmm, I don't think this approach is valid anymore...

If we have 12 dashboards viewed every day during the last 90 days, they'll show up in the first page and we won't remove others...

I wonder if we should change this to group per retention days (domainId: 'dashboard' and updated_at < retentionPeriodDays vs. not domainId: 'dashboard' and updated_at < USAGE_COUNTERS_KEEP_DOCS_FOR_DAYS).

[gsoldevila]

Good catch! we'll have to think about a better strategy.

[gsoldevila]

After discussion, we can do a search by domainId, and filter by updated_at < (now - retentionPeriodDays).
We can then loop through the different domain IDs.
Will assess bulkDelete following @TinaHeiligers's latest comment.

[gsoldevila]

@afharo updated with 357b474

[afharo]

nit: we can do it in a follow up: but we can now use the bulkDelete API :)

[gsoldevila]

We'll see about that, cause I believe the bulkDelete does not allow deleting from multiple namespaces.

[TinaHeiligers]

It will when you add 'force'. Please check that the option hasn't changed.

[afharo]

should we use PIT search?

[gsoldevila]

Totally, added with #187665 (comment)

[afharo]

nit: I don't see us defaulting it anywhere here.

How about defaulting retentionPeriodDays = USAGE_COUNTERS_KEEP_DOCS_FOR_DAYS if not provided?

[gsoldevila]

It's defaulted in the rollup logic, this way the in-memory UsageCounter's are lighter (no need to store the property if it matches the default value).

[afharo]

IMO, aggregations might better achieve what we're after here.

I imagine an API like:

usageCounters.search({
  filters: {
    domainId,
    counterName,
    counterType,
    source,
    timestamp: { [lt|lte|gt|gte]: Date }
  },
  aggregation_keys: [
    'domainId',
    'counterName',
    'counterType',
    'source',
    'timestamp'
  ]
});

That we internally map to an aggregation call and return flattened.

So... for a counter with this structure:

{
  "domainId": "dashboard",
  "counterName": "<dashboard-id>",
  "counterType": "views",
  "source": "server"
}

1. If I only care about the grand total of my N-day retention period for all my dashboards, I'd call the API like

usageCounters.search({ 
  filters: {
    domainId: "dashboard",
    counterType: "views",
    source: "server",
  },
  aggregation_keys: []
});

And I'd get a response like

{
  counters: [
    { 
      domainId: "dashboard",
      counterType: "views",
      source: "server",
      count: 9999999,
    }
  ]
}

2. If I want the grand total for each dashboard, I'd call the API like

usageCounters.search({ 
  filters: {
    domainId: "dashboard",
    counterType: "views",
    source: "server",
  },
  aggregation_keys: [
    'counterName'
  ]
});

And I'd get a response like

{
  counters: [
    { 
      domainId: "dashboard",
      counterType: "views",
      source: "server",
      counterName: "dashboard-1"
      count: 10,
    },
    { 
      domainId: "dashboard",
      counterType: "views",
      source: "server",
      counterName: "dashboard-2"
      count: 9999989,
    }
  ]
}

3. If I want the histogram for a specific dashboard and time range, I'd call the API like

usageCounters.search({ 
  filters: {
    domainId: "dashboard",
    counterType: "views",
    counterName: "dashboard-2",
    source: "server",
    timestamp: {
      gte: "2024-07-01T00:00:00.000Z",
      lte: "2024-07-05T00:00:00.000Z",
    }
  },
  aggregation_keys: [
    'timestamp'
  ]
});

And I'd get a response like

{
  counters: [
    { 
      domainId: "dashboard",
      counterType: "views",
      counterName: "dashboard-2"
      source: "server",
      timestamp: "2024-07-01T00:00:00.000Z"
      count: 10,
    },
    { 
      domainId: "dashboard",
      counterType: "views",
      counterName: "dashboard-2"
      source: "server",
      timestamp: "2024-07-03T00:00:00.000Z"
      count: 9999989,
    }
  ]
}

The benefits of the aggregations are:

• No pagination required
• Can be requested at any level

WDYT?

[gsoldevila]

It's an interesting proposal!

In the current use case, Anton still has to retrieve individual days counters to show them in the UI, so I went for the simplest approach possible, and let him aggregate counts on his side (saving 1 call in the process).

But I agree that it seems desirable to perform the aggregations on our side long term, makes for a more elegant API. Regarding pagination, when retrieving individual results you might still have plenty, but we currently circumvent this by allowing the from: string parameter. This way we can filter and only get counters that are more recent than a certain date (e.g. now - 90d).

Let's discuss this offline!

[afharo]

Regarding pagination, when retrieving individual results you might still have plenty, but we currently circumvent this by allowing the from: string parameter. This way we can filter and only get counters that are more recent than a certain date (e.g. now - 90d).

AFAIK, the recommended way to paginate is via PIT for various reasons:

1. The from + size technique is limited to 10_000 entries: https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html (no matter if they are 10 pages of 1000 or 1000 pages of 10)
2. If updates occur in the process, the list will be resorted, so getting the 2nd+ pages may return previously fetched documents if new documents are indexed during the pagination.

AFAIK, we'll always retrieve all values queried, since the intention is not to show these values in a table that the user can paginate. So I don't think we're saving ourselves from any potential issues.

Anton still has to retrieve individual days counters to show them in the UI

@Dosant, just FMI, will you retrieve all days for the histogram, and add them up to get the total? Or will the histogram be of the last 7 days and the total will account for the entire retention period (30d? 90d?)

Let's discuss this offline!

Sure! Happy to meet when you're back :)

[Dosant]

@afharo, I retrieve last 90 days using from. Sum them up to get a "total" to display "Views in last 90 days" and bucket into weeks to display a weekly histogram #187993

[gsoldevila]

After discussion with @afharo, we agreed that:

• We can leave aggregations for a later phase. I prepared the ground by encapsulating current filters under filters property.
• We must implement PIT search.

These changes have been implemented in d8e86e4

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 0240e5716a7a9547af04d37c8b598c068dcf59db

Failed CI Steps

• Jest Integration Tests #5

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
usageCollection	16	14	-2

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
usageCollection	2	4	+2

Unknown metric groups

API count

id	before	after	diff
usageCollection	56	50	-6

History

• 💚 Build #222027 succeeded 54c6a1252d2fc537072540eacedd2634a64a1b47
• 💚 Build #221977 succeeded b18505344cc7c115388daa9a5bb2dff9253c081b
• 💔 Build #221842 failed 27676f964e053139b5abb1ecca44df3eb525b3d4
• 💚 Build #221380 succeeded f9c4b4fcdee5c35a085e7a9c3e11c2cda4aae1ec
• 💔 Build #221330 failed f4e8e60e6bb69907d8b88cbdcdb428ea4fc759c4
• 💔 Build #221124 failed bd5f093367e32b29e1acc80b3c98096cb623d6fe

[afharo]

LGTM

[afharo]

nit: I think this is still UsageCollectionSetup. It's getting the entire plugin contract.
Prob what triggered is the name ICollectorSet. It sounds a bit confusing for a CollectorSet to be needed to create a usage collector...

WDYT?

[gsoldevila]

I thought about that for a while, and I think the root problem is "naming is hard".

IMO it would make more sense to name it CollectorManager instead of ICollectorSet.
Then, UsageCollectionSetup would have both the CollectorManager & UsageCountersServiceSetup.
NB the code above does not need anything about the UsageCounters.

The problem is that lots of plugins are already using the global UsageCollectionSetup, so changing all the references to the more specific CollectorManager would take some work + codeowners approvals.
I only changed this one cause it falls under our ownership.

I can rollback these changes and we can tackle this on a separate PR.

UPDATE: rolled back with b8abddb

[afharo]

🧡

[davismcphee]

Data Discovery changes LGTM

[Dosant]

x-pack/plugins/reporting/server/routes/ change due to interface changes lgtm.

I haven't rebased my frontend work yet to check if everything is still looking good, I'll try today or tomorrow. but don't want to block, up to you if you'd like to wait Looks good!

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: ea10c24

Failed CI Steps

• FTR Configs #19

Test Failures

• [job] [logs] FTR Configs #19 / Dataset Quality Dataset quality table filters shows full dataset names when toggled

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
usageCollection	16	14	-2

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
usageCollection	2	4	+2

Unknown metric groups

API count

id	before	after	diff
usageCollection	56	51	-5

History

• 💛 Build #225538 was flaky 9b9f007
• 💔 Build #225497 failed c62399f
• 💔 Build #225386 failed 357b474
• 💛 Build #225307 was flaky f58e23b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream