feat: limit tags and tag values search

[javiermolinar]

What this PR does:
It adds a new parameter to control the number of tag values returned. This allows the requester to limit the request, speeding up the response.

For the tag/values it limits the max number of values per tag:
http://localhost:3200/api/v2/search/tag/.http.status_code/values?limit=1

For the tag endpoint, the limit controls how many tags are returned per scope. Intrinsic scope is unbounded:

http://localhost:3200/api/v2/search/tags?limit=2

{
  "scopes": [
    {
      "name": "event",
      "tags": [
        "exception.escape"
      ]
    },
    {
      "name": "intrinsic",
      "tags": [
        "duration",
        "event:name",
        "event:timeSinceStart",
        "instrumentation:name",
      ....
      ]
    },
    {
      "name": "link",
      "tags": [
        "k6.97IMph1UKjxwJmE"
      ]
    },
  ....
}

Which issue(s) this PR fixes:
Fixes #4288

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[joe-elliott]

i'd also like to get your thoughts on a different kind of parameter. perhaps it could be added here as well?

what about a parameter that stopped searching early if N strings were added to the combiner and none were new? so the combiner would keep a running count of the number of added strings since a new one was found and stop searching once we crossed threshold.

consider /api/v2/search/tag/resource.cluster/values?q=<some very expensive query>. there are likely very few values for resource.cluster but we still spend a lot of resources to exhaustively evaluate <some very expensive query>. a parameter that allowed the ingester to just stop searching after N found values and nothing new could be a powerful way to dynamically circuit break.

[joe-elliott]

+1 :)

[joe-elliott]

the one concerning behavior in this PR is how it interacts with multiple scopes. if the user queries /api/v2/search/tags?limit=100 it's quite possible they don't have 100 resource tags, but have thousands of span level tags. the search would still be exhaustive b/c the 100 limit for the resource scope would never be hit.

can we come up with some rules about how to apply the limit 100 across all scopes? for instance maybe resource tags are always added first and then span tags next?

maybe we just don't add the limit parameter to the tag name endpoints? they tend to be far faster so the limit is less of a concern?

i know we discussed this behavior and agreed to this, but i'm struggling with it now that i'm looking through the PR. wdyt?

[javiermolinar]

I thought about that as well, if we split the limit between tags (in the same way the maxSize is working right now) we could end with an incomplete list of span tags and none of the resource tags. The way this limit works now cannot be worse than the current implementation but in the worst-case scenario it will end with an exhaustive search.

I'm not opposed of having a deterministic search by scope.

[joe-elliott]

The best pattern I can come up with for a tag names limit is to stop searching if ANY scope exceeds the limit instead of ALL scopes.

[knylander-grafana]

Suggested change

	Optional. Limits the maximum number of tags
	Optional. Limits the maximum number of tags.

[knylander-grafana]

Suggested change

	Optional. Limits the maximum number of tags per scope
	Optional. Limits the maximum number of tags per scope.

[knylander-grafana]

Suggested change

	Optional. Limits the maximum number of tags values
	Optional. Limits the maximum number of tags values.

[knylander-grafana]

Thank you for updating the docs! I have a few minor suggestions to correct missing periods.

[joe-elliott]

The more I think about this idea the more I want it:

what about a parameter that stopped searching early if N strings were added to the combiner and none were new? so the combiner would keep a running count of the number of added strings since a new one was found and stop searching once we crossed threshold.

The change here and the current max_bytes_per_tag_values_query protect us when a tag value query finds A LOT of matches. We currently have no protection for an expensive tag values query that find a very low number of values.