ES|QL: Add (stack) telemetry for views by luigidellaquila · Pull Request #143745 · elastic/elasticsearch

luigidellaquila · 2026-03-06T14:07:15Z

Adding Telemetry (stack) for views usage.

We just increment the counter every time a query uses views, ie. we don't track how many views a single query is using.
Developed using AI-assisted tooling

elasticsearchmachine · 2026-03-06T14:08:23Z

Hi @luigidellaquila, I've created a changelog YAML for you.

…ack' into esql/views_telemetry_stack

elasticsearchmachine · 2026-03-09T09:43:15Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/telemetry/VerifierMetricsTests.java

astefan · 2026-03-13T11:18:36Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Subquery.java

        return child();
    }
+
+    public String name() {


If the original code kept name private and, also, to have an easier check for this - why not create a boolean method in Subquery that checks the name existence? isView or something similar.

Wondering also why Subquery was not extended with a new class - View - that added the name. It feels very fragile to check name inside Subquery to discover if a plan is or is not a View. My 2c.
Maybe @craigtaverner can weigh in on this aspect (unrelated to this PR).

Thanks @astefan, I'm adding an isView() method, it will make the code much more readable, and maybe it will simplify a future refactoring.
I agree that having a subclass specific for views could make things more clear

It reminds me of this PR, it adds ViewUnionAll that extends UnionAll. I wonder if it affects view and subquery's telemetry?

I'm updating the PR at #143564 to remove the name field from Subquery. Instead the name is being saved inside the ViewUnionAll class, as @fang-xing-esql mentioned. I do actually also have a NamedSubquery class extending Subquery, but this is only ephemeral, and should not exist in the final plan.

That PR merged, and so Subquery no longer has names.

We should remove these changes to Subquery.

astefan · 2026-03-13T11:19:34Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

+        if (metrics == null || viewResolution.viewQueries().isEmpty()) {
+            return;
+        }
+        // If any views were used, increment the views metric


I don't think this comment helps the readability of the code. I think it is obvious what the next line does :-).

astefan · 2026-03-13T11:30:46Z

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/telemetry/VerifierMetricsTests.java

+    public void testSubqueryTelemetry() {
+        Row innerPlan = new Row(Source.EMPTY, List.of());
+
+        // A named Subquery represents a view - it should be excluded from plan traversal telemetry


Imho, this method also has too many comments.

craigtaverner

I think the changes to Subquery will make this easier. Let's get that in and simplify this one.

craigtaverner · 2026-03-13T13:29:34Z

docs/changelog/143745.yaml

+issues: []
+pr: 143745
+summary: Add (stack) telemetry for views
+type: enhancement


I'm wondering, do we add changelog entries for unreleased (snapshot-ony) features? Perhaps the label should be >non-issue?

craigtaverner · 2026-03-13T13:31:01Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

    COMPLETION(Completion.class::isInstance),
    SAMPLE(Sample.class::isInstance),
-    SUBQUERY(Subquery.class::isInstance),
+    SUBQUERY(plan -> plan instanceof Subquery s && s.isView() == false),


I'm removing the use of Subquery in views, so we won't need to use the isView method.

Subquery is now entirely removed from views.

craigtaverner · 2026-03-13T13:31:37Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

    SAMPLE(Sample.class::isInstance),
-    SUBQUERY(Subquery.class::isInstance),
+    SUBQUERY(plan -> plan instanceof Subquery s && s.isView() == false),
+    VIEWS(plan -> false), // Views are counted in EsqlSession.gatherViewMetrics, not via plan traversal


Well, actually we could use plan traversal once the ViewUnionAll is implemented.

Changed my mind. The EsqlSession.gatherViewMetrics is the way to go.

craigtaverner · 2026-03-13T13:31:59Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

    }

    private static boolean explicitlyExcluded(LogicalPlan plan) {
+        if (plan instanceof Subquery s && s.isView()) {


We can probably drop this too.

Yes, drop this.

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.yml

craigtaverner

I approve of the EsqlSession.gatherViewMetrics approach. But we need to revert the changes to Subquery, since that is no longer used in Views at all.

craigtaverner · 2026-03-24T11:04:24Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

    }

+    private void gatherViewMetrics(ViewResolver.ViewResolutionResult viewResolution) {
+        if (metrics == null || viewResolution.viewQueries().isEmpty()) {


This does look like the best approach for now, since we cannot rely on Subquery.name() nor on the existence of ViewUnionAll, both of which can be empty/missing due to query simplification/compaction.

craigtaverner · 2026-03-24T11:04:50Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Subquery.java

        return child();
    }
+
+    public String name() {


We should remove these changes to Subquery.

craigtaverner · 2026-03-24T11:06:07Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

    COMPLETION(Completion.class::isInstance),
    SAMPLE(Sample.class::isInstance),
-    SUBQUERY(Subquery.class::isInstance),
+    SUBQUERY(plan -> plan instanceof Subquery s && s.isView() == false),


Subquery is now entirely removed from views.

craigtaverner · 2026-03-24T11:06:46Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

    SAMPLE(Sample.class::isInstance),
-    SUBQUERY(Subquery.class::isInstance),
+    SUBQUERY(plan -> plan instanceof Subquery s && s.isView() == false),
+    VIEWS(plan -> false), // Views are counted in EsqlSession.gatherViewMetrics, not via plan traversal


Changed my mind. The EsqlSession.gatherViewMetrics is the way to go.

craigtaverner · 2026-03-24T11:07:03Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

    }

    private static boolean explicitlyExcluded(LogicalPlan plan) {
+        if (plan instanceof Subquery s && s.isView()) {


Yes, drop this.

craigtaverner · 2026-03-24T11:07:47Z

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/telemetry/VerifierMetricsTests.java

+    public void testSubqueryTelemetry() {
+        Row innerPlan = new Row(Source.EMPTY, List.of());
+
+        Subquery view = new Subquery(Source.EMPTY, innerPlan, "my_view");


Subquery is no longer used for views, so this should be changed.

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.yml

craigtaverner

LGTM, although I am wondering about the use of a plural name while all other counters use singular?

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.yml

craigtaverner · 2026-03-25T09:24:58Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

    COMPLETION(Completion.class::isInstance),
    SAMPLE(Sample.class::isInstance),
    SUBQUERY(Subquery.class::isInstance),
+    VIEWS(plan -> false), // Views are counted in EsqlSession.gatherViewMetrics, not via plan traversal


Wondering about the use of plural here, while others mostly use singular.

Fixed, thanks!

craigtaverner · 2026-03-25T09:25:53Z

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_usage.yml

  - set: { esql.features.completion: completion_counter }
  - set: { esql.features.sample: sample_counter }
  - set: { esql.features.subquery: subquery_counter }
+  - set: { esql.features.views: views_counter }


I just noticed almost all other counters use the singular, like subquery_counter, and we are almost the only one using plural. Should we switch to singular?

ES|QL: Add (stack) telemetry for views

66d3e8a

luigidellaquila added >enhancement test-release Trigger CI checks against release build :Analytics/ES|QL AKA ESQL labels Mar 6, 2026

elasticsearchmachine added the v9.4.0 label Mar 6, 2026

Update docs/changelog/143745.yaml

a4217fa

luigidellaquila added 3 commits March 6, 2026 16:05

Merge branch 'main' into esql/views_telemetry_stack

83fe22a

Merge branch 'main' into esql/views_telemetry_stack

bcb419d

Merge branch 'main' into esql/views_telemetry_stack

d9a8710

luigidellaquila removed the test-release Trigger CI checks against release build label Mar 9, 2026

luigidellaquila added 2 commits March 9, 2026 10:41

Fix integration tests

3a6c047

Merge remote-tracking branch 'luigidellaquila/esql/views_telemetry_st…

6999a9c

…ack' into esql/views_telemetry_stack

luigidellaquila marked this pull request as ready for review March 9, 2026 09:42

luigidellaquila requested a review from craigtaverner March 9, 2026 09:42

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 9, 2026

Merge branch 'main' into esql/views_telemetry_stack

c3beb16

astefan self-requested a review March 13, 2026 10:41

idegtiarenko approved these changes Mar 13, 2026

View reviewed changes

ivancea approved these changes Mar 13, 2026

View reviewed changes

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/telemetry/VerifierMetricsTests.java Outdated Show resolved Hide resolved

Fix tests

a995853

astefan reviewed Mar 13, 2026

View reviewed changes

Implement review suggestions

25fe966

craigtaverner requested changes Mar 13, 2026

View reviewed changes

craigtaverner requested changes Mar 24, 2026

View reviewed changes

luigidellaquila added 3 commits March 24, 2026 17:45

Merge branch 'main' into esql/views_telemetry_stack

ed64865

No subquery

97f54f8

Merge branch 'main' into esql/views_telemetry_stack

5100237

luigidellaquila requested a review from craigtaverner March 25, 2026 08:42

Capability

85d83ef

luigidellaquila added >non-issue and removed >enhancement labels Mar 25, 2026

luigidellaquila added 2 commits March 25, 2026 10:12

Delete docs/changelog/143745.yaml

7b044cf

Capabilities

3029134

craigtaverner approved these changes Mar 25, 2026

View reviewed changes

singular feature name

08da79f

luigidellaquila enabled auto-merge (squash) March 25, 2026 09:37

Merge branch 'main' into esql/views_telemetry_stack

9867da6

luigidellaquila merged commit e805d5d into elastic:main Mar 25, 2026
36 checks passed

seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 27, 2026

ES|QL: Add (stack) telemetry for views (elastic#143745)

d2bfbd3

Conversation

luigidellaquila commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 6, 2026

Uh oh!

elasticsearchmachine commented Mar 9, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

craigtaverner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

craigtaverner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

craigtaverner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

luigidellaquila commented Mar 6, 2026 •

edited

Loading