[Obs Agent] add observability get_trace_change_points tool by arturoliduena · Pull Request #247810 · elastic/kibana

arturoliduena · 2026-01-05T12:35:31Z

Closes https://github.com/elastic/obs-ai-team/issues/418

Summary

Add observability get_trace_change_points tool

Get trace change points tool

tool id: observability.get_trace_change_points
This tool analyzes traces to detect statistically significant change points in latency, throughput, and failure rate group by field example (default 'service.name'):
Service level: 'service.name', 'service.environment', 'service.version'
Transaction level: 'transaction.name', 'transaction.type'
Infrastructure level: 'host.name', 'container.id', 'kubernetes.pod.name'

Trace metrics:

Latency: avg/p95/p99 response time.
Throughput: requests per minute.
Failure rate: percentage of failed transactions.

Supports optional KQL filtering

elasticmachine · 2026-01-05T12:35:36Z

Pinging @elastic/obs-ai-team (Team:obs-ai)

sorenlouv · 2026-01-05T13:29:05Z

I also asked this in the issue: do we need this tool, when we have get_metric_change_points added in #242423?

The only reason for having a tool dedicate to traces is if it makes it (much) easier for the LLM to retrieve change points for trace metrics aka red metrics (latency, throughout, error rate).

Can you list how someone would do that using the get_metric_change_points and get_trace_change_points respectively?

sorenlouv · 2026-01-06T11:34:08Z

@arturoliduena Please take a look at @viduni94's PR in #247474. Specifically look at how getTraceMetrics is implemented. getPreferredDocumentSource and apmEventClient are used to query the right metric sets. getDurationFieldForTransactions and getOutcomeAggregation are used to query the right fields.

You should also read this comment to get a good understanding of the APM data model.

...servability/plugins/observability_agent_builder/server/tools/get_trace_change_points/tool.ts

elasticmachine · 2026-01-08T16:12:25Z

Pinging @elastic/obs-presentation-team (Team:obs-presentation)

...ck/solutions/observability/plugins/apm/server/agent_builder/tools/get_trace_metrics/index.ts

sorenlouv · 2026-01-08T21:42:34Z

...utions/observability/plugins/apm/server/agent_builder/tools/get_trace_change_points/index.ts

+                avg: {
+                  field: durationField,
+                },


You should also support p95 and p99

Done, updated here add support for avg|p95|p99 latency agg type

sorenlouv · 2026-01-09T16:31:47Z

...utions/observability/plugins/apm/server/agent_builder/tools/get_trace_change_points/index.ts

+              latency:
+                // Avoid unsupported aggregation on downsampled index, see example error:
+                // "reason": {
+                //   "type": "unsupported_aggregation_on_downsampled_index",
+                //   "reason": "Field [transaction.duration.summary] of type [aggregate_metric_double] is not supported for aggregation [percentiles]"
+                // }
+                durationField !== 'transaction.duration.summary' &&
+                (latencyType === 'p95' || latencyType === 'p99')
+                  ? {
+                      percentiles: {
+                        field: durationField,
+                        percents: [Number(`${latencyType.split('p')[1]}.0`)],
+                        keyed: true,
+                      },
+                    }
+                  : {
+                      avg: {
+                        field: durationField,
+                      },
+                    },


you can't calculate percentiles on the summary field. You need histogram or numeric latency field for that.
We already have this logic elsewhere. Can you re-use that?

couldn't find the logic that checks if dorationField is transaction.duration.summary can't use percentiles aggregation. I found some reusable code getLatencyAggregationType, getLatencyAggregation to make the code simpler:

const latencyAggregationType = getLatencyAggregationType(latencyType); durationField === TRANSACTION_DURATION_SUMMARY ? { avg: { field: durationField, }, } : getLatencyAggregation(latencyAggregationType, durationField)

sorenlouv · 2026-01-12T15:56:35Z

...utions/observability/plugins/apm/server/agent_builder/tools/get_trace_change_points/index.ts

+                durationField === TRANSACTION_DURATION_SUMMARY
+                  ? {
+                      avg: {
+                        field: durationField,
+                      },
+                    }
+                  : getLatencyAggregation(latencyAggregationType, durationField).latency,


This seems backwards.
Currently the logic is: if summary field is being used, calculate average regardless of what user requested.
Correct logic: if user requested percentiles, use histogram or numeric field.

it was that way before, but to reuse getLatencyAggregation and getLatencyAggregationType I changed the order, but isn't it the same? durationField can only be transaction.duration.summary, transaction.duration.histogram, or transaction.duration.us.
can't calculate percentiles for summary field, so only avg.
and getLatencyAggregation resolves the rest:

export function getLatencyAggregation( latencyAggregationType: LatencyAggregationType, field: string ) { return { latency: { ...(latencyAggregationType === LatencyAggregationType.avg ? { avg: { field } } : { percentiles: { field, percents: [latencyAggregationType === LatencyAggregationType.p95 ? 95 : 99], }, }), }, }; }

can't calculate percentiles for summary field, so only avg. and getLatencyAggregation resolves the rest:

I don't understand what this means.

We can’t calculate percentiles on thetransaction.duration.summary field. So, when using the summary field, the only supported aggregation is avg, right?

durationField === TRANSACTION_DURATION_SUMMARY ? { avg: { field: durationField, }, }

The other latency fields, transaction.duration.histogram and transaction.duration.us, support both avg and percentiles. It's handled by getLatencyAggregation.

We can’t calculate percentiles on thetransaction.duration.summary field. So, when using the summary field, the only supported aggregation is avg, right?

Yes, so if the user/LLM requests percentiles, you should not use the summary field.

got it, instead of defaulting to avg, we should use one of the other fields, change added here: c20786f

sorenlouv · 2026-01-13T10:51:05Z

...utions/observability/plugins/apm/server/agent_builder/tools/get_trace_change_points/index.ts

+  const useDurationSummaryField =
+    hasDurationSummaryField &&
+    latencyAggregationType !== LatencyAggregationType.p95 &&
+    latencyAggregationType !== LatencyAggregationType.p99;


Yes, this is the way to go :)

only nit: you could shorten this like:

Suggested change

const useDurationSummaryField =

hasDurationSummaryField &&

latencyAggregationType !== LatencyAggregationType.p95 &&

latencyAggregationType !== LatencyAggregationType.p99;

const useDurationSummaryField =

hasDurationSummaryField &&

latencyAggregationType === LatencyAggregationType.avg

But the other might be more clear

sorenlouv · 2026-01-13T11:00:11Z

...is/observability_agent_builder/utils/synthtrace_scenarios/create_trace_change_points_data.ts

+    });
+
+  await apmSynthtraceEsClient.index([traceStream]);
+}


Please move this to src/platform/packages/shared/kbn-synthtrace/src/scenarios/agent_builder/tools. It should also be available for CLI usage like:

export default createCliScenario(({ range, clients: { logsEsClient } }) => generateTraceChangePointsData({ range, logsEsClient }) );

Will do it in a follow-up issue: https://github.com/elastic/obs-ai-team/issues/467

sorenlouv · 2026-01-13T11:03:29Z

@arturoliduena can you hold off merging this until @viduni94 has merged #248513? Then you should be able to move the implementation of the get_trace_metrics tool to observability_agent_builder plugin

rmyz

presentation changes LGTM

viduni94 · 2026-01-13T13:51:19Z

@arturoliduena can you hold off merging this until @viduni94 has merged #248513? Then you should be able to move the implementation of the get_trace_metrics tool to observability_agent_builder plugin

@arturoliduena My PR is merged.

elasticmachine · 2026-01-15T10:28:16Z

💔 Build Failed

Buildkite Build
Commit: 2599523

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #81 / discover/esql discover esql controls when adding an ES|QL panel with controls in dashboards and exploring it in discover should retain the controls and their state
[job] [logs] FTR Configs #80 / Endpoint plugin spaces support "after all" hook in "Endpoint plugin spaces support"
[job] [logs] FTR Configs #80 / Endpoint plugin spaces support "after all" hook in "Endpoint plugin spaces support"
[job] [logs] FTR Configs #80 / Endpoint plugin spaces support "after all" hook in "Endpoint plugin spaces support"
[job] [logs] FTR Configs #80 / Endpoint plugin spaces support "before all" hook in "Endpoint plugin spaces support"
[job] [logs] FTR Configs #80 / Endpoint plugin spaces support "before all" hook in "Endpoint plugin spaces support"
[job] [logs] FTR Configs #80 / Endpoint plugin spaces support "before all" hook in "Endpoint plugin spaces support"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`apm`	2089	2090	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`apmDataAccess`	86	89	+3

Unknown metric groups

API count

id	before	after	diff
`apmDataAccess`	86	89	+3

History

sorenlouv · 2026-01-15T15:01:01Z

x-pack/solutions/observability/plugins/apm_data_access/common/latency_aggregation_types.ts

+export enum LatencyAggregationType {
+  avg = 'avg',
+  p99 = 'p99',
+  p95 = 'p95',
+}
+
+export const getLatencyAggregationType = (
+  latencyAggregationType: string | null | undefined
+): LatencyAggregationType => {
+  return (latencyAggregationType ?? LatencyAggregationType.avg) as LatencyAggregationType;
+};


If it's not too much work, please remove so they are not duplicated

kibana/x-pack/solutions/observability/plugins/apm/common/latency_aggregation_types.ts

Lines 10 to 26 in 8899fb8

export enum LatencyAggregationType {

avg = 'avg',

p99 = 'p99',

p95 = 'p95',

}

export const latencyAggregationTypeRt = t.union([

t.literal(LatencyAggregationType.avg),

t.literal(LatencyAggregationType.p95),

t.literal(LatencyAggregationType.p99),

]);

export const getLatencyAggregationType = (

latencyAggregationType: string | null | undefined

): LatencyAggregationType => {

return (latencyAggregationType ?? LatencyAggregationType.avg) as LatencyAggregationType;

};

Also ok to do that in a follow-up if that's easier. Might cause a bunch of changes in APM

removed: ad0be75

sorenlouv · 2026-01-15T15:02:06Z

...s/observability/plugins/apm_data_access/server/lib/helpers/latency_aggregation_type/index.ts

+
+import { LatencyAggregationType } from '../../../../common/latency_aggregation_types';
+
+export function getLatencyAggregation(


Same

kibana/x-pack/solutions/observability/plugins/apm/server/lib/helpers/latency_aggregation_type/index.ts

Lines 10 to 26 in d263ce5

export function getLatencyAggregation(

latencyAggregationType: LatencyAggregationType,

field: string

) {

return {

latency: {

...(latencyAggregationType === LatencyAggregationType.avg

? { avg: { field } }

: {

percentiles: {

field,

percents: [latencyAggregationType === LatencyAggregationType.p95 ? 95 : 99],

},

}),

},

};

}

remove: ad0be75

...rvability/plugins/observability_agent_builder/server/tools/get_trace_change_points/README.md

...vability/plugins/observability_agent_builder/server/tools/get_trace_change_points/handler.ts

…lder/server/tools/get_trace_change_points/README.md Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>

kibanamachine · 2026-01-16T11:29:45Z

Starting backport for target branches: 9.3

https://github.com/elastic/kibana/actions/runs/21065163474

kibanamachine · 2026-01-16T11:37:15Z

💔 All backports failed

Status	Branch	Result
❌	9.3	Backport failed because of merge conflicts You might need to backport the following PRs to 9.3: - [Obs AI] Replace `get_data_sources` with `get_index_info` tool (#248234) - [Obs AI] Extend `get_services` tool and add `get_trace_metrics` tool (#247474)

Manual backport

To create the backport manually run:

node scripts/backport --pr 247810

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2026-01-19T11:48:57Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 247810 locally
cc: @arturoliduena

kibanamachine · 2026-01-20T13:49:14Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 247810 locally
cc: @arturoliduena

kibanamachine · 2026-01-21T14:49:20Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 247810 locally
cc: @arturoliduena

arturoliduena requested review from a team as code owners January 5, 2026 12:35

arturoliduena added release_note:feature Makes this part of the condensed release notes backport:version Backport to applied version labels v9.3.0 Team:obs-ai Observability AI team labels Jan 5, 2026

sorenlouv reviewed Jan 6, 2026

View reviewed changes

...servability/plugins/observability_agent_builder/server/tools/get_trace_change_points/tool.ts Outdated Show resolved Hide resolved

sorenlouv reviewed Jan 6, 2026

View reviewed changes

...servability/plugins/observability_agent_builder/server/tools/get_trace_change_points/tool.ts Outdated Show resolved Hide resolved

arturoliduena force-pushed the obs-agent-418-trace-change-points-tool branch from 09208a9 to a276773 Compare January 8, 2026 16:12

arturoliduena requested a review from a team as a code owner January 8, 2026 16:12

botelastic bot added the Team:obs-presentation Focus: APM UI, Infra UI, Hosts UI, Universal Profiling, Obs Overview and left Navigation label Jan 8, 2026

sorenlouv reviewed Jan 8, 2026

View reviewed changes

...ck/solutions/observability/plugins/apm/server/agent_builder/tools/get_trace_metrics/index.ts Outdated Show resolved Hide resolved

sorenlouv reviewed Jan 8, 2026

View reviewed changes

sorenlouv reviewed Jan 9, 2026

View reviewed changes

sorenlouv reviewed Jan 12, 2026

View reviewed changes

sorenlouv reviewed Jan 13, 2026

View reviewed changes

sorenlouv approved these changes Jan 13, 2026

View reviewed changes

sorenlouv mentioned this pull request Jan 13, 2026

Move helpers required for trace metrics to the apm-data-access plugin #248513

Merged

2 tasks

arturoliduena changed the title ~~[Obs Agent] add observability get_trace_change_points tool~~ [Obs Agent] add observability get_trace_change_points tool Jan 13, 2026

arturoliduena changed the title ~~[Obs Agent] add observability get_trace_change_points tool~~ [Obs Agent] add observability get_trace_change_points tool Jan 13, 2026

pgayvallet approved these changes Jan 13, 2026

View reviewed changes

rmyz approved these changes Jan 13, 2026

View reviewed changes

joemcelroy approved these changes Jan 13, 2026

View reviewed changes

kibanamachine and others added 6 commits January 14, 2026 11:49

Changes from node scripts/lint_ts_projects --fix

751d5dd

Changes from node scripts/regenerate_moon_projects.js --update

7f4520f

Merge branch 'main' into obs-agent-418-trace-change-points-tool

da284e1

Merge branch 'main' into obs-agent-418-trace-change-points-tool

5faa5c3

Changes from node scripts/eslint_all_files --no-cache --fix

b8f2245

fix circular dependency

2599523

arturoliduena enabled auto-merge (squash) January 15, 2026 09:54

Merge branch 'main' into obs-agent-418-trace-change-points-tool

186cfa7

sorenlouv reviewed Jan 15, 2026

View reviewed changes

...rvability/plugins/observability_agent_builder/server/tools/get_trace_change_points/README.md Outdated Show resolved Hide resolved

arturoliduena disabled auto-merge January 15, 2026 15:05

sorenlouv reviewed Jan 15, 2026

View reviewed changes

...vability/plugins/observability_agent_builder/server/tools/get_trace_change_points/handler.ts Outdated Show resolved Hide resolved

sorenlouv reviewed Jan 15, 2026

View reviewed changes

...vability/plugins/observability_agent_builder/server/tools/get_trace_change_points/handler.ts Outdated Show resolved Hide resolved

arturoliduena and others added 2 commits January 16, 2026 09:56

Update x-pack/solutions/observability/plugins/observability_agent_bui…

ffd8127

…lder/server/tools/get_trace_change_points/README.md Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>

remove duplicated code

ad0be75

arturoliduena enabled auto-merge (squash) January 16, 2026 09:54

arturoliduena merged commit 5cfdbab into elastic:main Jan 16, 2026
13 checks passed

kibanamachine added the v9.4.0 label Jan 16, 2026

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jan 19, 2026

arturoliduena removed the v9.3.0 label Jan 19, 2026

arturoliduena added backport:skip This PR does not require backporting and removed backport missing Added to PRs automatically when the are determined to be missing a backport. backport:version Backport to applied version labels labels Jan 22, 2026

	export enum LatencyAggregationType {
	avg = 'avg',
	p99 = 'p99',
	p95 = 'p95',
	}

	export const latencyAggregationTypeRt = t.union([
	t.literal(LatencyAggregationType.avg),
	t.literal(LatencyAggregationType.p95),
	t.literal(LatencyAggregationType.p99),
	]);

	export const getLatencyAggregationType = (
	latencyAggregationType: string \| null \| undefined
	): LatencyAggregationType => {
	return (latencyAggregationType ?? LatencyAggregationType.avg) as LatencyAggregationType;
	};


		import { LatencyAggregationType } from '../../../../common/latency_aggregation_types';

		export function getLatencyAggregation(

	export function getLatencyAggregation(
	latencyAggregationType: LatencyAggregationType,
	field: string
	) {
	return {
	latency: {
	...(latencyAggregationType === LatencyAggregationType.avg
	? { avg: { field } }
	: {
	percentiles: {
	field,
	percents: [latencyAggregationType === LatencyAggregationType.p95 ? 95 : 99],
	},
	}),
	},
	};
	}

Conversation

arturoliduena commented Jan 5, 2026 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Get trace change points tool

Uh oh!

elasticmachine commented Jan 5, 2026

Uh oh!

sorenlouv commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sorenlouv commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jan 8, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arturoliduena Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sorenlouv Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arturoliduena Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sorenlouv commented Jan 13, 2026

Uh oh!

rmyz left a comment

Choose a reason for hiding this comment

Uh oh!

viduni94 commented Jan 13, 2026

Uh oh!

elasticmachine commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

API count

History

Uh oh!

sorenlouv Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

arturoliduena commented Jan 5, 2026 •

edited by kibanamachine

Loading

sorenlouv commented Jan 5, 2026 •

edited

Loading

sorenlouv commented Jan 6, 2026 •

edited

Loading

arturoliduena Jan 9, 2026 •

edited

Loading

sorenlouv Jan 13, 2026 •

edited

Loading

arturoliduena Jan 16, 2026 •

edited

Loading

elasticmachine commented Jan 15, 2026 •

edited

Loading

sorenlouv Jan 15, 2026 •

edited

Loading