[Inference] Instrument inference with OpenTelemetry#218694
Merged
dgieselaar merged 18 commits intoelastic:mainfrom May 7, 2025
Merged
[Inference] Instrument inference with OpenTelemetry#218694dgieselaar merged 18 commits intoelastic:mainfrom
dgieselaar merged 18 commits intoelastic:mainfrom
Conversation
dgieselaar
force-pushed
the
inference-output-langfuse
branch
10 times, most recently
from
f3ab323 to
be2ebc6
Compare
dgieselaar
force-pushed
the
inference-output-langfuse
branch
2 times, most recently
from
adbf4b7 to
90412a2
Compare
dgieselaar
added
backport:version
dgieselaar
force-pushed
the
inference-output-langfuse
branch
from
90412a2 to
9f5c364
Compare
dgieselaar
requested review from
a team,
kibanamachine and
vigneshshanmugam
as code owners
botelastic
bot
added
ci:project-deploy-observability
| * @param serviceName The service name used in resource attributes | ||
| * @returns A function that can be called on shutdown and allows exporters to flush their queue. | ||
| */ | ||
| export const initTelemetry = ( |
Member
There was a problem hiding this comment.
@dgieselaar Is there a problem you see when we have both OTel SDK and APM Node.js agent active at the same time? Should we set active flag to false when we enable OTel SDK?
Contributor
Author
There was a problem hiding this comment.
All auto-instrumentations for @opentelemetry should be disabled. Only spans in the context of Inference calls (chatComplete) should be recorded and exported, which are relatively low in volume. The elasticsearch-js client always creates spans via Transport, so those will be recorded as well - kind of accidentally, I'm not sure that's how it should work, but in this case also convenient. So they should be able to run in parallel without any problems, but if we migrate more stuff to OTel I think we should consider one or the other.
Member
There was a problem hiding this comment.
Gotcha, thanks for the context and agree we need to move towards the OTel SDK soonish.
Contributor
Author
|
@elasticmachine merge upstream |
afharo
approved these changes
May 6, 2025
Contributor
⏳ Build in-progress
History
|
Contributor
|
Starting backport for target branches: 8.19 https://github.com/elastic/kibana/actions/runs/14880279077 |
Contributor
💔 Backport failedThe pull request could not be backported due to the following error: Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
dgieselaar
added a commit
to dgieselaar/kibana
that referenced
this pull request
May 7, 2025
Instrument the inference chatComplete API with OpenTelemetry, and export helper functions to create spans w/ the right semconv attributes. Additionally, optionally export to Langfuse or Phoenix. As this is the first instance of OpenTelemetry based _tracing_ (we already have metrics in the MonitoringCollection plugin), some bootstrapping code is necessary to centrally configure OpenTelemetry. To this end, I've added the following config settings: - `telemetry.tracing.enabled`: whether OpenTelemetry tracing is enabled (defaults to undefined, if undefined, falls back to `telemetry.enabled`) - `telemetry.tracing.sample_rate` (defaults to 1) The naming of these configuration settings is mostly in-line with [the Elasticsearch tracing settings](https://github.com/elastic/elasticsearch/blob/main/TRACING.md). The following packages (containing bootstrapping logic, utility functions, types and config schemas) were added: - `@kbn/telemetry` - `@kbn/telemetry-config` - `@kbn/tracing` The OpenTelemetry bootstrapping depends on @kbn/apm-config-loader, as it has the same constraints - it needs to run before any other code, and it needs to read the raw config. Additionally, a root `telemetry` logger was added that captures OpenTelemetry logs. Note that there is no default exporter for spans, which means that although spans are being recorded, they do not get exported. Calls to `chatComplete` now create OpenTelemetry spans, roughly following semantic conventions (which for GenAI are very much in flux). Some helper functions were added to create other inference spans. These helper functions use baggage to determine whether the created inference span is the "root" of an inference trace. This allows us to export these spans as if it were root spans - something that is needed to be able to easily visualize these in other tools. Leveraging these inference spans, two exporters are added. One for [Phoenix](https://github.com/Arize-ai/phoenix) and one for [Langfuse](https://github.com/langfuse/langfuse/tree/main): two open-source LLM Observability suites. This allows engineers that use the Inference plugin to be able to inspect and improve their LLM-based workflows with much less effort. For both Phoenix and Langfuse, two service scripts were added. Run `node scripts/phoenix` or `node scripts/langfuse` to get started. Both scripts work with zero-config - they will log generated Kibana config to stdout. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 2387e3b)
dgieselaar
added a commit
to dgieselaar/kibana
that referenced
this pull request
May 7, 2025
Instrument the inference chatComplete API with OpenTelemetry, and export helper functions to create spans w/ the right semconv attributes. Additionally, optionally export to Langfuse or Phoenix. ## Centralizes OpenTelemetry setup As this is the first instance of OpenTelemetry based _tracing_ (we already have metrics in the MonitoringCollection plugin), some bootstrapping code is necessary to centrally configure OpenTelemetry. To this end, I've added the following config settings: - `telemetry.tracing.enabled`: whether OpenTelemetry tracing is enabled (defaults to undefined, if undefined, falls back to `telemetry.enabled`) - `telemetry.tracing.sample_rate` (defaults to 1) The naming of these configuration settings is mostly in-line with [the Elasticsearch tracing settings](https://github.com/elastic/elasticsearch/blob/main/TRACING.md). The following packages (containing bootstrapping logic, utility functions, types and config schemas) were added: - `@kbn/telemetry` - `@kbn/telemetry-config` - `@kbn/tracing` The OpenTelemetry bootstrapping depends on @kbn/apm-config-loader, as it has the same constraints - it needs to run before any other code, and it needs to read the raw config. Additionally, a root `telemetry` logger was added that captures OpenTelemetry logs. Note that there is no default exporter for spans, which means that although spans are being recorded, they do not get exported. ## Instrument chatComplete calls Calls to `chatComplete` now create OpenTelemetry spans, roughly following semantic conventions (which for GenAI are very much in flux). Some helper functions were added to create other inference spans. These helper functions use baggage to determine whether the created inference span is the "root" of an inference trace. This allows us to export these spans as if it were root spans - something that is needed to be able to easily visualize these in other tools. Leveraging these inference spans, two exporters are added. One for [Phoenix](https://github.com/Arize-ai/phoenix) and one for [Langfuse](https://github.com/langfuse/langfuse/tree/main): two open-source LLM Observability suites. This allows engineers that use the Inference plugin to be able to inspect and improve their LLM-based workflows with much less effort. For both Phoenix and Langfuse, two service scripts were added. Run `node scripts/phoenix` or `node scripts/langfuse` to get started. Both scripts work with zero-config - they will log generated Kibana config to stdout. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 2387e3b) # Conflicts: # .github/CODEOWNERS # renovate.json # src/cli/tsconfig.json # src/platform/plugins/shared/telemetry/server/config/config.ts # x-pack/platform/plugins/shared/observability_ai_assistant/server/service/client/index.ts # yarn.lock
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
This was referenced May 7, 2025
kibanamachine
added
the
backport missing
Contributor
|
Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync. |
dgieselaar
added a commit
that referenced
this pull request
May 9, 2025
…220349) # Backport This will backport the following commits from `main` to `8.19`: - [[Inference] Instrument inference with OpenTelemetry (#218694)](#218694) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Dario Gieselaar","email":"dario.gieselaar@elastic.co"},"sourceCommit":{"committedDate":"2025-05-07T09:44:29Z","message":"[Inference] Instrument inference with OpenTelemetry (#218694)\n\nInstrument the inference chatComplete API with OpenTelemetry, and export\nhelper functions to create spans w/ the right semconv attributes.\nAdditionally, optionally export to Langfuse or Phoenix.\n\n## Centralizes OpenTelemetry setup\n\nAs this is the first instance of OpenTelemetry based _tracing_ (we\nalready have metrics in the MonitoringCollection plugin), some\nbootstrapping code is necessary to centrally configure OpenTelemetry. To\nthis end, I've added the following config settings:\n\n- `telemetry.tracing.enabled`: whether OpenTelemetry tracing is enabled\n(defaults to undefined, if undefined, falls back to `telemetry.enabled`)\n- `telemetry.tracing.sample_rate` (defaults to 1)\n\nThe naming of these configuration settings is mostly in-line with [the\nElasticsearch tracing\nsettings](https://github.com/elastic/elasticsearch/blob/main/TRACING.md).\n\nThe following packages (containing bootstrapping logic, utility\nfunctions, types and config schemas) were added:\n- `@kbn/telemetry`\n- `@kbn/telemetry-config` \n- `@kbn/tracing`\n\nThe OpenTelemetry bootstrapping depends on @kbn/apm-config-loader, as it\nhas the same constraints - it needs to run before any other code, and it\nneeds to read the raw config.\n\nAdditionally, a root `telemetry` logger was added that captures\nOpenTelemetry logs.\n\nNote that there is no default exporter for spans, which means that\nalthough spans are being recorded, they do not get exported.\n\n## Instrument chatComplete calls\n\nCalls to `chatComplete` now create OpenTelemetry spans, roughly\nfollowing semantic conventions (which for GenAI are very much in flux).\nSome helper functions were added to create other inference spans. These\nhelper functions use baggage to determine whether the created inference\nspan is the \"root\" of an inference trace. This allows us to export these\nspans as if it were root spans - something that is needed to be able to\neasily visualize these in other tools.\n\nLeveraging these inference spans, two exporters are added. One for\n[Phoenix](https://github.com/Arize-ai/phoenix) and one for\n[Langfuse](https://github.com/langfuse/langfuse/tree/main): two\nopen-source LLM Observability suites. This allows engineers that use the\nInference plugin to be able to inspect and improve their LLM-based\nworkflows with much less effort.\n\nFor both Phoenix and Langfuse, two service scripts were added. Run `node\nscripts/phoenix` or `node scripts/langfuse` to get started. Both scripts\nwork with zero-config - they will log generated Kibana config to stdout.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"2387e3b88c83317ddd5354dff751a1ac0fb62bb9","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Obs AI Assistant","ci:project-deploy-observability","backport:version","v9.1.0","v8.19.0"],"title":"[Inference] Instrument inference with OpenTelemetry","number":218694,"url":"https://github.com/elastic/kibana/pull/218694","mergeCommit":{"message":"[Inference] Instrument inference with OpenTelemetry (#218694)\n\nInstrument the inference chatComplete API with OpenTelemetry, and export\nhelper functions to create spans w/ the right semconv attributes.\nAdditionally, optionally export to Langfuse or Phoenix.\n\n## Centralizes OpenTelemetry setup\n\nAs this is the first instance of OpenTelemetry based _tracing_ (we\nalready have metrics in the MonitoringCollection plugin), some\nbootstrapping code is necessary to centrally configure OpenTelemetry. To\nthis end, I've added the following config settings:\n\n- `telemetry.tracing.enabled`: whether OpenTelemetry tracing is enabled\n(defaults to undefined, if undefined, falls back to `telemetry.enabled`)\n- `telemetry.tracing.sample_rate` (defaults to 1)\n\nThe naming of these configuration settings is mostly in-line with [the\nElasticsearch tracing\nsettings](https://github.com/elastic/elasticsearch/blob/main/TRACING.md).\n\nThe following packages (containing bootstrapping logic, utility\nfunctions, types and config schemas) were added:\n- `@kbn/telemetry`\n- `@kbn/telemetry-config` \n- `@kbn/tracing`\n\nThe OpenTelemetry bootstrapping depends on @kbn/apm-config-loader, as it\nhas the same constraints - it needs to run before any other code, and it\nneeds to read the raw config.\n\nAdditionally, a root `telemetry` logger was added that captures\nOpenTelemetry logs.\n\nNote that there is no default exporter for spans, which means that\nalthough spans are being recorded, they do not get exported.\n\n## Instrument chatComplete calls\n\nCalls to `chatComplete` now create OpenTelemetry spans, roughly\nfollowing semantic conventions (which for GenAI are very much in flux).\nSome helper functions were added to create other inference spans. These\nhelper functions use baggage to determine whether the created inference\nspan is the \"root\" of an inference trace. This allows us to export these\nspans as if it were root spans - something that is needed to be able to\neasily visualize these in other tools.\n\nLeveraging these inference spans, two exporters are added. One for\n[Phoenix](https://github.com/Arize-ai/phoenix) and one for\n[Langfuse](https://github.com/langfuse/langfuse/tree/main): two\nopen-source LLM Observability suites. This allows engineers that use the\nInference plugin to be able to inspect and improve their LLM-based\nworkflows with much less effort.\n\nFor both Phoenix and Langfuse, two service scripts were added. Run `node\nscripts/phoenix` or `node scripts/langfuse` to get started. Both scripts\nwork with zero-config - they will log generated Kibana config to stdout.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"2387e3b88c83317ddd5354dff751a1ac0fb62bb9"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/218694","number":218694,"mergeCommit":{"message":"[Inference] Instrument inference with OpenTelemetry (#218694)\n\nInstrument the inference chatComplete API with OpenTelemetry, and export\nhelper functions to create spans w/ the right semconv attributes.\nAdditionally, optionally export to Langfuse or Phoenix.\n\n## Centralizes OpenTelemetry setup\n\nAs this is the first instance of OpenTelemetry based _tracing_ (we\nalready have metrics in the MonitoringCollection plugin), some\nbootstrapping code is necessary to centrally configure OpenTelemetry. To\nthis end, I've added the following config settings:\n\n- `telemetry.tracing.enabled`: whether OpenTelemetry tracing is enabled\n(defaults to undefined, if undefined, falls back to `telemetry.enabled`)\n- `telemetry.tracing.sample_rate` (defaults to 1)\n\nThe naming of these configuration settings is mostly in-line with [the\nElasticsearch tracing\nsettings](https://github.com/elastic/elasticsearch/blob/main/TRACING.md).\n\nThe following packages (containing bootstrapping logic, utility\nfunctions, types and config schemas) were added:\n- `@kbn/telemetry`\n- `@kbn/telemetry-config` \n- `@kbn/tracing`\n\nThe OpenTelemetry bootstrapping depends on @kbn/apm-config-loader, as it\nhas the same constraints - it needs to run before any other code, and it\nneeds to read the raw config.\n\nAdditionally, a root `telemetry` logger was added that captures\nOpenTelemetry logs.\n\nNote that there is no default exporter for spans, which means that\nalthough spans are being recorded, they do not get exported.\n\n## Instrument chatComplete calls\n\nCalls to `chatComplete` now create OpenTelemetry spans, roughly\nfollowing semantic conventions (which for GenAI are very much in flux).\nSome helper functions were added to create other inference spans. These\nhelper functions use baggage to determine whether the created inference\nspan is the \"root\" of an inference trace. This allows us to export these\nspans as if it were root spans - something that is needed to be able to\neasily visualize these in other tools.\n\nLeveraging these inference spans, two exporters are added. One for\n[Phoenix](https://github.com/Arize-ai/phoenix) and one for\n[Langfuse](https://github.com/langfuse/langfuse/tree/main): two\nopen-source LLM Observability suites. This allows engineers that use the\nInference plugin to be able to inspect and improve their LLM-based\nworkflows with much less effort.\n\nFor both Phoenix and Langfuse, two service scripts were added. Run `node\nscripts/phoenix` or `node scripts/langfuse` to get started. Both scripts\nwork with zero-config - they will log generated Kibana config to stdout.\n\n---------\n\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"2387e3b88c83317ddd5354dff751a1ac0fb62bb9"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
kibanamachine
removed
the
backport missing
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
May 29, 2025
Instrument the inference chatComplete API with OpenTelemetry, and export helper functions to create spans w/ the right semconv attributes. Additionally, optionally export to Langfuse or Phoenix. ## Centralizes OpenTelemetry setup As this is the first instance of OpenTelemetry based _tracing_ (we already have metrics in the MonitoringCollection plugin), some bootstrapping code is necessary to centrally configure OpenTelemetry. To this end, I've added the following config settings: - `telemetry.tracing.enabled`: whether OpenTelemetry tracing is enabled (defaults to undefined, if undefined, falls back to `telemetry.enabled`) - `telemetry.tracing.sample_rate` (defaults to 1) The naming of these configuration settings is mostly in-line with [the Elasticsearch tracing settings](https://github.com/elastic/elasticsearch/blob/main/TRACING.md). The following packages (containing bootstrapping logic, utility functions, types and config schemas) were added: - `@kbn/telemetry` - `@kbn/telemetry-config` - `@kbn/tracing` The OpenTelemetry bootstrapping depends on @kbn/apm-config-loader, as it has the same constraints - it needs to run before any other code, and it needs to read the raw config. Additionally, a root `telemetry` logger was added that captures OpenTelemetry logs. Note that there is no default exporter for spans, which means that although spans are being recorded, they do not get exported. ## Instrument chatComplete calls Calls to `chatComplete` now create OpenTelemetry spans, roughly following semantic conventions (which for GenAI are very much in flux). Some helper functions were added to create other inference spans. These helper functions use baggage to determine whether the created inference span is the "root" of an inference trace. This allows us to export these spans as if it were root spans - something that is needed to be able to easily visualize these in other tools. Leveraging these inference spans, two exporters are added. One for [Phoenix](https://github.com/Arize-ai/phoenix) and one for [Langfuse](https://github.com/langfuse/langfuse/tree/main): two open-source LLM Observability suites. This allows engineers that use the Inference plugin to be able to inspect and improve their LLM-based workflows with much less effort. For both Phoenix and Langfuse, two service scripts were added. Run `node scripts/phoenix` or `node scripts/langfuse` to get started. Both scripts work with zero-config - they will log generated Kibana config to stdout. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
qn895
pushed a commit
to qn895/kibana
that referenced
this pull request
Jun 3, 2025
Instrument the inference chatComplete API with OpenTelemetry, and export helper functions to create spans w/ the right semconv attributes. Additionally, optionally export to Langfuse or Phoenix. ## Centralizes OpenTelemetry setup As this is the first instance of OpenTelemetry based _tracing_ (we already have metrics in the MonitoringCollection plugin), some bootstrapping code is necessary to centrally configure OpenTelemetry. To this end, I've added the following config settings: - `telemetry.tracing.enabled`: whether OpenTelemetry tracing is enabled (defaults to undefined, if undefined, falls back to `telemetry.enabled`) - `telemetry.tracing.sample_rate` (defaults to 1) The naming of these configuration settings is mostly in-line with [the Elasticsearch tracing settings](https://github.com/elastic/elasticsearch/blob/main/TRACING.md). The following packages (containing bootstrapping logic, utility functions, types and config schemas) were added: - `@kbn/telemetry` - `@kbn/telemetry-config` - `@kbn/tracing` The OpenTelemetry bootstrapping depends on @kbn/apm-config-loader, as it has the same constraints - it needs to run before any other code, and it needs to read the raw config. Additionally, a root `telemetry` logger was added that captures OpenTelemetry logs. Note that there is no default exporter for spans, which means that although spans are being recorded, they do not get exported. ## Instrument chatComplete calls Calls to `chatComplete` now create OpenTelemetry spans, roughly following semantic conventions (which for GenAI are very much in flux). Some helper functions were added to create other inference spans. These helper functions use baggage to determine whether the created inference span is the "root" of an inference trace. This allows us to export these spans as if it were root spans - something that is needed to be able to easily visualize these in other tools. Leveraging these inference spans, two exporters are added. One for [Phoenix](https://github.com/Arize-ai/phoenix) and one for [Langfuse](https://github.com/langfuse/langfuse/tree/main): two open-source LLM Observability suites. This allows engineers that use the Inference plugin to be able to inspect and improve their LLM-based workflows with much less effort. For both Phoenix and Langfuse, two service scripts were added. Run `node scripts/phoenix` or `node scripts/langfuse` to get started. Both scripts work with zero-config - they will log generated Kibana config to stdout. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instrument the inference chatComplete API with OpenTelemetry, and export helper functions to create spans w/ the right semconv attributes. Additionally, optionally export to Langfuse or Phoenix.
Centralizes OpenTelemetry setup
As this is the first instance of OpenTelemetry based tracing (we already have metrics in the MonitoringCollection plugin), some bootstrapping code is necessary to centrally configure OpenTelemetry. To this end, I've added the following config settings:
telemetry.tracing.enabled: whether OpenTelemetry tracing is enabled (defaults to undefined, if undefined, falls back totelemetry.enabled)telemetry.tracing.sample_rate(defaults to 1)The naming of these configuration settings is mostly in-line with the Elasticsearch tracing settings.
The following packages (containing bootstrapping logic, utility functions, types and config schemas) were added:
@kbn/telemetry@kbn/telemetry-config@kbn/tracingThe OpenTelemetry bootstrapping depends on @kbn/apm-config-loader, as it has the same constraints - it needs to run before any other code, and it needs to read the raw config.
Additionally, a root
telemetrylogger was added that captures OpenTelemetry logs.Note that there is no default exporter for spans, which means that although spans are being recorded, they do not get exported.
Instrument chatComplete calls
Calls to
chatCompletenow create OpenTelemetry spans, roughly following semantic conventions (which for GenAI are very much in flux). Some helper functions were added to create other inference spans. These helper functions use baggage to determine whether the created inference span is the "root" of an inference trace. This allows us to export these spans as if it were root spans - something that is needed to be able to easily visualize these in other tools.Leveraging these inference spans, two exporters are added. One for Phoenix and one for Langfuse: two open-source LLM Observability suites. This allows engineers that use the Inference plugin to be able to inspect and improve their LLM-based workflows with much less effort.
For both Phoenix and Langfuse, two service scripts were added. Run
node scripts/phoenixornode scripts/langfuseto get started. Both scripts work with zero-config - they will log generated Kibana config to stdout.