[Streams] Replay loghub data with synthtrace by dgieselaar · Pull Request #212120 · elastic/kibana

dgieselaar · 2025-02-21T16:31:09Z

Download, parse and replay loghub data with Synthtrace, for use in the Streams project. In summary:

adds a @kbn/sample-log-parser package which parses Loghub sample data, creates valid parsers for extracting and replacing timestamps, using the LLM
add a sample_logs scenario which uses the parsed data sets to replay Loghub data continuously as if it were live data
refactor some parts of Synthtrace (follow-up work captured in [Synthtrace] Consolidate clients #212179)

Synthtrace changes

Replace custom Logger object with Kibana-standard ToolingLog
Report progress and estimated time to completion for long-running jobs
Simplify scenarioOpts (allow comma-separated key-value pairs instead of just JSON)
Simplify client initialization
When using workers, only bootstrap once (in the main thread)
Allow workers to gracefully shutdown
Downgrade some logging levels for less noise

flash1293

Implementation-wise this looks pretty good to me. Some meta questions:

Should we rely on the public loghub repo or fork it off? I'm a little worried this breaking at some point because loghub changes its layout. This would also make it easier to expand it by our own means. In both cases we should cite loghub and the paper somewhere appropriate (like a readme file) as by the license
I'm not so sure about the different speeds. I'm running via node scripts/synthtrace.js sample_logs --live --kibana=http://localhost:5601 --target=http://localhost:9200 --liveBucketSize=1000 and the liveBucketSize is essentially not considered because it computes its own speed. Can we make it taken into account? Different speeds for different data sets are a nice touch as it mirrors reality, but I would like to control the factor of data intake (and speed everything up by a factor of 1k for example). Maybe that's already possible and I just don't know the right command
I spot-checked some aspects of the refactoring and it makes sense to me, but I didn't dig through everything and as I'm not super familiar with the code base it's likely I'm missing something in there

github-actions · 2025-02-24T15:24:38Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

dgieselaar · 2025-02-24T15:38:32Z

Should we rely on the public loghub repo or fork it off? I'm a little worried this breaking at some point because loghub changes its layout. This would also make it easier to expand it by our own means. In both cases we should cite loghub and the paper somewhere appropriate (like a readme file) as by the license

I'm fine with either - but maybe good to do that as a follow-up, I'm not sure what the legal ramifications are.

I'm not so sure about the different speeds. I'm running via node scripts/synthtrace.js sample_logs --live --kibana=http://localhost:5601/ --target=http://localhost:9200/ --liveBucketSize=1000 and the liveBucketSize is essentially not considered because it computes its own speed. Can we make it taken into account? Different speeds for different data sets are a nice touch as it mirrors reality, but I would like to control the factor of data intake (and speed everything up by a factor of 1k for example). Maybe that's already possible and I just don't know the right command

Yes, totally forgot about this setting, I should be able to use it. Would we use a constant indexing rate for each generator, or keep the relative rate per generator (e.g. Android indexes at a way higher rate than Macbook)?

flash1293 · 2025-02-24T17:48:29Z

I'm fine with either - but maybe good to do that as a follow-up, I'm not sure what the legal ramifications are.

Sounds good, then we should add a backlink to the repo and paper and follow up later.

Would we use a constant indexing rate for each generator, or keep the relative rate per generator (e.g. Android indexes at a way higher rate than Macbook)?

I would prefer the latter, in practice this kind of thing happens all the time.

elasticmachine · 2025-02-25T10:47:27Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

elasticmachine · 2025-03-11T11:32:15Z

💚 Build Succeeded

Buildkite Build
Commit: ed40be5
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-212120-ed40be5a5c4a

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/apm-synthtrace`	97	98	+1
`@kbn/apm-synthtrace-client`	272	274	+2
`@kbn/sample-log-parser`	-	18	+18
total			+21

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/sample-log-parser`	-	1	+1

Unknown metric groups

API count

id	before	after	diff
`@kbn/apm-synthtrace`	97	98	+1
`@kbn/apm-synthtrace-client`	272	274	+2
`@kbn/sample-log-parser`	-	18	+18
total			+21

ESLint disabled in files

id	before	after	diff
`@kbn/apm-synthtrace`	2	3	+1

ESLint disabled line counts

id	before	after	diff
`@kbn/apm-synthtrace`	6	2	-4

Total ESLint disabled count

id	before	after	diff
`@kbn/apm-synthtrace`	8	5	-3

History

💛 Build #281694 was flaky 4a851b1
💚 Build #281314 succeeded ca6805e
💔 Build #281170 failed 8d977ef
💔 Build #279401 failed 38a6cd6
💔 Build #279048 failed 74681f2

cc @dgieselaar

kibanamachine · 2025-03-11T12:30:27Z

Starting backport for target branches: 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13788048345

kibanamachine · 2025-03-11T12:36:55Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts
❌	9.0	Backport failed because of merge conflicts You might need to backport the following PRs to 9.0: - [Lens] [Dashboard] Remove metric chart progress bar rounded border (#211189) - [Security Assistant] Fix timeout during Knowledge Base setup (#213738) - [APM] Breakdown Top dependencies API (#211441) - [SLO] file reorg, improvements to badge (#213597) - [Discover] Exclude Elasticsearch metadata fields from Display in Content Column (#213255)

Manual backport

To create the backport manually run:

node scripts/backport --pr 212120

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-03-13T12:49:33Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 212120 locally

kibanamachine · 2025-03-14T12:51:53Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 212120 locally

kibanamachine · 2025-03-17T13:49:23Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 212120 locally

Download, parse and replay loghub data with Synthtrace, for use in the Streams project. In summary: - adds a `@kbn/sample-log-parser` package which parses Loghub sample data, creates valid parsers for extracting and replacing timestamps, using the LLM - add a `sample_logs` scenario which uses the parsed data sets to replay Loghub data continuously as if it were live data - refactor some parts of Synthtrace (follow-up work captured in elastic#212179) - Replace custom Logger object with Kibana-standard ToolingLog - Report progress and estimated time to completion for long-running jobs - Simplify scenarioOpts (allow comma-separated key-value pairs instead of just JSON) - Simplify client initialization - When using workers, only bootstrap once (in the main thread) - Allow workers to gracefully shutdown - Downgrade some logging levels for less noise --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit ba13e86)

dgieselaar · 2025-03-18T08:59:46Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

Download, parse and replay loghub data with Synthtrace, for use in the Streams project. In summary: - adds a `@kbn/sample-log-parser` package which parses Loghub sample data, creates valid parsers for extracting and replacing timestamps, using the LLM - add a `sample_logs` scenario which uses the parsed data sets to replay Loghub data continuously as if it were live data - refactor some parts of Synthtrace (follow-up work captured in elastic#212179) ## Synthtrace changes - Replace custom Logger object with Kibana-standard ToolingLog - Report progress and estimated time to completion for long-running jobs - Simplify scenarioOpts (allow comma-separated key-value pairs instead of just JSON) - Simplify client initialization - When using workers, only bootstrap once (in the main thread) - Allow workers to gracefully shutdown - Downgrade some logging levels for less noise --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit ba13e86) # Conflicts: # .github/CODEOWNERS # src/platform/packages/shared/kbn-apm-synthtrace/src/cli/utils/get_apm_es_client.ts # src/platform/packages/shared/kbn-apm-synthtrace/src/cli/utils/get_entities_es_client.ts # src/platform/packages/shared/kbn-apm-synthtrace/src/cli/utils/get_infra_es_client.ts # src/platform/packages/shared/kbn-apm-synthtrace/src/cli/utils/get_logs_es_client.ts # src/platform/packages/shared/kbn-apm-synthtrace/src/cli/utils/get_otel_es_client.ts # src/platform/packages/shared/kbn-apm-synthtrace/src/cli/utils/get_synthetics_es_client.ts # src/platform/packages/shared/kbn-apm-synthtrace/src/lib/apm/client/apm_synthtrace_kibana_client.ts

kibanamachine · 2025-03-19T09:48:53Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

# Backport This will backport the following commits from `main` to `8.x`: - [[Streams] Replay loghub data with synthtrace (#212120)](#212120)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Download, parse and replay loghub data with Synthtrace, for use in the Streams project. In summary: - adds a `@kbn/sample-log-parser` package which parses Loghub sample data, creates valid parsers for extracting and replacing timestamps, using the LLM - add a `sample_logs` scenario which uses the parsed data sets to replay Loghub data continuously as if it were live data - refactor some parts of Synthtrace (follow-up work captured in elastic#212179) ## Synthtrace changes - Replace custom Logger object with Kibana-standard ToolingLog - Report progress and estimated time to completion for long-running jobs - Simplify scenarioOpts (allow comma-separated key-value pairs instead of just JSON) - Simplify client initialization - When using workers, only bootstrap once (in the main thread) - Allow workers to gracefully shutdown - Downgrade some logging levels for less noise --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

dgieselaar added 2 commits February 21, 2025 17:30

[Streams] Replay loghub data with synthtrace

57bbd8e

Add tests & ensure repo for synthtrace

fbf8063

dgieselaar mentioned this pull request Feb 22, 2025

[Synthtrace] Consolidate clients #212179

Closed

dgieselaar and others added 5 commits February 22, 2025 14:18

Fix type issues

b17ae1c

Merge branch 'main' of github.com:elastic/kibana into sample-log-parser

57909f0

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

be38355

Fix type issues

40ca31b

Use kibanaClient.fetch for entities client

f02eafb

dgieselaar added v9.0.0 v9.1.0 v8.19.0 Feature:Streams This is the label for the Streams Project labels Feb 23, 2025

dgieselaar self-assigned this Feb 23, 2025

dgieselaar marked this pull request as ready for review February 23, 2025 14:22

dgieselaar requested review from a team as code owners February 23, 2025 14:22

dgieselaar requested a review from a team February 23, 2025 14:22

dgieselaar requested a review from a team as a code owner February 23, 2025 14:22

dgieselaar added backport:version Backport to applied version labels release_note:skip Skip the PR/issue when compiling release notes labels Feb 23, 2025

flash1293 reviewed Feb 24, 2025

View reviewed changes

botelastic bot added the ci:project-deploy-observability Create an Observability project label Feb 24, 2025

dgieselaar added 2 commits February 24, 2025 19:20

Merge branch 'main' of github.com:elastic/kibana into sample-log-parser

0d6ade8

Distribute RPM amongst scenarios

6d8253a

botelastic bot added the Team:obs-ux-infra_services - DEPRECATED DEPRECATED - Use Team:obs-presentation. label Feb 25, 2025

dgieselaar merged commit ba13e86 into elastic:main Mar 11, 2025
10 checks passed

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 13, 2025

kibanamachine mentioned this pull request Mar 13, 2025

Update dependency @hapi/hapi to ^21.4.0 (main) #214209

Merged

1 task

This was referenced Mar 14, 2025

[Space time] extending Scout with perfTracker fixture #212397

Merged

Upgrade to Storybook 8 #195148

Merged

kibanamachine mentioned this pull request Mar 17, 2025

Upgrade xml-crypto 6.0.0 → 6.0.1 #214792

Merged

dgieselaar mentioned this pull request Mar 18, 2025

[9.0] [Streams] Replay loghub data with synthtrace (#212120) #214908

Closed

dgieselaar mentioned this pull request Mar 18, 2025

[8.x] [Streams] Replay loghub data with synthtrace (#212120) #214916

Merged

dgieselaar removed the v9.0.0 label Mar 18, 2025

kibanamachine mentioned this pull request Mar 18, 2025

[Streams] [Dedicated Grok UI] Package / highlight matched parts / suggestions #213278

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 19, 2025

Conversation

dgieselaar commented Feb 21, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Synthtrace changes

Uh oh!

flash1293 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 24, 2025

🤖 GitHub comments

Uh oh!

dgieselaar commented Feb 24, 2025

Uh oh!

flash1293 commented Feb 24, 2025

Uh oh!

elasticmachine commented Feb 25, 2025

Uh oh!

elasticmachine commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Uh oh!

kibanamachine commented Mar 11, 2025

Uh oh!

kibanamachine commented Mar 11, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented Mar 13, 2025

Uh oh!

kibanamachine commented Mar 14, 2025

Uh oh!

kibanamachine commented Mar 17, 2025

Uh oh!

dgieselaar commented Mar 18, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Mar 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

dgieselaar commented Feb 21, 2025 •

edited by kibanamachine

Loading

flash1293 left a comment •

edited

Loading

elasticmachine commented Mar 11, 2025 •

edited

Loading