[Inference] regex worker for anonymization by neptunian · Pull Request #227113 · elastic/kibana

neptunian · 2025-07-08T16:56:01Z

Summary

Followup to #225539

Adds an anonymization-regex worker pool to run user-defined regex patterns out-of-process, protecting the Node event-loop from catastrophic back-tracking.

For internal use we have config to adjust or turn off the worker:
• enabled – turn worker on/off
• minThreads / maxThreads – worker size
• idleTimeout – how long a worker thread is allowed to be idle
• taskTimeout – per-regex task limit

Contributions by windsurf(o3).

elasticmachine · 2025-07-08T23:38:25Z

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

pgayvallet

just passing by

pgayvallet · 2025-07-09T12:43:49Z

...platform/plugins/shared/inference/server/chat_complete/anonymization/regex_worker_wrapper.js

+// eslint-disable-next-line @kbn/imports/no_boundary_crossing
+require('../../../../../../../../src/setup_node_env');


🙈 we really have zero supports for workers today

x-pack/platform/plugins/shared/inference/server/config.ts

x-pack/platform/plugins/shared/inference/server/plugin.ts

dgieselaar · 2025-07-09T12:58:57Z

x-pack/platform/plugins/shared/inference/tsconfig.json

  ],
  "kbn_references": [
+    {
+      "path": "../../../../../src/setup_node_env/tsconfig.json"


Why do we need this?

setup_node_env causes type errors without it.

…nian/kibana into anonymization-regex-worker

...platform/plugins/shared/inference/server/chat_complete/anonymization/regex_worker_wrapper.js

…nian/kibana into anonymization-regex-worker

...platform/plugins/shared/inference/server/chat_complete/anonymization/regex_worker_wrapper.js

dgieselaar

approving ahead of time, but please take a look at x-pack/platform/plugins/shared/inference/server/chat_complete/anonymization/regex_worker_wrapper.js - specifically the /dist import. thanks for getting this done so quickly!

…nian/kibana into anonymization-regex-worker

elasticmachine · 2025-07-11T20:37:44Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b9ba0e0

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #94 / GlobalSearch API GlobalSearch providers "before all" hook in "GlobalSearch providers"

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/inference-common`	161	164	+3

Unknown metric groups

API count

id	before	after	diff
`@kbn/inference-common`	302	305	+3

ESLint disabled in files

id	before	after	diff
`inference`	2	4	+2

ESLint disabled line counts

id	before	after	diff
`inference`	1	2	+1

Total ESLint disabled count

id	before	after	diff
`inference`	3	6	+3

History

💔 Build #318242 failed fe3fabb
💛 Build #317862 was flaky e340e12
💚 Build #317737 succeeded 3bc8a97
💛 Build #317478 was flaky 1da8481
💚 Build #317414 succeeded a67c8e0
💔 Build #317326 failed bcbc382

kibanamachine · 2025-07-11T20:52:48Z

Starting backport for target branches: 8.19, 9.1

https://github.com/elastic/kibana/actions/runs/16229533879

## Summary Followup to elastic#225539 Adds an anonymization-regex worker pool to run user-defined regex patterns out-of-process, protecting the Node event-loop from catastrophic back-tracking. For internal use we have config to adjust or turn off the worker: • `enabled` – turn worker on/off • `minThreads` / `maxThreads` – worker size • `idleTimeout` – how long a worker thread is allowed to be idle • `taskTimeout` – per-regex task limit Contributions by windsurf(o3). (cherry picked from commit 1b9063a)

kibanamachine · 2025-07-11T20:59:17Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19
✅	9.1

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `9.1`: - [[Inference] regex worker for anonymization (#227113)](#227113)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Sandra G <neptunian@users.noreply.github.com>

# Backport This will backport the following commits from `main` to `8.19`: - [[Inference] regex worker for anonymization (#227113)](#227113)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Sandra G <neptunian@users.noreply.github.com>

## Summary Followup to elastic#225539 Adds an anonymization-regex worker pool to run user-defined regex patterns out-of-process, protecting the Node event-loop from catastrophic back-tracking. For internal use we have config to adjust or turn off the worker: • `enabled` – turn worker on/off • `minThreads` / `maxThreads` – worker size • `idleTimeout` – how long a worker thread is allowed to be idle • `taskTimeout` – per-regex task limit Contributions by windsurf(o3).

neptunian added 3 commits July 8, 2025 12:55

add piscina worker

138b810

remove log

ab0061f

add tests

be5e0c4

neptunian marked this pull request as ready for review July 8, 2025 23:37

neptunian requested a review from a team as a code owner July 8, 2025 23:37

neptunian added backport:version Backport to applied version labels release_note:skip Skip the PR/issue when compiling release notes v8.19.0 v9.1.0 Team:Obs AI Assistant Observability AI Assistant labels Jul 8, 2025

neptunian and others added 3 commits July 8, 2025 21:26

setup_node_env path

880feaa

fix jest tests

a6d8a1a

Merge branch 'main' into anonymization-regex-worker

f4a0f99

pgayvallet reviewed Jul 9, 2025

View reviewed changes

dgieselaar reviewed Jul 9, 2025

View reviewed changes

neptunian added 6 commits July 9, 2025 09:29

fix setup_node_env path

330d36a

Merge branch 'anonymization-regex-worker' of https://github.com/neptu…

eb58584

…nian/kibana into anonymization-regex-worker

remove tsconfig path

97adb50

remove unused tracing related code

bcbc382

remove setup_node_env

f2e5572

update comment

a67c8e0

dgieselaar reviewed Jul 9, 2025

View reviewed changes

...platform/plugins/shared/inference/server/chat_complete/anonymization/regex_worker_wrapper.js Outdated Show resolved Hide resolved

neptunian and others added 7 commits July 9, 2025 14:43

change to anonymizationRegexWorker

2513288

add back setup_node_env with diff paths

1da8481

change config naming

58e28fa

Merge branch 'main' into anonymization-regex-worker

3bc8a97

update test

5cab80a

Merge branch 'anonymization-regex-worker' of https://github.com/neptu…

1fc7035

…nian/kibana into anonymization-regex-worker

Merge branch 'main' into anonymization-regex-worker

e340e12

dgieselaar reviewed Jul 10, 2025

View reviewed changes

...platform/plugins/shared/inference/server/chat_complete/anonymization/regex_worker_wrapper.js Outdated Show resolved Hide resolved

dgieselaar reviewed Jul 10, 2025

View reviewed changes

...platform/plugins/shared/inference/server/chat_complete/anonymization/regex_worker_wrapper.js Outdated Show resolved Hide resolved

dgieselaar reviewed Jul 10, 2025

View reviewed changes

...platform/plugins/shared/inference/server/chat_complete/anonymization/regex_worker_wrapper.js Outdated Show resolved Hide resolved

dgieselaar approved these changes Jul 10, 2025

View reviewed changes

neptunian added 2 commits July 11, 2025 09:19

add skipCloud to anonymization tests due to llm proxy

f39bc7b

Merge branch 'anonymization-regex-worker' of https://github.com/neptu…

5959304

…nian/kibana into anonymization-regex-worker

neptunian requested a review from a team as a code owner July 11, 2025 13:19

try same path setup_node_env

fe3fabb

dgieselaar approved these changes Jul 11, 2025

View reviewed changes

neptunian added 3 commits July 11, 2025 10:41

fix path

f851324

try direct path

dcf01c8

try REPO_ROOT

b9ba0e0

neptunian merged commit 1b9063a into elastic:main Jul 11, 2025
14 checks passed

kibanamachine added the v9.2.0 label Jul 11, 2025

kibanamachine mentioned this pull request Jul 11, 2025

[8.19] [Inference] regex worker for anonymization (#227113) #227722

Merged

kibanamachine mentioned this pull request Jul 11, 2025

[9.1] [Inference] regex worker for anonymization (#227113) #227723

Merged

		// eslint-disable-next-line @kbn/imports/no_boundary_crossing
		require('../../../../../../../../src/setup_node_env');

Comments

Conversation

neptunian commented Jul 8, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Jul 8, 2025

Uh oh!

pgayvallet left a comment

Choose a reason for hiding this comment

Uh oh!

pgayvallet Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dgieselaar Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

neptunian Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgieselaar left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 11, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Uh oh!

kibanamachine commented Jul 11, 2025

Uh oh!

kibanamachine commented Jul 11, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

neptunian commented Jul 8, 2025 •

edited by kibanamachine

Loading

pgayvallet Jul 9, 2025 •

edited

Loading

neptunian Jul 9, 2025 •

edited

Loading