Load huggingface content datasets by dgieselaar · Pull Request #224543 · elastic/kibana

dgieselaar · 2025-06-19T10:22:17Z

Implements a huggingface dataset loader for RAG evals - see x-pack/platform/packages/shared/kbn-ai-tools-cli/src/hf_dataset_loader/README.md. Additionally, a @kbn/cache-cli tool was added that allows tooling authors to cache to disk (possibly remote storage later).

Used o3 for finding datasets on HuggingFace and doing an initial pass on a line-by-line dataset processor (see conversation)

Libraries added:

cache-manager, cache-manager-fs-hash, keyv, @types/cache-manager-fs-hash: caching libraries and plugins. could not find any existing caching libraries in the repo.
@huggingface/hub: api client for HF.

kibanamachine · 2025-06-19T10:22:33Z

Dependency Review Bot Analysis 🔍

Found 5 new third-party dependencies:

Package	Version	Vulnerabilities
cache-manager	^7.0.0	🔴 C: 0, 🟠 H: 0, 🟡 M: 0, 🟢 L: 0
keyv	^5.3.4	🔴 C: 0, 🟠 H: 0, 🟡 M: 0, 🟢 L: 0
@huggingface/hub	^2.2.0	🔴 C: 0, 🟠 H: 0, 🟡 M: 0, 🟢 L: 0
@types/cache-manager-fs-hash	^0.0.5	🔴 C: 0, 🟠 H: 0, 🟡 M: 0, 🟢 L: 0
cache-manager-fs-hash	^2.0.0	🔴 C: 0, 🟠 H: 0, 🟡 M: 0, 🟢 L: 0

Self Checklist

To help with the review, please update the PR description to address the following points for each new third-party dependency listed above:

Purpose: What is this dependency used for? Briefly explain its role in your changes.
Justification: Why is adding this dependency the best approach?
Alternatives explored: Were other options considered (e.g., using existing internal libraries/utilities, implementing the functionality directly)? If so, why was this dependency chosen over them?
Existing dependencies: Does Kibana have a dependency providing similar functionality? If so, why is the new one preferred?

Thank you for providing this information!

…ocs'

… hf-dataset-loader

elena-shostak

Since @huggingface/hub is intended only for local usage can we move it to dev dependencies? Same for caching libraries

elena-shostak · 2025-06-19T13:14:24Z

.github/CODEOWNERS

 src/platform/packages/shared/kbn-avc-banner @elastic/security-defend-workflows
 src/platform/packages/shared/kbn-axe-config @elastic/appex-qa
 src/platform/packages/shared/kbn-babel-register @elastic/kibana-operations
+src/platform/packages/shared/kbn-cache-cli @elastic/kibana-operations


Could we please add it to CodeQL ignore paths?

elena-shostak · 2025-06-19T13:14:38Z

.github/CODEOWNERS

 x-pack/platform/packages/shared/index-lifecycle-management/index_lifecycle_management_common_shared @elastic/kibana-management
 x-pack/platform/packages/shared/index-management/index_management_shared_types @elastic/kibana-management
 x-pack/platform/packages/shared/kbn-ai-assistant @elastic/search-kibana @elastic/obs-ai-assistant
+x-pack/platform/packages/shared/kbn-ai-tools-cli @elastic/appex-ai-infra


Could we please add it to CodeQL ignore paths?

what does that do and why should this be excluded? is it a blanket policy for CLI tools?

I've briefly covered the reason in your other PR - #218694 (review). Ideally, we wouldn't exclude or ignore anything at all, and hopefully, we'll get there eventually. However, due to our current constraints (CodeQL is slow, resource-hungry, and not super friendly to incremental tests/changes), we're trying to be pragmatic and only cover non-dev/non-test code. At the same time, I'm not yet confident it'll become a permanent policy (at least I hope so), so we haven't documented this anywhere and are handling it on a case-by-case basis.

@azasypkin thanks Oleg and apologies for missing that. Is this something we can automate? The package has the devOnly prop. If we want to do it manually, does it make sense to exclude these packages as they are very small? I'm ok with adding these manually but doing this every time is probably not sustainable.

In the interest of time I've added them to the CodeQL ignore paths – it'd be great if we can automate this but I also understand it might not be worth it if we are not sure what the long term plan is.

Yeah, we definitely want and need to add more automation here, and we're moving (albeit slowly) in that direction.

dgieselaar · 2025-06-19T14:07:33Z

Since @huggingface/hub is intended only for local usage can we move it to dev dependencies? Same for caching libraries

Done!

darnautov · 2025-06-19T14:18:44Z

x-pack/platform/packages/shared/kbn-kibana-api-cli/src/proxy_transport.ts

-      super(options);
+      super({
+        ...options,
+        productCheck: undefined,


could you please leave a comment why we need to disable it?

…t --include-path /api/status --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/dashboards --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/maintenance_window --update'

elena-shostak

LGTM

dgieselaar · 2025-06-26T13:41:12Z

@elasticmachine merge upstream

elasticmachine · 2025-06-26T15:19:10Z

💚 Build Succeeded

Buildkite Build
Commit: 025cd81

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/ai-tools-cli`	-	11	+11
`@kbn/cache-cli`	-	9	+9
total			+20

Any counts in public APIs

Total count of every any typed public API. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats any for more detailed information.

id	before	after	diff
`@kbn/ai-tools-cli`	-	1	+1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/cache-cli`	-	2	+2

Unknown metric groups

API count

id	before	after	diff
`@kbn/ai-tools-cli`	-	18	+18
`@kbn/cache-cli`	-	9	+9
total			+27

History

💚 Build #311971 succeeded 4a479b4
💚 Build #310858 succeeded dd99ec4
💔 Build #310850 failed ca667ff
💔 Build #310548 failed 35cb56f
💔 Build #310484 failed b0d6ba8
💔 Build #310372 failed 1abf55a

kibanamachine · 2025-06-26T15:25:05Z

Starting backport for target branches: 8.19

https://github.com/elastic/kibana/actions/runs/15905968267

kibanamachine · 2025-06-26T15:32:11Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 224543

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2025-06-30T15:52:00Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 224543 locally
cc: @dgieselaar

kibanamachine · 2025-07-01T16:49:17Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 224543 locally
cc: @dgieselaar

dgieselaar · 2025-07-02T13:51:26Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

Implements a huggingface dataset loader for RAG evals - see [x-pack/platform/packages/shared/kbn-ai-tools-cli/src/hf_dataset_loader/README.md](https://github.com/dgieselaar/kibana/blob/hf-dataset-loader/x-pack/platform/packages/shared/kbn-ai-tools-cli/src/hf_dataset_loader/README.md). Additionally, a `@kbn/cache-cli` tool was added that allows tooling authors to cache to disk (possibly remote storage later). Used o3 for finding datasets on HuggingFace and doing an initial pass on a line-by-line dataset processor ([see conversation](https://chatgpt.com/share/6853e49a-e870-8000-9c65-f7a5a3a72af0)) Libraries added: - `cache-manager`, `cache-manager-fs-hash`, `keyv`, `@types/cache-manager-fs-hash`: caching libraries and plugins. could not find any existing caching libraries in the repo. - `@huggingface/hub`: api client for HF. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 7d20301) # Conflicts: # .github/CODEOWNERS # tsconfig.base.json # yarn.lock

kibanamachine · 2025-07-02T16:50:29Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.
cc: @dgieselaar

# Backport This will backport the following commits from `main` to `8.19`: - [Load huggingface content datasets (#224543)](#224543)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

dgieselaar added 2 commits June 19, 2025 11:59

Load huggingface content datasets

69f5893

Check for HF access token

e8fb7a2

dgieselaar requested a review from a team as a code owner June 19, 2025 10:22

dgieselaar added release_note:skip Skip the PR/issue when compiling release notes v9.1.0 v8.19.0 labels Jun 19, 2025

kibanamachine requested a review from a team June 19, 2025 10:22

kibanamachine and others added 3 commits June 19, 2025 10:34

[CI] Auto-commit changed files from 'node scripts/build_plugin_list_d…

1abf55a

…ocs'

Update README.md

f7cfc25

Merge branch 'hf-dataset-loader' of github.com:dgieselaar/kibana into…

4a32350

… hf-dataset-loader

dgieselaar added the backport:version Backport to applied version labels label Jun 19, 2025

Add dep owners

99dc7dc

elena-shostak reviewed Jun 19, 2025

View reviewed changes

Move to devDependencies

b0d6ba8

kibanamachine requested a review from a team June 19, 2025 14:07

darnautov reviewed Jun 19, 2025

View reviewed changes

Add comment about skipping productCheck

35cb56f

darnautov approved these changes Jun 19, 2025

View reviewed changes

dgieselaar and others added 4 commits June 20, 2025 13:32

Make packages devOnly

ca667ff

Merge branch 'main' of github.com:elastic/kibana into hf-dataset-loader

d9691a9

Add -cli packages to CodeQL ignore paths

4a479b4

elena-shostak approved these changes Jun 24, 2025

View reviewed changes

Merge branch 'main' into hf-dataset-loader

025cd81

dgieselaar merged commit 7d20301 into elastic:main Jun 26, 2025
13 checks passed

dgieselaar deleted the hf-dataset-loader branch June 26, 2025 15:24

This was referenced Jun 26, 2025

Upgrade parcel/watcher to 2.5.1 #225512

Merged

Update dependency msw to ~2.10.2 (main) #223986

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jun 30, 2025

kibanamachine mentioned this pull request Jul 1, 2025

[Inference] Move tracing config to @kbn/tracing #225417

Merged

dgieselaar mentioned this pull request Jul 2, 2025

[8.19] Load huggingface content datasets (#224543) #226219

Merged

kibanamachine mentioned this pull request Jul 2, 2025

[Streams] Add Streamlang package #224915

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jul 3, 2025

joemcelroy mentioned this pull request Jul 15, 2025

[OneChat] Use Huggingface Data Loader to seed Elasticsearch #227993

Closed

3 tasks

Comments

Conversation

dgieselaar commented Jun 19, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kibanamachine commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Self Checklist

Uh oh!

elena-shostak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgieselaar commented Jun 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elena-shostak left a comment

Choose a reason for hiding this comment

Uh oh!

dgieselaar commented Jun 26, 2025

Uh oh!

elasticmachine commented Jun 26, 2025

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Any counts in public APIs

Public APIs missing exports

API count

History

Uh oh!

Uh oh!

kibanamachine commented Jun 26, 2025

Uh oh!

kibanamachine commented Jun 26, 2025

💔 All backports failed

Manual backport

Questions ?

Uh oh!

kibanamachine commented Jun 30, 2025

Uh oh!

kibanamachine commented Jul 1, 2025

Uh oh!

dgieselaar commented Jul 2, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kibanamachine commented Jul 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dgieselaar commented Jun 19, 2025 •

edited by kibanamachine

Loading

kibanamachine commented Jun 19, 2025 •

edited

Loading