Skip to content

Add base FTR test coverage for inference APIs#198000

Merged
pgayvallet merged 36 commits intoelastic:mainfrom
pgayvallet:kbn-193470-inference-integration-test
Dec 4, 2024
Merged

Add base FTR test coverage for inference APIs#198000
pgayvallet merged 36 commits intoelastic:mainfrom
pgayvallet:kbn-193470-inference-integration-test

Conversation

@pgayvallet
Copy link
Copy Markdown
Contributor

@pgayvallet pgayvallet commented Oct 28, 2024 •
edited by kibanamachine
Loading

Summary

Part of https://github.com/elastic/kibana-team/issues/1271

This PR introduces the first set of end to end integration test for the inference APIs, and the tooling required to do so (see issue for more context)

  • Add a dedicated pipeline for ai-infra GenAI tests. pipeline is triggered when:
    • genAI stack connectors, or ai-infra owned code is changed
    • when the ci:all-gen-ai-suites label is present on a PR
    • on merge
  • adapt the ftr_configs.sh script to load GenAI connector configuration from vault when a specific var env is set
  • create the @kbn/gen-ai-functional-testing package, which for now only contains utilities to load the GenAI connector configuration in FTR tests
  • Add FTR integration tests for the chatComplete API of the inference plugin

@pgayvallet pgayvallet added release_note:skip Skip the PR/issue when compiling release notes Team:AI Infra Platform AppEx AI Infrastructure Team t// v8.17.0 backport:prev-minor labels Oct 28, 2024
pgayvallet
pgayvallet commented Oct 28, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor Author

@pgayvallet pgayvallet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quick comments for @pheyos

@pgayvallet
Copy link
Copy Markdown
Contributor Author

pgayvallet commented Nov 22, 2024

/ci

@pgayvallet
Copy link
Copy Markdown
Contributor Author

pgayvallet commented Nov 22, 2024

/ci

@pgayvallet
Copy link
Copy Markdown
Contributor Author

pgayvallet commented Nov 22, 2024

/ci

@pgayvallet
Copy link
Copy Markdown
Contributor Author

pgayvallet commented Nov 25, 2024

/ci

@pgayvallet
Copy link
Copy Markdown
Contributor Author

pgayvallet commented Nov 25, 2024

/ci

@pgayvallet pgayvallet merged commit 14ad13b into elastic:main Dec 4, 2024
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Dec 4, 2024

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12159929564

@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Dec 4, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/gen-ai-functional-testing - 8 +8
@kbn/sse-utils-server 2 3 +1
total +9
Unknown metric groups

API count

id before after diff
@kbn/gen-ai-functional-testing - 10 +10
@kbn/sse-utils-server 2 4 +2
total +12

ESLint disabled line counts

id before after diff
@kbn/gen-ai-functional-testing - 2 +2
@kbn/test-suites-xpack 727 729 +2
total +4

Total ESLint disabled count

id before after diff
@kbn/gen-ai-functional-testing - 2 +2
@kbn/test-suites-xpack 752 754 +2
total +4

History

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Dec 4, 2024

💔 All backports failed

Status Branch Result
❌ 8.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 198000

Questions ?

Please refer to the Backport tool documentation

@pgayvallet pgayvallet mentioned this pull request Dec 4, 2024
Merged
@pgayvallet
Copy link
Copy Markdown
Contributor Author

pgayvallet commented Dec 4, 2024

💚 All backports created successfully

Status Branch Result
✅ 8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

pgayvallet added a commit to pgayvallet/kibana that referenced this pull request Dec 4, 2024
@pgayvallet
Add base FTR test coverage for inference APIs (elastic#198000)
81c15ed 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
(cherry picked from commit 14ad13b)

# Conflicts:
#	.buildkite/pipelines/on_merge.yml
#	.github/CODEOWNERS
@kibanamachine kibanamachine mentioned this pull request Dec 4, 2024
Merged
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 5, 2024
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Dec 5, 2024

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

pgayvallet added a commit that referenced this pull request Dec 5, 2024
@pgayvallet @kibanamachine
[8.x] Add base FTR test coverage for inference APIs (#198000) (#202916)
4ffcaed 
# Backport

This will backport the following commits from `main` to `8.x`:
- [Add base FTR test coverage for inference APIs
(#198000)](#198000)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Pierre
Gayvallet","email":"pierre.gayvallet@elastic.co"},"sourceCommit":{"committedDate":"2024-12-04T12:39:45Z","message":"Add
base FTR test coverage for inference APIs (#198000)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana-team/issues/1271\r\n\r\nThis PR
introduces the first set of end to end integration test for
the\r\ninference APIs, and the tooling required to do so (see issue for
more\r\ncontext)\r\n\r\n- Add a dedicated pipeline for ai-infra GenAI
tests. pipeline is\r\ntriggered when:\r\n - genAI stack connectors, or
ai-infra owned code is changed\r\n - when the `ci:all-gen-ai-suites`
label is present on a PR\r\n - on merge\r\n- adapt the `ftr_configs.sh`
script to load GenAI connector\r\nconfiguration from vault when a
specific var env is set\r\n- create the `@kbn/gen-ai-functional-testing`
package, which for now\r\nonly contains utilities to load the GenAI
connector configuration in FTR\r\ntests\r\n- Add FTR integration tests
for the `chatComplete` API of the\r\n`inference`
plugin\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"14ad13b6a3d7a87f4dfa9d7c1ed6c055242c9808","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:version","Team:AI
Infra","v8.18.0","ci:all-gen-ai-suites"],"number":198000,"url":"https://github.com/elastic/kibana/pull/198000","mergeCommit":{"message":"Add
base FTR test coverage for inference APIs (#198000)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana-team/issues/1271\r\n\r\nThis PR
introduces the first set of end to end integration test for
the\r\ninference APIs, and the tooling required to do so (see issue for
more\r\ncontext)\r\n\r\n- Add a dedicated pipeline for ai-infra GenAI
tests. pipeline is\r\ntriggered when:\r\n - genAI stack connectors, or
ai-infra owned code is changed\r\n - when the `ci:all-gen-ai-suites`
label is present on a PR\r\n - on merge\r\n- adapt the `ftr_configs.sh`
script to load GenAI connector\r\nconfiguration from vault when a
specific var env is set\r\n- create the `@kbn/gen-ai-functional-testing`
package, which for now\r\nonly contains utilities to load the GenAI
connector configuration in FTR\r\ntests\r\n- Add FTR integration tests
for the `chatComplete` API of the\r\n`inference`
plugin\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"14ad13b6a3d7a87f4dfa9d7c1ed6c055242c9808"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/198000","number":198000,"mergeCommit":{"message":"Add
base FTR test coverage for inference APIs (#198000)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana-team/issues/1271\r\n\r\nThis PR
introduces the first set of end to end integration test for
the\r\ninference APIs, and the tooling required to do so (see issue for
more\r\ncontext)\r\n\r\n- Add a dedicated pipeline for ai-infra GenAI
tests. pipeline is\r\ntriggered when:\r\n - genAI stack connectors, or
ai-infra owned code is changed\r\n - when the `ci:all-gen-ai-suites`
label is present on a PR\r\n - on merge\r\n- adapt the `ftr_configs.sh`
script to load GenAI connector\r\nconfiguration from vault when a
specific var env is set\r\n- create the `@kbn/gen-ai-functional-testing`
package, which for now\r\nonly contains utilities to load the GenAI
connector configuration in FTR\r\ntests\r\n- Add FTR integration tests
for the `chatComplete` API of the\r\n`inference`
plugin\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<42973632+kibanamachine@users.noreply.github.com>","sha":"14ad13b6a3d7a87f4dfa9d7c1ed6c055242c9808"}},{"branch":"8.x","label":"v8.18.0","labelRegex":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 5, 2024
SoniaSanzV pushed a commit to SoniaSanzV/kibana that referenced this pull request Dec 9, 2024
@pgayvallet @kibanamachine @SoniaSanzV
Add base FTR test coverage for inference APIs (elastic#198000)
8bdbfdc 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@pgayvallet pgayvallet mentioned this pull request Dec 9, 2024
Merged
pgayvallet added a commit that referenced this pull request Dec 9, 2024
@pgayvallet
GenAI FTR tests: switch to daily job (#203369)
1f2a76a 
## Summary

Part of elastic/kibana-team#1271
Follow-up of #198000

Fix #203205
Fix #203062

- Fix the problem with the Gemini token count events 
- Unskip the test suite
- Remove the suite from the `on-merge` pipeline
- Add a dedicated pipeline to run the tests daily
pgayvallet added a commit to pgayvallet/kibana that referenced this pull request Dec 9, 2024
@pgayvallet
GenAI FTR tests: switch to daily job (elastic#203369)
fa30b5e 
## Summary

Part of elastic/kibana-team#1271
Follow-up of elastic#198000

Fix elastic#203205
Fix elastic#203062

- Fix the problem with the Gemini token count events
- Unskip the test suite
- Remove the suite from the `on-merge` pipeline
- Add a dedicated pipeline to run the tests daily

(cherry picked from commit 1f2a76a)

# Conflicts:
#	.buildkite/pipeline-resource-definitions/locations.yml
#	.buildkite/pipelines/on_merge.yml
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
@pgayvallet @kibanamachine @CAWilson94
Add base FTR test coverage for inference APIs (elastic#198000)
6dee752 
## Summary

Part of elastic/kibana-team#1271

This PR introduces the first set of end to end integration test for the
inference APIs, and the tooling required to do so (see issue for more
context)

- Add a dedicated pipeline for ai-infra GenAI tests. pipeline is
triggered when:
  - genAI stack connectors, or ai-infra owned code is changed
  - when the `ci:all-gen-ai-suites` label is present on a PR
  - on merge
- adapt the `ftr_configs.sh` script to load GenAI connector
configuration from vault when a specific var env is set
- create the `@kbn/gen-ai-functional-testing` package, which for now
only contains utilities to load the GenAI connector configuration in FTR
tests
- Add FTR integration tests for the `chatComplete` API of the
`inference` plugin

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Dec 12, 2024
@pgayvallet @CAWilson94
GenAI FTR tests: switch to daily job (elastic#203369)
b9e299c 
## Summary

Part of elastic/kibana-team#1271
Follow-up of elastic#198000

Fix elastic#203205
Fix elastic#203062

- Fix the problem with the Gemini token count events 
- Unskip the test suite
- Remove the suite from the `on-merge` pipeline
- Add a dedicated pipeline to run the tests daily
@bhapas bhapas mentioned this pull request Feb 8, 2025
Open
@pgayvallet pgayvallet mentioned this pull request Nov 17, 2025
Merged
pgayvallet added a commit that referenced this pull request Nov 19, 2025
@pgayvallet
[Agent Builder] add basic smoke tests (#243248)
f4b00f3 
## Summary
Fix elastic/search-team#11697
Inspired by #198000

Add basic "real LLM" smoke tests to make sure agent builder's basic
functionalities are working as expected with our list of officially
supported models.

Smoke tests are triggered when the `agent-builder:run-smoke-tests` tag
is present on the PR, or when any of the following code has changes:
- the GenAI stack connectors
- the `inference` plugin and packages
- the agent builder plugin or packages

This PR re-uses the set of "CI" LLM connectors which was introduced in
#198000 for the AI-Infra inference
tests.
andrimal pushed a commit to andrimal/kibana that referenced this pull request Nov 20, 2025
@pgayvallet @andrimal
[Agent Builder] add basic smoke tests (elastic#243248)
3470e8c 
## Summary
Fix elastic/search-team#11697
Inspired by elastic#198000

Add basic "real LLM" smoke tests to make sure agent builder's basic
functionalities are working as expected with our list of officially
supported models.

Smoke tests are triggered when the `agent-builder:run-smoke-tests` tag
is present on the PR, or when any of the following code has changes:
- the GenAI stack connectors
- the `inference` plugin and packages
- the agent builder plugin or packages

This PR re-uses the set of "CI" LLM connectors which was introduced in
elastic#198000 for the AI-Infra inference
tests.
eokoneyo pushed a commit to eokoneyo/kibana that referenced this pull request Dec 2, 2025
@pgayvallet @eokoneyo
[Agent Builder] add basic smoke tests (elastic#243248)
29324c7 
## Summary
Fix elastic/search-team#11697
Inspired by elastic#198000

Add basic "real LLM" smoke tests to make sure agent builder's basic
functionalities are working as expected with our list of officially
supported models.

Smoke tests are triggered when the `agent-builder:run-smoke-tests` tag
is present on the PR, or when any of the following code has changes:
- the GenAI stack connectors
- the `inference` plugin and packages
- the agent builder plugin or packages

This PR re-uses the set of "CI" LLM connectors which was introduced in
elastic#198000 for the AI-Infra inference
tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@legrego legrego legrego approved these changes

@delanni delanni delanni approved these changes

@dmlemeshko dmlemeshko dmlemeshko approved these changes

+1 more reviewer

@dgieselaar dgieselaar dgieselaar approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport:version Backport to applied version labels ci:all-gen-ai-suites release_note:skip Skip the PR/issue when compiling release notes Team:AI Infra Platform AppEx AI Infrastructure Team t// v8.18.0 v9.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@pgayvallet @elasticmachine @kibanamachine @dgieselaar @legrego @delanni @dmlemeshko