[Console] Use ES specification for autocomplete definitions by yuliacech · Pull Request #159241 · elastic/kibana

yuliacech · 2023-06-07T16:56:29Z

Summary

This PR adds a new package kbn-generate-console-definitions that will eventually replace the package kbn-spec-to-console. It also adds a new command to use the script in the new package. The new command can be used as following:

node scripts/generate_console_definitions.js --source <PATH_TO_ES_SPECIFICATION_REPO> where PATH_TO_ES_SPECIFICATION_FOLDER is the absolute path to the root of the ES specification repo, for example /Users/yulia/elastic/elasticsearch-specification. This command will generate autocomplete definitions in the folder KIBANA_ROOT/src/plugins/console/server/lib/json/generated.
Optionally --dest parameter can be passed to generate definitions in a different folder, relative to KIBANA_ROOT.

Basic script functionality was implemented in this PR:

Create the folder if doesn't exist yet
Remove all files in the folder before generating definitions
Load the specification schema and parse each endpoint
Create a file for each endpoint with the endpoint name, methods, patterns and doc urls.

Functionality that will be added in follow up PRs:

Url paramaters
Request body parameters
Availability property
Unit test for script functions

How to test

Checkout ES specification repo
Run the command with node scripts/generate_console_definitions.js --source <ES_SPECIFICATION_REPO> --emptyDest where <ES_SPECIFICATION_REPO> is the absolute path to the root of the ES specification repo
Check the changes to the generated files in the folder /KIBANA_REPO/src/plugins/console/server/lib/spec_definitions/json/generated and make sure they have a correct endpoint name, patterns, methods and doc links. We are not generating any url params, request body params or availability property for now.
Change the constant in the file KIBANA_REPO/src/plugins/console/common/constants/autocomplete_definitions.ts to a non-existent folder. Run the script node scripts/generate_console_definitions.js --source <ES_SPECIFICATION_REPO> and check that the folder has been created successfully
Re-run the command without --emptyDest flag targeting a folder that already contain some files. Check that the script fails and doesn't silently remove existing files
Run the help command node scripts/generate_console_definitions.js --help and check if the help message makes sense

…efinitions generation

…fix'

…se it in the new script. Fix the patterns with leading forward slash and doc urls.

…fix'

yuliacech · 2023-06-14T16:51:09Z

I will hold off on merging this until after the 8.9 FF.

…'t exist and to clean the folder if it already contains files

…fix'

yuliacech · 2023-06-15T16:07:43Z

packages/kbn-generate-console-definitions/src/generate_console_definitions.ts

+import fs from 'fs';
+import Path, { join } from 'path';
+import { ToolingLog } from '@kbn/tooling-log';
+


This file is mostly WIP and probably will change a lot in follow up PRs. But still feel free to review as usual

elasticmachine · 2023-06-15T16:08:34Z

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

alisonelizabeth

Nice job @yuliacech 🎉 This looks like a great start. I followed your testing instructions and everything worked as expected. I reviewed a handful of the generated autocomplete definitions and the definitions appeared correct. The only thing I noticed is the script seemed to remove the new line at the end of each file, which previously existed.

alisonelizabeth · 2023-06-15T19:09:39Z

packages/kbn-generate-console-definitions/package.json

+  "private": true,
+  "version": "1.0.0",
+  "license": "SSPL-1.0 OR Elastic License 2.0"
+}


[nit] add new line

packages/kbn-generate-console-definitions/src/cli.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

yuliacech · 2023-06-21T09:44:02Z

Thanks a lot for your review, @alisonelizabeth!
I added back missing new lines (had them disabled in my diffs view) in a90111b

alisonelizabeth

Thanks for addressing my feedback! Reviewed the commit a90111b and lgtm.

kibana-ci · 2023-06-23T19:05:26Z

💚 Build Succeeded

Buildkite Build
Commit: 927cb73

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`console`	204	205	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/generate-console-definitions`	-	1	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`console`	418.5KB	418.5KB	+31.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`console`	27.2KB	27.4KB	+179.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/generate-console-definitions`	-	1	+1

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	14	16	+2
`securitySolution`	416	420	+4
total			+6

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	15	17	+2
`securitySolution`	497	501	+4
total			+6

History

💛 Build #136853 was flaky a90111b
💛 Build #135699 was flaky 7804ccf
💚 Build #135602 succeeded a237de0
💔 Build #135563 failed 2340672
💛 Build #135295 was flaky 1b63e1e
💔 Build #134898 failed 52051aa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

## Summary Fixes #160528 Follow up to #159241 This PR adds logic for query parameters to the new script generating Console autocomplete definitions from ES specification. The logic is explained in details in the [file](https://github.com/elastic/kibana/pull/160515/files#diff-b6853462c38db4e237dbb3cdec9d9f6659aa3fdc5f96a6193f2c4bac1439db43) but here is a short version: - Currently, the autocomplete engine only works with url params of 2 types: boolean (`__flag__`) and a list of options (for example `['all', 'open', 'hidden']`). The script will convert all of the url params from the specification into this format: a boolean or a list. If there are no set options for a parameter, but a default value is known, then it will be converted into a list with 1 option, for example `['random']` so that the autocomplete engine will display it as a single suggestion. We also need to convert any numbers to strings, because they won't be displayed otherwise. - Endpoints in the specification have a property `request` which in turn has 2 properties describing url params: `attachedBehaviours` and `query`. Both object contain an array of `property`'s each describing a url param. `property` is configured with either a built in type (`string`, `number`, `boolean`) or defined type, for example `ExpandWildcards`. By finding the type `ExpandWildcards` in the specification, we can convert this type to a list of options `['open', 'all', 'none', 'hidden']`. ### How to test Similar to #159241, you need to re-generenate the definitions and see if local changes to definitions files make sense. 1. Checkout the ES specification [repo](https://github.com/elastic/elasticsearch-specification) 2. Run the command `node scripts/generate_console_definitions.js --source <ES_SPECIFICATION_REPO> --emptyDest` 3. Check the changes in the folder `KIBANA_REPO/src/plugins/console/server/lib/spec_definitions/json/generated` #### Intended changes to the definitions files - Most of endpoints have 4 default url params that previously were not in the definitions files but added to all endpoints in this [file](https://github.com/elastic/kibana/blob/main/src/plugins/console/public/lib/autocomplete/url_params.js). These params are configured in the interface `CommonQueryParameters` in the specification (see this [file](https://github.com/elastic/elasticsearch-specification/blob/main/specification/_spec_utils/behaviors.ts)). <details> The interface in the specification ```js export interface CommonQueryParameters { error_trace?: boolean filter_path?: string | string[] human?: boolean pretty?: boolean } ``` The converted url params ```json "error_trace": "__flag__", "filter_path": [], "human": "__flag__", "pretty": "__flag__", ``` </details> - Previously existing `url_components` property in the definitions is deleted and this change will be addressed separately. (not sure it is currently working but added a task to the meta issue) - Previously numbers were configured as `0` or `0.0` but that is not currently displayed in suggestions. Instead, the new script converts numbers to strings and if any default value is present, it will be displayed as a suggestion. --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…63301) ## Summary This PR uses the new script to generate autocomplete definitions for Dev Tools Console from the ES specification repo. #### Definitions changes - New property `availability` is added to filter out endpoints that are not available in Serverless - Some endpoints' query parameters have more details now, for example common query params are now defined in definitions ```json "url_params": { "error_trace": "__flag__", "filter_path": [], "human": "__flag__", "pretty": "__flag__" }, ``` - Url components in few endpoints are removed, but those were added to overrides files in #163096 - Documentation links contain `{branch}` instead of `master` (fix for that added in #159241) #### Script changes - The logic for generating `availability` for endpoint has been updated based on the feedback from the Clients team. Details added to the script file. - Added a few "safe guards" to the spots in the script where an unexpected type of data might be coming from the ES specification schema #### Console changes - Fixed the autocomplete request on Serverless (we might need a proper fix for that, details in #163318) Also updates to readme files both in Console and the new script. I will remove the old script in a separate PR. ## Screenshots "ILM" autocomplete suggestions displayed on stateful <img width="583" alt="Screenshot 2023-08-07 at 17 47 48" src="https://github.com/elastic/kibana/assets/6585477/641a48b0-fb1a-4d3b-a8c9-99eab8795510"> "ILM" autocomplete suggestions not displayed on serverless <img width="572" alt="Screenshot 2023-08-07 at 17 35 16" src="https://github.com/elastic/kibana/assets/6585477/a1ee5468-eb9f-4f52-81d5-c661b06f8ceb"> ## How to test - Start Kibana on stateful (`yarn start`) and check that autocomplete suggestions are working as before (no changes) - Start Kibana on serverless (`yarn start --serverless`) and check that autocomplete suggestions are not displayed for endpoints blocked/internal on serverless. ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

yuliacech added 2 commits June 7, 2023 12:23

[Console] Create a new package to generate autocomplete definitions

2cc1e5f

[Console] Add Kibana a script command and implementation basics for d…

bd76003

…efinitions generation

yuliacech added Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more t// release_note:skip Skip the PR/issue when compiling release notes v8.9.0 Feature:Console Dev Tools Console Feature labels Jun 7, 2023

kibanamachine and others added 6 commits June 7, 2023 17:03

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

9b58966

…fix'

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

fdcddb6

[Console] Remove the new folder

e6754ee

[Console] Add options for source and dest folders

52051aa

[Console] Export a constant for autocomplete definitions folder and u…

e0b3c18

…se it in the new script. Fix the patterns with leading forward slash and doc urls.

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

1b63e1e

…fix'

yuliacech added v8.10.0 and removed v8.9.0 labels Jun 14, 2023

yuliacech and others added 4 commits June 15, 2023 13:23

[Console] Add functionality to create the definitions folder if doesn…

751ac76

…'t exist and to clean the folder if it already contains files

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

2340672

…fix'

Merge branch 'main' into console/new_definitions

a237de0

[Console] Add description and usage to the command

7804ccf

yuliacech commented Jun 15, 2023

View reviewed changes

yuliacech marked this pull request as ready for review June 15, 2023 16:08

yuliacech requested a review from a team as a code owner June 15, 2023 16:08

alisonelizabeth reviewed Jun 15, 2023

View reviewed changes

yuliacech and others added 5 commits June 21, 2023 11:08

Update packages/kbn-generate-console-definitions/src/cli.ts

5b66edd

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

Update packages/kbn-generate-console-definitions/src/cli.ts

ad59531

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

Update packages/kbn-generate-console-definitions/src/cli.ts

70045a0

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

Merge branch 'main' into console/new_definitions

ddc250f

[Console] Add back a new line at the end of the definitions files

a90111b

yuliacech force-pushed the console/new_definitions branch from 94e3e07 to a90111b Compare June 21, 2023 09:43

yuliacech mentioned this pull request Jun 21, 2023

[Console] Filter autocomplete suggestions for unavailable endpoints in Serverless #160160

Closed

alisonelizabeth approved these changes Jun 22, 2023

View reviewed changes

yuliacech and others added 2 commits June 23, 2023 20:03

Merge branch 'main' into console/new_definitions

aa781d5

[Console] Fix the new line

927cb73

yuliacech enabled auto-merge (squash) June 23, 2023 18:04

yuliacech merged commit 0a7ee08 into elastic:main Jun 23, 2023

kibanamachine added the backport:skip This PR does not require backporting label Jun 23, 2023

This was referenced Jun 26, 2023

[Console] Implement the logic for query params #160528

Closed

[Console] Implement the logic for body params #160533

Open

[Console] Update the logic for variables #160539

Closed

[Console] Add logic for query params #160515

Merged

yuliacech mentioned this pull request Jul 11, 2023

[Console] Implement logic to generate body params #161684

Closed

yuliacech mentioned this pull request Aug 7, 2023

[Console] Generate autocomplete definitions from ES specification #163301

Merged

8 tasks

yuliacech deleted the console/new_definitions branch February 15, 2024 12:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Console] Use ES specification for autocomplete definitions#159241

[Console] Use ES specification for autocomplete definitions#159241
yuliacech merged 19 commits intoelastic:mainfrom
yuliacech:console/new_definitions

yuliacech commented Jun 7, 2023 •

edited

Loading

Uh oh!

yuliacech commented Jun 14, 2023

Uh oh!

yuliacech Jun 15, 2023

Uh oh!

elasticmachine commented Jun 15, 2023

Uh oh!

alisonelizabeth left a comment •

edited

Loading

Uh oh!

alisonelizabeth Jun 15, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuliacech commented Jun 21, 2023

Uh oh!

alisonelizabeth left a comment

Uh oh!

kibana-ci commented Jun 23, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

yuliacech commented Jun 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test

Uh oh!

yuliacech commented Jun 14, 2023

Uh oh!

yuliacech Jun 15, 2023

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jun 15, 2023

Uh oh!

alisonelizabeth left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alisonelizabeth Jun 15, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuliacech commented Jun 21, 2023

Uh oh!

alisonelizabeth left a comment

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Jun 23, 2023

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yuliacech commented Jun 7, 2023 •

edited

Loading

alisonelizabeth left a comment •

edited

Loading