[8.15] [HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler` (#189262) by lcawl · Pull Request #190541 · elastic/kibana

lcawl · 2024-08-14T18:26:40Z

Backport

This will backport the bundler aspects of the following commits from main to 8.15:

Questions ?

Please refer to the Backport tool documentation

obltmachine · 2024-08-14T18:26:53Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

lcawl · 2024-08-14T21:11:04Z

It seems like #188110 and all subsequent PRs in https://github.com/elastic/kibana/commits/main/packages/kbn-openapi-bundler must be backported to 8.15 to address the build issues.

…88110) **Addresses:** elastic#186356 ## Summary This PR adds OpenAPI spec files merger utility (programmatic API). It provides a similar functionality as `npx @redocly/cli join` does and takes into account [discussion results](elastic#183019 (comment)) - provides a simple way to produce a single Kibana OpenAPI bundle - extends `requestBody` and `responses` MIME types with a version parameters `Elastic-Api-Version=<version>` to avoid different API endpoint versions conflicts - has flexibility to adjust Kibana needs The utility is exposed from `kbn-openapi-bundler` package. ## Details **OpenAPI merger** is a tool for merging multiple OpenAPI specification files. It's useful to merge already processed specification files to produce a result bundle. **OpenAPI bundler** uses the merger under the hood to merge bundled OpenAPI specification files. Exposed externally merger is a wrapper of the bundler's merger but extended with an ability to parse JSON files and forced to produce a single result file. It is able to read OpenAPI spec files defined in JSON and YAML formats. The result file is always written in YAML format where every `requestBody` and response in `responses` extended with document's `info.version` value added as a MIME type parameter `Elastic-Api-Version=<version>`. Currently package supports only programmatic API. As the next step you need to create a JavaScript script file like below ```ts require('../../src/setup_node_env'); const { resolve } = require('path'); const { merge } = require('@kbn/openapi-bundler'); const { REPO_ROOT } = require('@kbn/repo-info'); (async () => { await merge({ sourceGlobs: [ `${REPO_ROOT}/my/path/to/spec1.json`, `${REPO_ROOT}/my/path/to/spec2.yml`, `${REPO_ROOT}/my/path/to/spec3.yaml`, ], outputFilePath: `${REPO_ROOT}/oas_docs/bundle.yaml`, mergedSpecInfo: { title: 'My merged OpenAPI specs', version: '1.0.0', }, }); })(); ``` Finally you should be able to run OpenAPI merger via ```bash node ./path/to/the/script.js ``` or it could be added to `package.json` and run via `yarn`. After running the script it will log different information and write a merged OpenAPI specification to a the provided path. ## Caveats Items below don't look critical at the moment and can be addressed later on. - It doesn't support merging of specs having different OpenAPI versions (Kibana's OpenAPI specs use version `3.0.x` but we should keep an eye on that) - It doesn't support top level `$ref` for - Path item - Request body - Responses

…unctionality (elastic#188812) **Resolves:** elastic#188817 This PR adds automatic shared components conflict resolution functionality for OpenAPI merger. It boils down to a similar result as `npx @redocly/cli join --prefix-components-with-info-prop title` produces by prefixing shared components with document's title in each source. OpenAPI bundler intentionally won't solve conflicts automatically since it's focused on bundling domain APIs where conflicts are usually indicators of upstream problems. While working with various OpenAPI specs it may happen that different specs use exactly the same name for some shared components but different definitions. It must be avoided inside one API domain but it's a usual situation when merging OpenAPI specs of different API domains. For example domains may define a shared `Id` or `404Response` schemas where `Id` is a string in one domain and a number in another. OpenAPI merger implemented in elastic#188110 and OpenAPI bundler implemented in elastic#171526 do not solve shared components related conflicts automatically. It works perfectly for a single API domain forcing engineers choosing shared schema names carefully. This PR adds automatic shared components conflict resolution for OpenAPI merger. It prefixes shared component names with a normalized document's title. OpenAPI bundler intentionally won't solve conflicts automatically since it's focused on bundling domain APIs where conflicts are usually indicators of upstream problems. Consider two following OpenAPI specs each defining local `MySchema` **spec1.schema.yaml** ```yaml openapi: 3.0.3 info: title: My endpoint version: '2023-10-31' paths: /api/some_api: get: operationId: MyEndpointGet responses: '200': content: application/json: schema: $ref: '#/components/schemas/MySchema' components: schemas: MySchema: type: string enum: - value1 ``` **spec2.schema.yaml** ```yaml openapi: 3.0.3 info: title: My another endpoint version: '2023-10-31' paths: /api/another_api: get: operationId: MyAnotherEndpointGet responses: '200': content: application/json: schema: $ref: '#/components/schemas/MySchema' components: schemas: MySchema: type: number ``` and a script to merge them ```js require('../../src/setup_node_env'); const { resolve } = require('path'); const { merge } = require('@kbn/openapi-bundler'); const { REPO_ROOT } = require('@kbn/repo-info'); (async () => { await merge({ sourceGlobs: [ `${REPO_ROOT}/oas_docs/spec1.schema.yaml`, `${REPO_ROOT}/oas_docs/spec2.schema.yaml`, ], outputFilePath: resolve(`${REPO_ROOT}/oas_docs/merged.yaml`), options: { mergedSpecInfo: { title: 'Merge result', version: 'my version', }, }, }); })(); ``` will be merged successfully to **merged.yaml** ```yaml openapi: 3.0.3 info: title: Merge result version: 'my version' paths: /api/another_api: get: operationId: MyAnotherEndpointGet responses: '200': content: application/json; Elastic-Api-Version=2023-10-31: schema: $ref: '#/components/schemas/My_another_endpoint_MySchema' /api/some_api: get: operationId: MyEndpointGet responses: '200': content: application/json; Elastic-Api-Version=2023-10-31: schema: $ref: '#/components/schemas/My_endpoint_MySchema' components: schemas: My_another_endpoint_MySchema: type: number My_endpoint_MySchema: enum: - value1 type: string ```

…ce (elastic#189472) **Resolves:** elastic#188817 ## Summary This PR handles OpenAPI discriminator `mapping` (missing in elastic#188812) field by prefixing local references with a namespace (see elastic#188812 for more namespace details). It throws an error If mapping uses external references. ## How to test? Let's consider the following OpenAPI spec **spec1.schema.yaml** ```yaml openapi: 3.0.3 info: title: Spec1 version: '2023-10-31' paths: /api/some_api: get: responses: 200: content: 'application/json': schema: oneOf: - $ref: '#/components/schemas/Cat' - $ref: '#/components/schemas/Dog' - $ref: '#/components/schemas/Lizard' discriminator: propertyName: petType mapping: dog: '#/components/schemas/Dog' components: schemas: Pet: type: object required: [petType] properties: petType: type: string Cat: allOf: - $ref: '#/components/schemas/Pet' - type: object properties: name: type: string Dog: allOf: - $ref: '#/components/schemas/Pet' - type: object properties: bark: type: string Lizard: allOf: - $ref: '#/components/schemas/Pet' - type: object properties: lovesRocks: type: boolean ``` and a merging script ```js const { merge } = require('@kbn/openapi-bundler'); (async () => { await merge({ sourceGlobs: [ `path/to/spec1.schema.yaml`, ], outputFilePath: 'output.yaml, }); })(); ``` After running it it will produce the following bundler with references in `mapping` property prefixed with the spec title. ```yaml openapi: 3.0.3 info: title: Some title version: 1 paths: /api/some_api: get: responses: '200': content: application/json: schema: discriminator: mapping: dog: '#/components/schemas/Spec1_Dog' propertyName: petType oneOf: - $ref: '#/components/schemas/Spec1_Cat' - $ref: '#/components/schemas/Spec1_Dog' - $ref: '#/components/schemas/Spec1_Lizard' components: schemas: Spec1_Cat: allOf: - $ref: '#/components/schemas/Spec1_Pet' - type: object properties: name: type: string Spec1_Dog: allOf: - $ref: '#/components/schemas/Spec1_Pet' - type: object properties: bark: type: string Spec1_Lizard: allOf: - $ref: '#/components/schemas/Spec1_Pet' - type: object properties: lovesRocks: type: boolean Spec1_Pet: type: object properties: petType: type: string required: - petType ```

…for the result OpenAPI bundle (elastic#189348) **Resolves:** elastic#189269 **Resolves:** elastic#189270 This PR adds an ability to specify OpenAPI `servers` and security requirements (`security`) to be used in the result bundle. `servers` and/or `security` in the source OpenAPI specs are be dropped when custom `servers` and/or `security` provided. Kibana is usually deployed at a single access point and manages authentication in a central way. That way it's much more convenient to have control on what `servers` and `security` are present in the result bundles. It will help to avoid conflicts, duplicates and update them in centralized way. This PR extends OpenAPI bundler configuration options with `prototypeDocument`. "Prototype" in the name means it's a prototype for the result. The bundler uses certain properties from that prototype OpenAPI document to add them to the result OpenAPI bundle. The following properties are used - `info` representing OpenAPI Info object (former `options.specInfo`) - `servers` OpenAPI Server Object Array - `security` + `components.securitySchemes` OpenAPI Security Requirement Object Array + OpenAPI Security Schemes Object (validation checks that both fields are set otherwise an error is thrown) For convenience `prototypeDocument` could be specified as a string path to a file containing prototype OpenAPI document. `prototypeDocument` can be specified for `bundle` and `merge` utilities like the following **bundle** ```js const { bundle } = require('@kbn/openapi-bundler'); (async () => { await bundle({ sourceGlob: 'source/glob/*.yaml', outputFilePath: 'output/bundle.yaml, options: { prototypeDocument: { info: { title: 'Some title', description: 'Some description', }, servers: [{ url: 'https://{kibana_url}', variables: { kibana_url: { default: 'localhost:5601', } } }], security: [{ ApiKeyAuth: [] }], components: { securitySchemes: { ApiKeyAuth: { type: 'apiKey', in: 'header', name: 'Authorization', } } } }, }, }); ``` **bundle** with external prototype document ```js const { bundle } = require('@kbn/openapi-bundler'); (async () => { await bundle({ sourceGlob: 'source/glob/*.yaml', outputFilePath: 'output/bundle.yaml, options: { prototypeDocument: 'path/to/prototype_document.yaml',, }, }); ``` **merge** ```js const { merge } = require('@kbn/openapi-bundler'); (async () => { await merge({ sourceGlobs: [ 'absolute/path/to/file.yaml`, 'some/glob/*.schema.yaml', ], outputFilePath: 'output/file/path/bundle.yaml', options: { prototypeDocument: { info: { title: 'Some title', description: 'Some description', }, servers: [{ url: 'https://{kibana_url}', variables: { kibana_url: { default: 'localhost:5601', } } }], security: [{ ApiKeyAuth: [] }], components: { securitySchemes: { ApiKeyAuth: { type: 'apiKey', in: 'header', name: 'Authorization', } } } }, }, }); })(); ``` **merge** with external prototype document ```js const { merge } = require('@kbn/openapi-bundler'); (async () => { await merge({ sourceGlobs: [ 'absolute/path/to/file.yaml`, 'some/glob/*.schema.yaml', ], outputFilePath: 'output/file/path/bundle.yaml', options: { prototypeDocument: 'path/to/prototype_document.yaml', }, }); })(); ``` The result bundles will contain specified `servers` and `security` while source `servers` and `security` will be dropped.

…e result bundle (elastic#189482) **Resolves:** elastic#189463 ## Summary This PR adds functionality to produce result bundle with by default sorted root level OpenAPI `tags` alphabetically by name as requested by the Docs Engineering team. ## Details Bump.sh (new API reference documentation platform) uses OpenAPI tags for grouping API endpoints together. It displays tags encountered order which isn't always a desired one. To streamline displaying tags we need them sorted alphabetically by name. ## Screenshots **Example API reference documentation page BEFORE** ![image](https://github.com/user-attachments/assets/f66f0488-2a94-4e40-be91-808782b489b2) **Example API reference documentation page AFTER** ![image](https://github.com/user-attachments/assets/24ef9844-3d9d-4309-a1f0-69a2a9a23f08)

**Resolves:** elastic#183375 This PR implements functionality assigning a provided tag to OpenAPI `Operation object`s in the result bundle. Specified tag is also added as the only root level OpenAPI tag. This approach allows to produce domain bundles having a single tag assigned. At the next step domain bundles are merged together into single Kibana bundle where tags will allow to properly display grouping at Bump.sh (API reference documentation platform). Bump.sh (our new API reference documentation platform) uses OpenAPI tags for grouping API endpoints. It supports only one tag per endpoint. This PR facilitates preparation of Kibana OpenAPI bundle to be uploaded to Bump.sh by implementing functionality assigning a provided tag to OpenAPI `Operation object`s in the result domain bundles. It's implemented by providing an optional configuration option `assignTag` whose format is OpenAPI Tag Object. When `assignTag` isn't specified the bundler merges existing tags. Consider the following bundling configuration ```js const { bundle } = require('@kbn/openapi-bundler'); bundle({ // ... options: { assignTag: { name: 'Some Domain API tag name', description: 'Some Domain API description', externalDocs: { url: 'https://some-external-documentation-url', description: 'External documentation description', } }, }); ``` and source OpenAPI specs **spec1.schema.yaml** ```yaml openapi: 3.0.3 info: title: Spec1 version: '2023-10-31' paths: /api/some_api: get: tags: ['Some local tag'] responses: 200: content: 'application/json': schema: type: string ``` **spec2.schema.yaml** ```yaml openapi: 3.0.3 info: title: Spec2 version: '2023-10-31' paths: /api/some_api: post: tags: ['Some global tag'] responses: 200: content: 'application/json': schema: type: string tags: - name: Some global tag ``` **spec2.schema.yaml** ```yaml openapi: 3.0.3 info: title: Spec3 version: '2023-10-31' paths: /api/another_api: get: responses: 200: content: 'application/json': schema: type: string ``` After bundling above OpenAPI specs with the provided bundling script we'll get the following **domain-bundle.schema.yaml** ```yaml openapi: 3.0.3 info: title: Bundled document version: '2023-10-31' paths: /api/some_api: get: tags: ['Some Domain API tag name'] responses: 200: content: 'application/json': schema: type: string post: tags: ['Some Domain API tag name'] responses: 200: content: 'application/json': schema: type: string /api/another_api: get: tags: ['Some Domain API tag name'] responses: 200: content: 'application/json': schema: type: string tags: - name: Some Domain API tag name description: Some Domain API description externalDocs: url: 'https://some-external-documentation-url' description: External documentation description ```

…c#189262) **Addresses:** elastic#186356 **Relates to:** elastic#184428 This PR adds a merging JS script based on the utility implemented in elastic#186356. Resulted OpenAPI bundle as committed in `oas_docs/output/kibana.serverless.bundled.yaml`. elastic#188110 implements and exposes `merge` utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below ```js const { merge } = require('@kbn/openapi-bundler'); (async () => { await merge({ sourceGlobs: [/* a list of source globs goes here */], outputFilePath: 'path/to/the/output/file.yaml', }); })(); ``` The JS script added in this PR includes source OpenAPI specs presented in `oas_docs/makefile` plus Security Solution OpenAPI specs based on elastic#184428. **To run** the script use the following command from Kibana root folder ```bash node ./oas_docs/scripts/merge_serverless_oas.js ``` Running Spectral OpenAPI linter on the result bundle shows a number of errors caused by `no-$ref-siblings` rule. This caused by the current code generator implementation which requires `default` property to be set next to `$ref` though it's not correct for OpenAPI `3.0.3` while it's allowed in `3.1`. It seems that Bump.sh handles such cases properly though by properly showing a default value. We need to analyze the problem and decide if/when we should fix it. The rest of warnings look fixable and will be addressed in the next stage after setting up linter rules. Since `@kbn/openapi-bundler` package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means `oas_docs/makefile` should be superseded by JS script(s) using `merge` utility form `@kbn/openapi-bundler` package. `@kbn/openapi-bundler` SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting `spectral-cli` for linting purposes.

maximpn

LGTM

I doubled checked for missing changes but everything looks good. Local testing didn't reveal any issues neither.

kibana-ci · 2024-08-20T18:43:01Z

💚 Build Succeeded

Buildkite Build
Commit: bc31c59

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/openapi-bundler`	6	12	+6

Unknown metric groups

API count

id	before	after	diff
`@kbn/openapi-bundler`	6	12	+6

ESLint disabled in files

id	before	after	diff
`@kbn/openapi-bundler`	0	2	+2

ESLint disabled line counts

id	before	after	diff
`@kbn/openapi-bundler`	0	2	+2

Total ESLint disabled count

id	before	after	diff
`@kbn/openapi-bundler`	0	4	+4

History

💔 Build #228598 failed 7f933d1
💛 Build #228432 was flaky 339f257
💔 Build #227750 failed 90c0c1e98a51c2f1ab8aeb6c3335b2ad8559c7c8

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

lcawl added the backport This PR is a backport of another PR label Aug 14, 2024

lcawl enabled auto-merge (squash) August 14, 2024 18:26

lcawl mentioned this pull request Aug 14, 2024

[HTTP/OAS] Merge OpenAPI specs by using kbn-openapi-bundler #189262

Merged

lcawl disabled auto-merge August 14, 2024 18:27

lcawl marked this pull request as draft August 15, 2024 14:44

maximpn and others added 8 commits August 19, 2024 16:18

Edit merge scripts

c31d41c

lcawl force-pushed the backport/8.15/pr-189262 branch from 90c0c1e to c31d41c Compare August 20, 2024 00:04

lcawl added 2 commits August 19, 2024 17:05

Regenerate output

2847000

Merge branch '8.15' into backport/8.15/pr-189262

339f257

lcawl marked this pull request as ready for review August 20, 2024 00:10

lcawl requested a review from maximpn August 20, 2024 00:10

maximpn approved these changes Aug 20, 2024

View reviewed changes

Merge branch '8.15' into backport/8.15/pr-189262

7f933d1

lcawl enabled auto-merge (squash) August 20, 2024 14:44

lcawl added 2 commits August 20, 2024 10:11

Merge branch '8.15' into backport/8.15/pr-189262

b5a78ce

Merge branch '8.15' into backport/8.15/pr-189262

bc31c59

lcawl merged commit 07194fd into elastic:8.15 Aug 20, 2024

lcawl deleted the backport/8.15/pr-189262 branch August 20, 2024 19:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.15] [HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler` (#189262)#190541

[8.15] [HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler` (#189262)#190541
lcawl merged 13 commits intoelastic:8.15from
lcawl:backport/8.15/pr-189262

lcawl commented Aug 14, 2024 •

edited

Loading

Uh oh!

obltmachine commented Aug 14, 2024

Uh oh!

lcawl commented Aug 14, 2024

Uh oh!

maximpn left a comment •

edited

Loading

Uh oh!

kibana-ci commented Aug 20, 2024

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

lcawl commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backport

Questions ?

Uh oh!

obltmachine commented Aug 14, 2024

🤖 GitHub comments

Uh oh!

lcawl commented Aug 14, 2024

Uh oh!

maximpn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Aug 20, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lcawl commented Aug 14, 2024 •

edited

Loading

maximpn left a comment •

edited

Loading