[HTTP/OAS] Merge OpenAPI specs by using kbn-openapi-bundler

[maximpn]

Addresses: #186356
Relates to: #184428

Summary

This PR adds a merging JS script based on the utility implemented in #186356. Resulted OpenAPI bundle as committed in oas_docs/output/kibana.serverless.bundled.yaml.

Details

#188110 implements and exposes merge utility design to merge source OpenAPI specs without processing. It's has only a programmatic API. To merge OpenAPI specs it's required to add a JS script like below

const { merge } = require('@kbn/openapi-bundler');

(async () => {
  await merge({
   sourceGlobs: [/* a list of source globs goes here */],
   outputFilePath: 'path/to/the/output/file.yaml',
  });
})();

The JS script added in this PR includes source OpenAPI specs presented in oas_docs/makefile plus Security Solution OpenAPI specs based on #184428.

To run the script use the following command from Kibana root folder

node ./oas_docs/scripts/merge_serverless_oas.js 

Known linting issues with Security Solution OpenAPI specs

Running Spectral OpenAPI linter on the result bundle shows a number of errors caused by no-$ref-siblings rule. This caused by the current code generator implementation which requires default property to be set next to $ref though it's not correct for OpenAPI 3.0.3 while it's allowed in 3.1. It seems that Bump.sh handles such cases properly though by properly showing a default value.

We need to analyze the problem and decide if/when we should fix it.

The rest of warnings look fixable and will be addressed in the next stage after setting up linter rules.

Next steps

Since @kbn/openapi-bundler package is tailored specifically for Kibana it should replace Redocly currently used to merge OpenAPI specs. It also means oas_docs/makefile should be superseded by JS script(s) using merge utility form @kbn/openapi-bundler package.

@kbn/openapi-bundler SHOULD NOT replace OpenAPI linters since it doesn't perform thorough linting. It's good if we continue adopting spectral-cli for linting purposes.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[jloleysens]

Thank you for driving this @maximpn ! I have previewed the output on bump.sh and it's looking mostly good. Left a few comments I'd like to get your thoughts on.

[jloleysens]

Can we replace this make command with the new bundler command? Or do we want to keep both? Maybe more a q for @lcawl , but curious to hear your thoughts too @maximpn

[maximpn]

I didn't want to remove existing command since @lcawl might use it. Eventually I got it reorganized to use OpenAPI bundler in api-docs command and renamed existing api-docs to api-docs-redocly. It's up to @lcawl to remove it if it's unused.

[lcawl]

I think it's good to leave for now since there are some outstanding issues (e.g. some operationIds that are not unique) that need to be resolved before we start publishing the security APIs. We can switch to the new command when those are fixed.

[jloleysens]

I see several entries of Elastic-Api-Version with values defined that don't fit the public API contract of a date string. I'm not sure where these values are coming from, could we just fallback to 2023-10-31 if not otherwise defined for all endpoints?

Full set of instances + line nr here

68:            application/json; Elastic-Api-Version=0.2:
87:          application/json; Elastic-Api-Version=0.2:
160:          application/json; Elastic-Api-Version=0.2:
198:          application/json; Elastic-Api-Version=0.2:
215:            application/json; Elastic-Api-Version=0.2:
247:          application/json; Elastic-Api-Version=0.2:
267:            application/json; Elastic-Api-Version=0.2:
303:            application/json; Elastic-Api-Version=0.2:
384:            application/json; Elastic-Api-Version=0.2:
413:          application/json; Elastic-Api-Version=0.2:
424:            application/json; Elastic-Api-Version=0.2:
444:          application/json; Elastic-Api-Version=0.2:
455:            application/json; Elastic-Api-Version=0.2:
483:          application/json; Elastic-Api-Version=0.2:
515:            application/json; Elastic-Api-Version=0.2:
579:            application/json; Elastic-Api-Version=0.2:
602:            application/json; Elastic-Api-Version=0.2:
647:          application/json; Elastic-Api-Version=1.0.0:
664:            application/json; Elastic-Api-Version=1.0.0:
696:          application/json; Elastic-Api-Version=1.0.0:
719:            application/json; Elastic-Api-Version=1.0.0:
790:            application/json; Elastic-Api-Version=1.0.0:
1068:            application/json; Elastic-Api-Version=0.1:
1095:            application/json; Elastic-Api-Version=0.1:
1109:          application/json; Elastic-Api-Version=0.1:
1119:            application/json; Elastic-Api-Version=0.1:
1125:            application/json; Elastic-Api-Version=0.1:
1145:            application/json; Elastic-Api-Version=0.1:
1159:            application/json; Elastic-Api-Version=0.1:
1168:            application/json; Elastic-Api-Version=0.1:
1182:          application/json; Elastic-Api-Version=0.1:
1192:            application/json; Elastic-Api-Version=0.1:
1198:            application/json; Elastic-Api-Version=0.1:
1216:          application/json; Elastic-Api-Version=0.1:
1232:            application/json; Elastic-Api-Version=0.1:
1241:            application/json; Elastic-Api-Version=0.1:
1256:          application/json; Elastic-Api-Version=0.1:
1278:            application/json; Elastic-Api-Version=0.1:
1298:          application/json; Elastic-Api-Version=0.1:
1320:            application/json; Elastic-Api-Version=0.1:
1333:            application/json; Elastic-Api-Version=0.1:
1351:            application/json; Elastic-Api-Version=0.1:
1366:            application/json; Elastic-Api-Version=0.1:
1382:            application/json; Elastic-Api-Version=0.1:
1396:          application/json; Elastic-Api-Version=0.1:
1420:            application/json; Elastic-Api-Version=0.1:
1433:            application/json; Elastic-Api-Version=0.1:
1446:            application/json; Elastic-Api-Version=0.1:
1459:          application/json; Elastic-Api-Version=0.1:
1483:            application/json; Elastic-Api-Version=0.1:
1492:            application/json; Elastic-Api-Version=0.1:
1510:          application/json; Elastic-Api-Version=0.1:
1520:            application/json; Elastic-Api-Version=0.1:
1556:          application/json; Elastic-Api-Version=0.1:
1567:            application/json; Elastic-Api-Version=0.1:
3255:            application/json; Elastic-Api-Version=0.1:
3289:            application/json; Elastic-Api-Version=0.1:
3295:            application/json; Elastic-Api-Version=0.1:
7281:            application/json; Elastic-Api-Version=1.0.1:
7290:            application/json; Elastic-Api-Version=1.0.1:
7820:          application/json; Elastic-Api-Version=0.1:
7829:            application/json; Elastic-Api-Version=0.1:
7835:            application/json; Elastic-Api-Version=0.1:
7869:          application/json; Elastic-Api-Version=0.1:
7878:            application/json; Elastic-Api-Version=0.1:
7887:            application/json; Elastic-Api-Version=0.1:
7902:          application/json; Elastic-Api-Version=0.1:
7911:            application/json; Elastic-Api-Version=0.1:
7917:            application/json; Elastic-Api-Version=0.1:
7940:          application/json; Elastic-Api-Version=0.1:
7949:            application/json; Elastic-Api-Version=0.1:
7958:            application/json; Elastic-Api-Version=0.1:
7974:          application/json; Elastic-Api-Version=0.1:
7983:            application/json; Elastic-Api-Version=0.1:
7992:            application/json; Elastic-Api-Version=0.1:
8024:          application/json; Elastic-Api-Version=0.1:
8058:            application/x-ndjson; Elastic-Api-Version=0.1:
8068:            application/json; Elastic-Api-Version=0.1:
8194:            application/json; Elastic-Api-Version=0.1:
8200:            application/json; Elastic-Api-Version=0.1:
8259:          multipart/form-data; Elastic-Api-Version=0.1:
8278:            application/json; Elastic-Api-Version=0.1:
8324:            application/json; Elastic-Api-Version=0.1:
8373:          multipart/form-data; Elastic-Api-Version=0.1:
8438:            application/json; Elastic-Api-Version=0.1:
8482:            application/json; Elastic-Api-Version=0.1:
8504:          application/json; Elastic-Api-Version=0.1:
8520:            application/json; Elastic-Api-Version=0.1:
8526:            application/json; Elastic-Api-Version=0.1:
8544:            application/json; Elastic-Api-Version=0.1:
8550:            application/json; Elastic-Api-Version=0.1:
8574:          application/json; Elastic-Api-Version=0.1:
8590:            application/json; Elastic-Api-Version=0.1:
8596:            application/json; Elastic-Api-Version=0.1:
8613:          application/json; Elastic-Api-Version=0.1:
8620:            application/json; Elastic-Api-Version=0.1:
8626:            application/json; Elastic-Api-Version=0.1:
8632:            application/json; Elastic-Api-Version=0.1:
8657:            application/json; Elastic-Api-Version=0.1:
8663:            application/json; Elastic-Api-Version=0.1:
10132:            application/json; Elastic-Api-Version=1.0:
10138:            application/json; Elastic-Api-Version=1.0:
10144:            application/json; Elastic-Api-Version=1.0:
10150:            application/json; Elastic-Api-Version=1.0:
10156:            application/json; Elastic-Api-Version=1.0:
10173:          application/json; Elastic-Api-Version=1.0:
10180:            application/json; Elastic-Api-Version=1.0:
10186:            application/json; Elastic-Api-Version=1.0:
10192:            application/json; Elastic-Api-Version=1.0:
10198:            application/json; Elastic-Api-Version=1.0:
10204:            application/json; Elastic-Api-Version=1.0:
10223:          application/json; Elastic-Api-Version=1.0:
10232:            application/json; Elastic-Api-Version=1.0:
10238:            application/json; Elastic-Api-Version=1.0:
10244:            application/json; Elastic-Api-Version=1.0:
10266:            application/json; Elastic-Api-Version=1.0:
10272:            application/json; Elastic-Api-Version=1.0:
10278:            application/json; Elastic-Api-Version=1.0:
10284:            application/json; Elastic-Api-Version=1.0:
10309:            application/json; Elastic-Api-Version=1.0:
10315:            application/json; Elastic-Api-Version=1.0:
10321:            application/json; Elastic-Api-Version=1.0:
10327:            application/json; Elastic-Api-Version=1.0:
10333:            application/json; Elastic-Api-Version=1.0:
10351:          application/json; Elastic-Api-Version=1.0:
10358:            application/json; Elastic-Api-Version=1.0:
10364:            application/json; Elastic-Api-Version=1.0:
10370:            application/json; Elastic-Api-Version=1.0:
10376:            application/json; Elastic-Api-Version=1.0:
10382:            application/json; Elastic-Api-Version=1.0:
10402:            application/json; Elastic-Api-Version=1.0:
10408:            application/json; Elastic-Api-Version=1.0:
10414:            application/json; Elastic-Api-Version=1.0:
10420:            application/json; Elastic-Api-Version=1.0:
10426:            application/json; Elastic-Api-Version=1.0:
10448:            application/json; Elastic-Api-Version=1.0:
10454:            application/json; Elastic-Api-Version=1.0:
10460:            application/json; Elastic-Api-Version=1.0:
10466:            application/json; Elastic-Api-Version=1.0:
10488:            application/json; Elastic-Api-Version=1.0:
10494:            application/json; Elastic-Api-Version=1.0:
10500:            application/json; Elastic-Api-Version=1.0:
10506:            application/json; Elastic-Api-Version=1.0:
12724:        application/json; Elastic-Api-Version=0.2:
12730:        application/json; Elastic-Api-Version=0.2:
12750:        application/json; Elastic-Api-Version=0.2:

[maximpn]

This is a known issue and there are multiple ways to solve it.

The right way to solve it is say the bundler what version source OpenAPI specs implement. One of the ideas is to use info.x-implementation-version in the source OpenAPI specs as in this example PR.

The simplest approach for now is to fallback to 2023-10-31 when it's impossible to detect an implementation version. It's not future proof since it won't work when new versions pop up.

Security Solution OpenAPI specs use info.version as an implementation version (it works for us since we consider document version equal to implementation version and wanna bump document version when we have incompatible changes). The bundler takes info.version (there are no other known version fields in OpenAPI specs) and uses it for Elastic-Api-Version.

@jloleysens what solution do you prefer?

[jloleysens]

The simplest approach for now is to fallback to 2023-10-31 when it's impossible to detect an implementation version. It's not future proof since it won't work when new versions pop up.

IMO this seems OK. In future we can find a way to provide version for a set of specs like a file naming my_api.2024-20-1.yaml or path convention /2024-20-1/my_api.yaml. The default would just be to use the "first" version. WDYT?

[maximpn]

Sounds good for me if @lcawl don't have any objections. I'll make adjustments in this PR (should be few lines change).

In future we can find a way to provide version for a set of specs like a file naming my_api.2024-20-1.yaml or path convention /2024-20-1/my_api.yaml. The default would just be to use the "first" version. WDYT?

Any option (or multiple options) works well if we agree on that across Kibana.

[lcawl]

I think this is difficult to visualize until we start having more than one Elastic-Api-Version value. However it's implemented in the code and the bundler, we'd ideally show all of the header values and their associated request and response bodies in a single document. As long as this PR is moving in that direction, it is sufficient for docs purposes.

[jbudz]

.buildkite

[banderror]

Rule Management-owned changes and PR overall LGTM 👍

One weird thing is that GitHub:

• requested a review from @elastic/security-solution which is probably caused by changes in:
  • x-pack/plugins/security_solution/docs/openapi/ess/security_solution_detections_api_2023_10_31.bundled.schema.yaml
• but didn't request a review from @elastic/security-detection-rule-management because of changes in:
  • packages/kbn-openapi-bundler/src/bundler/merge_documents/enrich_with_version_mime_param.ts
  • x-pack/plugins/security_solution/common/api/detection_engine/rule_management/bulk_crud/bulk_delete_rules/bulk_delete_rules_route.schema.yaml

@maximpn Could you please check if there's anything wrong with the CODEOWNERS file?

Apart from that, thanks for your hard work on Security OpenAPI specs and all the automation tools you created for us and other teams in Security. The progress on that is amazing and the quality of the solution is top-notch, especially considering the tight timelines and the cope of work that we had. Thank you! 👏

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 244de4f

Metrics [docs]

✅ unchanged

History

• 💔 Build #227255 failed bc01448
• 💛 Build #227243 was flaky fc7903e0805f519d696ed853d5f9d8a5cc35e604
• 💔 Build #227215 failed f6772be7373608a892e0d3c32248fba3dfebe861
• 💚 Build #227001 succeeded 131ac8283c0dc4936a2e8a87c1e84a78ad5a92ab
• 💚 Build #226726 succeeded 9385e559d4d9641b220d9d39a1f17b67bf4b1afb
• 💛 Build #226263 was flaky 5e8af020d28e796fd08674052bc48c00b1a3518d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @maximpn

[maximpn]

@banderror

I checked CODEOWNERS file and haven't found anything suspicious. packages/kbn-openapi-bundler is assigned to @elastic/security-detection-rule-management as expected. The only thing we need to change there is assign domain OpenAPI bundles to appropriate teams instead of @elastic/security-solution .

Most probably GitHub's algorithm didn't handle review properly since I made changes to packages/kbn-openapi-bundler/src/bundler/merge_documents/enrich_with_version_mime_param.ts and
x-pack/plugins/security_solution/common/api/detection_engine/rule_management/bulk_crud/bulk_delete_rules/bulk_delete_rules_route.schema.yaml in the last commit. I'll keep an eye on that.

[lcawl]

💚 All backports created successfully

Status	Branch	Result
✅	8.15

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation