Skip to content

[Security Solution] Allow to specify custom servers and security for the result OpenAPI bundle#189348

Merged
maximpn merged 8 commits intoelastic:mainfrom
maximpn:implement-explicit-overrides-in-oas-bundler
Jul 31, 2024
Merged

[Security Solution] Allow to specify custom servers and security for the result OpenAPI bundle#189348
maximpn merged 8 commits intoelastic:mainfrom
maximpn:implement-explicit-overrides-in-oas-bundler

Conversation

@maximpn
Copy link
Copy Markdown
Contributor

@maximpn maximpn commented Jul 29, 2024

Resolves: #189269
Resolves: #189270

Summary

This PR adds an ability to specify OpenAPI servers and security requirements (security) to be used in the result bundle. servers and/or security in the source OpenAPI specs are be dropped when custom servers and/or security provided.

Details

Kibana is usually deployed at a single access point and manages authentication in a central way. That way it's much more convenient to have control on what servers and security are present in the result bundles. It will help to avoid conflicts, duplicates and update them in centralized way.

This PR extends OpenAPI bundler configuration options with prototypeDocument. "Prototype" in the name means it's a prototype for the result. The bundler uses certain properties from that prototype OpenAPI document to add them to the result OpenAPI bundle. The following properties are used

  • info representing OpenAPI Info object (former options.specInfo)
  • servers OpenAPI Server Object Array
  • security + components.securitySchemes OpenAPI Security Requirement Object Array + OpenAPI Security Schemes Object (validation checks that both fields are set otherwise an error is thrown)

For convenience prototypeDocument could be specified as a string path to a file containing prototype OpenAPI document.

How to test?

prototypeDocument can be specified for bundle and merge utilities like the following

bundle

const { bundle } = require('@kbn/openapi-bundler');

(async () => {
  await bundle({
    sourceGlob: 'source/glob/*.yaml',
    outputFilePath: 'output/bundle.yaml,
    options: {
      prototypeDocument: {
        info: {
          title: 'Some title',
          description: 'Some description',
        },
        servers: [{
          url: 'https://{kibana_url}',
          variables: {
            kibana_url: {
              default: 'localhost:5601',
            }
          }
        }],
        security: [{ ApiKeyAuth: [] }],
        components: {
          securitySchemes: {
            ApiKeyAuth: {
              type: 'apiKey',
              in: 'header',
              name: 'Authorization',
            }
          }
        }
      },
    },
  });

bundle with external prototype document

const { bundle } = require('@kbn/openapi-bundler');

(async () => {
  await bundle({
    sourceGlob: 'source/glob/*.yaml',
    outputFilePath: 'output/bundle.yaml,
    options: {
      prototypeDocument: 'path/to/prototype_document.yaml',,
    },
  });

merge

const { merge } = require('@kbn/openapi-bundler');

(async () => {
  await merge({
    sourceGlobs: [
      'absolute/path/to/file.yaml`,
      'some/glob/*.schema.yaml',
    ],
    outputFilePath: 'output/file/path/bundle.yaml',
    options: {
      prototypeDocument: {
        info: {
          title: 'Some title',
          description: 'Some description',
        },
        servers: [{
          url: 'https://{kibana_url}',
          variables: {
            kibana_url: {
              default: 'localhost:5601',
            }
          }
        }],
        security: [{ ApiKeyAuth: [] }],
        components: {
          securitySchemes: {
            ApiKeyAuth: {
              type: 'apiKey',
              in: 'header',
              name: 'Authorization',
            }
          }
        }
      },
    },
  });
})();

merge with external prototype document

const { merge } = require('@kbn/openapi-bundler');

(async () => {
  await merge({
    sourceGlobs: [
      'absolute/path/to/file.yaml`,
      'some/glob/*.schema.yaml',
    ],
    outputFilePath: 'output/file/path/bundle.yaml',
    options: {
      prototypeDocument: 'path/to/prototype_document.yaml',
    },
  });
})();

The result bundles will contain specified servers and security while source servers and security will be dropped.

@maximpn maximpn added release_note:skip Skip the PR/issue when compiling release notes docs Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team v8.16.0 labels Jul 29, 2024
@maximpn maximpn self-assigned this Jul 29, 2024
@maximpn maximpn changed the title [Security Solution] Allow to specify custom servers and security for the result bundle [Security Solution] Allow to specify custom servers and security for the result OpenAPI bundle Jul 29, 2024
@maximpn maximpn force-pushed the implement-explicit-overrides-in-oas-bundler branch 2 times, most recently from e3d3681 to d22c1d7 Compare July 29, 2024 20:27
@maximpn maximpn marked this pull request as ready for review July 30, 2024 05:34
@maximpn maximpn requested review from a team as code owners July 30, 2024 05:34
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@maximpn maximpn requested a review from e40pud July 30, 2024 05:34
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@maximpn maximpn requested review from xcrzx and removed request for dplumlee July 30, 2024 05:35
Copy link
Copy Markdown
Contributor

@e40pud e40pud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DE changes LGTM

Copy link
Copy Markdown
Contributor

@xcrzx xcrzx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 Left just a couple of nits.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these two bundler calls also be wrapped in an IIFE?

Comment on lines 45 to 50
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks the same as in bundler, tet's deduplicate this logic

Comment on lines 21 to 23
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this condition relevant? It doesn't look like anything other than an object can be returned from readFile.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally JSON/YAML files may contain an array, primitive values or an empty string. These files are invalid OpenAPI documents. Even not all objects are valid OpenAPI documents but that validation is omitted here since it looks for specific keys in that object.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't version be overridable by the prototypeDoc.info.version?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bundle utility groups OpenAPI specs by info.version. blankOasDocumentFactory constructs a blank OpenAPI document with a desired info.vesion.

merge utility on the other hand should produce result bundles with specified version.

@maximpn maximpn force-pushed the implement-explicit-overrides-in-oas-bundler branch from 2b555f2 to 9fe2267 Compare July 31, 2024 06:32
@kibana-ci
Copy link
Copy Markdown

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id before after diff
@kbn/openapi-bundler 1 2 +1

Total ESLint disabled count

id before after diff
@kbn/openapi-bundler 3 4 +1

History

  • 💛 Build #224517 was flaky d22c1d7c121a6a4667368d5ead9768360b3b7e05
  • 💔 Build #224308 failed e3d3681d484259ab0516ca10e408512d5b81a449

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @maximpn

@maximpn maximpn merged commit 51c8949 into elastic:main Jul 31, 2024
@kibanamachine kibanamachine added the backport:skip This PR does not require backporting label Jul 31, 2024
@maximpn maximpn deleted the implement-explicit-overrides-in-oas-bundler branch July 31, 2024 08:29
lcawl pushed a commit to lcawl/kibana that referenced this pull request Aug 20, 2024
…for the result OpenAPI bundle (elastic#189348)

**Resolves:** elastic#189269
**Resolves:** elastic#189270

This PR adds an ability to specify OpenAPI `servers` and security requirements (`security`) to be used in the result bundle. `servers` and/or `security` in the source OpenAPI specs are be dropped when custom  `servers` and/or `security` provided.

Kibana is usually deployed at a single access point and manages authentication in a central way. That way it's much more convenient to have control on what `servers` and `security` are present in the result bundles. It will help to avoid conflicts, duplicates and update them in centralized way.

This PR extends OpenAPI bundler configuration options with `prototypeDocument`. "Prototype" in the name means it's a prototype for the result. The bundler uses certain properties from that prototype OpenAPI document to add them to the result OpenAPI bundle. The following properties are used

- `info` representing OpenAPI Info object (former `options.specInfo`)
- `servers` OpenAPI Server Object Array
- `security` + `components.securitySchemes` OpenAPI Security Requirement Object Array + OpenAPI Security Schemes Object (validation checks that both fields are set otherwise an error is thrown)

For convenience `prototypeDocument` could be specified as a string path to a file containing prototype OpenAPI document.

`prototypeDocument` can be specified for `bundle` and `merge` utilities like the following

**bundle**
```js
const { bundle } = require('@kbn/openapi-bundler');

(async () => {
  await bundle({
    sourceGlob: 'source/glob/*.yaml',
    outputFilePath: 'output/bundle.yaml,
    options: {
      prototypeDocument: {
        info: {
          title: 'Some title',
          description: 'Some description',
        },
        servers: [{
          url: 'https://{kibana_url}',
          variables: {
            kibana_url: {
              default: 'localhost:5601',
            }
          }
        }],
        security: [{ ApiKeyAuth: [] }],
        components: {
          securitySchemes: {
            ApiKeyAuth: {
              type: 'apiKey',
              in: 'header',
              name: 'Authorization',
            }
          }
        }
      },
    },
  });
```

**bundle** with external prototype document
```js
const { bundle } = require('@kbn/openapi-bundler');

(async () => {
  await bundle({
    sourceGlob: 'source/glob/*.yaml',
    outputFilePath: 'output/bundle.yaml,
    options: {
      prototypeDocument: 'path/to/prototype_document.yaml',,
    },
  });
```

**merge**
```js
const { merge } = require('@kbn/openapi-bundler');

(async () => {
  await merge({
    sourceGlobs: [
      'absolute/path/to/file.yaml`,
      'some/glob/*.schema.yaml',
    ],
    outputFilePath: 'output/file/path/bundle.yaml',
    options: {
      prototypeDocument: {
        info: {
          title: 'Some title',
          description: 'Some description',
        },
        servers: [{
          url: 'https://{kibana_url}',
          variables: {
            kibana_url: {
              default: 'localhost:5601',
            }
          }
        }],
        security: [{ ApiKeyAuth: [] }],
        components: {
          securitySchemes: {
            ApiKeyAuth: {
              type: 'apiKey',
              in: 'header',
              name: 'Authorization',
            }
          }
        }
      },
    },
  });
})();
```

**merge** with external prototype document
```js
const { merge } = require('@kbn/openapi-bundler');

(async () => {
  await merge({
    sourceGlobs: [
      'absolute/path/to/file.yaml`,
      'some/glob/*.schema.yaml',
    ],
    outputFilePath: 'output/file/path/bundle.yaml',
    options: {
      prototypeDocument: 'path/to/prototype_document.yaml',
    },
  });
})();
```

The result bundles will contain specified `servers` and `security` while source `servers` and `security` will be dropped.
lcawl added a commit that referenced this pull request Aug 20, 2024
…189262) (#190541)

# Backport

This will backport the bundler aspects of the following commits from
`main` to `8.15`:
- [[HTTP/OAS] Merge OpenAPI specs by using `kbn-openapi-bundler`
(#189262)](#189262)
 - #189621
 - #189482
 - #189348
 - #189472
 - #188110
 - #188812

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Maxim
Palenov","email":"maxim.palenov@elastic.co"},"sourceCommit":{"committedDate":"2024-08-13T10:45:35Z","message":"[HTTP/OAS]
Merge OpenAPI specs by using `kbn-openapi-bundler`
(#189262)\n\n**Addresses:**
https://github.com/elastic/kibana/issues/186356\r\n**Relates to:**
https://github.com/elastic/kibana/issues/184428\r\n\r\n##
Summary\r\n\r\nThis PR adds a merging JS script based on the utility
implemented in #186356. Resulted
OpenAPI bundle as committed in
`oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n##
Details\r\n\r\nhttps://github.com//pull/188110 implements
and exposes `merge` utility design to merge source OpenAPI specs without
processing. It's has only a programmatic API. To merge OpenAPI specs
it's required to add a JS script like below\r\n\r\n```js\r\nconst {
merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n
await merge({\r\n sourceGlobs: [/* a list of source globs goes here
*/],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n
});\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes
source OpenAPI specs presented in `oas_docs/makefile` plus Security
Solution OpenAPI specs based on
https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the
script use the following command from Kibana root
folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js
\r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI
specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows
a number of errors caused by `no-$ref-siblings` rule. This caused by the
current code generator implementation which requires `default` property
to be set next to `$ref` though it's not correct for OpenAPI `3.0.3`
while it's allowed in `3.1`. It seems that Bump.sh handles such cases
properly though by properly showing a default value.\r\n\r\nWe need to
analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest
of warnings look fixable and will be addressed in the next stage after
setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince
`@kbn/openapi-bundler` package is tailored specifically for Kibana it
should replace Redocly currently used to merge OpenAPI specs. It also
means `oas_docs/makefile` should be superseded by JS script(s) using
`merge` utility form `@kbn/openapi-bundler`
package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI
linters since it doesn't perform thorough linting. It's good if we
continue adopting `spectral-cli` for linting
purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","backport:skip","docs","Team:
SecuritySolution","Feature:OAS","v8.16.0"],"number":189262,"url":"https://github.com/elastic/kibana/pull/189262","mergeCommit":{"message":"[HTTP/OAS]
Merge OpenAPI specs by using `kbn-openapi-bundler`
(#189262)\n\n**Addresses:**
https://github.com/elastic/kibana/issues/186356\r\n**Relates to:**
https://github.com/elastic/kibana/issues/184428\r\n\r\n##
Summary\r\n\r\nThis PR adds a merging JS script based on the utility
implemented in #186356. Resulted
OpenAPI bundle as committed in
`oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n##
Details\r\n\r\nhttps://github.com//pull/188110 implements
and exposes `merge` utility design to merge source OpenAPI specs without
processing. It's has only a programmatic API. To merge OpenAPI specs
it's required to add a JS script like below\r\n\r\n```js\r\nconst {
merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n
await merge({\r\n sourceGlobs: [/* a list of source globs goes here
*/],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n
});\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes
source OpenAPI specs presented in `oas_docs/makefile` plus Security
Solution OpenAPI specs based on
https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the
script use the following command from Kibana root
folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js
\r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI
specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows
a number of errors caused by `no-$ref-siblings` rule. This caused by the
current code generator implementation which requires `default` property
to be set next to `$ref` though it's not correct for OpenAPI `3.0.3`
while it's allowed in `3.1`. It seems that Bump.sh handles such cases
properly though by properly showing a default value.\r\n\r\nWe need to
analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest
of warnings look fixable and will be addressed in the next stage after
setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince
`@kbn/openapi-bundler` package is tailored specifically for Kibana it
should replace Redocly currently used to merge OpenAPI specs. It also
means `oas_docs/makefile` should be superseded by JS script(s) using
`merge` utility form `@kbn/openapi-bundler`
package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI
linters since it doesn't perform thorough linting. It's good if we
continue adopting `spectral-cli` for linting
purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/189262","number":189262,"mergeCommit":{"message":"[HTTP/OAS]
Merge OpenAPI specs by using `kbn-openapi-bundler`
(#189262)\n\n**Addresses:**
https://github.com/elastic/kibana/issues/186356\r\n**Relates to:**
https://github.com/elastic/kibana/issues/184428\r\n\r\n##
Summary\r\n\r\nThis PR adds a merging JS script based on the utility
implemented in #186356. Resulted
OpenAPI bundle as committed in
`oas_docs/output/kibana.serverless.bundled.yaml`.\r\n\r\n##
Details\r\n\r\nhttps://github.com//pull/188110 implements
and exposes `merge` utility design to merge source OpenAPI specs without
processing. It's has only a programmatic API. To merge OpenAPI specs
it's required to add a JS script like below\r\n\r\n```js\r\nconst {
merge } = require('@kbn/openapi-bundler');\r\n\r\n(async () => {\r\n
await merge({\r\n sourceGlobs: [/* a list of source globs goes here
*/],\r\n outputFilePath: 'path/to/the/output/file.yaml',\r\n
});\r\n})();\r\n```\r\n\r\nThe JS script added in this PR includes
source OpenAPI specs presented in `oas_docs/makefile` plus Security
Solution OpenAPI specs based on
https://github.com/elastic/kibana/issues/184428.\r\n\r\n**To run** the
script use the following command from Kibana root
folder\r\n\r\n```bash\r\nnode ./oas_docs/scripts/merge_serverless_oas.js
\r\n```\r\n\r\n## Known linting issues with Security Solution OpenAPI
specs\r\n\r\nRunning Spectral OpenAPI linter on the result bundle shows
a number of errors caused by `no-$ref-siblings` rule. This caused by the
current code generator implementation which requires `default` property
to be set next to `$ref` though it's not correct for OpenAPI `3.0.3`
while it's allowed in `3.1`. It seems that Bump.sh handles such cases
properly though by properly showing a default value.\r\n\r\nWe need to
analyze the problem and decide if/when we should fix it.\r\n\r\nThe rest
of warnings look fixable and will be addressed in the next stage after
setting up linter rules.\r\n\r\n## Next steps\r\n\r\nSince
`@kbn/openapi-bundler` package is tailored specifically for Kibana it
should replace Redocly currently used to merge OpenAPI specs. It also
means `oas_docs/makefile` should be superseded by JS script(s) using
`merge` utility form `@kbn/openapi-bundler`
package.\r\n\r\n`@kbn/openapi-bundler` SHOULD NOT replace OpenAPI
linters since it doesn't perform thorough linting. It's good if we
continue adopting `spectral-cli` for linting
purposes.","sha":"7a2e7bef9644eaf04fd820d16b8ee137b3c00f2b"}}]}]
BACKPORT-->

---------

Co-authored-by: Maxim Palenov <maxim.palenov@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting docs release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.16.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants