Fix for #3295 to ensure custom httpjson input with oauth2 auth and explicit scopes definition works by colin-stubbs · Pull Request #3297 · elastic/integrations

colin-stubbs · 2022-05-09T08:50:17Z

Render of auth.oauth2.scopes fails for custom httpjson input, this will fix the issue

Bug

What does this PR do?

Render of auth.oauth2.scopes fails for custom httpjson input, this will fix the issue #3295

Checklist

[X ] I have reviewed tips for building integrations and this pull request is aligned with them.
[ X] I have verified that all data streams collect metrics or logs.
[ X] I have added an entry to my package's changelog.yml file.
[ X] I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

None - custom httpjson input with oauth2 explicit scope requirement simply does not work.

How to test this PR locally

Use Custom HTTPJSON input against any Oauth2 authenticated API where scopes must be explicitly defined, e.g. Microsoft Graph Security API.

Related issues

Closes [httpjson] Render of auth.oauth2.scopes fails #3295

Screenshots

Example API response when input not accepted because YAML format is not detected,

{
    "statusCode": 400,
    "error": "Bad Request",
    "message": "Package policy is invalid: inputs.httpjson.streams.httpjson.generic.vars.oauth_scopes: Invalid format"
}

Example API response when input is accepted because YAML is detected, but due to rendering fail nothing is set, note auth.oauth2.scopes is null.

{
    "item": {
        "id": "9eab7f8f-c2a6-4f95-9330-dbf3c5287675",
        "version": "WzE5NzUzODY4LDJd",
        "name": "msgraph-security-api-alerts-equate-blah",
        "namespace": "equate",
        "description": "Ingests alerts from Microsoft MSGraph Security API for equate",
        "package": {
            "name": "httpjson",
            "title": "Custom HTTPJSON Input",
            "version": "1.2.0"
        },
        "enabled": true,
        "policy_id": "b9acba50-5cdf-11ec-802d-9b5a8de450fb",
        "output_id": "",
        "inputs": [
            {
                "policy_template": "generic",
                "type": "httpjson",
                "enabled": true,
                "streams": [
                    {
                        "data_stream": {
                            "type": "logs",
                            "dataset": "httpjson.generic"
                        },
                        "vars": {
                            "cursor": {
                                "type": "yaml",
                                "value": "#last_requested_at:\n#  value: '[[now]]'\n"
                            },
                            "request_proxy_url": {
                                "type": "text"
                            },
                            "request_timeout": {
                                "type": "text"
                            },
                            "request_retry_max_attempts": {
                                "type": "text"
                            },
                            "request_method": {
                                "type": "text",
                                "value": "GET"
                            },
                            "processors": {
                                "type": "yaml",
                                "value": "- add_fields:\r\n    target: ''\r\n    fields:\r\n        event.module: 'msgraph'\r\n        event.dataset: 'msgraph.unknown'"
                            },
                            "request_retry_wait_min": {
                                "type": "text"
                            },
                            "request_ssl": {
                                "type": "yaml",
                                "value": "#verification_mode: none\n"
                            },
                            "response_decode_as": {
                                "type": "text"
                            },
                            "oauth_endpoint_params": {
                                "type": "yaml",
                                "value": "#Param1:\n#  - ValueA\n#  - ValueB\n#Param2:\n#  - Value\n"
                            },
                            "password": {
                                "type": "password"
                            },
                            "request_rate_limit_limit": {
                                "type": "text"
                            },
                            "oauth_scopes": {
                                "type": "yaml",
                                "value": "- https://graph.microsoft.com/.default"
                            },
                            "oauth_azure_tenant_id": {
                                "type": "text"
                            },
                            "request_transforms": {
                                "type": "yaml",
                                "value": ""
                            },
                            "response_transforms": {
                                "type": "yaml",
                                "value": ""
                            },
                            "oauth_provider": {
                                "type": "text"
                            },
                            "oauth_id": {
                                "type": "text",
                                "value": "CLIENT_ID"
                            },
                            "request_redirect_max_redirects": {
                                "type": "text"
                            },
                            "request_rate_limit_remaining": {
                                "type": "text"
                            },
                            "oauth_google_credentials_json": {
                                "type": "text"
                            },
                            "request_redirect_headers_ban_list": {
                                "type": "text",
                                "value": []
                            },
                            "request_retry_wait_max": {
                                "type": "text"
                            },
                            "oauth_google_jwt_file": {
                                "type": "text"
                            },
                            "request_encode_as": {
                                "type": "text"
                            },
                            "oauth_token_url": {
                                "type": "text",
                                "value": "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token"
                            },
                            "request_url": {
                                "type": "text",
                                "value": "https://graph.microsoft.com/v1.0/security/alerts"
                            },
                            "request_redirect_forward_headers": {
                                "type": "bool"
                            },
                            "response_split": {
                                "type": "yaml",
                                "value": "target: body.value"
                            },
                            "tags": {
                                "type": "text",
                                "value": [
                                    "forwarded,msgraph"
                                ]
                            },
                            "pipeline": {
                                "type": "text",
                                "value": "msgraph-security-alert-pipeline"
                            },
                            "request_rate_limit_reset": {
                                "type": "text"
                            },
                            "request_body": {
                                "type": "yaml",
                                "value": ""
                            },
                            "data_stream.dataset": {
                                "type": "text",
                                "value": "msgraph.alert"
                            },
                            "request_interval": {
                                "type": "text",
                                "value": "5m"
                            },
                            "oauth_google_credentials_file": {
                                "type": "text"
                            },
                            "response_request_body_on_pagination": {
                                "type": "bool"
                            },
                            "oauth_azure_resource": {
                                "type": "text"
                            },
                            "oauth_secret": {
                                "type": "password",
                                "value": "SECRET"
                            },
                            "response_pagination": {
                                "type": "yaml",
                                "value": ""
                            },
                            "username": {
                                "type": "text"
                            }
                        },
                        "enabled": true,
                        "id": "httpjson-httpjson.generic-9eab7f8f-c2a6-4f95-9330-dbf3c5287675",
                        "compiled_stream": {
                            "config_version": 2,
                            "data_stream": {
                                "dataset": "msgraph.alert"
                            },
                            "interval": "5m",
                            "pipeline": "msgraph-security-alert-pipeline",
                            "auth.oauth2.client.id": "CLIENT_ID",
                            "auth.oauth2.client.secret": "SECRET",
                            "auth.oauth2.token_url": "https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token",
                            "auth.oauth2.scopes": null,
                            "auth.oauth2.endpoint_params": null,
                            "request.url": "https://graph.microsoft.com/v1.0/security/alerts",
                            "request.method": "GET",
                            "request.ssl": null,
                            "response.split": {
                                "target": "body.value"
                            },
                            "cursor": null,
                            "tags": [
                                "forwarded,msgraph"
                            ],
                            "processors": [
                                {
                                    "add_fields": {
                                        "target": "",
                                        "fields": {
                                            "event.module": "msgraph",
                                            "event.dataset": "msgraph.unknown"
                                        }
                                    }
                                }
                            ]
                        }
                    }
                ]
            }
        ],
        "revision": 1,
        "created_at": "2022-05-09T08:42:02.361Z",
        "created_by": "username",
        "updated_at": "2022-05-09T08:42:02.361Z",
        "updated_by": "username"
    }
}

Render of auth.oauth2.scopes fails for custom httpjson input, this will fix the issue

elasticmachine · 2022-05-09T08:53:04Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-05-12T10:13:58.343+0000
Duration: 16 min 58 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7
Skipped	0
Total	7

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

efd6 · 2022-05-09T10:02:36Z

packages/httpjson/data_stream/generic/manifest.yml

        required: false
      - name: oauth_scopes
-        type: yaml
+        type: text


Can you add a test to ensure that this is working as expected? You can construct an endpoint in the packages/httpjson/_dev/deploy/docker/files/config.yml file and make a test in the packages/httpjson/data_stream/generic/_dev/test/system/ directory to check that the scopes are correctly set.

Hi @efd6 - I've added the variable as per my commit no. 3 ( 3105bd5 ) ... there's nothing to add to the docker config file, I don't know if the stream container which provides a HTTP interface to test against is capable of echo'ing back the requested scopes as a "scope" value in the response, and my quick look at it suggests it isn't... and "scope" in an Oauth2 token response is optional. This should do the trick in terms of testing render of a list of scopes as would be generated by the Kibana/Fleet webUI to the Fleet API though.

Hmm, actually, my bad, seems like it can

Still an optional part of the response though... OAuth2 token endpoint doesn't need to include scopes, the assumption is that if a code comes back from a request that include scopes, then the token is valid for those.

So what you will need to do is write some rules into the packages/httpjson/_dev/deploy/docker/files/config.yml file. This is the configuration for the stream tool.

That's lovely, where's the actual doco, or is the paying customer who's bug fixing Elastic's product (yes that means me.) supposed to reverse engineer stream tool too? Right now, the product I'm paying Elastic for - multiple platinum and enterprise level subscription Elastic Cloud deployments - can't do what it says it can do. The tests are clearly non-existent at present and I'm not going to burn a bunch of my time creating tests for Elastic's commercial product that I'm paying for, creating something that doesn't currently exist, based on documentation that doesn't exist.

I can pick this up if you would like.

TBH, noting "Contributor"... if you're not getting paid for this I don't feel you should burn your time either... is there no Elastic support team that this can be assigned to?

I am an Elastic employee, so this is something I'm happy to do. If you'd like me to take this on, I can pick it up.

colin-stubbs · 2022-05-09T10:41:51Z

/test

elasticmachine · 2022-05-09T20:20:04Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 · 2022-05-10T04:16:30Z

/test

elasticmachine · 2022-05-10T04:33:56Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚 3.589
Classes	100.0% (`0/0`)	💚 3.589
Methods	100.0% (`3/3`)	💚 11.675
Lines	100.0% (`0/0`)	💚 10.972
Conditionals	100.0% (`0/0`)	💚

efd6 · 2022-05-10T22:34:59Z

/test

efd6 · 2022-05-12T10:13:26Z

/test

colin-stubbs · 2022-05-16T00:02:13Z

@efd6 - Finally getting time to check status here, thank you Dan

efd6 · 2022-05-16T00:09:25Z

Just waiting on review.

colin-stubbs added 2 commits May 9, 2022 18:35

Fix for issue #3295

2174b5c

Render of auth.oauth2.scopes fails for custom httpjson input, this will fix the issue

Update changelog related to #3295

7b61c16

colin-stubbs requested a review from a team as a code owner May 9, 2022 08:50

efd6 reviewed May 9, 2022

View reviewed changes

Add oauth_scopes test variable

3105bd5

efd6 self-assigned this May 9, 2022

andrewkroh added bug Something isn't working, use only for issues Team:Security-External Integrations Integration:httpjson Custom API labels May 9, 2022

add testing and bump stream version

cc65a63

efd6 requested a review from a team May 10, 2022 06:53

adriansr approved these changes May 17, 2022

View reviewed changes

efd6 merged commit 2dacc67 into elastic:main May 17, 2022

Conversation

colin-stubbs commented May 9, 2022

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented May 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

efd6 May 9, 2022

Choose a reason for hiding this comment

Uh oh!

colin-stubbs May 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

colin-stubbs May 9, 2022

Choose a reason for hiding this comment

Uh oh!

colin-stubbs May 9, 2022

Choose a reason for hiding this comment

Uh oh!

efd6 May 9, 2022

Choose a reason for hiding this comment

Uh oh!

colin-stubbs May 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 May 9, 2022

Choose a reason for hiding this comment

Uh oh!

colin-stubbs May 9, 2022

Choose a reason for hiding this comment

Uh oh!

efd6 May 9, 2022

Choose a reason for hiding this comment

Uh oh!

colin-stubbs commented May 9, 2022

Uh oh!

elasticmachine commented May 9, 2022

Uh oh!

efd6 commented May 10, 2022

Uh oh!

elasticmachine commented May 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

efd6 commented May 10, 2022

Uh oh!

efd6 commented May 12, 2022

Uh oh!

colin-stubbs commented May 16, 2022

Uh oh!

efd6 commented May 16, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented May 9, 2022 •

edited

Loading

colin-stubbs May 9, 2022 •

edited

Loading

colin-stubbs May 9, 2022 •

edited

Loading

elasticmachine commented May 10, 2022 •

edited

Loading