[Filebeat] Add URI Parts Processor to multiple modules by legoguy1000 · Pull Request #24699 · elastic/beats

legoguy1000 · 2021-03-23T05:03:11Z

What does this PR do?

Updates Ingest Pipelines for the below modules:

Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, ZScaler

With the below changes

Add uri_parts processor to parse URIs (includes URL decoding) to add url.path, url.extension, url.query....
URL Decodes http.request.referrer (when applicable) to make them human readable

Why is it important?

Parses URLs to break up the URL into the different parts and URL decodes them.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

cd beats/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=apache,nginx,iis,traefik mage -v pythonIntegTest
cd beats/x-pack/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=s3access,cisco,f5,fortinet,google_workspace,imperva,microsoft,netscout,o365,sophos,squid,suricata,zeek,zia,zoom,zscaler mage -v pythonIntegTest

Related issues

Closes Default nginx pipelines should decode url.original field #19088

Use cases

Screenshots

Logs

elasticmachine · 2021-03-23T05:09:45Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: andrewstucki commented: jenkins run tests
Start Time: 2021-04-27T16:37:55.516+0000
Duration: 140 min 57 sec
Commit: b01d893

Test stats 🧪

Test	Results
Failed	0
Passed	13707
Skipped	2285
Total	15992

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13707
Skipped	2285
Total	15992

elasticmachine · 2021-03-29T15:38:52Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

legoguy1000 · 2021-03-30T19:37:18Z

@andrewstucki This should be ready review and CI tests

andrewstucki · 2021-04-02T16:54:12Z

jenkins run tests

andrewstucki · 2021-04-05T12:30:02Z

@legoguy1000 so it looks to me like the new log entries that were generated for the nginx module are missing user_agent.device.type which was added recently to Elasticsearch in elastic/elasticsearch#69322 -- can you make sure your development Elasticsearch instance is up-to-date with the latest snapshot before regeneration? I believe you should be able to grab the latest docker container with:

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT

and then run it with:

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "logger.org.elasticsearch.transport=trace" docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT

prior to re-running the regeneration for nginx.

legoguy1000 · 2021-04-05T12:31:26Z

@legoguy1000 so it looks to me like the new log entries that were generated for the nginx module are missing user_agent.device.type which was added recently to Elasticsearch in elastic/elasticsearch#69322 -- can you make sure your development Elasticsearch instance is up-to-date with the latest snapshot before regeneration? I believe you should be able to grab the latest docker container with:
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT
and then run it with:
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "logger.org.elasticsearch.transport=trace" docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT
prior to re-running the regeneration for nginx.

Not a problem

andrewstucki

Question about the backslash behavior. Not entirely sure if this is behavior based off of a combination of using urldecode + uri_parts and ordering or if this is just some strange behavior/potentially a bug in the processor itself. I'll try and check this out locally and play around with it to see what's going on.

legoguy1000 · 2021-04-05T16:22:10Z

Ran the pipelines to update the user_agent.device.type taking a look at the extra escapes.

legoguy1000 · 2021-04-06T16:15:19Z

@andrewstucki @andrewkroh If you guys are good, can u run the pipeline?

andrewstucki · 2021-04-06T16:19:23Z

jenkins run tests

andrewstucki

So, based off of #19088 (comment) I want to see if @ebeahan can chime in about whether the best course is to keep the url.original field in url-encoded form or if it should be decoded.

andrewstucki · 2021-04-26T19:36:52Z

jenkins run tests

andrewstucki · 2021-04-26T19:37:29Z

@legoguy1000 In general, this looks fine to me, I'll try and get it merged assuming that the tests pass

legoguy1000 · 2021-04-26T19:46:50Z

@legoguy1000 In general, this looks fine to me, I'll try and get it merged assuming that the tests pass

I suspect its going to fail since they made the changes to the geo IP database, i'm re-generating the data now. I'll hold off pushing to see if it passes.

legoguy1000 · 2021-04-27T12:48:00Z

@andrewstucki I update the generated data files to account for the changes in the pipeline. If you rerun the tests, we should be good.

andrewstucki · 2021-04-27T16:37:32Z

jenkins run tests

legoguy1000 · 2021-04-27T19:00:59Z

@andrewstucki all passed

andrewstucki · 2021-04-27T19:32:55Z

@legoguy1000 thanks for the additions, I'll backport this for the 7.14 release

legoguy1000 · 2021-04-27T19:38:36Z

@legoguy1000 thanks for the additions, I'll backport this for the 7.14 release

Ya, that was definitely a long one. Definitely good conversations with the ECS team clarifying the fields.

@andrewstucki

* Update Nginx pipelines * Update Apache, Nginx, IIS, Traefik pipelines * Update AWS S3 * Update Cisco * Update F5 * Update Fortinet * Update Imperva, Netscout, O365, Sophos, Squid, Suricata, Zscaler * additional fixes * update pipelines * unescape \ * remove urldecodes for url.original * updates after rebase * update zeek SIP * update changelog as requested by @andrewstucki * remove `url_decode` for `http.request.referrer` * update generated data (cherry picked from commit f1fea95)

andrewstucki · 2021-04-27T19:50:48Z

@legoguy1000 backport opened at #25353

@andrewstucki

* Update Nginx pipelines * Update Apache, Nginx, IIS, Traefik pipelines * Update AWS S3 * Update Cisco * Update F5 * Update Fortinet * Update Imperva, Netscout, O365, Sophos, Squid, Suricata, Zscaler * additional fixes * update pipelines * unescape \ * remove urldecodes for url.original * updates after rebase * update zeek SIP * update changelog as requested by @andrewstucki * remove `url_decode` for `http.request.referrer` * update generated data (cherry picked from commit f1fea95) Co-authored-by: Alex Resnick <adr8292@gmail.com>

botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 23, 2021

legoguy1000 force-pushed the 19088-nginx-pipelines branch from 08dda87 to 800299e Compare March 23, 2021 05:04

legoguy1000 marked this pull request as ready for review March 23, 2021 05:04

legoguy1000 marked this pull request as draft March 23, 2021 13:28

legoguy1000 changed the title ~~[Filebeat] Update Nginx pipelines~~ [Filebeat] Update Nginx & Apache pipelines Mar 23, 2021

legoguy1000 force-pushed the 19088-nginx-pipelines branch from 800299e to dcd06c2 Compare March 23, 2021 23:53

legoguy1000 changed the title ~~[Filebeat] Update Nginx & Apache pipelines~~ [Filebeat] Update URI decoding and parsing across multiple modules Mar 25, 2021

legoguy1000 force-pushed the 19088-nginx-pipelines branch 4 times, most recently from 5c94728 to 50cc14b Compare March 26, 2021 15:23

legoguy1000 marked this pull request as ready for review March 27, 2021 21:36

andrewkroh added Filebeat Filebeat Team:Security-External Integrations labels Mar 29, 2021

botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 29, 2021

andrewkroh assigned andrewstucki Mar 29, 2021

legoguy1000 force-pushed the 19088-nginx-pipelines branch from 00493ed to 4d466fe Compare March 30, 2021 13:16

legoguy1000 force-pushed the 19088-nginx-pipelines branch from 4d466fe to dd1fdc8 Compare April 2, 2021 11:57

andrewstucki reviewed Apr 5, 2021

View reviewed changes

Comment thread filebeat/module/iis/error/test/iis_error_url.log-expected.json Outdated

legoguy1000 force-pushed the 19088-nginx-pipelines branch from dd1fdc8 to 8afd9c3 Compare April 5, 2021 18:24

andrewstucki reviewed Apr 6, 2021

View reviewed changes

Comment thread filebeat/module/apache/access/test/ssl-request.log-expected.json Outdated

Comment thread filebeat/module/iis/access/test/test-x-forward-for.log-expected.json Outdated

legoguy1000 added 10 commits April 26, 2021 18:37

Update Fortinet

3b213b6

Update Imperva, Netscout, O365, Sophos, Squid, Suricata, Zscaler

0cc1bf4

additional fixes

d90b7d4

update pipelines

2af6e09

unescape \

24d5811

remove urldecodes for url.original

e936522

updates after rebase

c42a73e

update zeek SIP

e49a4c4

update changelog as requested by @andrewstucki

e4fe9b8

remove url_decode for http.request.referrer

aabbfa3

legoguy1000 force-pushed the 19088-nginx-pipelines branch from e930be4 to aabbfa3 Compare April 26, 2021 18:39

update generated data

b01d893

andrewstucki approved these changes Apr 27, 2021

View reviewed changes

andrewstucki merged commit f1fea95 into elastic:master Apr 27, 2021

andrewstucki added the backport-v7.14.0 Automated backport with mergify label Apr 27, 2021

legoguy1000 deleted the 19088-nginx-pipelines branch April 27, 2021 19:34

andrewstucki mentioned this pull request Apr 27, 2021

Cherry-pick #24699 to 7.x: [Filebeat] Add URI Parts Processor to multiple modules #25353

Merged

6 tasks

andrewstucki added the v7.13.0 label Apr 27, 2021

andresrc mentioned this pull request Jul 30, 2021

docs: Prepare Changelog for 7.14.0 #27150

Merged

Conversation

legoguy1000 commented Mar 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Mar 23, 2021 • edited by jenkins-beats-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

elasticmachine commented Mar 29, 2021

Uh oh!

legoguy1000 commented Mar 30, 2021

Uh oh!

andrewstucki commented Apr 2, 2021

Uh oh!

andrewstucki commented Apr 5, 2021

Uh oh!

legoguy1000 commented Apr 5, 2021

Uh oh!

andrewstucki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

legoguy1000 commented Apr 5, 2021

Uh oh!

legoguy1000 commented Apr 6, 2021

Uh oh!

andrewstucki commented Apr 6, 2021

Uh oh!

andrewstucki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

andrewstucki commented Apr 26, 2021

Uh oh!

andrewstucki commented Apr 26, 2021

Uh oh!

legoguy1000 commented Apr 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

legoguy1000 commented Apr 27, 2021

Uh oh!

andrewstucki commented Apr 27, 2021

Uh oh!

legoguy1000 commented Apr 27, 2021

Uh oh!

andrewstucki commented Apr 27, 2021

Uh oh!

legoguy1000 commented Apr 27, 2021

Uh oh!

andrewstucki commented Apr 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

legoguy1000 commented Mar 23, 2021 •

edited

Loading

elasticmachine commented Mar 23, 2021 •

edited by jenkins-beats-ci Bot

Loading

legoguy1000 commented Apr 26, 2021 •

edited

Loading