Skip to content

[Filebeat] sophos.xg - Handle new log field names#31388

Merged
andrewkroh merged 6 commits intoelastic:mainfrom
andrewkroh:feature/fb/sync-sophos-integration
Apr 25, 2022
Merged

[Filebeat] sophos.xg - Handle new log field names#31388
andrewkroh merged 6 commits intoelastic:mainfrom
andrewkroh:feature/fb/sync-sophos-integration

Conversation

@andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Apr 21, 2022 •
edited
Loading

What does this PR do?

Sync the Fleet integration pipeline into the Filebeat module.

Based on elastic/integrations@919fe81.

Fixes #29002

Why is it important?

New versions of Sophos (XG) Firewall logs use different field names so the pipeline needed updated.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@andrewkroh andrewkroh requested review from a team as code owners April 21, 2022 19:43
@andrewkroh andrewkroh requested review from belimawr and faec and removed request for a team April 21, 2022 19:43
@elasticmachine
Copy link
Contributor

elasticmachine commented Apr 21, 2022

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 21, 2022
@andrewkroh andrewkroh mentioned this pull request Apr 21, 2022
Closed
@elasticmachine
Copy link
Contributor

elasticmachine commented Apr 21, 2022 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-04-25T21:09:10.055+0000

  • Duration: 43 min 56 sec

Test stats 🧪

Test Results
Failed 0
Passed 1962
Skipped 394
Total 2356

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@andrewkroh andrewkroh force-pushed the feature/fb/sync-sophos-integration branch from 00dea68 to 95617fc Compare April 21, 2022 20:31
@andrewkroh andrewkroh mentioned this pull request Apr 21, 2022
Closed
@andrewkroh
Remove duplicate syslog_server_name field
f954580
@andrewkroh
Pipeline modifications for Beats
6811492 
Add event.ingested.

Change to Go template format.
@andrewkroh
Update generated files
66863aa 
[git-generate]
mage -d filebeat update
cd x-pack/filebeat
mage update
PYTEST_ADDOPTS="-k sophos" TESTING_FILEBEAT_MODULES=sophos GENERATE=true mage -v pythonIntegTest
@andrewkroh
Add changelog
3d4654d
@andrewkroh andrewkroh force-pushed the feature/fb/sync-sophos-integration branch from 95617fc to 3d4654d Compare April 23, 2022 03:22
belimawr
belimawr approved these changes Apr 25, 2022
View reviewed changes
Copy link
Contributor

@belimawr belimawr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's quite hard to do a detailed review, but in general it looks good. Once CI is green, it's good to merge.

@andrewkroh
Copy link
Member Author

andrewkroh commented Apr 25, 2022

/test

@andrewkroh andrewkroh merged commit ef833ae into elastic:main Apr 25, 2022
kush-elastic pushed a commit to kush-elastic/beats that referenced this pull request May 2, 2022
@andrewkroh @kush-elastic
[Filebeat] sophos.xg - Handle new log field names (elastic#31388)
c8b253a 
* Sort fields.yml by name before modifying

* sophos.xg - Handle new log field names

Sync the Fleet integration pipeline into the Filebeat module.

Based on elastic/integrations@919fe81.

Fixes elastic#29002

* Remove duplicate syslog_server_name field

* Pipeline modifications for Beats

Add event.ingested.

Change to Go template format.

* Update generated files

[git-generate]
mage -d filebeat update
cd x-pack/filebeat
mage update
PYTEST_ADDOPTS="-k sophos" TESTING_FILEBEAT_MODULES=sophos GENERATE=true mage -v pythonIntegTest

* Add changelog
@jamiehynds
Copy link

jamiehynds commented May 12, 2022

@andrewkroh has this pipeline enhancement been backported to 7.17? A user is running 7.17 and the Filebeat module, and wondering if they can avail of the newer XG log support? (cc: @tiaanwest)

@andrewkroh andrewkroh added the backport-7.17 Automated backport to the 7.17 branch with mergify label May 16, 2022
@mergify mergify bot mentioned this pull request May 16, 2022
Merged
andrewkroh added a commit that referenced this pull request May 16, 2022
@andrewkroh
[Filebeat] sophos.xg - Handle new log field names (#31388)
852bd50 
* Sort fields.yml by name before modifying

* sophos.xg - Handle new log field names

Sync the Fleet integration pipeline into the Filebeat module.

Based on elastic/integrations@919fe81.

Fixes #29002

* Remove duplicate syslog_server_name field

* Pipeline modifications for Beats

Add event.ingested.

Change to Go template format.

* Update generated files

[git-generate]
mage -d filebeat update
cd x-pack/filebeat
mage update
PYTEST_ADDOPTS="-k sophos" TESTING_FILEBEAT_MODULES=sophos GENERATE=true mage -v pythonIntegTest

* Add changelog
@tiaanwest
Copy link

tiaanwest commented May 18, 2022 •
edited by andrewkroh
Loading

@andrewkroh has this pipeline enhancement been backported to 7.17? A user is running 7.17 and the Filebeat module, and wondering if they can avail of the newer XG log support? (cc: @tiaanwest)

Hey @andrewkroh, would you be able to give me an indication whether or not this will be backported to 7.17? Or should the user rather upgrade to 8.x where the change has been implemented?

@andrewkroh andrewkroh added the backport-v8.3.0 Automated backport with mergify label May 18, 2022
@andrewkroh
Copy link
Member Author

andrewkroh commented May 18, 2022

I opened a 7.17 backport at #31637, but I'm not sure if it's going to mergeable. I will need to look over the failures that occurred. It might be that because this new work is based on ECS 8.x that some of the fields don't exists in ECS 1.x that is being used in Filebeat 7.17.

I think the 8.3.0 release will be out sooner than the next 7.17 minor. So that will get this in their hands faster. Or if they try Elastic Agent they could use the Fleet integration now. Or another option could be to install the Fleet integration and wire up Filebeat to send data into data stream that it creates.

andrewkroh added a commit that referenced this pull request May 24, 2022
@andrewkroh
[Filebeat] sophos.xg - Handle new log field names (#31388)
e1263f2 
* Sort fields.yml by name before modifying

* sophos.xg - Handle new log field names

Sync the Fleet integration pipeline into the Filebeat module.

Based on elastic/integrations@919fe81.

Fixes #29002

* Remove duplicate syslog_server_name field

* Pipeline modifications for Beats

Add event.ingested.

Change to Go template format.

* Update generated files

[git-generate]
mage -d filebeat update
cd x-pack/filebeat
mage update
PYTEST_ADDOPTS="-k sophos" TESTING_FILEBEAT_MODULES=sophos GENERATE=true mage -v pythonIntegTest

* Add changelog
andrewkroh added a commit that referenced this pull request May 25, 2022
@mergify @andrewkroh
[7.17](backport #31388) [Filebeat] sophos.xg - Handle new log field n…
6d6be4a 
…ames (#31637)

* [Filebeat] sophos.xg - Handle new log field names (#31388)

* Sort fields.yml by name before modifying

* sophos.xg - Handle new log field names

Sync the Fleet integration pipeline into the Filebeat module.

Based on elastic/integrations@919fe81.

Fixes #29002

* Remove duplicate syslog_server_name field

* Pipeline modifications for Beats

Add event.ingested.

Change to Go template format.

* Update generated files

[git-generate]
mage -d filebeat update
cd x-pack/filebeat
mage update
PYTEST_ADDOPTS="-k sophos" TESTING_FILEBEAT_MODULES=sophos GENERATE=true mage -v pythonIntegTest

* Add default_field: false to sophos.xg fields

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@andrewkroh @chrisberkhout
[Filebeat] sophos.xg - Handle new log field names (#31388)
3922b9c 
* Sort fields.yml by name before modifying

* sophos.xg - Handle new log field names

Sync the Fleet integration pipeline into the Filebeat module.

Based on elastic/integrations@919fe81.

Fixes #29002

* Remove duplicate syslog_server_name field

* Pipeline modifications for Beats

Add event.ingested.

Change to Go template format.

* Update generated files

[git-generate]
mage -d filebeat update
cd x-pack/filebeat
mage update
PYTEST_ADDOPTS="-k sophos" TESTING_FILEBEAT_MODULES=sophos GENERATE=true mage -v pythonIntegTest

* Add changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@belimawr belimawr belimawr approved these changes

@faec faec Awaiting requested review from faec faec was automatically assigned from elastic/elastic-agent-data-plane

Assignees

@andrewkroh andrewkroh

Labels

backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.3.0 Automated backport with mergify Filebeat Filebeat

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Sophos XG improvements

5 participants

@andrewkroh @elasticmachine @jamiehynds @tiaanwest @belimawr