Tychon Agentless version 56

[joeperuzzi]

• Enhancement

Proposed commit message

• WHAT: TYCHON endpoint gathers datasets from Windows and Linux for the purpose of historical compliance and current state tracking.
• WHY: This is the first major release to be approved into the Elastic Integrations Stack.

Checklist

• [ X] I have reviewed tips for building integrations and this pull request is aligned with them.
• [ X] I have verified that all data streams collect metrics or logs.
• [ X] I have added an entry to my package's changelog.yml file.
• [ X] I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• Fully tested with Elasticsearch and Kibana versions 8, 9, 10

How to test this PR locally

Please reach out to support@tychon.io to get a copy of TYCHON Agentless if testing is needed.

[elasticmachine]

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-12T11:49:51.589+0000

• Duration: 3 min 9 sec

Steps errors

Expand to view the steps failures

Load a resource file from a library

• Took 0 min 0 sec . View more details here
• Description: approval-list/elastic/integrations.yml

Error signal

• Took 0 min 0 sec . View more details here
• Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/joeperuzzi return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/joeperuzzi : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/joeperuzzi : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

/test

[efd6]

This PR looks to be a revival of #6701 with a large number of additions. Ideally, I would like a fresh start based on the current main with some separation of the data streams that are being added. My suggestion would be to start with a single data stream and its associated dashboard and then add subsequent datastreams/dashboards when we have a good success with the first and we are in our stride.

[efd6]

My proposed approach would be to:

1. Close this PR and delete the remote branch, and rename the local tychon_agentless for your safety.
2. Copy the package code here out of tree.
3. git checkout main && git pull origin main && git checkout -b tychon_agentless
4. Copy back the package from you out-of-tree stash including only one datastream and its associated dashboard files, and edit ".github/CODEOOWNERS" to add the line for the package

   /packages/tychon @elastic/security-external-integrations
   

5. git add .github/CODEOWNERS packages/tychon && git commit -m "tychon: add new package"
6. Open a new PR with the resulting branch.

Then we can focus on a limited set of changes to get things right and then progress from there.

[efd6]

I have pushed a copy of the code here, squashed/rebase onto main to #8141. I have also cleaned up typographic issues with the code (you can see what was done in the last three commits, but briefly: changed line endings from dos crlf to unix lf only; removed trailing whitespace on lines; and added final new lines to yaml and hbs files). Please use the state there as a starting point.

[joeperuzzi]

@efd6 thanks for working this with us, quick question, should the versioning move to 1.0? This is already in production and going back to a .0.0.1 will not allow customers to upgrade in the future.

[efd6]

@joeperuzzi The version of the package is independent of the version of the product that is being integrated. The tychon package here is new, so we can start at any version. When you say that it is already in production, do you mean that there are users that have this package installed external to EPR?

[joeperuzzi]

@joeperuzzi The version of the package is independent of the version of the product that is being integrated. The tychon package here is new, so we can start at any version. When you say that it is already in production, do you mean that there are users that have this package installed external to EPR?

@efd6 we tried to get the first version through review before deploying to customer sites but it took longer than expected so I closed that pull request for this one with all the new datasets. We have deployed this plugin to multiple sites for pilots and existing customers. We installed via air gapped registration servers as we were unable to install via normal methods.

There are more datasets on the way so I’d like to see this deploy as 1.0 so we can iterate higher as we release more source data.

[efd6]

OK. So given how I would like this to proceed, I'm happy for that to happen, but I would like to use a sub-1.0.0 version until we have everything sorted and then bump to 1.

[joeperuzzi]

@efd6 I'm struggling a bit trying to break this code up and have it still install/function. We did not intend for this to be pieced together and therefore its making me very nervous to break this code up into pieces. This is a well-tested integration and my fear is that I'm taking production ready code and breaking it up for ease of review putting a ton of risk for defects once released.

Would it be better to leave everything intact and then give you a break-down of each stream and their associated parts (dashboards, siem rules, transforms, etc..). Then you can focus the data streams one at a time without having a lot of different branches between internal testing, code review, and this pull request.

TYCHON Plugin Mapping.docx

[efd6]

Sorry, I won't open a docx. Are you saying that there is interaction between the different datastreams?

I'm not entirely sure what to do here. The package does not pass when I test it locally (the branch at the PR I linked above), and I would like to ensure that our code standards are met when we add this. To do that over a 20kSLOC addition in one go is not going to work well. I think it would be much better to add a single datastream, iron out the issues in that and then allow you to take what is learned there and apply that to other datastreams.

Run static tests for the package
--- Test results for package: tychon - START ---
FAILURE DETAILS:
tychon/tychon_cve Verify sample_event.json:
[0] field "tychon.campaign" is undefined
[1] field "tychon.realm" is undefined
[2] parsing field value failed: field "event.outcome"'s value "fail" is not one of the allowed values (failure, success, unknown)
tychon/tychon_epp Verify sample_event.json:
[0] field "package.updateid" is undefined
[1] field "tychon.realm" is undefined
[2] field "package.revision" is undefined
[3] field "package.installed" is undefined
[4] field "package.product" is undefined
[5] field "tychon.campaign" is undefined
tychon/tychon_networkadapter Verify sample_event.json:
[0] parsing field value failed: field "script.current_duration"'s Go type, string, does not match the expected field type: long (field value: 1150.99)
[1] field "host.ip" is not normalized as expected: expected array, found "10.154.5.200,fe80::c2c9:f4e0:eb65:2c33,192.168.56.1,fe80::5bed:2433:ff9d:efdb,172.16.0.1,fe80::e9d4:5969:ce85:2c87" (string)
[2] parsing field value failed: field "host.adapter.link_speed"'s Go type, string, does not match the expected field type: long (field value: 100 Mbps)
[3] field "host.mac" is not normalized as expected: expected array, found "60:E3:2B:4B:40:E2,0A:00:27:00:00:08,0A:00:27:00:00:0D" (string)
tychon/tychon_stig Verify sample_event.json:
[0] field "package.updateid" is undefined
[1] field "package.product" is undefined
[2] field "tychon.realm" is undefined
[3] field "package.installed" is undefined
[4] field "package.revision" is undefined
[5] field "tychon.campaign" is undefined


╭─────────┬───────────────────────┬───────────┬──────────────────────────┬────────────────────────────────────────────┬──────────────╮
│ PACKAGE │ DATA STREAM           │ TEST TYPE │ TEST NAME                │ RESULT                                     │ TIME ELAPSED │
├─────────┼───────────────────────┼───────────┼──────────────────────────┼────────────────────────────────────────────┼──────────────┤
│ tychon  │ tychon_cve            │ static    │ Verify sample_event.json │ FAIL: one or more errors found in document │  71.321974ms │
│ tychon  │ tychon_epp            │ static    │ Verify sample_event.json │ FAIL: one or more errors found in document │  64.837559ms │
│ tychon  │ tychon_networkadapter │ static    │ Verify sample_event.json │ FAIL: one or more errors found in document │  61.086539ms │
│ tychon  │ tychon_stig           │ static    │ Verify sample_event.json │ FAIL: one or more errors found in document │  60.743941ms │
╰─────────┴───────────────────────┴───────────┴──────────────────────────┴────────────────────────────────────────────┴──────────────╯

╭─────────┬──────────────────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────┬──────────────╮
│ PACKAGE │ DATA STREAM              │ TEST TYPE │ TEST NAME                                                                         │ RESULT                              │ TIME ELAPSED │
├─────────┼──────────────────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────────────┼──────────────┤
│ tychon  │                          │ asset     │ dashboard tychon-078edb40-d137-11e9-a2af-693b633cf871-stig is loaded              │ PASS                                │      5.453µs │
│ tychon  │                          │ asset     │ dashboard tychon-0c036be0-3de5-11ee-9610-15dee918f31a-exposedservice is loaded    │ PASS                                │        142ns │
│ tychon  │                          │ asset     │ dashboard tychon-19325010-4597-11ee-83e4-c92ed141b9e5-hardware is loaded          │ PASS                                │         93ns │
│ tychon  │                          │ asset     │ dashboard tychon-1af57010-41b6-11ee-83e4-c92ed141b9e5-networkadapter is loaded    │ PASS                                │        123ns │
│ tychon  │                          │ asset     │ dashboard tychon-267716e0-e9d8-11ed-9d4a-9513ae375d2b-epp is loaded               │ PASS                                │        116ns │
│ tychon  │                          │ asset     │ dashboard tychon-2bd4ca50-3dfd-11ee-9610-15dee918f31a-softwareinventory is loaded │ PASS                                │        118ns │
│ tychon  │                          │ asset     │ dashboard tychon-2de7a3c0-3e08-11ee-9610-15dee918f31a-cve is loaded               │ PASS                                │        174ns │
│ tychon  │                          │ asset     │ dashboard tychon-380b6c10-3dbd-11ee-9610-15dee918f31a-harddrive is loaded         │ PASS                                │        154ns │
│ tychon  │                          │ asset     │ dashboard tychon-3cb855d0-3c5e-11ee-8557-a7ea91123f8b-networkadapter is loaded    │ PASS                                │        155ns │
│ tychon  │                          │ asset     │ dashboard tychon-6165bf50-3dbf-11ee-9610-15dee918f31a-host is loaded              │ PASS                                │        191ns │
│ tychon  │                          │ asset     │ dashboard tychon-75c383c0-e508-11ed-8a95-ab70156d4b18-cve is loaded               │ PASS                                │        172ns │
│ tychon  │                          │ asset     │ dashboard tychon-8082ac00-3d41-11ee-9610-15dee918f31a-harddrive is loaded         │ PASS                                │        178ns │
│ tychon  │                          │ asset     │ dashboard tychon-8c858ea0-3c74-11ee-8557-a7ea91123f8b-cpu is loaded               │ PASS                                │        179ns │
│ tychon  │                          │ asset     │ dashboard tychon-993e07a0-3e02-11ee-9610-15dee918f31a-hardware is loaded          │ PASS                                │        192ns │
│ tychon  │                          │ asset     │ dashboard tychon-b85e87c0-41ab-11ee-83e4-c92ed141b9e5-epp is loaded               │ PASS                                │        183ns │
│ tychon  │                          │ asset     │ dashboard tychon-cb312af0-3d4c-11ee-9610-15dee918f31a-arp is loaded               │ PASS                                │        182ns │
│ tychon  │                          │ asset     │ dashboard tychon-e1c9c490-41a5-11ee-83e4-c92ed141b9e5-stig is loaded              │ PASS                                │        227ns │
│ tychon  │                          │ asset     │ dashboard tychon-e24ce070-3c85-11ee-9610-15dee918f31a-exposedservice is loaded    │ PASS                                │        202ns │
│ tychon  │                          │ asset     │ dashboard tychon-e3cbb1a0-112a-11ee-af86-538da1394f27-log is loaded               │ PASS                                │        207ns │
│ tychon  │                          │ asset     │ dashboard tychon-f3f86a20-3d47-11ee-9610-15dee918f31a-host is loaded              │ PASS                                │        208ns │
│ tychon  │                          │ asset     │ visualization tychon-837878a0-c3cb-11eb-8956-0b1a70e695fd is loaded               │ FAIL: could not find expected asset │     446.18µs │
│ tychon  │                          │ asset     │ visualization tychon-d954bdb0-3298-11ec-b058-cf4fefc29658 is loaded               │ PASS                                │        429ns │
│ tychon  │                          │ asset     │ visualization tychon-e6c0e460-c3da-11eb-8956-0b1a70e695fd is loaded               │ PASS                                │        167ns │
│ tychon  │                          │ asset     │ visualization tychon-ee4b44b0-40e6-11ee-8111-21f5f34f6dfc is loaded               │ PASS                                │        154ns │
│ tychon  │                          │ asset     │ lens tychon-1d1b99c0-c3e4-11eb-8956-0b1a70e695fd is loaded                        │ PASS                                │        296ns │
│ tychon  │ tychon_arp               │ asset     │ index_template logs-tychon.tychon_arp is loaded                                   │ PASS                                │        270ns │
│ tychon  │ tychon_arp               │ asset     │ ingest_pipeline logs-tychon.tychon_arp-0.0.1 is loaded                            │ PASS                                │        233ns │
│ tychon  │ tychon_cpu               │ asset     │ index_template logs-tychon.tychon_cpu is loaded                                   │ PASS                                │        331ns │
│ tychon  │ tychon_cpu               │ asset     │ ingest_pipeline logs-tychon.tychon_cpu-0.0.1 is loaded                            │ PASS                                │        245ns │
│ tychon  │ tychon_cve               │ asset     │ index_template logs-tychon.tychon_cve is loaded                                   │ PASS                                │        281ns │
│ tychon  │ tychon_cve               │ asset     │ ingest_pipeline logs-tychon.tychon_cve-0.0.1 is loaded                            │ PASS                                │        255ns │
│ tychon  │ tychon_epp               │ asset     │ index_template logs-tychon.tychon_epp is loaded                                   │ PASS                                │        299ns │
│ tychon  │ tychon_epp               │ asset     │ ingest_pipeline logs-tychon.tychon_epp-0.0.1 is loaded                            │ PASS                                │        259ns │
│ tychon  │ tychon_exposedservice    │ asset     │ index_template logs-tychon.tychon_exposedservice is loaded                        │ PASS                                │        347ns │
│ tychon  │ tychon_exposedservice    │ asset     │ ingest_pipeline logs-tychon.tychon_exposedservice-0.0.1 is loaded                 │ PASS                                │        286ns │
│ tychon  │ tychon_harddrive         │ asset     │ index_template logs-tychon.tychon_harddrive is loaded                             │ PASS                                │        328ns │
│ tychon  │ tychon_harddrive         │ asset     │ ingest_pipeline logs-tychon.tychon_harddrive-0.0.1 is loaded                      │ PASS                                │        263ns │
│ tychon  │ tychon_hardware          │ asset     │ index_template logs-tychon.tychon_hardware is loaded                              │ PASS                                │        312ns │
│ tychon  │ tychon_hardware          │ asset     │ ingest_pipeline logs-tychon.tychon_hardware-0.0.1 is loaded                       │ PASS                                │        267ns │
│ tychon  │ tychon_host              │ asset     │ index_template logs-tychon.tychon_host is loaded                                  │ PASS                                │        325ns │
│ tychon  │ tychon_host              │ asset     │ ingest_pipeline logs-tychon.tychon_host-0.0.1 is loaded                           │ PASS                                │        285ns │
│ tychon  │ tychon_networkadapter    │ asset     │ index_template logs-tychon.tychon_networkadapter is loaded                        │ PASS                                │        375ns │
│ tychon  │ tychon_networkadapter    │ asset     │ ingest_pipeline logs-tychon.tychon_networkadapter-0.0.1 is loaded                 │ PASS                                │        316ns │
│ tychon  │ tychon_patch             │ asset     │ index_template logs-tychon.tychon_patch is loaded                                 │ PASS                                │        343ns │
│ tychon  │ tychon_patch             │ asset     │ ingest_pipeline logs-tychon.tychon_patch-0.0.1 is loaded                          │ PASS                                │        292ns │
│ tychon  │ tychon_softwareinventory │ asset     │ index_template logs-tychon.tychon_softwareinventory is loaded                     │ PASS                                │        368ns │
│ tychon  │ tychon_softwareinventory │ asset     │ ingest_pipeline logs-tychon.tychon_softwareinventory-0.0.1 is loaded              │ PASS                                │        289ns │
│ tychon  │ tychon_stig              │ asset     │ index_template logs-tychon.tychon_stig is loaded                                  │ PASS                                │        375ns │
│ tychon  │ tychon_stig              │ asset     │ ingest_pipeline logs-tychon.tychon_stig-0.0.1 is loaded                           │ PASS                                │        300ns │
│ tychon  │ tychon_volume            │ asset     │ index_template logs-tychon.tychon_volume is loaded                                │ PASS                                │        368ns │
│ tychon  │ tychon_volume            │ asset     │ ingest_pipeline logs-tychon.tychon_volume-0.0.1 is loaded                         │ PASS                                │        285ns │
╰─────────┴──────────────────────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────┴──────────────╯

[joeperuzzi]

@efd6 yes visuals and rules have overlap between the datasreams.

Looking at your build output it seems like the sample_event.json files are not correct in those few items, we'll remove the sample_event.json files as they are going unused and have no bearing on the functionality of the product.

One error I'm seeing is "[2] parsing field value failed: field "event.outcome"'s value "fail" is not one of the allowed values (failure, success, unknown)". This dataset is handled in the pipeline, and when looking at the _dev/test/pipeline folder you'll notice the expected outcome matches failure/success/unknown. But once we remove the sample_event.json that should take care of this error.

- script: source: | if(ctx.vulnerability?.result == 'fail'){ ctx.event.outcome = "failure" }else if(ctx.vulnerability?.result == 'pass'){ ctx.event.outcome = "success" }else{ ctx.event.outcome = "unknown" }

[joeperuzzi]

Removed the 4 sample_event.json files from datastreams "cve, epp, networkadapter, and stig". This should correct the build errors on data being undefined. The sample_event.json files were not being used as we felt these would clutter the readme file and this data is better defined in the TYCHON documentation.

[joeperuzzi]

@efd6 we're going to push a new PR with the first stand-alone datastream in it, but I'm going to leave this PR open. We couldn't get it to build correctly as there are overlapping artifacts between streams so hopefully you're good with pull requests that have build errors that are not recoverable. Unfortunately with this process, it's not possible to get it to build without having all the needed artifacts from other streams. We'll have the first push later this afternoon for review and I'll update both tickets to point to each-other.

[efd6]

I'm going to push back on leaving this PR open; there are 82 commit here making review communication more difficult. I would really like to have this simplified by starting from a base of main, with the state that is at the PR that I linked as it fixes things that will need to be fixed in the package before it can move forward.

@efd6 yes visuals and rules have overlap between the datasreams.

OK, so how about we start with adding just the pipelines and tests (including system tests which are absent here). Then, when they are all in, we can add the cross-dependent layers.

[joeperuzzi]

Closing for PR: #8212