[Zeek] Add additional data sets

[legoguy1000]

What does this PR do?

Add new data sets for known_hosts, known_certs, known_services, & software logs files.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Relates Additional zeek module log files #3288

Screenshots

[elasticmachine]

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-05-11T15:47:16.897+0000

• Duration: 2 min 23 sec

Steps errors

Expand to view the steps failures

Google Storage Download

• Took 0 min 0 sec . View more details here

Error signal

• Took 0 min 0 sec . View more details here
• Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/legoguy1000 return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/legoguy1000 : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/legoguy1000 : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/reference/orgs#check-organization-membership-for-a-user"}

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-28T01:49:12.499+0000

• Duration: 71 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	219
Skipped	0
Total	219

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

/test

[legoguy1000]

@efd6 can we retest

[efd6]

/test

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (43/43)	💚
Files	96.341% (79/82)	👎 -0.218
Classes	96.341% (79/82)	👎 -0.218
Methods	86.267% (647/750)	👎 -2.856
Lines	92.968% (8620/9272)	👍 2.933
Conditionals	100.0% (0/0)	💚

[efd6]

/test

[legoguy1000]

fixed the test error. Ready to retest.

[efd6]

/test

[legoguy1000]

@efd6 Any idea about the error from Elastic Machine or from Jenkins found 0 hits in logs-zeek.known_certs-ep data stream: index_not_found_exception: no such index [logs-zeek.known_certs-ep] Status=404??

[efd6]

Incidental to looking into the issue:

--- Test results for package: zeek - START ---
FAILURE DETAILS:
zeek/dns test-dns.log:
[0] parsing field value failed: the IP "40.126.31.143" is not one of the allowed test IPs (see: https://github.com/elastic/elastic-package/blob/main/internal/fields/_static/allowed_geo_ips.txt)
zeek/known_certs test-known_certs.log:
[0] parsing field value failed: field "event.kind"'s value "info" is not one of the allowed values (alert, enrichment, event, metric, state, pipeline_error, signal)
zeek/known_hosts test-known_hosts.log:
[0] parsing field value failed: field "event.kind"'s value "info" is not one of the allowed values (alert, enrichment, event, metric, state, pipeline_error, signal)
zeek/known_services test-known_services.log:
[0] parsing field value failed: field "event.kind"'s value "info" is not one of the allowed values (alert, enrichment, event, metric, state, pipeline_error, signal)
zeek/software test-software.log:
[0] parsing field value failed: field "event.kind"'s value "info" is not one of the allowed values (alert, enrichment, event, metric, state, pipeline_error, signal)

[efd6]

The non-progress issue comes down to the absence of terminating new-lines in the test inputs. There are bunch of other inputs that suffer the same issue, but these all have more than one test line, so they pass because only a single event is required to allow a system test to pass (they would have been found if this were in place).

[legoguy1000]

🤦‍♂️ ya I've had this issue before, good catch. I'll fix this in the AM and then I think we should be good.

[efd6]

The other inputs that have the issue are

• sample_logs/capture_loss.log
• sample_logs/dhcp.log
• sample_logs/files.log
• sample_logs/notice.log
• sample_logs/sip.log
• sample_logs/snmp.log
• sample_logs/ssh.log
• sample_logs/ssl.log

[legoguy1000]

@efd6 fixed

[efd6]

/test

[legoguy1000]

@efd6 I think i got it now 🤦

[efd6]

That will fix the non-progress, but host is already populated. You will need to move the host assignments to other fields. I'm not sure where at this stage. I'm looking into this.

[efd6]

/test

[legoguy1000]

The forwarded tag needs to be added so that host is only set by zeek, not by the agent.

[efd6]

/test