[Auditbeat] Socket metricset

[cwurm]

This sockets metricset is mostly based on Metricbeat's socket metricset. It uses the same code in a number of places:

1. Netlink to get a list of sockets from the kernel (code refactored into a new NetlinkSession)
2. Enrichment by username (UserCache), direction (ListenerTable), and process (ProcTable).

Otherwise, it pretty much follows the conventions of the existing metricsets in the system module.

There are a few things we might want to do in the future:

1. I'll be opening a PR after this for collecting user information in the host metricset. With a list of users available over there we might want to access that instead of doing new lookups here.
2. Similarly, we already have process information in the processes metricset, so we might want to use that instead of relying on ProcTable. We'll still have to scan /proc to find the PID owning a socket though.
3. Some sockets are not owned by a user process, and so their inode does not appear in /proc (e.g. NFS). At least some can be owned by RPC services, and I have a working prototype for enriching such sockets with information about the RPC service that owns them. It's a bit tricky to get a list of RPC services in a good way so we might not want to do it now. I'll open an issue after this.

[webmat]

LGTM overall.

I have a few comments about leveraging ECS more, mostly :-)

[cwurm]

Should be ready for another review.

High-level changes since the last:

• To be able to reuse some of the code from Metricbeat's system/socket I moved listeners.go and ptable.go to metricbeat/helper/.
• Fields should now be ECS compliant (including using network/source/destination)
• Regular state reporting.

[tsg]

Left one question I wasn't sure about, but otherwise LGTM.

[andrewkroh]

Overall this looks pretty good and I'm glad you were able to reuse some Metricbeat parts.

I'm curious how strongly others feel about the utility of persisting state to disk for this metricset. And I think we need some changes to more closely match ECS.

[cwurm]

Changes:

• removed TCP state
• changed entirely to top-level ECS fields
• removed storing state timestamp to disk, and changed default state.period to 1h

@andrewkroh Let me know if this looks good, please.

[andrewkroh]

LGTM

[cwurm]

Adding network.type to fields.ecs.yml opened pandora's box, with all fields.asciidoc needing an update. Something I think we should rather do in master - so for now I've merged this after backing out the change to fields.ecs.yml and catching the "field not documented" exception in the socket system test. To be removed when the field is in - I'll create a follow-up issue.

[webmat]

@cwurm Yes, when I did a big push on Filebeat modules, the MO was to add the fields as locally as possible (e.g. module's overall _meta/fields.yml, or Filebeat level).

Fully defining all fields in libbeat has some issues.

One of which is that there used to be only a few common fields. So each Beat would list them right alongside their local field definitions in the documentation, and it wasn't an issue.

When introducing ECS' 100+ fields, this means the Beat fields are drowned in common field definitions 😆 It's fine that they're documented explicitly in each Beat, but it should be made clear which are common fields and which are specific fields.

@ruflin is working on improving documentation generation and other wide ranging issues like this.

[ruflin]

@cwurm If you are targeting 6.x, please don't add all ECS fields and only the ones you need ;-)

[cwurm]

@ruflin Which fields did you have in mind?

This will go into master first, and then be backported to 6.x. I assume we might have to adjust some fields for the backport.

[ruflin]

@cwurm Can you ping me on the backport? I'm worried that some fields might conflict with fields we have set as alias in 6.x (hopefully not).

[cwurm]

@ruflin Backport PR is open: #9581