packetbeat: add support for NFS v3 and v4 protocols

[kofemann]

This pull request is more an RFC. We will run it bit in production go get more experience. Comments are welcome. The example output:

{
  "@timestamp": "2016-03-28T06:18:18.431Z",
  "beat": {
    "hostname": "localhost",
    "name": "localhost"
  },
  "count": 1,
  "dst": "127.0.0.1",
  "dst_port": 2049,
  "nfs": {
    "minor_version": 1,
    "opcode": "GETATTR",
    "status": "NFSERR_NOENT",
    "tag": "",
    "version": 4
  },
  "rpc": {
    "auth_flavor": "unix",
    "call_size": 200,
    "cred": {
      "gid": 500,
      "gids": [
        491,
        499,
        500
      ],
      "machinename": "localhost",
      "stamp": 4597002,
      "uid": 500
    },
    "reply_size": 96,
    "status": "success",
    "time": 25631000,
    "time_str": "25.631ms",
    "xid": "2cf0c876"
  },
  "src": "127.0.0.1",
  "src_port": 975,
  "type": "nfs"
}

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

[kofemann]

The one of the questions I would like to know: the other beats use field names with dots, but elasticsearch 2.0 does not allow that. So is it ok in beats?

[kofemann]

Ok. Now it's in the shape, that others can use it as well, IOW - this is a real pull request now.

[kofemann]

Dashboard example

[andrewkroh]

I would recommend not using a pointer here. Maps are reference types so they are always passed by reference.

[kofemann]

fixed.

[andrewkroh]

Thanks for the contribution! Very cool screen shot. A couple miscellaneous points:

• Try running your package against goimports. It should change the ordering on some of the imports.
• Some of the variable names are using underscores. The Go convention is mixedCase. Try running golint against your package.
• I see you added some pcaps. To ensure that there are no regressions we like to have some system tests in-place that replay the pcaps and check the outputs. Could you add some test cases like these ones that use those pcaps?
• And finally, to make sure that everyone understands what the fields mean it would be great if you could document their meanings. We put each field into fields.yml then we generate documentation from it by running make update.

[kofemann]

I have addresses all issues you pointed out.. With style it was not clear, the code used mixed style in various places, like tcptuple.Dst_ip.String(). Any way, Tests added, rebased to current master, branch updated.

[andrewkroh]

The one of the questions I would like to know: the other beats use field names with dots, but elasticsearch 2.0 does not allow that. So is it ok in beats?

Because field names containing dots are not allowed in ES 2.0 we do not use them in Beats. We use nested maps which gives the appearance of field names with dots because you can use dot-notation to Kibana and ES to reference the fields. For example:

event = common.MapStr{}
nfs = common.MapStr{}
event["nfs"] = nfs
nfs["version"] = 4

It appears that you figured this out. 👍 I neglected to see the question until now.

[andrewkroh]

There appears to be a few tabs here that are throwing off the indentation. Please convert them to spaces. Thanks

[kofemann]

sorry about that. fixed.

[andrewkroh]

Looks good. I would like to get at least one more reviewer on this. @elastic/beats

[kofemann]

I will create a separate PR for dashboard later on.

[andrewkroh]

Please run make update to re-generate the packetbeat/docs/fields.asciidoc document.

[monicasarbu]

@kofemann Thank you for the complete PR and the nice dashboard. I have only a small concern.

I think it would be easier for the users to understand if we rename the Packetbeat module to nfs to be the same with the name of the configuration section nfs. In this case, we can move all the RPC information from rpc under nfs.rpc.

[kofemann]

@monicasarbu I had the same idea during prototyping was renaming it already several times :). The reason to call it rpc is that it will be easy to add other oncrpc/sunrpc based protocols as well. Nevertheless, I am fine to rename it. @andrewkroh any strong opinion?

[andrewkroh]

I would rename to nfs and re-organize event structure now. If in the future new RPC based protocols are added we can refactor a bit to reuse this code.

[kofemann]

updated. The fields are still rpc.xxx and nfs.xxx. This makes more sense. You don't put tcp relates staff into http namspace

[monicasarbu]

@kofemann Ok, I agree with you that it makes more sense to have rpc and nfs inside the nfs event type.

[kofemann]

thanks! About dashboard: shold it go into packetbeat/etc/kibana ? Is there any conventions?

[monicasarbu]

@kofemann I am working now at a simple way to import and export the Kibana dashboard. For now, you can only export all the Kibana dashboards and dependencies (visualizations, searches, index-patterns) using the python script from https://github.com/elastic/beats-dashboards/tree/master/save. Please send us a PR against the beats-dashboards repo with all the exported dashboards and dependencies. I can help you to sort them out once the PR is opened. Thanks!