Update filebeat registry configuration

[urso]

Change registry config and directory layout, preparing for future
changes.

• remove settings filebeat.registry_file and replace with
  filebeat.registry.path. Path will be a directory, and the
  actual contents will be stored under
  ${filebeat.registry.path}/filebeat/data.json.
• introduce <registry path>/filebeat/meta.json with version number of
  the current directories layout.
• move filebeat.registry_flush to filebeat.registry.flush
• move filebeat.registry_file_permission to
  filebeat.registry.file_permission
• update tests

This PR introduces breaking changes now, so to reduce the impact for when we introduce the new registry format.

Note: I did keep the registry settings under filebeat.registry, but I'm still considering to change to top-level setting registry, as the future registry might become interesting for some auditbeat use-cases as well.

[houndci-bot]

exported type Registry should have comment or be unexported

[urso]

Jenkins, test this.

[ruflin]

As the new registry is a libbeat feature, should the config be prefixed by filebeat or not?

[urso]

As the new registry is a libbeat feature, should the config be prefixed by filebeat or not?

I'm thinking the same. It's a little ugly though. Filebeat itself needs to install the registry with the publisher pipeline, but filebeat does not have access to the root configuration (without introducing even more hacks).

The core functionality will be provided by libbeat. This is basically a kvstore interface. but the actual registrar will stay in filebeat as well, as the registrar is very much coupled with the log input. Out of the box integration of registry with publisher pipeline and required refactorings in filebeat is more likely a 8.0 target.

[ph]

@urso good looks good and thanks for taking care of updating the doc, the failures on the CI are maybe related, they all look to be a synchronization issue.

[urso]

fixed filebeat tests. Was another rebase error.