Commits on Oct 31, 2016

1. Improve prospector state handling …

   Previously each prospector was holding old states initially loaded by the registry file. This was changed to that each prospector only loads the states with a path that matches the glob pattern. This reduces the number of states handled by each prospector and reduces overlap.
   
   One consequence of this is that in case a log file "moves" from one prospector to an other through renaming during the runtime of filebeat, no state will be found. But this behaviour is not expected and would lead to other issues already now. The expectation is that a file stays inside the prospector over its full lifetime.
   
   In case of a filebeat restart including a config change, it can happen that some states are not managed anymore by a prospector. These "obsolete" states will stay in the registrar until a further config change when a glob pattern would again include these states. As an example:
   
   Filebeat is started with the following config.
   
   ```
   filebeat.prospectors:
   - paths: ['*.log']
     clean_removed: true
   ```
   
   There is a log file `a.log` and `b.log`. Both files are harvested and states for `a.log` and `b.log` are persisted. Then filebeat is stopped and the config file is modified to.
   
   ```
   filebeat.prospectors:
   - paths: ['a.log']
     clean_removed: true
   ```
   
   Filebeat is started again. The prospector will now only handle the state for `a.log`. In case `b.log` is removed from disk, the state for `b.log` will stay in the registry file.
   
   In case the config file was with `clean_inactive: 5min`, all TTL are reset on restart to `-2` by the registry and `-1` the prospector or the new `clean_inactive` value. Using `-2` in the registrar can be useful in the future to detect which states are not managed anymore by any prospector. As all TTL are reset on restart, persisting of the TTL is not required anymore but can become useful for cleanup so the information is kept in the registry file.
   
   Further changes:
   
   * Add tests for matchFile method
   * Add tests for prospector state init filtering
   * states are passed to `Prospector.Init` on startup instead of setting it directly in the object
   * `Prospector.Init` and `Prospectorer.Init` now return an error and filebeat exits on startup problems
   * Remove lastClean as not needed anymore
   * Have one log file for each bom file test
   * Update to new vagrant box with Golang 1.7.3

   

   ruflin committed Oct 31, 2016
   Configuration menu

   • View commit details
   • Copy full SHA for 33e2a8c
   • Browse repository at this point

   Copy the full SHA

   33e2a8c View commit details

   Browse the repository at this point in the history