Root directory per worker process #1374
Conversation
tagomoris
added
feature request
tagomoris
changed the title
tagomoris
force-pushed
the
root-directory-per-worker-process
branch
from
ad8fda7 to
176baaa
Compare
tagomoris
force-pushed
the
root-directory-per-worker-process
branch
from
176baaa to
d52686b
Compare
|
@repeatedly could you review this change? |
| @@ -56,6 +51,14 @@ def initialize | |||
| def configure(conf) | |||
| super | |||
|
|
|||
| unless @path | |||
There was a problem hiding this comment.
In the future, should we validate @path value for avoding conflict trouble?
Fixed path should not allowed with multiprocess configuration or
support %{worker_id}/%{plugin_id} placeholder?
There was a problem hiding this comment.
Buffer chunks have unique_id, which are generated with time and random value.
So it SHOULD NOT conflict with other chunks in other processes. No need to do such thing.
There was a problem hiding this comment.
Ahhh, there are some problem about resuming at startup...
There was a problem hiding this comment.
Anyway, plugin_id can't help that situation. plugin_id is (should be) unique in a worker process, not server global.
|
|
||
| def plugin_root_dir | ||
| return @_plugin_root_dir if @_plugin_root_dir | ||
| return nil unless system_config.root_dir |
There was a problem hiding this comment.
Put log.warn is better?
Maybe, users hard to find the cause why buffer_path/storage is not configured automatically.
There was a problem hiding this comment.
No, it's not about users.
This code is called by plugins, and plugin should check whether plugin_root_dir is available or not by return value, and raise errors with messages which shows how to configure that plugin.
There was a problem hiding this comment.
The reason of this implementation is, plugin MAY use any default values (paths or any others) if root directory is unavailable.
| def plugin_root_dir | ||
| return @_plugin_root_dir if @_plugin_root_dir | ||
| return nil unless system_config.root_dir | ||
| return nil unless plugin_id_configured? |
|
Can I merge this? |
|
LGTM |
This change introduces system-wide "root directory" and root directory per plugins.
It's enabled when:
root_dirspecified in<system>section@idconfigured in each plugin configuration sectionsThe root directories per plugin instances must be unique in a filesystem (between many workers of fluentd). So it uses
worker_idprovided by supervisor process via environment variable.The path of directories are:
/system-wide-root/workerN/plugin-id(generated and provided viaFluent::Plugin::Base#plugin_root_dir). Plugins will creates files or directories under that.This change also adds 2 implementations to use this new feature:
buf_file: configure buffer filepathautomaticallystorage_local: configure storage filepathautomaticallyI have an additional plan to fix
in_tailto use storage plugin as replacement ofpos_file, but it's will be added by another pull-request.