V1 Roadmap

Roadmap for v1

v0.12 (2014 winter)

Filter & Label

• Goal: Don't rewrite tags!
  • Encourage the use of a tag as an identifier of a data source, NOT as a mechanism to filter/modify data. Use filter_label for the latter.
• Design:
  • See filter_label
• Implementation:
  • See filter_label

Maybe including partially Error Stream support because Error Stream is a part of Label.

v0.13

Odd number is development version

v0.14 (2016 summer)

Actor

• Goal: Provide a API set to start/stop threads and receive/send data through sockets to plugin developers
• Design:
  • Create new class
  • Initialize/start/stop the class automatically at a base class or mix-in module.
• Necessary refactoring:
  • Add #close method to all plugins to not cause "already closed" exception
    • #shutdown stops emitting new records
    • #close stops receiving new records
    • Actor stops threads at shutdown and closes sockets at #close
• Implementation:
  • Actor should use an IO library which supports JRuby and Windows.
  • Otherwise, Actor can use threads.

ServerEngine

• Purpose: Support Windows and simplify multi-process code
• Design:
  • Replace supervisor and server code with ServerEngine
  • Including daemonize, logging, shutdown/restart, and signal handling.
• Implementation:
  • Implement worker_type=spawn to ServerEngine

SocketManager

• Purpose: Zero-downtime restart
• Design:
  • Add a new class named SocketManager
  • SocketManager listens TCP/UDP sockets in the parent process of ServerEngine
    • Child processes requests the parent process to listen a TCP/UDP socket
    • The parent process listens a new socket and passes its file descriptor to the child process

https://github.com/naritta/fluentd/tree/socket-manager

New Buffer API

• Purpose: Buffer plugins should be easier to create
• Design:
  • Not designed yet
• Implementation:
  • Create a new Buffer class
  • Create a new buffered output class (in Fluentd::plugin namespace?)
  • Create an adaptor code into old buffered output class to use the new Buffer class

https://github.com/fluent/fluentd/pull/562

Sub-second support

https://github.com/fluent/fluentd/pull/653

Windows support

https://github.com/fluent/fluentd/pull/674

v1.0 (2017 earlier)

Finalized v0.12 and v0.14 changes, fixed critical bugs, v1.0 will be released.

TODO items

Error stream

• Goal: Provide standardized concept and config syntax to handle broken events to users
• Design:
  • Route broken events (which caused a exception in emit of output plugins) to a special built-in label
  • Provide a base class or mix-in for plugin developers
  • Don't use chain any more
• Implementation:
  • EventRouter catches an exception and calls @error_handle_collector.emit
  • Optionally, output plugins can call directly @error_handle_collector.emit.
    • In this case, output plugins should not throw an exception.
    • This is necessary if only part of multiple events in a EventStream are broken.
    • In other words, output plugins need to call handle exceptions appropriately if they have optimization to handle multiple events at once.