Error stream API

[repeatedly]

I hope this is the last discussion for v0.12.
I am now working for adding error emit to send invalid record to @ERROR.

Basic design is here: https://github.com/fluent/fluentd/wiki/V1-Roadmap#error-stream

My plan is adding emit_error_event to EventRouter.
I need a feedback for API design.
@ERROR is optional so we should care no @ERROR case.

Considering record_transformer plugin as an example:

    def filter_stream(tag, es)
      new_es = MultiEventStream.new
      # ...
      last_record = nil
      es.each do |time, record|
        last_record = record # for debug log
        begin
          new_record = reform(time, record, placeholders)
          new_es.add(time, new_record)
        rescue => e
          # Need good API
        end
      end
      new_es
    rescue => e
      # should change to more better message.
      log.warn "failed to reform records", :error_class => e.class, :error => e.message
    end

In my plan, emit_error_event signature is below:

emit_error_event(tag, time, record, error)

Without @ERROR

emit_error_event dumps logs with received arguments.

  es.each do |time, record|
    begin
      new_record = reform(time, record, placeholders)
      new_es.add(time, new_record)
    rescue => e
      router.emit_error_event(tag, time, record, error)  # dump "emit failed. tag = foo, time = bar, record = {...}, error = message" like log
    end
  end

or callback:

router.emit_error_event(tag, time, record, error)  {
  log.warn "map is hoge"
}

Former is easy to implement. Latter can dump plugin specific log message.

With @ERROR

Event are routed to @ERROR's match.
Should we dump additional "emit failed" like messages with warn or debug level?
Using warn, we can use <match fluent**> approach to send fluentd logs to monitor system.

[tagomoris]

Are there any idea note or specification for error stream?

[tagomoris]

I found this url: https://github.com/fluent/fluentd/wiki/V1-Roadmap#error-stream

[repeatedly]

Ah, I forgot to link the document. I added it to the description.

[repeatedly]

I implemented it in v0.12.
We will open an another issue for improvement.