v0.14 migration

[cosmo0920]

Why not migrate to Fluentd v0.14 API?

• Pros/Cons

  • https://gist.github.com/okkez/05ee6af7b8bb7de852a36eb2f679c344

• Updating plugin for v0.14 from v0.12

  • http://docs.fluentd.org/v0.14/articles/plugin-update-from-v12

• Other Google Cloud Platform services:

  • fluent-plugin-gcs: done
  • fluent-plugin-gcloud-pubsub: will be merged soon
  • fluent-plugin-gcloud-pubsub-custom: done

And 100+ plugins have already been migrated to Fluentd v0.14 plugin API.

If you have questions, feel free to mention to me.

[igorpeshansky]

We are waiting for fluent/fluentd#1555 to be fixed before we fully commit to 0.14.

[cosmo0920]

fluent/fluentd#1623 should fix fluent/fluentd#1555.
Is it enough to start v0.14 migration?

[igorpeshansky]

We just updated our packaging to pull in v0.14.19. We'd like to release that and observe it in the wild for a couple of weeks — if all goes fine, we should be ready to migrate. Ideally, 0.14 would be marked "stable" on the fluentd side first...

[cosmo0920]

Fluentd v0.14.22 which is marked as stable release has been released.
https://www.fluentd.org/blog/fluentd-v0.14.22-has-been-released

Is this really enough for v0.14 migration?

[igorpeshansky]

Some of our clients are still on v0.12. We're working to address the issues they're having with v0.14. Once we have no more clients using the latest fluent-plugin-google-cloud with v0.12, we'll consider migrating to v0.14.

[derekperkins]

Will this work with 1.x?

[shusugmt]

This seems to be already done in 906df5e
I confirmed with fluent-plugin-google-cloud 0.7.0 on fluentd 1.2.5

[cosmo0920]

@shusugmt No, this plugin still doesn't use Fluentd v1 native API.

[qingling128]

The changes recommended in https://docs.fluentd.org/v1.0/articles/plugin-update-from-v0.12#buffered-output-plugins will need to wait till our next major release to avoid disruptions on the client side. Probably some time next year.

[apu1111]

@qingling128 @igorpeshansky
Any update/timeline for this? I am particularly interested for the multi-worker support.

[qingling128]

@apu1111 - Thanks for checking! The fluent-plugin-google-cloud output plugin itself already supports multi-worker:

fluent-plugin-google-cloud/lib/fluent/plugin/out_google_cloud.rb

Line 716 in 975357e

def multi_workers_ready?

As to the next major release cut, it is being evaluated / prioritized in the roadmap now. We are not sure how soon it would be yet though.

[apu1111]

@qingling128 - Thanks for the info. We wanted to upgrade to fluentd 1.4 to use <worker N-M> directive. Anyway will wait for the next major release.

[jkohen]

@qingling128 can you please triage? What value does this provide?

[qingling128]

@jkohen The values of this migration is described here: https://gist.github.com/okkez/05ee6af7b8bb7de852a36eb2f679c344.

We have to cut a major release for this though, which we already agreed to do. It's a matter of prioritization.

[jkohen]

Are any of the "pros" listed there and the migration itself particularly valuable for Stackdriver users in order to justify doing the migration sooner? It sounds like no, so just confirming. If that's the case, then I think that we should close this ticket.

[qingling128]

This is now obsolete. See https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent for collecting logs from GCE.