Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can I get s3 output format like y=2016/m=01/d=25/h=17/M=00 (M=15,M=30,M=45)? #784

Closed
git-bernie opened this issue Jan 26, 2016 · 7 comments
Labels
feature request *Deprecated Label* Use enhancement label in general v0.14

Comments

@git-bernie
Copy link

I am being asked to log directly to s3. The requested s3 format would look like s3-bucket/logs/y=2016/m=01/d=25/h=17/M=00/ info.log-xxxx-x
where m=00, m=15, m=30, and m=45.

I can't think of a way of getting this without having four separate match/store blocks. Maybe I am missing something obvious with tags?

The file structure now is currently /mnt/logs/2016/01/25/info.log but it could even change on the file system to /mnt/logs/2016/01/25/h=17/M=15.

Thanks for your help.

@git-bernie git-bernie changed the title Can I get s3 output format like y=2016/m=01/d=25/h=27/M=00 (M=15,M=30,M=45)? Can I get s3 output format like y=2016/m=01/d=25/h=17/M=00 (M=15,M=30,M=45)? Jan 26, 2016
@repeatedly
Copy link
Member

where m=00, m=15, m=30, and m=45.

There is no way for now.
If you want to do it, extending S3 plugin and change @time_slicer is better.
https://github.com/fluent/fluentd/blob/master/lib/fluent/output.rb#L516-L528
https://github.com/fluent/fluentd/blob/master/lib/fluent/output.rb#L565

@tagomoris
Copy link
Member

timekey_range configuration parameter for Fluentd v0.14 will solve this problem.

@tagomoris tagomoris added the v0.14 label Apr 6, 2016
@repeatedly
Copy link
Member

@tagomoris From the master, timekey_range is already supported. Can w close this issue?

@repeatedly
Copy link
Member

Or close after release fluentd v0.14.0.pre2?

@tagomoris
Copy link
Member

tagomoris commented Apr 27, 2016

No. There's no way to instantiate & run plugins with new APIs.
It depends on #912 .

@repeatedly
Copy link
Member

Ah, I see. Okay.

@repeatedly repeatedly added the feature request *Deprecated Label* Use enhancement label in general label Apr 27, 2016
@tagomoris
Copy link
Member

#912 was merged into master.
I know fluent-plugin-s3 doesn't support that feature yet, but it's the issue of that plugin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request *Deprecated Label* Use enhancement label in general v0.14
Projects
None yet
Development

No branches or pull requests

3 participants