Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timestamp is always parsed as local time #377

Open
wonnyz opened this issue Nov 11, 2020 · 1 comment
Open

Timestamp is always parsed as local time #377

wonnyz opened this issue Nov 11, 2020 · 1 comment
Labels
enhancement Feature request

Comments

@wonnyz
Copy link

wonnyz commented Nov 11, 2020

We use UTC timestamps when we write logs in Kafka, and date partitioning on S3.
When I tried to parse a timestamp column like { "@t": "2020-11-11T07:58:58.7291622Z" }, generated filename was 20201110_22xx (GMT+9), not 20201111_07xx!

in-kafka-group creates TimeParser with just time_format, since there is no parameter like utc.

if @time_source == :record and @time_format
if defined?(Fluent::TimeParser)
@time_parser = Fluent::TimeParser.new(@time_format)
else
@time_parser = Fluent::TextParser::TimeParser.new(@time_format)
end
end

TimeParser uses local time by default. (localtime = true)
https://github.com/fluent/fluentd/blob/6757d70b7fdfcf5e4a6316b1ecd802dfeb0e326e/lib/fluent/time.rb#L208

Can we have utc parameter like parsers? When I set time_source as kafka, it worked properly. The timestamp was UTC.

@kenhys kenhys added the enhancement Feature request label Jul 5, 2021
@kenhys
Copy link
Contributor

kenhys commented Jul 5, 2021

It seems that supporting utc may be good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature request
Projects
None yet
Development

No branches or pull requests

2 participants