Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fluentから、Amazon S3への書き込み方法について #1

Closed
CkReal opened this issue Oct 6, 2011 · 2 comments
Closed

fluentから、Amazon S3への書き込み方法について #1

CkReal opened this issue Oct 6, 2011 · 2 comments

Comments

@CkReal
Copy link

CkReal commented Oct 6, 2011

初めまして、CkRealこと本間と申します。
現在、fluentを試験的に使っています。

fluentからAmazon S3への書き込みテストを行った際にエラーを発生したので、
どなたか心当たりがある方がいれば、ご教示いただけないでしょうか。

転送先サーバから転送元サーバへのローカルファイル転送までは確認できています。

以下、試験環境とスタックトレースになります。

■試験環境
AWS上で、以下のサーバ構成でテストを行なっています。
転送元サーバのアクセスログを転送先サーバに送信した後、
Amazon S3に書きこもうとしています。

【転送元サーバ】
OS：CentOS5.6
Apache：2.2.3
fluent：0.9.18

□設定ファイル
[source]
type tail
format apache
path /var/log/httpd/【アクセスログ名】
tag apache.access

[match apache.access]
type tcp
host 【ホスト名】
port 24224
buffer_type file
buffer_path /tmp/fluent_buf_myforward
flush_interval 10s

【転送先サーバ】
OS：CentOS6.0
fluent：0.9.18
fluent-plugin-s3：0.1.1

□設定ファイル
[source]
type tcp
port 24224
bind 0.0.0.0
[match apache.access]
type s3
aws_key_id 【AWSのアクセスキー ID】
aws_sec_key 【AWSのシークレットアクセスキー】
s3_bucket 【バケット名】
buffer_path /tmp/hoge
path /fluent/fluent.log

■スタックトレース
2011-10-06 17:53:53 +0900: fluent/engine.rb:119:rescue in emit_stream: emit transaction faild error="undefined method iso8601' for 2011-10-06 17:53:51 +0900:Time" 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.1.1/lib/fluent/plugin/out_s3.rb:46:inblock in configure'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.1.1/lib/fluent/plugin/out_s3.rb:64:in call' 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.1.1/lib/fluent/plugin/out_s3.rb:64:informat'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/output.rb:407:in block in emit' 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/event.rb:81:ineach'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/event.rb:81:in each' 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/output.rb:398:inemit'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/match.rb:33:in emit' 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/engine.rb:117:inemit_stream'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/engine.rb:106:in emit_array' 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/plugin/in_stream.rb:89:inon_message'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/plugin/in_stream.rb:119:in each' 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/plugin/in_stream.rb:119:inon_read'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/io.rb:108:in on_readable' 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/io.rb:170:inon_readable'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in run_once' 2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:inrun'
2011-10-06 17:53:53 +0900: fluent/engine.rb:120:rescue in emit_stream: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/plugin/in_stream.rb:44:in run' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:46:rescue in run: unexpected error error="undefined methodiso8601' for 2011-10-06 17:53:51 +0900:Time"
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.1.1/lib/fluent/plugin/out_s3.rb:46:in block in configure' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.1.1/lib/fluent/plugin/out_s3.rb:64:incall'
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.1.1/lib/fluent/plugin/out_s3.rb:64:in format' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/output.rb:407:inblock in emit'
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/event.rb:81:in each' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/event.rb:81:ineach'
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/output.rb:398:in emit' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/match.rb:33:inemit'
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/engine.rb:117:in emit_stream' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/engine.rb:106:inemit_array'
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/plugin/in_stream.rb:89:in on_message' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/plugin/in_stream.rb:119:ineach'
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/plugin/in_stream.rb:119:in on_read' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/io.rb:108:inon_readable'
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/io.rb:170:in on_readable' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:inrun_once'
2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.0.0/lib/cool.io/loop.rb:96:in run' 2011-10-06 17:53:53 +0900: plugin/in_stream.rb:47:rescue in run: /usr/local/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-0.9.18/lib/fluent/plugin/in_stream.rb:44:inrun'
@repeatedly
Copy link
Member

This is a bug.

    if @localtime
      @formatter = Proc.new {|tag,event|
        "#{Time.at(event.time).iso8601}\t#{tag}\t#{event.record.to_json}\n"
      }
    else
      @formatter = Proc.new {|tag,event|
        "#{Time.at(event.time).utc.iso8601}\t#{tag}\t#{event.record.to_json}\n"
      }
    end

Time#iso8601 is not default method, needs "require 'time'".

I wrote a following patch. @frsyuki ?

diff --git a/lib/fluent/plugin/out_s3.rb b/lib/fluent/plugin/out_s3.rb
index 7f4e2ae..e245ef1 100644
--- a/lib/fluent/plugin/out_s3.rb
+++ b/lib/fluent/plugin/out_s3.rb
@@ -8,6 +8,7 @@ class S3Output < Fluent::TimeSlicedOutput
     super
     require 'aws-sdk'
     require 'zlib'
+    require 'time'
   end

   def configure(conf)

@repeatedly
Copy link
Member

I pushed a patch. Please retry!

パッチコミットして0.1.2にバージョンあげました，
後でrubygems.orgに反映されるはずなので，その後もう一度試して見て下さい．

chillaxd pushed a commit to chillaxd/fluent-plugin-s3 that referenced this issue Dec 19, 2016
@chillaxd
# This is a combination of 2 commits.
af5d40a 
# This is the 1st commit message:

# This is a combination of 3 commits.
# This is the 1st commit message:
# This is a combination of 3 commits.
# This is the 1st commit message:
Out S3 change

so that user can use minimal policy at AWS, to work with fluentd
like S3:PutObject

# This is the commit message fluent#2:

Spacing handled in out-S3 plugin

# This is the commit message fluent#3:

Identation modified

# This is the commit message fluent#4:

Introduce check_bucket paramater
To skip the using of auto_create_bucket for bucket existence checking

# This is the commit message fluent#5:

Fixes:

Test cases added.
README has been modified with check_bucket's usage and sample AWS policy.

# This is the commit message fluent#6:

Fixes:

Issues for extra spoaces. Modified

# This is the commit message fluent#7:

Spacing issues

# This is the commit message fluent#8:

Update test_out_s3.rb

# This is the commit message fluent#9:

time slicer modified for utc also

# This is the commit message fluent#10:

removing test cases

# This is the commit message fluent#1:

renaming time_slicer variable

# This is the commit message fluent#2:

Test casees added

Test cases for out_s3 plugin has been added. This feature will work with hardened
S3 policy like with only PutObject permission. As it will not check bucket existence
not even object existence, as for each object creation it will name it with timestamp.
So anyhow it will be unique everytime.

Only thing it willl assume that bucket is created already.

# This is the commit message fluent#3:

Some variable renamed

# This is the commit message fluent#2:

Clean up dependencies

# This is the commit message fluent#3:

Update README for placeholders

# This is the commit message fluent#2:

v0.8.0.rc1
repeatedly pushed a commit that referenced this issue Jan 28, 2017
@cmarodrigues
Merge pull request #1 from cmarodrigues/add-option-ssl_verify_peer
6d786a3 
Update out_s3.rb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@repeatedly @CkReal