kinesis-to-s3

Application for copying Amazon Kinesis data to S3.

Usage

1. Download the app
2. Configure your application (start with the template KinesisToS3.SAMPLE.properties)
3. Configure AWS access credentials:
   1. Locally, set either
      1. the environment variables AWS_ACCESS_KEY_ID=xxx and AWS_SECRET_KEY=yyy
      2. or the JVM options -Daws.accessKeyId=xxx and -D=aws.secretKey=yyy
   2. On Amazon EC2
      1. No configuration needed if you start the server instance with a proper IAM role (highly recommended!)
4. Run the app: java -jar kinesis-to-s3-[version].jar configuration.properties

Development

The main class is com.github.codeflows.aws.KinesisToS3

Building

Build with Maven. Running

mvn clean package

will create a runnable jar at ./target/kinesis-to-s3-[version].jar

Notes

Licensed under the MIT license.

Uses the Amazon Kinesis Client Library, the Amazon Kinesis Connector Library and the AWS SDK for Java. These are also embedded in the runnable jar. See the AWS licensing terms.

Rationale

The Kinesis Connector sample code contains code for copying Kinesis data to S3, but is not easily reusable. This application contains the minimal code for the copying process and nothing else.