added example with parquet

Author: hifly81

README.adoc

@@ -2074,8 +2074,9 @@ A S3 sink connector will be created with this link:kafka-connect-sink-s3/config/
 }
 ----
 
-Sink will read messages from topic _gaming-player-activity_ and store in S3 bucket _gaming-player-activity-bucket_.
-Sink will generate a new object storage entry every 100 messages (_flush_size_).
+Sink connector will read messages from topic _gaming-player-activity_ and store in S3 bucket _gaming-player-activity-bucket_ using _io.confluent.connect.s3.format.avro.AvroFormat_ as format class.
+
+Sink connector will generate a new object storage entry every 100 messages (_flush_size_).
 
 To generate random records for topic _gaming-player-activity_ we will use link:https://github.com/ugol/jr[jr] tool.
 
@@ -2086,7 +2087,7 @@ Send 1000 messages to _gaming-player-activity_ topic using jr:
 docker exec -it -w /home/jr/.jr jr jr template run gaming_player_activity -n 1000 -o kafka -t gaming-player-activity -s --serializer avro-generic
 ----
 
-Verify that 10 entries are stored in Minio into _gaming-player-activity-bucket_ bucket, connecting to MiniIO web console, http://localhost:9000 (admin/minioadmin):
+Verify that 10 entries are stored in MinIO into _gaming-player-activity-bucket_ bucket, connecting to MiniIO web console, http://localhost:9000 (admin/minioadmin):
 
 image::images/minio.png[gaming-player-activity-bucket]
 
@@ -2097,6 +2098,65 @@ Teardown:
 scripts/tear-down-connect-sink-s3.sh
 ----
 
+==== Parquet format
+
+Same example but Sink connector will read Avro messages from topic _gaming-player-activity_ and store them in S3 bucket _gaming-player-activity-bucket_ using _io.confluent.connect.s3.format.parquet.ParquetFormat_ as format class.
+
+The format of data stored in MinIO will be Parquet.
+
+Run the example:
+
+[source,bash]
+----
+scripts/bootstrap-connect-sink-s3-parquet.sh
+----
+
+A S3 sink connector will be created with this link:kafka-connect-sink-s3/config/s3_parquet_sink.json[config]:
+
+[source,bash]
+----
+{
+  "name": "sink-parquet-s3",
+  "config":
+  {
+    "topics": "gaming-player-activity",
+    "tasks.max": "1",
+    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
+    "store.url": "http://minio:9000",
+    "s3.region": "us-west-2",
+    "s3.bucket.name": "gaming-player-activity-bucket",
+    "s3.part.size": "5242880",
+    "flush.size": "100",
+    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
+    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
+    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
+    "parquet.codec": "snappy",
+    "schema.registry.url": "http://schema-registry:8081",
+    "value.converter": "io.confluent.connect.avro.AvroConverter",
+    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
+    "value.converter.schema.registry.url": "http://schema-registry:8081"
+  }
+}
+----
+
+Send 1000 messages to _gaming-player-activity_ topic using jr:
+
+[source,bash]
+----
+docker exec -it -w /home/jr/.jr jr jr template run gaming_player_activity -n 1000 -o kafka -t gaming-player-activity -s --serializer avro-generic
+----
+
+Verify that 10 entries are stored in MinIO into _gaming-player-activity-bucket_ bucket, connecting to MiniIO web console, http://localhost:9000 (admin/minioadmin):
+
+image::images/minio2.png[gaming-player-activity-bucket]
+
+Teardown:
+
+[source,bash]
+----
+scripts/tear-down-connect-sink-s3.sh
+----
+
 === SAP HANA Source Connector example
 
 Folder: link:kafka-connect-source-sap-hana/[kafka-connect-source-sap-hana]

images/minio2.png

@@ -0,0 +1,22 @@
+{
+  "name": "sink-parquet-s3",
+  "config":
+  {
+    "topics": "gaming-player-activity",
+    "tasks.max": "1",
+    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
+    "store.url": "http://minio:9000",
+    "s3.region": "us-west-2",
+    "s3.bucket.name": "gaming-player-activity-bucket",
+    "s3.part.size": "5242880",
+    "flush.size": "100",
+    "storage.class": "io.confluent.connect.s3.storage.S3Storage",
+    "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
+    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
+    "parquet.codec": "snappy",
+    "schema.registry.url": "http://schema-registry:8081",
+    "value.converter": "io.confluent.connect.avro.AvroConverter",
+    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
+    "value.converter.schema.registry.url": "http://schema-registry:8081"
+  }
+}

kafka-connect-sink-s3/config/s3_parquet_sink.json

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+sh scripts/tear-down-connect-sink-s3.sh
+
+echo "Starting Kafka cluster..."
+docker-compose -f kafka-connect-sink-s3/docker-compose.yml --env-file .env up -d
+
+echo "Wait 20 seconds..."
+
+sleep 20
+
+echo "Create gaming-player-activity..."
+docker exec -it broker kafka-topics --bootstrap-server broker:9092 --create --topic gaming-player-activity --replication-factor 1 --partitions 1
+
+echo "Installing s3 parquet format sink..."
+curl -X POST -H Accept:application/json -H Content-Type:application/json http://localhost:8083/connectors/ -d @kafka-connect-sink-s3/config/s3_parquet_sink.json
+
+echo "Wait 3 seconds..."
+
+sleep 3
+
+echo "connectors status..."
+curl -v http://localhost:8083/connectors?expand=status