Skip to content

[SPARK-12177][Streaming][Kafka] Update KafkaDStreams to new Kafka 0.10 Consumer API #11863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
koeninger wants to merge 42 commits into from
Closed

[SPARK-12177][Streaming][Kafka] Update KafkaDStreams to new Kafka 0.10 Consumer API #11863

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
aaef0fc
[SPARK-12177][Streaming][Kafka] separate project for starting work on…
koeninger Mar 3, 2016
d4e504a
[SPARK-12177][Streaming][Kafka] WIP modify KafkaRDD to use cached new…
koeninger Mar 4, 2016
37cca1e
[SPARK-12177][Streaming][Kafka] WIP modify DirectKafkaInputDStream to…
koeninger Mar 7, 2016
1c0699d
Merge branch 'master' into kafka-0.9
koeninger Mar 7, 2016
f37717c
[SPARK-12177][Streaming][Kafka] WIP merge backpressure fixes from master
koeninger Mar 7, 2016
e01daea
[SPARK-12177][Streaming][Kafka] WIP remove KafkaUtils until interface…
koeninger Mar 8, 2016
20b59e1
[SPARK-12177][Streaming][Kafka] fix handling of kafkaParams
koeninger Mar 8, 2016
b55a3d4
[SPARK-12177][Streaming][Kafka] better handling of dynamic topic subs…
koeninger Mar 8, 2016
d2949f8
[SPARK-12177][Streaming][Kafka] WIP cached consumers basically workin…
koeninger Mar 9, 2016
47209c0
[SPARK-12177][Streaming][Kafka] WIP remove unused class
koeninger Mar 9, 2016
dea8ec0
[SPARK-12177][Streaming][Kafka] WIP add java convenience constructors…
koeninger Mar 10, 2016
43c27e9
[SPARK-12177][Streaming][Kafka] WIP allow user configuration of consu…
koeninger Mar 11, 2016
c04bc68
[SPARK-12177][Streaming][Kafka] WIP allow manual commit of offsets to…
koeninger Mar 15, 2016
546246e
[SPARK-12177][Streaming][Kafka] WIP pause topicpartitions to prevent …
koeninger Mar 15, 2016
8f2da8c
[SPARK-12177][Streaming][Kafka] scalastyle cleanup
koeninger Mar 21, 2016
477055c
Merge branch 'master' of https://github.com/apache/spark into kafka-0.9
koeninger Mar 21, 2016
ba41956
[SPARK-12177][Streaming][Kafka] scalastyle test cleanup
koeninger Mar 21, 2016
e559183
[SPARK-12177][Streaming][Kafka] workaround performance issues, error …
koeninger Apr 7, 2016
47d3a9f
[SPARK-12177][Streaming][Kafka] keep executor consumer group distinct…
koeninger Apr 8, 2016
f77288b
Merge branch 'master' into kafka-0.9
koeninger Apr 8, 2016
ec095fb
[SPARK-12177][Streaming][Kafka] code review changes
koeninger Apr 18, 2016
544bf88
[SPARK-12177][Streaming][Kafka] prevent NPE
koeninger May 6, 2016
44d3434
[SPARK-12177][Streaming][Kafka] update to kafka 0.10 RC3
koeninger May 6, 2016
a26160f
Merge branch 'master' into kafka-0.9
koeninger May 27, 2016
7cf5495
[SPARK-12177][Streaming][Kafka] exclude new kafka beta from mima
koeninger May 27, 2016
82067c0
[SPARK-12177][Streaming][Kafka] rename kafka-beta to kafka-0-10
koeninger Jun 14, 2016
c715872
[SPARK-12177][Streaming][Kafka] make test less flaky
koeninger Jun 15, 2016
ed6df9d
[SPARK-12177][Streaming][Kafka] better test error messages
koeninger Jun 15, 2016
fbaf186
Merge branch 'master' into kafka-0.9
koeninger Jun 15, 2016
3a95969
[SPARK-12177][Streaming][Kafka] more tweaks to flaky test
koeninger Jun 15, 2016
9dd7bce
[SPARK-12177][Streaming][Kafka] doc cleanup, add experimental annotation
koeninger Jun 15, 2016
b404304
[SPARK-12177][Streaming][Kafka] when recovering from checkpoint, disa…
koeninger Jun 15, 2016
0f15bd1
[SPARK-12177][Streaming][Kafka] code + doc cleanup
koeninger Jun 26, 2016
0610e9b
[SPARK-12177][Streaming][Kafka] hide DirectKafkaInputDStream and Kafk…
koeninger Jun 26, 2016
498585a
[SPARK-12177][Streaming][Kafka] make strategy for preferred location …
koeninger Jun 26, 2016
83c9c5b
[SPARK-12177][Streaming][Kafka] OOP style interface for configuring c…
koeninger Jun 26, 2016
db95290
[SPARK-12177][Streaming][Kafka] review feedback
koeninger Jun 28, 2016
b1eec57
[SPARK-12177][Streaming][Kafka] review feedback
koeninger Jun 29, 2016
31502d9
[SPARK-12177][Streaming][Kafka] review feedback
koeninger Jun 29, 2016
f863369
[SPARK-12177][Streaming][Kafka] review feedback
koeninger Jun 29, 2016
4caca18
[SPARK-12177][Streaming][Kafka] fix docstring
koeninger Jun 30, 2016
cffb0e0
[SPARK-12177][Streaming][Kafka] move package name to include kafka 0.…
koeninger Jun 30, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
176 changes: 176 additions & 0 deletions external/kafka-0-10-assembly/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10-assembly_2.11</artifactId>
<packaging>jar</packaging>
<name>Spark Integration for Kafka 0.10 Assembly</name>
<url>http://spark.apache.org/</url>

<properties>
<sbt.project.name>streaming-kafka-0-10-assembly</sbt.project.name>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<!--
Demote already included in the Spark assembly.
-->
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>net.jpountz.lz4</groupId>
<artifactId>lz4</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
<classifier>${avro.mapred.classifier}</classifier>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>net.java.dev.jets3t</groupId>
<artifactId>jets3t</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>

<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
<resource>log4j.properties</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheLicenseResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.ApacheNoticeResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

98 changes: 98 additions & 0 deletions external/kafka-0-10/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
<artifactId>spark-parent_2.11</artifactId>
<version>2.0.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<properties>
<sbt.project.name>streaming-kafka-0-10</sbt.project.name>
</properties>
<packaging>jar</packaging>
<name>Spark Integration for Kafka 0.10</name>
<url>http://spark.apache.org/</url>

<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_${scala.binary.version}</artifactId>
<version>0.10.0.0</version>
<exclusions>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>net.sf.jopt-simple</groupId>
<artifactId>jopt-simple</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>net.sf.jopt-simple</groupId>
<artifactId>jopt-simple</artifactId>
<version>3.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.binary.version}</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
</dependency>
</dependencies>
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
</build>
</project>
Loading