KAFKA-6670: Implement a Scala wrapper library for Kafka Streams by debasishg · Pull Request #4756 · apache/kafka

debasishg · 2018-03-22T13:19:09Z

This PR implements a Scala wrapper library for Kafka Streams. The library is implemented as a project under streams, namely :streams:streams-scala. The PR contains the following:

the library implementation of the wrapper abstractions
the test suite
the changes in build.gradle to build the library jar

The library has been tested running the tests as follows:

$ ./gradlew -Dtest.single=StreamToTableJoinScalaIntegrationTestImplicitSerdes streams:streams-scala:test
$ ./gradlew -Dtest.single=StreamToTableJoinScalaIntegrationTestImplicitSerdesWithAvro streams:streams-scala:test
$ ./gradlew -Dtest.single=WordCountTest streams:streams-scala:test

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

mjsax · 2018-03-22T18:22:51Z

\cc @bbejeck @vvcephei

guozhangwang · 2018-03-23T05:16:18Z

retest this please

ijuma · 2018-03-23T05:14:32Z

We don't use logback for anything else. I'd suggest keeping it consistent with the project.

ijuma · 2018-03-23T05:15:57Z

scalaLogging is already defined in this file.

ijuma · 2018-03-23T05:16:15Z

This is already defined in this file.

guozhangwang

Thanks for the PR. Could you rebase?

Also a meta question about the avro-dependent test StreamToTableJoinScalaIntegrationTestImplicitSerdesWithAvro.scala, do people think it's worth adding Avro dependency for demonstrating this test?

guozhangwang · 2018-03-23T16:56:25Z

Do we need to import these two dependencies? Could we use Kafka's own EmbeddedKafkaCluster?

This library scalatestEmbeddedKafka has a nice integration with scalatest. Hence it makes writing tests easier and we don't have to bother starting / managing the embedded kafka instance. The test code becomes very concise.

Generally speaking AK repo tend to avoid dependencies unless it is necessary. I'm wondering if we can improve on Kafka's own EmbeddedKafkaCluster to have the same functionalities as the net.manub:scalatest-embedded-kafka-streams.

We can definitely use Kafka's own EmbeddedKafkaCluster to integrate with ScalaTest. In net.manub:scalatest-embedded-kafka-streams, the main value add is integration with ScalaTest and hence you don't have to explicitly start / stop server as part of the test.

Also with Kafka Streams it has very nice constructs like the one we use here .. https://github.com/lightbend/kafka/blob/scala-streams/streams/streams-scala/src/test/scala/org/apache/kafka/streams/scala/StreamToTableJoinScalaIntegrationTestImplicitSerdes.scala#L77-L99 .. Note you just have to define the transformations and do the publish and consume as part of a closure. No need to start / stop the topology. Hence the test code becomes very concise.

Of course it depends on the opinion of the committee but I think this would be a great addition to the dependency.

Here's a suggestion ..

We use net.manub:scalatest-embedded-kafka-streams for now. After all it's a test dependency. And work on a separate PR to make the integration between EmbeddedKafkaCluster and Scalatest better and in line with the functionalities offered by the library.

WDYAT ?

I'm a fan of net.manub:scalatest-embedded-kafka-streams, but I understand the concern about bringing in more deps. I only mentioned it on the dev-kafka list because I didn't think there was much value in improving the embedded Kafka implementation in kafka-stream-scala and making it a public interface because scalatest-embedded-kafka already existed. I wasn't aware of EmbeddedKafkaCluster.

If scalatest-embedded-kafka were brought into the project then there will be drift between the version of Kafka broker in code and whatever this test lib references.

@debasishg I like your suggestion:

work on a separate PR to make the integration between EmbeddedKafkaCluster and Scalatest better and in line with the functionalities offered by the library.

Perhaps we can do this now for this PR, but keep it simple. I could work on it if you're busy.

I have used net.manub:scalatest-embedded-kafka-streams before, and it is very nice.

But I also worry about adding dependencies to the core project, even test deps. If our tests become a bit uglier, or if we have some test-util class that duplicates some of the functionality you're using, I would consider that to be a worthy tradeoff for dropping the dependency.

I would absolutely support planning to come back in a follow-up PR to build out support for testing scala code and then terraforming these tests to use the new support. Or even delaying this PR until a test-support one is available.

@seglo @vvcephei @debasishg thanks for your thoughts. We do have plans to publish testing-util artifacts inside AK in the future. And in fact we have been doing so for kafka-streams module as a first step and going to do that for kafka-core and kafka-clients soon. In kafka-clients testing utils we are going to include some improved version of embeddedkafkacluster for users to easily write their integration tests that involve interacting with a mock a kafka cluster.

So I'd suggest we stay with the uglier implementation with the existing embedded kafka cluster and not bring in the dependency.

Thanks for your thoughts @guozhangwang .. we will remove the dependency on net.manub:scalatest-embedded-kafka-streams and use EmbeddedKafkaCluster instead.

@guozhangwang - Removed all dependencies on net.manub:scalatest-embedded-kafka and net.manub:scalatest-embedded-kafka-streams. Now using EmbeddedKafkaCluster instead for tests. Also removed the test that used avro - hence dependency on avros eliminated.

Thanks @debasishg !

guozhangwang · 2018-03-23T16:57:34Z

Does it worth to include this dependency at test runtime? cc @ewencp @ijuma .

It's only to demonstrate custom Serdes. We picked up Avro since (AFAIR) @guozhangwang suggested this example in one of the earlier PR discussions. This example goes to show that custom serdes can be handled as seamlessly as primitive ones.

Cool. In that case, maybe we should also add 'streams:streams-scala:examples' and put it there?

I'm a bit on the fence for introducing Avro as "the one" serde in our demonstration examples rather than keeping Kafka and Avro separate, since there are many protobufs / etc fans in the community.

How about adding avro examples in eco-system repos, e.g. in Lightbend / Confluent / etc's own examples repo They can add their own example? cc @ijuma @gwenshap

+1 Sounds reasonable to me.

@mjsax - Cool .. then we can remove the test StreamToTableJoinScalaIntegrationTestImplicitSerdesWithAvro and the dependencies from build.gradle .. ok ?

Sounds good.

guozhangwang · 2018-03-23T16:59:42Z

Could we add the default for Short and ByteBuffer as well?

guozhangwang · 2018-03-23T17:02:47Z

I'm wondering if we could provide default serdes for windowed key as well? See o.a.k.streams.kstream.WindowedSerdes for Java code.

guozhangwang · 2018-03-23T17:05:08Z

nit: how about rename to FlatValueMapperFromFunction for better understanding? Ditto below.

guozhangwang · 2018-03-23T17:53:26Z

Is it a syntax sugar as branch? I'd prefer to keep Java and Scala interfaces consistent, so that if we think it is worthwhile we'd better add it in Java APIs as well, otherwise we should remove it from Scala APIs.

WDYT? @mjsax @bbejeck @vvcephei

I agree that both APIs should be consistent. Does a split add much value compare to branch? Btw, this might be related to https://issues.apache.org/jira/browse/KAFKA-5488

I am also open to add a split() if we think it's useful.

For the time being, we can remove split from the Scala API and rethink if / when it's implemented as a Java API

Sounds good.

ok, will remove split from KStream for now.

guozhangwang · 2018-03-23T17:57:29Z

This is a deprecated API in java, we should replace it with Materialized.

We can add the following:

def count(store: String, materialized: Materialized[K, Long, WindowStore[Bytes, Array[Byte]]]): KTable[Windowed[K], Long] = { //..

But this one also may be useful when the user just needs to pass in the keySerde. She need not construct any Materialized which is abstracted within the implementation of the API.

Suggestions ?

I'd vote for keeping java / scala API consistent, and we are going to remove deprecated APIs in future releases anyway.

In current API we'd only have one additional overload:

def count(materialized: Materialized[K, Long, WindowStore[Bytes, Array[Byte]]]): KTable[Windowed[K], Long] = { //..

I think for users who do not want to specify the store name at all, they can rely on

static Materialized<K, V, S> with(final Serde<K> keySerde, final Serde<V> valueSerde)

to still hide the materialized parameter with implicit conversion.

For users who do want to specify the store name, but want to rely on type conversion, we could call withKeySerde and withValueSerde internally in the implicit conversion so that user only need to give Materialized.as(storeName)

Does that work?

+1 .. will remove this overload for count.

guozhangwang · 2018-03-23T18:01:58Z

nit: default to INFO?

guozhangwang · 2018-03-23T18:03:20Z

The file logs/kafka-server.log seems not appropriate as it is not for kafka broker logs right?

Will change the name to kafka-streams-scala.log

guozhangwang · 2018-03-23T18:05:54Z

This is a meta comment: I'd suggest we consider adding logback generally for Kafka, instead of sneaking in for Streams Scala wrapper. We can still use log4j for now. See https://issues.apache.org/jira/browse/KAFKA-2717. cc @ijuma

Yes, let's stick to log4j in this PR.

should we remove logback.xml ?

Yup, please remove that file as well.

debasishg · 2018-03-25T14:32:28Z

Added commit with changes for code review feedback. Waiting for more feedback and some of the still unresolved questions. Should I rebase now or wait till all questions resolved ?

mjsax

I am not a Scala person -- so take my review with a grain of salt...

Did not look at the tests yet.

mjsax · 2018-03-26T05:49:58Z

nit: those are actually sorted alphabetically -- can we clean this up? Thx.

mjsax · 2018-03-26T05:50:06Z

mjsax · 2018-03-26T05:54:10Z

Should we add those exclusions? I know that we put exclusions when introducing findbugs because it is not possible to introduce it and rewrite all the code -- but for new code, we should consider changing the code. I am not a Scale person though -- can you elaborate on this?

This seems to be a false positive. FindBugs is reporting that Serializer and Deserializer should be defined as a different type name than what it's inheriting. IIRC the consensus earlier is that we want type names the same as the base types they're wrapping (which includes traits and interfaces IMO). I've updated the FindBugs rule exclusion to be specific to the types generating the violation, rather than the entire ScalaSerde file.

mjsax · 2018-03-26T05:58:09Z

Should this be ByteArrayKeyValueStore? -- we don't use abbreviations in the Java code base.

mjsax · 2018-03-26T06:05:20Z

Why do we need an asInstanceOf here? (same below)

I think it's because we're presenting the Serde[java.lang.Long] as a Serde[scala.lang.Long], but casting the Serde won't automatically cast the parameters and returns of its methods. I'm surprised you don't get cast class exceptions trying to use the Java long serde as a Scala long serde. Unless I'm wrong about what this is for...

There is no such thing as a scala.Long at runtime, Scala changed to use the same classes as Java for boxing around the 2.8 timeframe if I remember correctly. Previously there was a RichLong, RichInt, etc.

In any case, this seems like a variance issue, but I didn't look into it.

In Scala there's an implicit conversion between scala.Long and java.lang.Long but not between Serde[scala.Long] and Serde[java.lang.Long] - hence the cast. In fact picked this trick from https://github.com/confluentinc/kafka-streams-examples/blob/4.0.0-post/src/test/scala/io/confluent/examples/streams/StreamToTableJoinScalaIntegrationTest.scala#L106

Huh! Well, that explains it. I have seen the conversions of the raw types, and also suffered from type errors that Serde[scala.Long] != Serde[java.lang.Long], so I just assumed that scala.Long was a different class than java.Long. Thanks for the explanation, @ijuma .

mjsax · 2018-03-26T18:09:25Z

Nit: I am not a fan of adding links in JavaDocs, because links might break; better reference to the corresponding class as JavaDoc cross reference?

Also: I am wondering if we should remove this method from the wrapper in the first place? IMHO, it's not a good idea to add deprecated API in new code?

+1 on not including this method in the wrapper. The code that would use this library is not written yet, so it's better if deprecated methods are simply not available.

Here, the deprecation is on punctuate, which is part of the contract of Transformer. How do we remove this ? We can only remove this when punctuate is removed from Transformer ? Or am I missing something ?

Oh, right, I thought this was one of your scala replacement classes.

What I have been doing for cases like this is throwing an UnsupportedOperationException in the body. It's not as good as not having the method, but it def. ensures it can't be used. And you don't have to maintain the code that's in the body.

I understand that you cannot change the Java Transformer interface and must implement the deprecated method when calling new Transformer -- what I was wondering is about scala.Transformer interface -- should we add one and remove punctuate from it?

FWIW, I think adding a new scala interface just to remove a method that we plan to remove from the java interface is not necessary. Better just to implement it and move on.

Also, it would be a bit trickier to swap in a scala replacement for Transformer than for the top-level DSL classes, since implementations of the java Transformer won't implement the scala Transfomer, so you wouldn't be able to plug them in via the scala DSL wrapper. But there's otherwise no reason this shouldn't work.

Ack. Was just an idea. I don't really speak Scala (yet) -- this is an exercise to learn something about it...

If we need to have it, I vote to throw an exception to forces users to use the new API.

mjsax · 2018-03-26T18:11:39Z

A ValueTransformer also have init(), punctuate() and close() method. Why is this code much simpler than the wrapper for transform() above?

We don't need to provide implementation of ValueTransformer here since the passed in () => ValueTransformer[V, VR] gets converted to ValueTransformerSupplier[V, VR] in the implementation through SAM type conversion. We could not do that for transform as the SAM type conversion will not be handled automatically in that case. Please have a look here for SAM type conversions in Scala.

mjsax · 2018-03-26T18:14:29Z

As above. What about init(), punctuate(), and close() ?

Same logic as ValueTransformer above.

mjsax · 2018-03-26T18:23:14Z

I agree that both APIs should be consistent. Does a split add much value compare to branch? Btw, this might be related to https://issues.apache.org/jira/browse/KAFKA-5488

I am also open to add a split() if we think it's useful.

mjsax · 2018-03-26T18:26:04Z

Why do we only allow to specify a keySerde but not replace the store with a different one?

Scala noob question: would it be possible to have a single count / reduce etc instead of overloads and use Option and implicits to infer optional arguments?

@mjsax -

I think I may be missing something here. We are allowing the user to pass in the store and keySerde. And we create the Materialized out of the 2 with Long as the value serde.

However we were allowing the user to pass in the keySerde optionally and in case the user does not supply one we assumed it will be taken from the config. This is actually a remnant from the earlier thoughts where we thought passing serdes through config may be a good idea. However in the current context, we should not make keySerde optional. Here's the suggestion for the changed API ..

def count(store: String, keySerde: Serde[K]): KTable[K, Long] = { val materialized = Materialized.as[K, java.lang.Long, ByteArrayKeyValueStore](store).withKeySerde(keySerde) val c: KTable[K, java.lang.Long] = inner.count(materialized) c.mapValues[Long](Long2long _) }

An alternative option could have been to allow the user to pass in the Materialized instance itself (like we do in the reduce function). The problem with that alternative is that the Java API expects java.lang.Long as the value serde, while the Scala API needs to take a scala.Long. And there is no way we can convert a Materialized.as[K, scala.Long, ByteArrayKeyValueStore] to Materialized.as[K, java.lang.Long, ByteArrayKeyValueStore] without a cast. Hence the divergence in the API signature between count and reduce.

Please share if u have any other thoughts.

That makes sense.

My 2 cents:
We're presenting the scala KGroupedStream basically as the java one, but not implementing the interface so that we can smooth over a couple of specific gaps. I think this is a good call, but it's also a huge risk for cognitive dissonance and developer confusion, since they will read the java version of the docs and try to use that knowledge in scala. Therefore, it's important to be super disciplined about making sure the methods available are as close to the java interface as possible.

Clearly, moving serdes, etc., to implicit params is the kind of thing we do want to do. But I think that presenting count(String,Serde[K]) instead of count(Materialized[K, Long, KeyValueStore[Bytes, Array[Byte]]] is too far off.

I do agree that the method should take a scala type. Apparently, it's perfectly fine to cast Serde[scala.Long] to Serde[java.Long]. Does that same logic apply here? Alternatively, we can actually convert the Materialized[java] to a Materialized[scala].

@vvcephei .. Sure we can do the following instead ..

def count(materialized: Materialized[K, Long, ByteArrayKeyValueStore]): KTable[K, Long] = { val c: KTable[K, java.lang.Long] = inner.count(materialized.asInstanceOf[Materialized[K, java.lang.Long, ByteArrayKeyValueStore]]) c.mapValues[Long](Long2long _) }

WDYT ?

Ok, I'm already a little nervous about the cast in one direction, so this feels super gross, but would this work?

def count(materialized: Materialized[K, Long, ByteArrayKeyValueStore]): KTable[K, Long] = { inner.count(materialized.asInstanceOf[Materialized[K, java.lang.Long, ByteArrayKeyValueStore]]).asInstanceOf[KTable[K, scala.Long]] }

Please understand I'm wincing as I type this.

Well, we allowing to pass in a store "name" (String) but not a store. Note, that Materialized allows to replace default RocksDB with an in-memory story, disable change-capture-logging or even use a custom store implementation.

How about the API that I suggested above ? It takes Materialized much like the Java API though we need a cast.

@vvcephei - in my implementation we have 1 cast and the other map for the long conversion in KTable.

@debasishg Sorry, I should have acked your implementation. I was actually proposing an evolution of it.

It just seems a bit unfortunate to have to add a real function invocation to the topology in order to do the cast back to scala.Long. The version I proposed just does a cast back out without adding anything new to the topology. Does that make sense?

At the risk of sounding like an idiot, if it's fine to do the cast on the way in, then it should be fine again on the way out, right?

@vvcephei - cast is a runtime operation and my philosophy is to minimize its use. And scala.Predef indeed uses Long2long to do such conversions. Hence I would like to prefer using proper functions when available instead of the cast.

vvcephei

hey @debasishg , thanks so much for this PR. I've been super frustrated in the past with how awkward it is to use KStreams in Scala vs. how awkward it should be.

My favorite features here are:

serdes for Scala types
DSL methods taking implicits for things that are easy to infer (like the serde implementation).

My personal preference would be to avoid renaming imports and just using fully-qualified package names inline. It makes our code longer, but easier to understand. For a "base-level" library, I'd prefer to have less pretty but more comprehensible code in general.

I'm also wondering if we really want to use the same class names as the Java DSL, this seems like it would be confusing to use in an IDE with code-completion. What do you think about using ScalaKStream, etc. instead? This would also get around the desire to rename imports.

Thanks again,
-John

vvcephei · 2018-03-26T19:25:44Z

I'd like to confirm that this option is actually safe. Is this a best practice at this point for targeting 2.11? Also, how can we know we're not dragging in other (potentially unwanted) experimental compiler features with this?

I agree it looks concerning, I'll need to check what other potential features this brings in, unfortunately there's no way to be more specific about just enabling SAM type conversion AFAIK.

We could remove this flag, but we would need to desugar all the places where conversions occur.

I'm also wondering if we really want to use the same class names as the Java DSL, this seems like it would be confusing to use in an IDE with code-completion. What do you think about using ScalaKStream, etc. instead? This would also get around the desire to rename imports.

Hi @vvcephei - regarding the above, we had a discussion on the 2 approaches on this thread only and the universal suggestion was to use the same name across Scala and Java APIs. In fact in the initial version that we posted, we had different names (KStream / KStreamS). The reasoning of using the same names is that the renaming of imports in the user code needs to be done only very occasionally when we mix usage of Scala and Java APIs.

@vvcephei There's no list of what's brought in with the experimental flag, unless you grep the compiler code. The purpose of the SAM type conversion feature in 2.11 was only to get early feedback and only properly finished in 2.12. As its name suggests it's not meant to be used in production code. Since Kafka is still targeting 2.11 it makes sense to not include this flag to build a releasable artifact. I'll remove the flag and desugar the conversions.

@debasishg Thanks for the explanation, I missed that discussion. Sorry to bring it back up!

@seglo Thanks! Am I correct in thinking that only affects our code, and not our users' code? I.e., if they are using 2.12, they can pass in lambdas as arguments, right?

@vvcephei Yes, this change won't affect end users at all. If Kafka drops Scala 2.11 support then we can bring back the SAM conversions as they're available without any ominous compiler flags.

vvcephei · 2018-03-26T19:27:21Z

Cool. In that case, maybe we should also add 'streams:streams-scala:examples' and put it there?

vvcephei · 2018-03-26T19:28:57Z

nit: this is a bit cumbersome. Can we do project(':streams:scala-wrapper') and archive: 'kafka-streams-scala-wrapper' or some such instead?

vvcephei · 2018-03-26T19:39:58Z

I have used net.manub:scalatest-embedded-kafka-streams before, and it is very nice.

But I also worry about adding dependencies to the core project, even test deps. If our tests become a bit uglier, or if we have some test-util class that duplicates some of the functionality you're using, I would consider that to be a worthy tradeoff for dropping the dependency.

I would absolutely support planning to come back in a follow-up PR to build out support for testing scala code and then terraforming these tests to use the new support. Or even delaying this PR until a test-support one is available.

vvcephei · 2018-03-26T19:46:08Z

I think it's because we're presenting the Serde[java.lang.Long] as a Serde[scala.lang.Long], but casting the Serde won't automatically cast the parameters and returns of its methods. I'm surprised you don't get cast class exceptions trying to use the Java long serde as a Scala long serde. Unless I'm wrong about what this is for...

vvcephei · 2018-03-26T19:58:11Z

+1 on not including this method in the wrapper. The code that would use this library is not written yet, so it's better if deprecated methods are simply not available.

vvcephei · 2018-03-26T20:00:27Z

mjsax · 2018-03-27T20:39:26Z

This and all other classes are public API. Thus, we should improve the JavaDocs for those classes and also add JavaDocs for all methods. I guess we can c&p from existing Java classes.

@mjsax - Looking for suggestions. Should we copy/paste Javadoc from Java classes or use @see annotation ? The problem with copy is maintenance - when one changes someone needs to be careful enough to change the other.

Good question. Not sure. I agree that maintaining JavaDocs twice is a hassle and error prone. But might be annoying for user if it's only linked on the other hand. Would be good to hear what others thing. \cc @guozhangwang @deanwampler @ijuma @vvcephei @bbejeck @seglo

Adding JavaDocs to all the public API methods in the PR is the same amount of work any way we do it. From an end user perspective I agree it would be nice to have the same (or slightly tweaked, as necessary) JavaDocs for all public API methods, plus a @see or @link tag to the corresponding Java API method. It will be a small burden to maintain it going forward so I defer to the AK committers to make the call on the format.

I'd vote for using @see and @link to avoid maintaining two copies, because we have some public classes following this pattern in the repo (like https://github.com/apache/kafka/blob/23d01c805bef7504abfa83ecac7e384d121a583a/clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java), and from the past I find most people would not remember or bother to update two places than one.

Additionally, this PR should include updates to the web docs in docs/streams/... and in "notable changes" in docs/upgrade.html ?

Meta Comment - Besides updating documentation and Javadoc, is there any outstanding item in this PR that needs to be addressed ?

I think the web docs (in docs/streams) needs to be updated as well, especially in the upgrade-guide.html page, as well as the streams-api page.

For streams-api page above, I meant https://github.com/apache/kafka/blob/23d01c805bef7504abfa83ecac7e384d121a583a/docs/api.html

Some thoughts:

I think the Consumer and KafkaConsumer pattern is bad. The documentation should have been on Consumer instead. The AdminClient follows the latter pattern.

I think it's a poor user experience to ask users to read the docs in the Java class. My recommendation would be to at least include a short description in the Scala docs along with a link to the relevant Java documentation. The short description is less likely to change and it helps users make progress without having to jump to the Java code all the time. However, for more detailed information (which is more likely to change), they can check the Java code.

I tend to agree with @ijuma comment about KafkaConsumer/Consumer pattern -- it's quite annoying to not get the JavaDocs directly. Thus, even if it's a burden it seems to be worth to maintain two copies.

@mjsax - I have started writing the Scaladocs in the commit 631ab9a .. Pls review if it's following the correct pattern

mjsax · 2018-03-27T20:42:00Z

Meta comment: For JavaDocs, so we need to set up some pipeline to get JavaDocs published? Or will this happen automatically?

Additionally, this PR should include updates to the web docs in docs/streams/... and in "notable changes" in docs/upgrade.html ?

debasishg · 2018-03-29T09:23:03Z

Meta Comment - Besides updating documentation and Javadoc, is there any outstanding item in this PR that needs to be addressed ?

seglo · 2018-03-31T16:43:04Z

For user documentation my plan is to update the following:

Streams homepage - Update the Scala example to use Scala DSL
Developer Guide
- Writing a Streams Application - Add a reference to the streams-scala artifact and add Scala examples
- Configuring a Streams Application - Note that default key/value serdes not required for Scala DSL.
- Streams DSL - Add a new section for the Scala DSL. Packages, examples, implicit serdes and other differences from Java DSL. The bulk of the content would go here.
- Data Types & Serialization - Reference implicit serdes from Streams DSL page which includes implicit serdes explanation and usage, or vice versa.

I could use some advice on how to actually go about updating the docs efficiently. The Contributing Website Documentation Changes doesn't go into detail about a workflow I can use while developing docs (make a change, preview, etc). I noticed the code samples for the streams docs are formatted as HTML, but I couldn't find the source for these examples in the project, are they generated somehow?

EDIT: I also added to the https://kafka.apache.org/documentation/#streamsapi section, to explain how to include the dependency in maven build file, e.g. to use the scala wrapper.

EDIT 2: I also updated the section 2.3 (streams api) to reference the new client library.

EDIT 3: I also updated the streams upgrade guide.

mjsax · 2018-04-01T05:29:57Z

@seglo Sounds great.

You are right that it is HTML -- just edit it directly. I have no concrete workflow suggestion. What code examples do you mean? It should all be in the HTML files. There is nothing in streams docs thats generated.

seglo · 2018-04-01T13:16:09Z

@mjsax I'm referring to the code snippets found in pages like the DSL API. They look like they've been generated with syntax highlightling.

                        <div class="highlight-java"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">org.apache.kafka.common.serialization.Serdes</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.streams.StreamsBuilder</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">org.apache.kafka.streams.kstream.GlobalKTable</span><span class="o">;</span>

<span class="n">StreamsBuilder</span> <span class="n">builder</span> <span class="o">=</span> <span class="k">new</span> <span class="n">StreamsBuilder</span><span class="o">();</span>

<span class="n">GlobalKTable</span><span class="o">&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">&gt;</span> <span class="n">wordCounts</span> <span class="o">=</span> <span class="n">builder</span><span class="o">.</span><span class="na">globalTable</span><span class="o">(</span>
    <span class="s">&quot;word-counts-input-topic&quot;</span><span class="o">,</span>
    <span class="n">Materialized</span><span class="o">.&lt;</span><span class="n">String</span><span class="o">,</span> <span class="n">Long</span><span class="o">,</span> <span class="n">KeyValueStore</span><span class="o">&lt;</span><span class="n">Bytes</span><span class="o">,</span> <span class="kt">byte</span><span class="o">[]&gt;&gt;</span><span class="n">as</span><span class="o">(</span>
      <span class="s">&quot;word-counts-global-store&quot;</span> <span class="cm">/* table/store name */</span><span class="o">)</span>
      <span class="o">.</span><span class="na">withKeySerde</span><span class="o">(</span><span class="n">Serdes</span><span class="o">.</span><span class="na">String</span><span class="o">())</span> <span class="cm">/* key serde */</span>
      <span class="o">.</span><span class="na">withValueSerde</span><span class="o">(</span><span class="n">Serdes</span><span class="o">.</span><span class="na">Long</span><span class="o">())</span> <span class="cm">/* value serde */</span>
    <span class="o">);</span>
</pre></div>
                        </div>

https://github.com/apache/kafka/blob/trunk/docs/streams/developer-guide/dsl-api.html#L189

guozhangwang · 2018-04-02T00:41:39Z

@seglo For the doc changes, as I mentioned before we should also add a new section in the https://kafka.apache.org/documentation/#streamsapi section, to explain how to include the dependency in maven build file, e.g. to use the scala wrapper.

mjsax · 2018-04-02T06:14:56Z

@seglo I see -- I guess, somebody used some special HTML editor... \cc @joel-hamill might be able to shed some light... Most people just use plain text editors. You can just add plain HTML without any syntax highlighting. Cf https://github.com/apache/kafka/blob/trunk/docs/streams/developer-guide/testing.html#L76-L93

(I was just not sure what you mean by "generated" because the configs HTML is generated from the source code directly -- cf. https://kafka.apache.org/documentation/#configuration and https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L1095)

seglo · 2018-04-03T19:14:01Z

I've implemented the user docs. However I don't know how to view the documentation in my browser. None of the HTML files in docs subdir render when I open them locally with Chrome (even before I made any edits). There are likely some formatting mistakes since they're difficult to spot in the markup alone. Can someone familiar with editing these docs please describe a workflow I can use to view rendered documentation locally?

In the meantime I encourage people to review the content.

/cc @mjsax @guozhangwang @joel-hamill

guozhangwang · 2018-04-03T21:54:42Z

@seglo you can read this wiki page for render the pages locally: https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+Documentation+Changes

To do that you'd need to first copy-paste the docs to kafka-site repo, and start the apache server locally in order to see the difference.

guozhangwang

@seglo please update the streams/upgrade-guide.html, under the

<h3><a id="streams_api_changes_120" href="#streams_api_changes_120">Streams API changes in 1.2.0</a></h3>

section, to talk about this addition to the public APIs.

guozhangwang · 2018-04-03T22:07:53Z

nit: period at the end of the sentence.

guozhangwang · 2018-04-03T22:08:35Z

Maybe mention again which artifact to include in order to import this package.

guozhangwang · 2018-04-03T22:11:09Z

We do not need indentation in the code block; the following code blocks are formatted correctly.

I copied the formatting from the streams main page which indented the WordCount examples: https://github.com/apache/kafka/blob/trunk/docs/streams/index.html#L155

I removed the initial indentation for this example on this page to make it consistent with the others.

guozhangwang · 2018-04-03T22:12:07Z

This is not introduced in this PR: duplicated provides.

I fixed the typos in this line, but I'm not sure what you mean by it not being introduced in this PR. This line is to indicate the presence of the Kafka Streams DSL for Scala library.

I meant the duplicated provides exist before this PR, so it is not a regression introduced from this PR.

seglo · 2018-04-04T17:34:10Z

@guozhangwang I was able to setup a local apache2 webserver (wow, I haven't done that in 10 years!) I fixed formatting issues, typos, added the blurb to the streams upgrade guide, and other misc. feedback you provided.

guozhangwang · 2018-04-04T20:16:22Z

@guozhangwang I was able to setup a local apache2 webserver (wow, I haven't done that in 10 years!) I fixed formatting issues, typos, added the blurb to the streams upgrade guide, and other misc. feedback you provided.

Thanks!

Could you also rebase your PR against trunk to resolve the conflicts as well?

seglo · 2018-04-05T01:44:33Z

@guozhangwang Done!

guozhangwang · 2018-04-05T18:37:56Z

retest this please

dguy

Thanks for the PR - do we need the integration tests? i.e., the scala api is just translating to the underlying java api, so we could write tests that ensure that the correct topology is built?

debasishg · 2018-04-06T14:39:48Z

@dguy - One of the reasons we wrote the tests was to demonstrate the idiomatic usage of the Scala APIs. If you think we should write additional tests to verify if the correct topology is built, what kind of tests can do this verification ? Is there any example test in the Java APIs that does this verification of building the correct topology ?

mjsax

Couple of nits.

mjsax · 2018-04-16T07:54:15Z

 	</pre>

+	<p>
+	When using Scala you may optionally include the <code>kafka-streams-scala</code> library.  Additional documentation on using the Kafka Streams DSL for Scala is available <a href="/{{version}}/documentation/streams/developer-guide/dsl-api.html#scala-dsl">here</a>.


"is available here" is bad phrasing.

here -> in the developer guide

mjsax · 2018-04-16T07:58:35Z

+              <li><code class="docutils literal"><span class="pre">org.apache.kafka.streams.scala.DefaultSerdes</span></code>: Module that brings into scope the implicit values of all primitive SerDes.</li>
+              <li><code class="docutils literal"><span class="pre">org.apache.kafka.streams.scala.ScalaSerde</span></code>: Base abstraction that can be used to implement custom SerDes in a type safe way.</li>
+            </ul>
+            <p>The library is cross-built with Scala 2.11 and 2.12.  To include it your maven <code>pom.xml</code> add the following:</p>


To include it your maven -- sounds weird

mjsax · 2018-04-16T08:00:05Z

+			&lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
+			&lt;artifactId&gt;kafka-streams-scala&lt;/artifactId&gt;
+			&lt;version&gt;{{fullDotVersion}}&lt;/version&gt;
+		&lt;/dependency&gt;


I am wondering if this is correct? Should the Scala version not be included here?

For different scala version compiled packages, their project name is actually the same. And here people only need to specify the version of the artifact itself, which will be the Kafka version.

Users can, indeed, build kafka-streams-scala with different scala versions other than the default one, but that is to be done before they include it in the dependency. For maven, it will always be whatever is uploaded to maven central.

I'm not very familiar with gradle, but it appears to not support cross building jars in the same manner as SBT. The build needs to be run for each Scala version you want a jar for, but the output won't encode the version into the filename. I think what we need to do is add a task to the gradle file, or some other build related packaging script, to pluck the generated kafka-streams-scala file, rename it to include the Scala version, and then publish it to maven central.

Ex) The built outputs this when specifying a 2.12 SCALA_VERSION (./gradlew -PscalaVersion=2.12 jar)

-rw-rw-r-- 1 seglo seglo 105423 Apr 18 14:17 kafka-streams-scala-1.2.0-SNAPSHOT.jar

When a release artifact is published we'll publish a file: kafka-streams-scala_2.12-1.2.0.jar with the Scala major version encoded into the artifact name. A maven user would reference the artifact with:

<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams-scala_2.12</artifactId> <version>1.1.0</version> </dependency>

Please let me know if I'm missing something here about the Kafka build system.

On a related note I found a gradle build plugin that handles cross building projects and referencing Scala dependencies in a SBT style here: https://github.com/ADTRAN/gradle-scala-multiversion-plugin

We already so the right thing for core jars. We just need to follow the same approach. And yes, the Scala version needs to be encoded in the artifact id. Not sure what @guozhangwang was trying to say, but doesn't seem correct to me.

@ijuma Ok, is the approach you refer to in the core project of the build.gradle? I'll take a closer look.

WRT the docs @mjsax is correct that we should update the maven dependency examples to include the Scala version.

What I was saying is that when we build the artifact we already chose which scala version to use compiling the jar and made the scala version as part of the artifact name, so users do not need to specify the scala version in declaring the dependency, but just:

<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.12</artifactId> <version>1.1.0</version> </dependency>

I'll correct this to include the Scala version (2.11) across all the maven pom.xml references.

@guozhangwang @seglo That's what I meant by my comment -- sorry for expressing myself unclear.

mjsax · 2018-04-16T08:00:56Z

+                &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
+                &lt;artifactId&gt;kafka-streams&lt;/artifactId&gt;
+                &lt;version&gt;{{fullDotVersion}}&lt;/version&gt;
+              &lt;/dependency&gt;


Don't we need the Scala version, here?

mjsax · 2018-04-16T08:01:54Z

+            </pre>
+            <p>If using SBT then you can add a reference to the library using the following:</p>
+            <pre class="brush: scala;">
+              libraryDependencies += "org.apache.kafka" %% "kafka-streams-scala" % "{{fullDotVersion}}"


Don't we need the Scala version here?

We don't need to specify the Scala version here. The %% operator in SBT will automatically determine the right artifact based on the running Scala version.

mjsax · 2018-04-16T08:36:53Z

+   * Transform the value of each input record into a new value (with possible new type) of the output record.
+   * A `ValueTransformer` (provided by the given `ValueTransformerSupplier`) is applied to each input
+   * record value and computes a new value for it.
+   *


mjsax · 2018-04-16T08:37:05Z

+   * Transform the value of each input record into a new value (with possible new type) of the output record.
+   * A `ValueTransformer` (provided by the given `ValueTransformerSupplier`) is applied to each input
+   * record value and computes a new value for it.
+   *


mjsax · 2018-04-16T08:37:38Z

+
+  /**
+   * Process all records in this stream, one record at a time, by applying a `Processor` (provided by the given
+   * `processorSupplier`).


mjsax · 2018-04-16T08:41:42Z

+      inner.outerJoin[VO, VR](otherStream.inner, joiner.asValueJoiner, windows, joined)
+
+  /**
+   * Merge this stream and the given stream into one larger stream.


Maybe explain, that there is not ordering guarantee for the merged result stream for records of different input streams? Relative order is only preserved for record of the same input stream?

mjsax · 2018-04-16T08:48:00Z

@@ -0,0 +1,35 @@
+## /*


Markup seems weird? Why do you have JavaDoc comment markup?

Would a single # not be sufficient?

mjsax

Couple of nits.

mjsax

Couple of nits.

mjsax

Couple of nits.

guozhangwang

Left some more comments.

guozhangwang · 2018-04-16T22:31:18Z

+			&lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
+			&lt;artifactId&gt;kafka-streams-scala&lt;/artifactId&gt;
+			&lt;version&gt;{{fullDotVersion}}&lt;/version&gt;
+		&lt;/dependency&gt;


For different scala version compiled packages, their project name is actually the same. And here people only need to specify the version of the artifact itself, which will be the Kafka version.

Users can, indeed, build kafka-streams-scala with different scala versions other than the default one, but that is to be done before they include it in the dependency. For maven, it will always be whatever is uploaded to maven central.

guozhangwang · 2018-04-16T22:57:47Z

-      </ol>
-</div>
-</div>
+      <div class="section" id="implementing-custom-serdes">


cc @joel-hamill @ewencp we are adding a few new sections in web docs regarding the streams scala API, which may be affecting #4536.

guozhangwang · 2018-04-16T23:00:09Z

+              <tr class="row-even"><td><code class="docutils literal"><span class="pre">org.apache.kafka</span></code></td>
+                  <td><code class="docutils literal"><span class="pre">kafka-streams-scala</span></code></td>
+                  <td><code class="docutils literal"><span class="pre">{{fullDotVersion}}</span></code></td>
+                  <td>(Optional) Scala DSL API library to write Scala Kafka Streams applications.</td>


I think the Scala version cannot be changed when specifying the kafka-streams-scala artifact, as it is encapsulated when that artifact is compiled already. @debasishg please correct me if I'm wrong.

guozhangwang · 2018-04-16T23:02:43Z

  apacheda: "1.0.0",
  apacheds: "2.0.0-M24",
  argparse4j: "0.7.0",
+  avro4sVersion: "1.8.3",


I think we do not need avro4sVersion any more? Same as line 86 here.

guozhangwang · 2018-04-16T23:02:56Z

  kafka_0110: "0.11.0.2",
  kafka_10: "1.0.1",
  lz4: "1.4.1",
+  mavenArtifact: "3.5.2",


This is not needed.

removed the duplicate entry ..

guozhangwang · 2018-04-16T23:06:10Z

@@ -0,0 +1,3 @@
+Kafka Streams Scala
+Copyright (C) 2018 Lightbend Inc. <https://www.lightbend.com>
+Copyright (C) 2017-2018 Alexis Seigneurin.


guozhangwang · 2018-04-16T23:11:08Z

@debasishg Ping on this comment again, could you elaborate if my concern is valid or not?

guozhangwang · 2018-04-16T23:12:59Z

Could we remove this line then?

guozhangwang · 2018-04-16T23:39:02Z

+
+import java.util.Properties
+
+import org.junit.Assert._


nit: move import org.junit.Assert._ after line 22, ditto below elsewhere.

guozhangwang · 2018-04-16T23:40:40Z

+
+  val pattern = Pattern.compile("\\W+", Pattern.UNICODE_CHARACTER_CLASS)
+
+  @Test def shouldBuildIdenticalTopologyInJavaNScala_1() = {


nit: replace the _1/2/3 suffix with some more meaningful name? E.g. simple, aggregate, join?

ijuma · 2018-04-18T13:29:10Z

@debasishg, for some reason GitHub doesn't let me reply inline. Your "more efficient version" is still allocating a tuple, right? To avoid it, mapper would have to return a KeyValue.

debasishg · 2018-04-18T13:37:43Z

@ijuma - The map function doesn't allocate the tuple. It takes a function of the form (K, V) => (KR, VR) and constructs the KeyValue from the tuple that mapper returns. Wanted to abstract the construction of KeyValue to avoid noise.

def map[KR, VR](mapper: (K, V) => (KR, VR)): KStream[KR, VR] = {
  val mapperJ: KeyValueMapper[K, V, KeyValue[KR, VR]] = ((k: K, v: V) => {
    val res = mapper(k, v)
    new KeyValue[KR, VR](res._1, res._2)
  }).asKeyValueMapper
  inner.map[KR, VR](mapperJ)
}

ijuma · 2018-04-18T13:46:42Z

My point is that there's still a throwaway tuple for each mapped item which is the issue @guozhangwang raised. The GC is generally good at dealing with such short lived objects, so not sure if it matters in this case, but it's clearly adding some overhead when compared to the Java implementation.

debasishg · 2018-04-18T14:13:16Z

Yeah .. the point is this tuple is generated within the map function through the call of the mapper. To avoid this, the way out will be to change the mapper to (K, V) => KeyValue[KR, VR]. Which can be done .. but makes the API contract a bit noisy. OTOH we can rely on the JIT and the ability of GC to deal with such short lived objects. Suggestions welcome :-)

guozhangwang · 2018-04-18T17:16:39Z

@debasish Maybe I was a bit paranoid on the GC pressure before, just raised it as a concern. Since you mentioned We did run some tests in bulk to check the diff in performance between the 2 versions. Couldn't find much of a difference though. I'm fine with keeping the API as elegant as of now and assume JIT doing the right thing.

guozhangwang

Thanks for the great effort @debasishg @seglo . I do not have any further comments, will leave to @mjsax if he has anything else.

debasishg · 2018-04-20T13:53:05Z

retest this please

ijuma · 2018-04-20T14:26:31Z

Out of curiosity, how long does it take to build (compile and run the tests) kafka-streams-scala.

debasishg · 2018-04-20T14:54:20Z

@ijuma

$ ./gradlew -Dtest.single=*Test* streams:streams-scala:test

> Configure project : 
Building project 'core' with Scala version 2.11.12
Building project 'streams-scala' with Scala version 2.11.12

> Task :streams:streams-scala:test 
org.apache.kafka.streams.scala.TopologyTest > shouldBuildIdenticalTopologyInJavaNScalaJoin PASSED

org.apache.kafka.streams.scala.TopologyTest > shouldBuildIdenticalTopologyInJavaNScalaSimple PASSED

org.apache.kafka.streams.scala.TopologyTest > shouldBuildIdenticalTopologyInJavaNScalaAggregate PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava PASSED

org.apache.kafka.streams.scala.WordCountMergeTest > testShouldCountWords PASSED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes > testShouldCountClicksPerRegionJava PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords PASSED

org.apache.kafka.streams.scala.StreamToTableJoinScalaIntegrationTestImplicitSerdes > testShouldCountClicksPerRegion PASSED

BUILD SUCCESSFUL in 37s

miguno · 2018-04-25T14:16:17Z

+<!-- Optionally include Kafka Streams DSL for Scala for Scala 2.11 -->
+<dependency>
+    <groupId>org.apache.kafka</groupId>
+    <artifactId>kafka-streams-scala_2.11</artifactId>


This doesn't look right to me. The latest trunk build of Kafka only generates the following artifact(s):

$ git log | head -n 1 commit c853ef75a11663cbee3160e64e45f46f5a8ac78c # Build Kafka trunk $ gradle && ./gradlew clean installAll # Find KIP-270 artifacts $ find . -name "kafka-streams-scala*.jar" ./streams/streams-scala/build/libs/kafka-streams-scala-2.0.0-SNAPSHOT.jar ./streams/streams-scala/build/libs/kafka-streams-scala-2.0.0-SNAPSHOT-javadoc.jar ./streams/streams-scala/build/libs/kafka-streams-scala-2.0.0-SNAPSHOT-test.jar ./streams/streams-scala/build/libs/kafka-streams-scala-2.0.0-SNAPSHOT-sources.jar ./streams/streams-scala/build/libs/kafka-streams-scala-2.0.0-SNAPSHOT-scaladoc.jar ./streams/streams-scala/build/libs/kafka-streams-scala-2.0.0-SNAPSHOT-test-sources.jar

The maven coordinates for the artifacts above have an artifactId of kafka-streams-scala, not kafka-streams-scala_2.11.

Looking at the build.gradle again, today we only build kafka-streams-scala with the default scala versions defined in dependencies.gradle, 2.11.12.

If we want to publish multiple artifacts with different scala versions we should follow the core project pattern, i.e. sth. like:

for ( sv in availableScalaVersions ) { String taskSuffix = sv.replaceAll("\\.", "_") tasks.create(name: "jar_core_${taskSuffix}", type: GradleBuild) { startParameter = project.getGradle().getStartParameter().newInstance() startParameter.projectProperties += [scalaVersion: "${sv}"] tasks = ['core:jar'] } tasks.create(name: "test_core_${taskSuffix}", type: GradleBuild) { startParameter = project.getGradle().getStartParameter().newInstance() startParameter.projectProperties += [scalaVersion: "${sv}"] tasks = ['core:test'] } tasks.create(name: "srcJar_${taskSuffix}", type: GradleBuild) { startParameter = project.getGradle().getStartParameter().newInstance() startParameter.projectProperties += [scalaVersion: "${sv}"] tasks = ['core:srcJar'] } tasks.create(name: "docsJar_${taskSuffix}", type: GradleBuild) { startParameter = project.getGradle().getStartParameter().newInstance() startParameter.projectProperties += [scalaVersion: "${sv}"] tasks = ['core:docsJar'] } tasks.create(name: "install_${taskSuffix}", type: GradleBuild) { startParameter = project.getGradle().getStartParameter().newInstance() startParameter.projectProperties += [scalaVersion: "${sv}"] tasks = ['install'] } tasks.create(name: "releaseTarGz_${taskSuffix}", type: GradleBuild) { startParameter = project.getGradle().getStartParameter().newInstance() startParameter.projectProperties += [scalaVersion: "${sv}"] tasks = ['releaseTarGz'] } tasks.create(name: "uploadCoreArchives_${taskSuffix}", type: GradleBuild) { startParameter = project.getGradle().getStartParameter().newInstance() startParameter.projectProperties += [scalaVersion: "${sv}"] tasks = ['core:uploadArchives'] } }

We could also consider just building one artifact with the default Scala version, in this case we would remove the suffix here and add the explanation which scala version users should be expected to use.

I recommend releasing versions of kafka-streams-scala for both major versions of Scala currently supported by Kafka. We should copy build and release conventions used by Kafka core so that both artifacts are produced. I'm not very familiar with gradle or the Kafka release process, so I wasn't sure how far to go with this, but now that that gradle snippet is right in front of me it's clear that the same should be done for this library. I believe @ijuma recommended this earlier, but I didn't make the appropriate update before the merge.

~~@ijuma how are multiple versions of Kafka core published at part of the release process? Is the build script called twice with appropriate scalaVersion parameter?~~ Nevermind, I see how it's done now.

Only making the library available to Scala 2.11 will leave behind a lot of users that are already on 2.12, which has been out for several years now. Cross building (building an artifact per version of Scala) will also make it a trivial matter to support future of versions of Scala in the release process.

I agree that we should generate artifacts for both 2.11 and 2.12, like we do for Kafka Core.

Sounds good. @seglo could you submit a follow-up PR to modify build.gradle for publishing multiple artifacts for different scala versions of kafka-streams-scala then?

Yes. I'm travelling ATM, but I'll make a new PR in the next few days.

miguno · 2018-04-25T14:34:24Z

+    p
+  }
+
+  val builder: StreamsBuilder = new StreamsBuilder()


This example code doesn't compile, because e.g. the import for StreamsBuilder is missing (from package org.apache.kafka.streams.scala).

I would probably double-check the other examples that are shown in the documentation, too.

@seglo could you take a look?

Yes. I'll test the snippets in the build PR.

ijuma · 2018-04-28T00:26:12Z

I have been unable to reply to this PR since it has continuously failed to load for me for 1 week. So I am resorting to "reply by email". :) "@ijuma <https://github.com/ijuma> how are multiple versions of Kafka core published at part of the release process? Is the build script called twice with appropriate scalaVersion parameter?" We currently invoke it twice with -PscalaVersion=2.12 for the second invocation because we haven't enabled Scala 2.12 builds by default. The reason is that Scala 2.12 requires Java 8. Once we switch our build to require Java 8 (should happen soon), we will enable 2.12 by default and then a single command with the *All suffix will do it for all Scala versions supported. To give an example, `test` will run the tests with the default Scala version while `testAll` will run the tests with all supported Scala versions. For what it's worth, I strongly agree that we should do the same we do for core, i.e. publish for both Scala 2.11 and Scala 2.12 and include the Scala version in the artifact id, as it's standard practice for Scala libraries. Ismael

…

On Wed, Apr 25, 2018 at 1:21 PM, Sean Glover ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In docs/streams/index.html <#4756 (comment)>: > +import org.apache.kafka.streams.kstream.Materialized +import org.apache.kafka.streams.scala.kstream._ +import org.apache.kafka.streams.{KafkaStreams, StreamsConfig} + +object WordCountApplication extends App { + import DefaultSerdes._ + import ImplicitConversions._ + + val config: Properties = { + val p = new Properties() + p.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-application") + p.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker1:9092") + p + } + + val builder: StreamsBuilder = new StreamsBuilder() Yes. I'll test the snippets in the build PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4756 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABgq2Xsaj25ViUqKl-yS6dxK1zDwkq2ks5tsNq4gaJpZM4S3B3-> .

miguno · 2018-05-07T09:37:32Z

+   * @return a [[KStream]] that contains more or less records with new key and value (possibly of different type)
+   * @see `org.apache.kafka.streams.kstream.KStream#transform`
+   */ 
+  def transform[K1, V1](transformerSupplier: Transformer[K, V, (K1, V1)],


The single parameter for transform() is a Transformer, not a TransformerSupplier. The variable needs renaming and the javadocs updating.

I raised https://issues.apache.org/jira/browse/KAFKA-6871 for this.

mjsax requested review from dguy, guozhangwang and ijuma March 22, 2018 18:22

mjsax added the streams label Mar 22, 2018

mjsax self-requested a review March 22, 2018 18:22

ijuma reviewed Mar 23, 2018

View reviewed changes

guozhangwang reviewed Mar 23, 2018

View reviewed changes

mjsax reviewed Mar 26, 2018

View reviewed changes

vvcephei reviewed Mar 26, 2018

View reviewed changes

mjsax reviewed Mar 27, 2018

View reviewed changes

guozhangwang reviewed Apr 3, 2018

View reviewed changes

seglo force-pushed the scala-streams branch from 88185aa to ee7b8f1 Compare April 5, 2018 01:41

dguy reviewed Apr 6, 2018

View reviewed changes

seglo force-pushed the scala-streams branch from 40c7691 to 453d4ab Compare April 12, 2018 13:13

mjsax reviewed Apr 16, 2018

View reviewed changes

guozhangwang reviewed Apr 16, 2018

View reviewed changes

Miscellaneous code review feedbacks

e91758e

Misc. userdocs updates

ee60ee5

guozhangwang approved these changes Apr 19, 2018

View reviewed changes

guozhangwang closed this in b2e4db0 Apr 23, 2018

miguno reviewed Apr 25, 2018

View reviewed changes

seglo mentioned this pull request Apr 30, 2018

MINOR: Build and code sample updates for Kafka Streams DSL for Scala #4949

Merged

3 tasks

miguno reviewed May 7, 2018

View reviewed changes

guozhangwang mentioned this pull request May 7, 2018

Make Serdes less confusing in Scala #4963

Merged

3 tasks

vvcephei mentioned this pull request Mar 31, 2021

KAFKA-12593: Fix Apache License headers #10452

Merged

3 tasks

FrankYang0529 mentioned this pull request Apr 1, 2025

KAFKA-19058 Running the streams/streams-scala module tests produces a streams-scala.log #19324

Merged


		val pattern = Pattern.compile("\\W+", Pattern.UNICODE_CHARACTER_CLASS)

		@Test def shouldBuildIdenticalTopologyInJavaNScala_1() = {

Uh oh!

Conversation

debasishg commented Mar 22, 2018

Committer Checklist (excluded from commit message)

Uh oh!

mjsax commented Mar 22, 2018

Uh oh!

guozhangwang commented Mar 23, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guozhangwang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guozhangwang Mar 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

debasishg Mar 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guozhangwang Mar 26, 2018 •

edited

Loading

debasishg Mar 27, 2018 •

edited

Loading