KAFKA-7609: Add Protocol Generator for Kafka by cmccabe · Pull Request #5893 · apache/kafka

cmccabe · 2018-11-08T18:42:12Z

No description provided.

cmccabe · 2018-11-08T18:59:44Z

Overview:

buildSrc/src
The message generator code is here. This code is automatically re-run by gradle when one of the schema files changes. The entire directory is processed at once to minimize the number of times we have to start a new JVM. We use Jackson to translate the JSON files into Java objects.

clients/src/main/java/org/apache/kafka/common/protocol/Message.java
This is the interface implemented by all automatically generated messages.

clients/src/main/java/org/apache/kafka/common/protocol/MessageUtil.java
Some utility functions used by the generated message code.

clients/src/main/java/org/apache/kafka/common/protocol/Readable.java, Writable.java, ByteBufferAccessor.java
The generated message code uses these classes for writing to a buffer.

clients/src/main/message/README.md
This README file explains how the JSON schemas work.

clients/src/main/message/*.json
The JSON files in this directory implement every supported version of every Kafka API. The unit tests automatically validate that the generated schemas match the hand-written schemas in our code. Additionally, there are some things like request and response headers that have schemas here.

clients/src/main/java/org/apache/kafka/common/utils/ImplicitLinkedHashSet.java
I added an optimization here for empty sets. This is useful here because I want all messages to start with empty sets by default prior to being loaded with data. This is similar to the "empty list" optimizations in the java.util.ArrayList class

core/src, connect/src, clients/src/main/java/org/apache/kafka/clients/admin, etc.
These changes are related to switching CreateTopicsRequest and CreateTopicsResponse to using the automatically generate classes rather than the hand-written ones.

bob-barrett

This looks good to me, Colin. I left a couple minor comments.

As a more general comment, I think we should include the docstrings in the schema files, so that we can eventually generate the protocol docs off of them. And if we're going to do that, we may actually want to be more precise about not changing field names from the current manual schemas, so that people don't get thrown off by changed names. I don't think either of these things necessarily has to happen in this patch, though.

cmccabe · 2018-11-29T18:34:24Z

As a more general comment, I think we should include the docstrings in the schema files, so that we can eventually generate the protocol docs off of them.

That's a good point. I added documentation to all JSON schema files, based on the original Struct documentation. In cases where this documentation was missing, I created protocol documentation for the fields. I also ported over the comments explaining the purpose of each protocol version change.

And if we're going to do that, we may actually want to be more precise about not changing field names from the current manual schemas, so that people don't get thrown off by changed names. I don't think either of these things necessarily has to happen in this patch, though.

I think it's best to decide about field names when doing the conversion for each RPC.

Are we no longer concerned about duplicate topics? Is that handled during deserialization?

I fixed this so that we handle duplicates the same way we do now: by setting an INVALID_REQUEST error for them. This is actually specified in the KIP that added CreateTopics.

In order to make this possible, deserialization needs to be able to represent duplicate topics in the set. I handled this by using multisets instead of sets for data received over the wire. I think this is the correct approach in general because it's flexible and will allow us to covert over the existing code more easily. Throwing a deserialization error when encountering duplicates would be very unfriendly-- deserialization errors just result in the broker disconnecting without an error message.

To make this easier to review, I have split the CreateTopics changes off into a separate PR, #5972 . I think with that stuff split out, this is a very low-impact change since it's just adding a code generator, and not modifying any existing broker code path.

bob-barrett

Thanks Colin! I agree about handling field names when we do the conversion. I added a couple more comments.

cmccabe · 2018-12-05T23:28:57Z

Rebase on trunk
Update message schemas for KIP-380

cmccabe · 2018-12-07T22:03:00Z

retest this please

hachikuji

Thanks, this looks great! I left a few minor comments and some questions.

hachikuji · 2019-01-02T21:47:15Z

I wonder if there should be an enum type in this schema. Looking at the generated code, this field is just represented as a byte, but it would be nice to have something friendlier to work with. Otherwise we'll just create the enum manually and do some conversion.

Yeah, enums would be a good addition. It's a bit trickier than it seems, though. For one thing, we already have a lot of public/stable enum types, and we can't break them. enums can't inherit from other classes, either, so that escape hatch is closed.

Perhaps a simple way to make it work would be generating simple constants

public static int NONE = 0; public static int OFFSET_OUT_OF_RANGE = 1; ... etc. etc. ...

Then the existing public/stable enums could be retrofitted to grab the values from these declarations. We could also generate classes in cases where there was no existing public thing.

cmccabe · 2019-01-05T01:22:24Z

The test failure was kafka.api.CustomQuotaCallbackTest.testCustomQuotaCallback, which is unrelated.

cmccabe · 2019-01-05T22:04:16Z

retest this please

hachikuji · 2019-01-09T23:26:26Z

Mentioned offline, but we need to move the few test cases under clients/src/test/java/org/apache/kafka/common/message/ to clients/src/test/java/org/apache/kafka/message/.

hachikuji

A few more small comments.

hachikuji · 2019-01-10T00:10:39Z

+        this.ignorable = ignorable;
+        this.about = about == null ? "" : about;
+        if (!this.struct.fields().isEmpty()) {
+            if (!this.type.isArray()) {


Couldn't we make this check stricter? Would we ever expect the type to be something other than STRUCT?

I refactored this code to clean it up a bit. Not all FieldSpecs describe arrays-- only some of them do. We should have a toStruct() method to make it convenient, but not embed a StructSpec unconditionally into the FieldSpec.

hachikuji · 2019-01-10T00:14:17Z

+import java.util.Optional;
+
+public interface FieldType {
+    static final String STRUCT_PREFIX = "[]";


nit: static final is redundant since this is an interface

Removed, thanks

hachikuji · 2019-01-10T00:19:53Z

+        this.imports.add(newImport);
+    }
+
+    public void generate() throws Exception {


nit: Exception not raised. A bunch of these in MessageDataGenerator and MessageFactoryGenerator as well. Probably fall out from removing the throws on printf.

Good point. I was able to remove quite a few of these.

hachikuji · 2019-01-10T00:29:18Z

+        return true;
+    }
+
+    private boolean generateNonNullCheck(Versions prevVersions, FieldSpec field) throws Exception {


This is unused.

hachikuji · 2019-01-10T00:43:09Z

+    /**
+     * Returns the API key of this message, or -1 if there is none.
+     */
+    short apiKey();


It's a little annoying that we have to expose this for all of the nested types. Have you considered having something like an ApiMessage which extends this?

That is a fair point. I will separate out Message as a subtype of the ApiMessage interface.

hachikuji · 2019-01-10T00:51:25Z

+
+    public void registerMessageType(MessageSpec spec) {
+        if (spec.type() == MessageSpecType.REQUEST) {
+            if (requestApis.containsKey(spec.apiKey())) {


Is it worth having any validation that apiKey is not -1? I was even considering whether MessageSpec.apiKey could return Optional<Short>.

I can use Optional there.

hachikuji · 2019-01-10T00:55:07Z

+        }
+        this.fields = Collections.unmodifiableList(fields == null ?
+            Collections.emptyList() : new ArrayList<>(fields));
+        this.hasKeys = fields == null ? false : fields.stream().anyMatch(f -> f.mapKey());


nit: fields != null && fields.stream().anyMatch(f -> f.mapKey())

hachikuji · 2019-01-11T17:41:06Z

 // limitations under the License.

 {
+  "apiKey": -1,


Why was this needed?

It shouldn't be needed-- I'll remove it.

In the long term, we'll want to generate other things besides ApiMessage from JSON files (enums, common structures, etc.) but we can hold off on that for now.

hachikuji · 2019-01-11T17:41:36Z

+/**
+ * A Message which is part of the top-level Kafka API.
+ */
+public interface ApiMessage extends Message {


Would it make any sense to move the version fields into ApiMessage?

Hmm... I think it makes sense to leave them in Message. All Messages have minimum and maximum supported versions, not just ApiMessages.

hachikuji · 2019-01-11T21:47:46Z

retest this please

hachikuji

Thanks @cmccabe, LGTM. This is a major improvement. I'll merge after the builds complete.

cmccabe · 2019-01-12T00:37:54Z

Failing test is kafka.api.UserQuotaTest.testThrottledProducerConsumer, which is definitely unrelated to this PR

junrao · 2019-01-22T18:50:15Z

@cmccabe : I could find clients/src/main/message/README.md mentioned in the PR. Does that still exist?

cmccabe · 2019-01-22T22:29:52Z

It was moved to ./clients/src/main/resources/common/message/README.md

junrao · 2019-01-25T23:40:45Z

+          "about": "The result offsets." },
+        { "name": "Timestamp", "type": "int64", "versions": "1+" },
+        { "name": "Offset", "type": "int64", "versions": "1+",
+          "about": "The timestamp associated with the returned offset." },


This comment is for Timestamp, not for Offset?

Good catch. I have filed a PR to fix this: #6214

This patch adds a framework to automatically generate the request/response classes for Kafka's protocol. The code will be updated to use the generated classes in follow-up patches. Below is a brief summary of the included components: **buildSrc/src** The message generator code is here. This code is automatically re-run by gradle when one of the schema files changes. The entire directory is processed at once to minimize the number of times we have to start a new JVM. We use Jackson to translate the JSON files into Java objects. **clients/src/main/java/org/apache/kafka/common/protocol/Message.java** This is the interface implemented by all automatically generated messages. **clients/src/main/java/org/apache/kafka/common/protocol/MessageUtil.java** Some utility functions used by the generated message code. **clients/src/main/java/org/apache/kafka/common/protocol/Readable.java, Writable.java, ByteBufferAccessor.java** The generated message code uses these classes for writing to a buffer. **clients/src/main/message/README.md** This README file explains how the JSON schemas work. **clients/src/main/message/\*.json** The JSON files in this directory implement every supported version of every Kafka API. The unit tests automatically validate that the generated schemas match the hand-written schemas in our code. Additionally, there are some things like request and response headers that have schemas here. **clients/src/main/java/org/apache/kafka/common/utils/ImplicitLinkedHashSet.java** I added an optimization here for empty sets. This is useful here because I want all messages to start with empty sets by default prior to being loaded with data. This is similar to the "empty list" optimizations in the `java.util.ArrayList` class. Reviewers: Stanislav Kozlovski <stanislav_kozlovski@outlook.com>, Ismael Juma <ismael@juma.me.uk>, Bob Barrett <bob.barrett@outlook.com>, Jason Gustafson <jason@confluent.io>

cmccabe force-pushed the KAFKA-7609 branch from 9cf77ae to 886658c Compare November 8, 2018 19:30

stanislavkozlovski reviewed Nov 9, 2018

View reviewed changes

Comment thread buildSrc/src/main/java/org/apache/kafka/message/JsonUtil.java Outdated

ijuma reviewed Nov 9, 2018

View reviewed changes

Comment thread clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java Outdated

cmccabe force-pushed the KAFKA-7609 branch from 03676b3 to f6ff824 Compare November 14, 2018 16:13

bob-barrett reviewed Nov 15, 2018

View reviewed changes

Comment thread core/src/main/scala/kafka/server/KafkaApis.scala Outdated

Comment thread core/src/main/scala/kafka/server/KafkaApis.scala Outdated

ijuma reviewed Nov 19, 2018

View reviewed changes

Comment thread clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsResponse.java Outdated

cmccabe force-pushed the KAFKA-7609 branch 2 times, most recently from 497b56e to 14d5e20 Compare November 24, 2018 19:16

cmccabe force-pushed the KAFKA-7609 branch 2 times, most recently from ad3de1a to 2aba254 Compare November 29, 2018 18:17

bob-barrett reviewed Dec 3, 2018

View reviewed changes

Comment thread clients/src/main/java/org/apache/kafka/common/utils/ImplicitLinkedHashSet.java Outdated

Comment thread buildSrc/src/main/java/org/apache/kafka/message/CodeBuffer.java Outdated

cmccabe force-pushed the KAFKA-7609 branch from 0e95526 to 9813bc5 Compare December 5, 2018 23:28

hachikuji reviewed Jan 2, 2019

View reviewed changes

cmccabe force-pushed the KAFKA-7609 branch 2 times, most recently from 47cb19c to b585449 Compare January 4, 2019 22:13

cmccabe added 7 commits January 7, 2019 11:23

KAFKA-7609: Add Protocol Generator for Kafka

8cd1846

Move some namespaces

b60abe2

Require struct versions to be set

a5c9bc4

More style fixes and etc.

b7cffd2

Finish renaming

baffbf3

Bump ListOffset for KIP-207

3ea2671

Fix some javadoc, use assertTrue instead of Assert.fail in a case

e1b2173

cmccabe force-pushed the KAFKA-7609 branch from 753b0cf to e1b2173 Compare January 7, 2019 19:29

Fix namespace for VersionsTest, MessageGeneratorTest

2091b26

hachikuji reviewed Jan 10, 2019

View reviewed changes

cmccabe added 5 commits January 10, 2019 14:59

Some syntax cleanups

f003e4d

Separate ApiMessage and Message, add Optional, etc.

6386c1a

StructSpec: stylistic improvement

aedc370

Versions: add JavaDoc

744b972

Don't embed Struct into FieldSpec

d0957a1

hachikuji reviewed Jan 11, 2019

View reviewed changes

Cleanups

6d8de7a

hachikuji approved these changes Jan 11, 2019

View reviewed changes

hachikuji merged commit 71e85f5 into apache:trunk Jan 12, 2019

tombentley mentioned this pull request Jan 16, 2019

KAFKA-5692: Change PreferredReplicaLeaderElectionCommand to use Admin… #3848

Merged

junrao reviewed Jan 25, 2019

View reviewed changes

gwenshap mentioned this pull request Feb 7, 2019

Kafka codec envoyproxy/envoy#4950

Merged

cmccabe deleted the KAFKA-7609 branch May 20, 2019 19:04

Uh oh!

Conversation

cmccabe commented Nov 8, 2018

Uh oh!

cmccabe commented Nov 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bob-barrett left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cmccabe commented Nov 29, 2018

Uh oh!

bob-barrett left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cmccabe commented Dec 5, 2018

Uh oh!

cmccabe commented Dec 7, 2018

Uh oh!

hachikuji left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cmccabe commented Jan 5, 2019

Uh oh!

cmccabe commented Jan 5, 2019

Uh oh!

hachikuji commented Jan 9, 2019

Uh oh!

hachikuji left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

cmccabe commented Nov 8, 2018 •

edited

Loading

bob-barrett left a comment •

edited

Loading

junrao Jan 25, 2019 •

edited

Loading