Add oneof support for Protobuf via Confluent Schema Registry

[adamjshook]

Description

This adds Kafka connector support for Protobuf oneOf data types by encoding them as a JSON object.

Additional context and related issues

Protobuf oneOf data types are a single field in the message that can hold one value of multiple fields (of varying types) per the schema definition. This adds support for these data types by encoding the entire oneOf field as a JSON object where each field name is a key in the JSON object where all values, except one, are null. The value that is set corresponds to the set value in the DynamicMessage that is processed in the message.

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(X) Release notes are required, with the following suggested text:

# Kafka Connector
* Added support for `oneof` Protobuf types when using the Confluent table description provider

[vlad-lyutenko]

I am new for this topic, but I made review, general implementation looks good to me, I have only question about dependencies in pom. A little bit missed with them, why we comment some protobuf plugins?

[adamjshook]

I am new for this topic, but I made review, general implementation looks good to me

Thanks for the review, I'm hoping to clean things up today and open this up for review.

I have only question about dependencies in pom. A little bit missed with them, why we comment some protobuf plugins?

I originally added the protobuf plugins to generate a TestOneOf object for unit testing, however by the end of implementation I was able to do everything using a DynamicMessage. I commented out the plugins to make sure they weren't needed in any of my tests. The final PR will not use these plugins (I'll be deleting all the commented code).

[hashhar]

skimmed, mostly editorial comments

Impl looks good.

[hashhar]

why do we need to sort?

[adamjshook]

I was sorting it based on this comment in JsonType. Maybe it isn't relevant if the built type is a Slice instead of a Block. (Back when I was using Block to build the JSON, it did require sorting as the comment suggested.)

[hashhar]

this is very interesting, I had no idea the Trino JsonType required keys to be sorted. I now see in JDBC connectors we don't need to sort because it's done by JsonTypeUtils#jsonParse/JsonTypeUtils#toJsonValue for us.

[wendigo]

Maybe we should encode this knowledge?

[adamjshook]

Since the JSON record we create will only have at most one field in it (since oneof can only have one set), sorting the column handles isn't necessary here.

[hashhar]

can we come up with better estimates, e.g. based on field count * some avg length?

[adamjshook]

I made a change to calculate the size based on the string values of the field name, value, plus the extra characters. Looking at the data type test case, there wouldn't be any resizing.

[hashhar]

nit: Let's consitently use oneof instead of oneOf since that's what the protobuf schema calls it.

[hashhar]

separate commit, just formatting changes

[adamjshook]

I'm just going to revert this; accidentally committed from when I was messing around in this class during implementation.

[hashhar]

extract to a serializeJson method?

[hashhar]

all ctors should ideally go via the JsonCreator, less chance of mistakes

[hashhar]

if this causes a large diff you can extract as the first commit with some reasons in commit message

[hashhar]

list equality depends on element order.

Does a protobuf schema consider differently ordered oneofs as different?
At-least within JSON the field order doesn't matter - so maybe this equality method should do the same?

[adamjshook]

Can't say the protobuf schema would treat different ordered oneof definitions as different objects. The order of the column names in this list matches the order of how the fields are defined in the schema (by name, not by tag number). Since order doesn't matter, and the final JSON object will have at most one field, how about we change this to a Set<String> instead of List? @hashhar

[hashhar]

Set sounds good if we're only going to keep the single field.

[hashhar]

yes

[hashhar]

if it's a large diff, extract a commit

[hashhar]

same applies to other ctor changes as well

[adamjshook]

Do we want all possible columns in the oneof to be in the JSON string, leaving all other values as null? Or just a single column e.g. {"stringColumn":"value"}?

[hashhar]

I think having a single field populated would be better because otherwise you can't differentiate that stringColumn=null because stringColumn was set to null or because some other field existed and stringColumn didn't exist.

[adamjshook]

Makes sense and I agree. I'll update to only write a field if it is present. Protobuf doesn't do normal null values, but I have added a test case for an empty message (which results in an empty JSON value {}).

[hashhar]

A test showcasing this doesn't work with the file-based table description supplier would be useful.

LGTM from me.

[hashhar]

nit: "{\"\":\"\"}".length (or a comment).

[adamjshook]

A test showcasing this doesn't work with the file-based table description supplier would be useful.

Added a test case asserting the column names are not found when using the file-based descriptor.

[adamjshook]

I've stumbled on a library com.google.protobuf:protobuf-java-util which has support for JSON-encoding protobuf oneof types. I'm going to flip this back to a draft because I think it may make the implementation a lot more simple.

[adamjshook]

Flipping this back to review for merge in the current state. We can follow up on any improvements later on.

[hashhar]

@mosabua / @jhlodin can you please review the last commit which includes documentation changes.

[Praveen2112]

Flipping this back to review for merge in the current state. We can follow up on any improvements later on.
Can we have a GH issue to track the followup ?

[Praveen2112]

Overall looks good to me.

But I would like to see how it serializes/deserializes if one of oneOf field is of a message type - From this docs - https://protobuf.dev/programming-guides/proto3/#using-oneof it looks like map and array are not supported but row should be supported by oneOf

And if it is possible for us to determine the oneOf fields when reading the message, do we need to propagate these information when we create KafkaColumnHandle ?

[Praveen2112]

nit: Can we return as KafkaTopicFieldDescription here so as to be unified with getOneofs.

[Praveen2112]

Can we also pass a message type and check on how DynamicMessage would be serialized.

[Praveen2112]

Any specific reason on why we should pass the oneofColumns from the KafkaColumnHandle ? Why can't we use message.getDescriptorForType() get oneOf types from here and handle them ? In this case we don't have to change other decoder right ?

[wendigo]

Yeah. You can collect oneof fields from FieldDescriptor:

.getContainingOneof().getName()

[adamjshook]

Great, thanks for the suggestion. I just couldn't find a way to get the names to where they needed to be, but this does the trick. This removes the need to include the oneof names in the column handles.

[Praveen2112]

nit: These can be extracted to a new commit.

[Praveen2112]

Can we add additional columnName with message type

[Praveen2112]

for timestamp ?

[mosabua]

A couple of fixes for docs necessary. Good once the suggestions are applied.

[mosabua]

Suggested change

	Protobuf specific type handling in Confluent table description supplier
	Protobuf-specific type handling in Confluent table description supplier

[mosabua]

Suggested change

	When using the Confluent table description supplier the following Protobuf
	When using the Confluent table description supplier, the following Protobuf

[mosabua]

Suggested change

	specific types are supported in addition to the :ref:`normally supported
	specific types are supported in addition to the :ref:`normally supported types

[mosabua]

Suggested change

	<kafka-protobuf-decoding>` types:
	<kafka-protobuf-decoding>`:

[mosabua]

Suggested change

	Protobuf schemas containing ``oneof`` fields get mapped to a ``JSON`` field in
	Protobuf schemas containing ``oneof`` fields are mapped to a ``JSON`` field in

[mosabua]

Suggested change

	For example given the following Protobuf schema:
	For example given the following Protobuf schema:

Suggested change

	For example given the following Protobuf schema:
	For example, given the following Protobuf schema:

[mosabua]

Suggested change

	The corresponding Trino row would have a ``JSON`` field ``test_oneof_column``
	The corresponding Trino row is a ``JSON`` field ``test_oneof_column``

[mosabua]

Suggested change

	containing a JSON object with a single key. The value of the key would match
	containing a JSON object with a single key. The value of the key matches

[mosabua]

Suggested change

	In the above example it means if the Protobuf message had the
	In the above example, if the Protobuf message has the

[mosabua]

Suggested change

	then the corresponding Trino row would have a column named
	then the corresponding Trino row includes a column named

[jhlodin]

One additional comment, +1 to Manfred that docs are good once changes are applied

[jhlodin]

Suggested change

	the name of the ``oneof`` type which was actually present.
	the name of the ``oneof`` type that is present.

[wendigo]

nit: getOneOfs ?

[adamjshook]

We've been styling as as oneof/Oneof to match how Protobuf styles it.

[wendigo]

Makes sense!

[wendigo]

Is the "at most one" limitation of the protobuf?

[wendigo]

https://protobuf.dev/programming-guides/proto3/#using-oneof as far as I can tell, there could be more then just one named oneofs in the protobuf message.

[wendigo]

We should rather than pass the oneofColumns enumerate those using DynamicMessage

[adamjshook]

You can specify more than one oneof fields in the message schema, but the field names across all oneof types have to be distinct so you'll only find one.

I've made the change to remove using oneofColumns in favor of enumerating the field names in DynamicMessage.

[hashhar]

but the field names across all oneof types have to be distinct so you'll only find one.

Is this verified by the protobuf schema parser or something else? AFAIK the protobuf schema is based on tags so the schema itself probably doesn't care.

[adamjshook]

Given a schema file:

syntax = "proto3";
message schema {
    oneof testOneofColumn {
        string stringColumn = 1;
    }
    string stringColumn = 2;
}

Loading the file-based schema parser doesn't result in any failure, but there is only one stringColumn field defined in the Descriptor (I'd say because it doesn't handle oneof types in the first place, so it is ignored).

The Confluent parser doesn't result in any failures, but there is only one stringColumn field in the resulting descriptor (it seems to flatten all of the oneof fields into a single field description, so having an additional field with the same name will be de-duplicated.

Meanwhile, protoc fails (specifically the Maven plugin which runs protoc) fails but this may just be an artifact of the code generation:

protoc-jar: executing: [/tmp/protocjar14095071959677501612/bin/protoc.exe, -I/tmp/protocjar6731110910677765846/include, -I/home/adamjshook/gitrepos/trino/lib/trino-record-decoder/src/test/resources/decoder/protobuf, --java_out=/home/adamjshook/gitrepos/trino/lib/trino-record-decoder/target/generated-test-sources, /home/adamjshook/gitrepos/trino/lib/trino-record-decoder/src/test/resources/decoder/protobuf/test_oneof.proto]
test_oneof.proto:7:12: "stringColumn" is already defined in "schema".

[hashhar]

Yeah, seems protobuf is under-specified for this case and behaviour depends on impl. Since messages will be produced using Confluent serializer i think the assumption holds true.

[hashhar]

Can we make it throw if we find multiple?

[wendigo]

add requireNonNull - oneofs is not a primitive field

[wendigo]

Yeah. You can collect oneof fields from FieldDescriptor:

.getContainingOneof().getName()

[adamjshook]

Pushed after applying the feedback here. Noteable changes are:

• Removed having the oneofs on the DecoderColumnHandle and the KafkaTopicFieldDescription. They are now sourced from the message descriptor in the ProtobufColumnDecoder
• Added the Row and NestedRow data types to the test_oneof.proto file
• Changed how the JSON value is being built in ProtobufColumnDecoder to use the JsonFormat.Printer from the protobuf-java-util

[hashhar]

just rebased on master for fresh ci results, will merge soon