-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
update confluent dependency version to 7.6.0 #30638
Conversation
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
There are test failures related to avro code generation dependency change:
and
see https://github.com/apache/beam/runs/22672196056 |
Thanks for the contribution! It involves avro dependency and things get tricky. As avro dependency in Beam is known to be hard to change. Is it possible to avoid avro dependency changes and still update confluent dependency? |
Thanks for reviewing! I'll try fixing the problems that the tests discovered, if it gets too tricky or risky I imagine we might want to update the confluent version gradually, ie instead of going straight for the 7.6.0 maybe try first with something less radical like 6.x or even 5.4 |
@@ -262,8 +262,7 @@ private static TableFieldSchema fieldDescriptorFromAvroField(Schema.Field field) | |||
elementType.getType() != Schema.Type.UNION, | |||
"Multiple non-null union types are not supported."); | |||
TableFieldSchema unionFieldSchema = | |||
fieldDescriptorFromAvroField( | |||
new Schema.Field(field.name(), elementType, field.doc(), field.defaultVal())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why defaultVal dropped here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the elementType
here will be a schema of the union, but with the null type filtered out - in the "extreme" case of an "optionable" union type of [null, SomeSchema]
it won't even be a union anymore, just the bare SomeSchema
- if the defaultVal
is null
, it will not correspond to this derived schema anymore, and Avro will throw an exception when constructing the Field
object - I'm not sure why this wasn't a problem before (maybe Avro didn't actually check this?)
avro : "org.apache.avro:avro:1.8.2", | ||
avro_tests : "org.apache.avro:avro:1.8.2:tests", | ||
avro : "org.apache.avro:avro:1.11.3", | ||
avro_tests : "org.apache.avro:avro:1.11.3:tests", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I remember there was issue upgrading avro version and that was why it sticked to 1.8.2 for a long time. Might be good to ask in devlist about this proposal
...tensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java
Show resolved
Hide resolved
also, it's good to run https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Avro_Versions.yml in this PR. You can add an empty file named |
Run Java_GCP_IO_Direct PreCommit |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tests now all passed.
wait for another day and will merge if no objection in the devlist mail thread
There is (only) one integration test failing, not too bad: testBigQueryStorageReadProjectionPushdown (org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOStorageReadIT) failed
See https://github.com/apache/beam/runs/22988286765 |
* update confluent dependency version to 7.6.0 * make avroSchema field non-transient as the Schema class is now Serializable * remove now unused org.codehaus.jackson package from GcpApiSurfaceTest * update expected exception class in PubsubClientTest * fix AvroGenericRecordToStorageApiProtoTest * move org.tukaani:xz:1.9 dependency to testImplementation * bring back the tukaani dep to the avroVersions configurations * trigger the beam_PostCommitJava_Avro_Versions workflow
fixes #30610
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123
), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>
instead.CHANGES.md
with noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.