Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve][connector] JDBC sinks: support Avro specific datatypes #15845

Merged
merged 8 commits into from
Jun 1, 2022
Merged

[improve][connector] JDBC sinks: support Avro specific datatypes #15845

merged 8 commits into from
Jun 1, 2022

Conversation

nicoloboschi
Copy link
Contributor

Motivation

JDBC sinks doesn't support Avro datatypes like Utf8.

2022-05-30T16:16:35,850+0000 [pool-5-thread-1] ERROR org.apache.pulsar.io.jdbc.JdbcAbstractSink - Got exception 
java.lang.Exception: Not support value type, need to add it. class org.apache.avro.util.Utf8
	at org.apache.pulsar.io.jdbc.BaseJdbcAutoSchemaSink.setColumnValue(BaseJdbcAutoSchemaSink.java:134) ~[pulsar-io-jdbc-core-2.10.0.5-SNAPSHOT.jar:?]
	at org.apache.pulsar.io.jdbc.BaseJdbcAutoSchemaSink.bindValue(BaseJdbcAutoSchemaSink.java:89) ~[pulsar-io-jdbc-core-2.10.0.5-SNAPSHOT.jar:?]
	at org.apache.pulsar.io.jdbc.JdbcAbstractSink.flush(JdbcAbstractSink.java:206) ~[pulsar-io-jdbc-core-2.10.0.5-SNAPSHOT.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Modifications

Added converted (inspired by the ElasticSearch one) to do proper conversion of data, from Avro generic record to JDBC compatible type.

  • doc-not-needed

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label May 30, 2022
@nicoloboschi nicoloboschi requested a review from eolivelli May 31, 2022 08:12
eolivelli
eolivelli approved these changes May 31, 2022
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eolivelli
Copy link
Contributor

@nicoloboschi checkstyle failed PTAL

shibd
shibd approved these changes May 31, 2022
View reviewed changes
Copy link
Member

@shibd shibd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@nicoloboschi
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

1 similar comment
@nicoloboschi
Copy link
Contributor Author

/pulsarbot rerun-failure-checks

@nicoloboschi nicoloboschi merged commit 3530349 into apache:master Jun 1, 2022
@nicoloboschi nicoloboschi added this to the 2.11.0 milestone Jun 1, 2022
nicoloboschi added a commit to datastax/pulsar that referenced this pull request Jun 3, 2022
@nicoloboschi
[improve][connector] JDBC sinks: support Avro specific datatypes (apa…
9f5e073 
…che#15845)

(cherry picked from commit 3530349)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eolivelli eolivelli eolivelli approved these changes

@coderzc coderzc coderzc approved these changes

@shibd shibd shibd approved these changes

Assignees

@nicoloboschi nicoloboschi

Labels
area/connector doc-not-needed Your PR changes do not impact docs
Projects
None yet
Milestone
2.11.0
Development

Successfully merging this pull request may close these issues.
4 participants
@nicoloboschi @eolivelli @coderzc @shibd