Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Functions]Support protobuf schema for pulsar function #11709

Merged
merged 3 commits into from
Aug 20, 2021

Conversation

tuteng
Copy link
Member

@tuteng tuteng commented Aug 19, 2021

Fixes #

(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)

Master Issue: #

Motivation

Some users have encountered the following exception when using the protobuf schema, so add the relevant dependencies to the instance:

ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - Sink open produced uncaught exception:
java.lang.IllegalArgumentException: com.google.protobuf.GeneratedMessageV3 is not assignable from bold.proto.PersonOuterClass$Person
	at org.apache.pulsar.client.impl.schema.ProtobufSchema.of(ProtobufSchema.java:110) ~

Modifications

  • Add the relevant dependencies to the instance

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

For contributor

For this PR, do we need to update docs?

  • If yes, please update docs or create a follow-up issue if you need help.

  • If no, please explain why.

For committer

For this PR, do we need to update docs?

  • If yes,

    • if you update docs in this PR, label this PR with the doc label.

    • if you plan to update docs later, label this PR with the doc-required label.

    • if you need help on updating docs, create a follow-up issue with the doc-required label.

  • If no, label this PR with the no-need-doc label and explain why.

@tuteng tuteng self-assigned this Aug 19, 2021
@tuteng tuteng added release/2.8.1 type/bug The PR fixed a bug or issue reported a bug area/function labels Aug 19, 2021
@tuteng tuteng added this to the 2.9.0 milestone Aug 19, 2021
@tuteng tuteng changed the title [pulsar-function]Support protobuf schema [Functions]Support protobuf schema Aug 19, 2021
@tuteng tuteng changed the title [Functions]Support protobuf schema [Functions]Support protobuf schema for pulsar function Aug 19, 2021
Copy link
Contributor

@gaoran10 gaoran10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

It would be better to follow up and include an integration test.
Because we are actively working on the classpath for functions and I wouldn't like to see this broken

@tuteng
Copy link
Member Author

tuteng commented Aug 19, 2021

Lgtm

It would be better to follow up and include an integration test.
Because we are actively working on the classpath for functions and I wouldn't like to see this broken

Thanks, I created an issue #11711 to track it and I will consider adding integration tests in the next pr.

@tuteng
Copy link
Member Author

tuteng commented Aug 19, 2021

/pulsarbot run-failure-checks

@tuteng tuteng merged commit f893c08 into apache:master Aug 20, 2021
hangc0276 pushed a commit that referenced this pull request Aug 25, 2021
### Motivation

Some users have encountered the following exception when using the protobuf schema, so add the relevant dependencies to the instance:

```
ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - Sink open produced uncaught exception:
java.lang.IllegalArgumentException: com.google.protobuf.GeneratedMessageV3 is not assignable from bold.proto.PersonOuterClass$Person
	at org.apache.pulsar.client.impl.schema.ProtobufSchema.of(ProtobufSchema.java:110) ~
```

### Modifications

* Add the relevant dependencies to the instance

### Verifying this change

- [x] Make sure that the change passes the CI checks.

(cherry picked from commit f893c08)
@hangc0276 hangc0276 added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Aug 25, 2021
@Anonymitaet
Copy link
Member

@tuteng Thanks for your contribution. For this PR, do we need to update docs?

(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

@tuteng
Copy link
Member Author

tuteng commented Sep 22, 2021

@tuteng Thanks for your contribution. For this PR, do we need to update docs?

(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

I think don't need to update the document, the document already exists.

@Anonymitaet Anonymitaet added the doc-not-needed Your PR changes do not impact docs label Sep 23, 2021
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
### Motivation

Some users have encountered the following exception when using the protobuf schema, so add the relevant dependencies to the instance:

```
ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - Sink open produced uncaught exception:
java.lang.IllegalArgumentException: com.google.protobuf.GeneratedMessageV3 is not assignable from bold.proto.PersonOuterClass$Person
	at org.apache.pulsar.client.impl.schema.ProtobufSchema.of(ProtobufSchema.java:110) ~
```

### Modifications

* Add the relevant dependencies to the instance

### Verifying this change

- [x] Make sure that the change passes the CI checks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/function cherry-picked/branch-2.8 Archived: 2.8 is end of life doc-not-needed Your PR changes do not impact docs release/2.8.1 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants