[#6003][pulsar-functions] Possibility to add builtin Functions #6895

oncodeit · 2020-05-06T14:09:32Z

Master Issue: #6003

Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors).

The builtin function must include a pulsar-io.yml file with the following content

name: <function-name>
description: <function-desciption>
functionClass: <function-class>

e.g.

name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala

val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

Function folder to be specified in the conf/functions_worker.yml conf file

e.g.
functionsDirectory: ./functions

Function package must be in *.nar format like for source/sink connectors

Modifications

I modified the pulsar-function-utils, pulsar-functions-worker and pulsar-common modules on the basis of the built in connectors implementation.
Also Function.proto has been modified in order to include the builtin property

What this MR does not include

modification of pulsar-admin to fetch the available buildin functions
the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? must be documented in the docs

sijie

The change looks great!

sijie · 2020-05-18T22:44:41Z

@oncodeit can you rebase this pull request to latest master?

nicolo-paganin · 2020-05-19T08:02:17Z

@sijie rebase done

codelipenghui · 2020-05-20T09:11:43Z

@nicolo-paganin Could you please take a look at the failed CI tests?

nicolo-paganin · 2020-05-21T00:08:45Z

@codelipenghui I think I need some help, there are 5 CI tests that I really don't know.. something that I never modified..

nicolo-paganin · 2020-05-22T17:01:12Z

@codelipenghui I rebased and some CI tests that were failing now are passing, but there are two that are not passing, it is nothing related with this PR.
It seems something wrong with the CI itself and I cannot restart the CI. Do you have any advice?

nicolo-paganin · 2020-05-22T17:05:24Z

one of the error is this one https://github.com/streamnative/pulsar/issues/662 already report by @sijie

nicolo-paganin · 2020-05-29T22:33:31Z

@codelipenghui any ideas on how to fix the CI?

codelipenghui · 2020-06-04T05:25:32Z

@nicolo-paganin You can merge the master branch into your branch, there are some conflicts fixed on the master branch.

…inside the function folder (apache#6003)

nicolo-paganin · 2020-06-04T20:48:32Z

@codelipenghui ok thanks, the pipeline finally passed

…te-update * 'website-update' of github.com:zeo1995/pulsar: (432 commits) Fixed ordering issue in KeyShared dispatcher when adding consumer (apache#7106) Fix Duplicated messages are sent to dead letter topic apache#6960 (apache#7021) [Issue 2793][Doc]--Update the TLS hostname verification for CPP and Python clients (apache#7162) [Doc]--set netty mex frame size (apache#7174) [Doc] Update for the maximum message size (apache#7171) Fixed KeyShared consumers getting stuck on delivery (apache#7105) [apache#6003][pulsar-functions] Possibility to add builtin Functions (apache#6895) [Issue 6921][pulsar-broker-common] Replaced "Paths.get(...).getParent()", because it's system dependent and uses '\' as path separator on Windows (apache#6992) Improve broker unit test CI (apache#7173) Fix typo in exception message (apache#7027) Support KeyValue Schema Use Null Key And Null Value (apache#7139) [Doc]--Update documents for support consumer priority level in failover mode (apache#7136) Add schema config to cpp and cgo docs. (apache#7137) [Doc]--Update for the maximum message size (apache#7160) [C++] Expose ZSTD and Snappy compression to C API (apache#7014) [pulsar-proxy] add proxyLogLevel into config file (apache#6948) Add multi-hosts example for bookkeeperMetadataServiceUri (apache#6998) support for termination of partitioned topic (apache#6126) Use pure-java Air-Compressor instead of JNI based libraries (apache#5390) [Issues 5709]remove the namespace checking (apache#5716) ... # Conflicts: # site2/website/scripts/split-swagger-by-version.js

…pache#6895) Master Issue: apache#6003 This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). The builtin function must include a `pulsar-io.yml` file with the following content ```yml name: <function-name> description: <function-desciption> functionClass: <function-class> ``` e.g. ```yml name: test-function description: test function description functionClass: it.oncode.pulsar.functions.TestFunction ``` it is possible to create a builtin function in the same way of the builtin sinks/sources. Example in scala ```scala val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder() val function = functionConfigBuilder .tenant("public") .namespace("default") .jar("builtin://test-function") .name("test-function-name") .className("it.oncode.pulsar.functions.TestFunction") .inputs(Seq("channel_in").asJava) .output("channel_out") .runtime(FunctionConfig.Runtime.JAVA) .build() Pulsar.admin.functions .createFunction(function, null) ``` Function folder to be specified in the `conf/functions_worker.yml` conf file e.g. `functionsDirectory: ./functions` Function package must be in `*.nar` format like for source/sink connectors I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation. Also `Function.proto` has been modified in order to include the `builtin` property - modification of pulsar-admin to fetch the available buildin functions - the related documentation This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.

…pache#6895) Master Issue: apache#6003 ### Motivation This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). The builtin function must include a `pulsar-io.yml` file with the following content ```yml name: <function-name> description: <function-desciption> functionClass: <function-class> ``` e.g. ```yml name: test-function description: test function description functionClass: it.oncode.pulsar.functions.TestFunction ``` it is possible to create a builtin function in the same way of the builtin sinks/sources. Example in scala ```scala val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder() val function = functionConfigBuilder .tenant("public") .namespace("default") .jar("builtin://test-function") .name("test-function-name") .className("it.oncode.pulsar.functions.TestFunction") .inputs(Seq("channel_in").asJava) .output("channel_out") .runtime(FunctionConfig.Runtime.JAVA) .build() Pulsar.admin.functions .createFunction(function, null) ``` Function folder to be specified in the `conf/functions_worker.yml` conf file e.g. `functionsDirectory: ./functions` Function package must be in `*.nar` format like for source/sink connectors ### Modifications I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation. Also `Function.proto` has been modified in order to include the `builtin` property #### What this MR does not include - modification of pulsar-admin to fetch the available buildin functions - the related documentation This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.

nicolo-paganin force-pushed the master branch 2 times, most recently from b96197a to 71b3819 Compare May 6, 2020 14:40

sijie requested review from jerrypeng, srkukarni, codelipenghui and jiazhai and removed request for jerrypeng May 7, 2020 05:55

sijie assigned oncodeit May 7, 2020

sijie added this to the 2.6.0 milestone May 7, 2020

sijie added area/function type/feature The PR added a new feature or issue requested a new feature labels May 7, 2020

sijie approved these changes May 18, 2020

View reviewed changes

nicolo-paganin force-pushed the master branch from 71b3819 to 8879a44 Compare May 19, 2020 08:00

sijie mentioned this pull request May 20, 2020

[discussion] Pulsar release 2.6.0 #5819

Closed

nicolo-paganin force-pushed the master branch from 743f64b to 70616e8 Compare May 20, 2020 22:58

nicolo-paganin force-pushed the master branch 5 times, most recently from 69e3046 to c261f2b Compare May 22, 2020 15:49

codelipenghui approved these changes May 25, 2020

View reviewed changes

nicolo-paganin force-pushed the master branch from c261f2b to 8abbe73 Compare May 27, 2020 07:18

nicolo-paganin force-pushed the master branch from 8abbe73 to 9a5b819 Compare June 2, 2020 22:56

nicolopaganin added 2 commits June 4, 2020 18:30

Possibility to add builtin Pulsar Functions with a shared jar copied …

2fa6309

…inside the function folder (apache#6003)

Added to Localrunner the possibility to execute builtin functions

ac756a3

nicolo-paganin force-pushed the master branch from 9a5b819 to ac756a3 Compare June 4, 2020 16:31

codelipenghui merged commit 9b0098a into apache:master Jun 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#6003][pulsar-functions] Possibility to add builtin Functions #6895

[#6003][pulsar-functions] Possibility to add builtin Functions #6895

oncodeit commented May 6, 2020 •

edited

Loading

sijie left a comment

sijie commented May 18, 2020

nicolo-paganin commented May 19, 2020

codelipenghui commented May 20, 2020

nicolo-paganin commented May 21, 2020

nicolo-paganin commented May 22, 2020

nicolo-paganin commented May 22, 2020

nicolo-paganin commented May 29, 2020

codelipenghui commented Jun 4, 2020

nicolo-paganin commented Jun 4, 2020

[#6003][pulsar-functions] Possibility to add builtin Functions #6895

[#6003][pulsar-functions] Possibility to add builtin Functions #6895

Conversation

oncodeit commented May 6, 2020 • edited Loading

Motivation

Modifications

What this MR does not include

Documentation

sijie left a comment

Choose a reason for hiding this comment

sijie commented May 18, 2020

nicolo-paganin commented May 19, 2020

codelipenghui commented May 20, 2020

nicolo-paganin commented May 21, 2020

nicolo-paganin commented May 22, 2020

nicolo-paganin commented May 22, 2020

nicolo-paganin commented May 29, 2020

codelipenghui commented Jun 4, 2020

nicolo-paganin commented Jun 4, 2020

oncodeit commented May 6, 2020 •

edited

Loading