[refactor][function][PIP-166] Function add MANUAL delivery semantics #16279

shibd · 2022-06-29T08:52:03Z

Motivation

#15560

Modifications

This PR includes java, go, and python instance.
Mark autoAck will be deprecated.
Add MANUAL processing guarantees.
Add when autoAck == false and processingGuarantees == ATMOST_ONCE verification.
Refactor delivery semantics of windowsFunction, refer: PIP[Implementation][4]
Change user docs.
For go and python instance, add processingGuarantees not support EFFECTIVELY_ONCE validate.

Verifying this change

This change added tests and can be verified as follows:

FunctionConfigUtils cover autoAck == fase and ATMOST_ONCE scenarios.
SinkConfigUtils cover autoAck == fase and ATMOST_ONCE scenarios.
FunctionConfigUtils cover WindowsFunctionConfig will set processingGuarantees.
ContextImplTest and PulsarFunctionRecordTest cover when processingGuarantees != MANUAL, call record.ack() ignore.
JavaInstanceRunnableTest cover when function result == null, scenarios in which ack messages are required under various configurations.
WindowsFunctionExecutorTest cover windows function support processing guarantees scenarios.
Other unit tests have not changed, compatible with the current version implementation.

Documentation

doc
(Your PR contains doc changes)

shibd · 2022-07-01T09:28:54Z

@asafm @nlu90 @freeznet @eolivelli @codelipenghui Please help to review this, I am coding on go and python function instances, and will open a new PR. Thank you

shibd · 2022-07-04T07:41:02Z

@asafm @nlu90 @freeznet @eolivelli @codelipenghui Please help to review this, I am coding on go and python function instances, and will open a new PR. Thank you

Modifications of the go and python instances have been submitted to this PR.

...ctions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java

...ons/instance/src/main/java/org/apache/pulsar/functions/windowing/WindowFunctionExecutor.java

pulsar-function-go/conf/conf.go

...unctions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarFunctionRecord.java

pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/FunctionConfigUtils.java

pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/sink/PulsarSinkTest.java

...ions/instance/src/test/java/org/apache/pulsar/functions/source/PulsarFunctionRecordTest.java

pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/FunctionConfigUtils.java

...functions/utils/src/test/java/org/apache/pulsar/functions/utils/FunctionConfigUtilsTest.java

pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/SinkConfigUtilsTest.java

shibd · 2022-07-05T00:58:25Z

/pulsarbot run-failure-checks

codelipenghui · 2022-07-08T02:37:57Z

pulsar-function-go/pf/instanceConf_test.go

+	assert.Panics(t, func() {
+		newInstanceConfWithConf(&cfg.Conf{ProcessingGuarantees: 0, AutoACK: false})
+	}, "Should have a panic")
+	assert.Panics(t, func() {
+		newInstanceConfWithConf(&cfg.Conf{ProcessingGuarantees: 1, AutoACK: false})
+	}, "Should have a panic")
+	assert.Panics(t, func() {
+		newInstanceConfWithConf(&cfg.Conf{ProcessingGuarantees: 2})
+	}, "Should have a panic")


Does this will introduce break change? If users upgrade to the new version but with the old configs.

It's more like a bug fix, In go instance, it does not support exactly-once. It was not checked at startup before, which will cause the function to not ack any message. python instance has the same problem.

shibd · 2022-07-08T07:23:07Z

@nahguam Thanks for your review, all suggestions are fixed, please look again. Thanks.

freeznet

Nice work, @shibd, LGTM.

pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/FunctionConfigUtils.java

site2/docs/functions-concepts.md

…tions/windowing/WindowFunctionExecutor.java Co-authored-by: Dave Maughan <[email protected]>

Co-authored-by: Dave Maughan <[email protected]>

…tions/source/PulsarFunctionRecord.java Co-authored-by: Dave Maughan <[email protected]>

…ns/utils/FunctionConfigUtils.java Co-authored-by: Dave Maughan <[email protected]>

Co-authored-by: Dave Maughan <[email protected]>

shibd · 2022-07-30T03:07:43Z

/pulsarbot run-failure-checks

shibd · 2022-07-31T03:32:04Z

/pulsarbot run-failure-checks

shibd · 2022-07-31T06:25:59Z

/pulsarbot run-failure-checks

shibd · 2022-07-31T08:01:57Z

/pulsarbot run-failure-checks

…pache#16279)

github-actions bot added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Jun 29, 2022

shibd force-pushed the pip-166 branch from a1adef1 to c5d6163 Compare June 29, 2022 10:41

shibd changed the title ~~Implement java runtime instance~~ [refactor][function][PIP-166] Function add MANUAL delivery semantics Jun 29, 2022

shibd force-pushed the pip-166 branch 2 times, most recently from 8e23f7e to 3f4dcbb Compare July 1, 2022 07:44

shibd marked this pull request as ready for review July 1, 2022 09:23

nahguam reviewed Jul 4, 2022

View reviewed changes

shibd force-pushed the pip-166 branch from 1c25dcd to 1c38de0 Compare July 5, 2022 02:35

codelipenghui requested review from freeznet and nlu90 July 8, 2022 02:33

codelipenghui assigned shibd Jul 8, 2022

codelipenghui added this to the 2.11.0 milestone Jul 8, 2022

codelipenghui added type/feature The PR added a new feature or issue requested a new feature area/function labels Jul 8, 2022

codelipenghui reviewed Jul 8, 2022

View reviewed changes

shibd force-pushed the pip-166 branch 2 times, most recently from 8c75395 to 5d6ce24 Compare July 22, 2022 02:28

codelipenghui modified the milestones: 2.11.0, 2.12.0 Jul 26, 2022

shibd force-pushed the pip-166 branch from 5d6ce24 to 91019f6 Compare July 27, 2022 15:11

freeznet approved these changes Jul 29, 2022

View reviewed changes

pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/FunctionConfigUtils.java Show resolved Hide resolved

site2/docs/functions-concepts.md Outdated Show resolved Hide resolved

shibd added 3 commits July 29, 2022 16:33

[PIP-166] Function add MANUAL delivery semantics

6a81c5a

Change docs.

548ccef

fix unit test

d8231ee

shibd and others added 14 commits July 29, 2022 16:33

Change python and go instance.

f6a68b7

Update pulsar-functions/instance/src/main/java/org/apache/pulsar/func…

9e68f1f

…tions/windowing/WindowFunctionExecutor.java Co-authored-by: Dave Maughan <[email protected]>

Update pulsar-function-go/conf/conf.go

d9e5af0

Co-authored-by: Dave Maughan <[email protected]>

Update pulsar-functions/instance/src/main/java/org/apache/pulsar/func…

413e47f

…tions/source/PulsarFunctionRecord.java Co-authored-by: Dave Maughan <[email protected]>

Update pulsar-functions/utils/src/main/java/org/apache/pulsar/functio…

9ae69ee

…ns/utils/FunctionConfigUtils.java Co-authored-by: Dave Maughan <[email protected]>

Apply suggestions from code review

3835535

Co-authored-by: Dave Maughan <[email protected]>

Update pulsar-function-go/pf/instanceConf.go

7e8c421

Co-authored-by: Dave Maughan <[email protected]>

Code format

f5151c1

Fix code style

bead847

Fix unit test

68eaa41

Add go unit test.

f0e265e

Fix go code style

c022c2b

Set SinkConfig.autoAck to deprecated.

3539668

Add WindowsFunction does not support MANUAL.

efe4804

shibd force-pushed the pip-166 branch from 91019f6 to efe4804 Compare July 29, 2022 08:33

Change code.

0846a74

nahguam approved these changes Jul 30, 2022

View reviewed changes

Delete protected method modify.

1429448

codelipenghui merged commit 5529301 into apache:master Aug 1, 2022

freeznet mentioned this pull request Aug 3, 2022

support MANUAL delivery semantics in Pulsar Functions streamnative/function-mesh#440

Open

Gleiphir2769 pushed a commit to Gleiphir2769/pulsar that referenced this pull request Aug 4, 2022

[refactor][function][PIP-166] Function add MANUAL delivery semantics (a…

ce66727

…pache#16279)

momo-jun mentioned this pull request Aug 5, 2022

[Doc] add explanations for function (autoack and nack) #13051

Closed

momo-jun added a commit to momo-jun/pulsar that referenced this pull request Aug 5, 2022

Fix minor issues introduced by apache#16279

9b0b0bd

momo-jun mentioned this pull request Aug 5, 2022

[improve][doc] Functions doc improvements Phase2 #16172

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[refactor][function][PIP-166] Function add MANUAL delivery semantics #16279

[refactor][function][PIP-166] Function add MANUAL delivery semantics #16279

shibd commented Jun 29, 2022 •

edited

Loading

shibd commented Jul 1, 2022

shibd commented Jul 4, 2022 •

edited

Loading

shibd commented Jul 5, 2022

codelipenghui Jul 8, 2022

shibd Jul 8, 2022

shibd commented Jul 8, 2022

freeznet left a comment

shibd commented Jul 30, 2022

shibd commented Jul 31, 2022

shibd commented Jul 31, 2022

shibd commented Jul 31, 2022

[refactor][function][PIP-166] Function add MANUAL delivery semantics #16279

[refactor][function][PIP-166] Function add MANUAL delivery semantics #16279

Conversation

shibd commented Jun 29, 2022 • edited Loading

Motivation

Modifications

Verifying this change

Documentation

shibd commented Jul 1, 2022

shibd commented Jul 4, 2022 • edited Loading

shibd commented Jul 5, 2022

codelipenghui Jul 8, 2022

Choose a reason for hiding this comment

shibd Jul 8, 2022

Choose a reason for hiding this comment

shibd commented Jul 8, 2022

freeznet left a comment

Choose a reason for hiding this comment

shibd commented Jul 30, 2022

shibd commented Jul 31, 2022

shibd commented Jul 31, 2022

shibd commented Jul 31, 2022

shibd commented Jun 29, 2022 •

edited

Loading

shibd commented Jul 4, 2022 •

edited

Loading