KAFKA-18026: KIP-1112 migrate KTableSuppressProcessorSupplier #18150
Conversation
| final ProcessorGraphNode<K, Change<V>> node = new TableSuppressNode<>( | ||
| name, | ||
| new ProcessorParameters<>(suppressionSupplier, name), | ||
| StoreBuilderWrapper.wrapStoreBuilder(storeBuilder) | ||
| new String[]{storeName} |
There was a problem hiding this comment.
Is this intentional? The callee seems not using the storeName below: https://github.com/apache/kafka/pull/18150/files#diff-c1133b80fb36ee7e4fbf7a26007f63e41a411e7f36e45aa96373afd9ebdc5559R22
There was a problem hiding this comment.
it's definitely not being used in the PR, but actually we do still need it so it should/will be used (see https://github.com/apache/kafka/pull/18150/files#r1885482579)
streams/src/main/java/org/apache/kafka/streams/kstream/internals/graph/TableSuppressNode.java
Show resolved
Hide resolved
| super(nodeName, processorParameters); | ||
|
|
||
| this.storeNames = storeNames; | ||
| this.storeNames = null; |
There was a problem hiding this comment.
I don't think we can change this yet since not all StatefulProcessorNode users have been migrated yet. For example the FKJ. Pretty sure this is what's causing the test failures.
There was a problem hiding this comment.
Mm yeah almost definitely the problem:
Caused by: org.apache.kafka.streams.errors.StreamsException: Processor KTABLE-FK-JOIN-SUBSCRIPTION-PROCESSOR-0000000022 has no access to StateStore KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-0000000021 as the store is not connected to the processor. If you add stores manually via '.addStateStore()' make sure to connect the added store to the processor by providing the processor name to '.addStateStore()' or connect them via '.connectProcessorAndStateStores()'. DSL users need to provide the store name to '.process()', '.transform()', or '.transformValues()' to connect the store to the corresponding operator, or they can provide a StoreBuilder by implementing the stores() method on the Supplier itself. If you do not add stores manually, please file a bug report at https://issues.apache.org/jira/projects/KAFKA.
basically setting this to null skips the #connectProcessorAndStateStore call, and not every operator that uses this constructor (including children of StatefulProcessorNode) have been migrated to implementing stores (which will result in the store being connected during the ProcessorParameters#addProcessorTo call)
|
failure is due to size limit on build so unrelated, merging to trunk |
…e-old-protocol-versions * apache-github/trunk: (25 commits) KAFKA-18270: FindCoordinator v0 incorrectly tagged as deprecated (apache#18262) KAFKA-18284: Add group coordinator records for Streams rebalance protocol (apache#18228) MINOR: Fix flaky state updater test (apache#18253) MINOR: improve StreamsResetter logging (apache#18237) KAFKA-18227: Ensure v2 partitions are not added to last transaction during upgrade (apache#18176) Add IT for share consumer with duration base offet auto reset (apache#18251) KAFKA-18283: Add StreamsGroupDescribe RPC definitions (apache#18230) KAFKA-18241: add docs check to CI (apache#18183) KAFKA-18223 Improve flaky test report (apache#18212) MINOR Remove triage label in nightly job (apache#18147) KAFKA-18294 Remove deprecated SourceTask#commitRecord (apache#18260) KAFKA-18264 Remove NotLeaderForPartitionException (apache#18211) KAFKA-13722: Refactor SerdeGetter (apache#18242) KAFKA-18094 Remove deprecated TopicListing(String, Boolean) (apache#18248) KAFKA-18282: Add StreamsGroupHeartbeat RPC definitions (apache#18227) KAFKA-18026: KIP-1112 migrate KTableSuppressProcessorSupplier (apache#18150) KAFKA-18026: transition KTable#filter impl to use processor wrapper (apache#18205) KAFKA-18293 Remove `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler` and `org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler` (apache#18244) MINOR: add assertion about groupEpoch and targetAssignmentEpoch to testConsumerGroups (apache#18203) KAFKA-17960; PlaintextAdminIntegrationTest.testConsumerGroups fails with CONSUMER group protocol (apache#18234) ...
…#18150) Migrates KTableSuppressProcessorSupplier to use the the ProcessorSupplier#stores() method Reviewers: Guozhang Wang <[email protected]>, Anna Sophie Blee-Goldman <[email protected]>
Migrates KTableSuppressProcessorSupplier to use the the ProcessorSupplier#stores() method Reviewers: Guozhang Wang <[email protected]>, Anna Sophie Blee-Goldman <[email protected]>
|
Forgot to cherrypick this earlier -- cherrypicked to 4.0 |
Migrates
KTableSuppressProcessorSupplierto use the theProcessorSupplier#stores()method