[DO NOT MERGE] Sync upstream trunk 28 Feb 2023#882
Merged
Rittika Adhikari (rittikaadhikari) merged 22 commits intoMar 21, 2023
Conversation
Details: * gradle upgrade: 7.6 -> 8.0.1 * spotbugs plugin upgrade: 5.0.9 -> 5.0.13 * tweaked the mechanics for `-release`/`-source`/`-target` to workaround idiosyncrasies in Gradle 8.0.1 and newer Scala 2.13 versions. * streams-scala `test` task no longer triggers the `spotless` task since a newer version is required for Gradle 8 support, but the newer version requires Java 11. Note: relates to apache#5479 Gradle upgrade highlights: * "Scala Incremental Compilation for Multi-Module projects broken in 7.x": gradle/gradle#20101 * "Incremental compilation of java modules is broken with Gradle 7.6": gradle/gradle#23067 Full release notes: https://docs.gradle.org/8.0/release-notes.html Reviewers: Ismael Juma <ismael@juma.me.uk>
…ionEnable (apache#13296) Reviewers: Luke Chen <showuon@gmail.com>
* Moved pausing-tasks logic out of the commit-interval loop to be on the top-level loop, similar to resuming tasks. * Added thread-level restoration metrics. * Related unit tests. Reviewers: Lucas Brutschy <lucasbru@users.noreply.github.com>, Matthias J. Sax <matthias@confluent.io>
…13252) Introduces the metered store layer for the new versioned key-value store introduced in KIP-889. This outermost, metered store layer handles all serialization/deserialization from VersionedKeyValueStore to a bytes-representation (VersionedBytesStore) so that all inner stores may operate only with bytes types. Reviewers: Matthias J. Sax <matthias@confluent.io>
… error (apache#13206) Reviewers: Rittika Adhikari <rittika.adhikari@gmail.com>, Luke Chen <showuon@gmail.com>, Satish Duggana <satishd@apache.org>, Alexandre Dupriez <alexandre.dupriez@gmail.com>, Jun Rao <junrao@gmail.com>
…autocommit requests (apache#13021) This pull request introduces a CommitRequestManager to efficiently manage commit requests from clients and the autocommit state. The manager utilizes a "staged" commit queue to store commit requests made by clients. A background thread regularly polls the CommitRequestManager, which then checks the queue for any outstanding commit requests. When permitted, the CommitRequestManager generates a PollResult which contains a list of UnsentRequests that are subsequently processed by the NetworkClientDelegate. In addition, a RequestManagerRegistry has been implemented to hold all request managers, including the new CommitRequestManager and the CoordinatorRequestManager. The registry is regularly polled by a background thread in each event loop, ensuring that all request managers are kept up to date and able to handle incoming requests Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Currently, the kafka.network:type=RequestMetrics,name=MessageConversionsTimeMs,request=Fetch will not get updated because the request metrics is recorded BEFORE the messageConversions metrics value updated. That means, even if we updated the messageConversions metrics value, the request metrics will never reflect the update. This patch fixes it by updating the request metric after callback completed, so that the messageConversions metric value can be updated correctly. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Divij Vaidya <diviv@amazon.com>
…3305) Reviewers: Divij Vaidya <diviv@amazon.com>, Chia-Ping Tsai <chia7712@gmail.com>
…e#13311) Also re-enable it in CI. We do this by adjusting the `Jenkinsfile` to use a more general task (`./gradlew check -x test`). Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Dejan Stojadinović <dejan2609@users.noreply.github.com>
- Remove unused methods - Cleanup syntax Reviewers: Josep Prat <josep.prat@aiven.io>, Christo Lolov <christololov@gmail.com>
Reviewers: Chris Egerton <fearthecellos@gmail.com>, Christo Lolov <christololov@gmail.com>
…orkerSourceTaskTest (apache#13191) Reviewers: Christo Lolov <christololov@gmail.com>, Chris Egerton <chrise@aiven.io>
…3308) Reviewers: Guozhang Wang <wangguoz@gmail.com>
Rittika Adhikari (rittikaadhikari)
requested review from
a team
as code owners
February 27, 2023 21:43
Matthew Wong (mattwong949)
approved these changes
Feb 27, 2023
Matthew Wong (mattwong949)
left a comment
Matthew Wong (mattwong949)
left a comment
There was a problem hiding this comment.
thanks Rittika Adhikari (@rittikaadhikari) , if you can add some details on the conflicts we discussed in the PR description, that would be helpful
let's wait for the tests and build too
…to fix flakey OOMEs (apache#13291) KAFKA-14742: Throttle connectors in ExactlyOnceSourceIntegrationTest to fix flakey OOMEs fixup: rename config constant, use meaningful constant names in EOSIT Signed-off-by: Greg Harris <greg.harris@aiven.io> Reviewers: Edoardo Comar <edocomar@gmail.com>
Reviewers: Chris Egerton <chrise@aiven.io>
…iltered records (apache#13193) Reviewers: Christo Lolov <christololov@gmail.com>, Chris Egerton <chrise@aiven.io>
This fixes the following `./gradlew install` issue:
```text
* What went wrong:
A problem was found with the configuration of task ':storage:srcJar' (type 'Jar').
- Gradle detected a problem with the following location: '/Users/ijuma/src/kafka/storage/src/generated/java'.
Reason: Task ':storage:srcJar' uses this output of task ':storage:processMessages' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
Possible solutions:
1. Declare task ':storage:processMessages' as an input of ':storage:srcJar'.
2. Declare an explicit dependency on ':storage:processMessages' from ':storage:srcJar' using Task#dependsOn.
3. Declare an explicit dependency on ':storage:processMessages' from ':storage:srcJar' using Task#mustRunAfter.
Please refer to https://docs.gradle.org/8.0.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
```
Reviewers: David Jacot <david.jacot@gmail.com>
Member
|
We also need apache#13316. |
…tream-trunk-27-Feb-2023 * commit 'dcc179995153c22c6248702976b60755b0b9fda8': MINOR: srcJar should depend on processMessages task (apache#13316) KAFKA-14659 source-record-write-[rate|total] metrics should exclude filtered records (apache#13193) MINOR: ExponentialBackoff Javadoc improvements (apache#13317) KAFKA-14742: Throttle connectors in ExactlyOnceSourceIntegrationTest to fix flakey OOMEs (apache#13291)
Rittika Adhikari (rittikaadhikari)
changed the title
Rittika Adhikari (rittikaadhikari)
changed the title
Rittika Adhikari (rittikaadhikari)
deleted the
sync-upstream-trunk-27-Feb-2023
branch
March 21, 2023 17:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixing conflicts in build.gradle and Jenkinsfile