update - #4
Merged
Merged
Conversation
This patch adds logging of topic config overrides during creation or during the handling of alter config requests. Also did some minor cleanup to avoid redundant validation logic when adding partitions. Author: Jason Gustafson <jason@confluent.io> Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> Closes #5812 from hachikuji/minor-log-topic-creation-configs
…res (#5800) Reviewers: Ismael Juma <ismael@juma.me.uk>
Author: Jason Gustafson <jason@confluent.io> Reviewers: Dong Lin <lindong28@gmail.com> Closes #5817 from hachikuji/add-zstandard-upgrade-notes
…g down ReplicaFetcherThread After KAFKA-6051, we close leaderEndPoint in replica fetcher thread initiateShutdown to try to preempt in-progress fetch request and accelerate repica fetcher thread shutdown. However, leaderEndpoint can throw an Exception when the replica fetcher thread is still actively fetching, which can cause ReplicaManager to fail to shutdown cleanly. This PR catches the exceptions thrown in "leaderEndpoint.close()" instead of letting it throw up in the call stack. Author: Zhanxiang (Patrick) Huang <hzxa21@hotmail.com> Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Dong Lin <lindong28@gmail.com> Closes #5808 from hzxa21/KAFKA-7464
Reviewers: Stanislav Kozlovski <stanislav_kozlovski@outlook.com>, Ismael Juma <ismael@juma.me.uk>
Decrease the lower bound for expected available memory, as thread scheduling entails that a variable amount of deallocation happens by the point of assertion. Also make minor clarifications to test logic and comments. The passing rate improved from 98% to 100% locally after these changes (100+ runs). Reviewers: Ismael Juma <ismael@juma.me.uk> ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes)
Just a doc change Author: John Eismeier <john.eismeier@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #4573 from jeis2497052/trunk
Author: Bibin Sebastian <bisebastian@DELC02QP51SG8WN.sea.corp.expecn.com> Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Matthias J. Sax <mjsax@apache.org>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #5572 from bibinss/release_mail
…sage too large error on expired batch (#5807) Minor clean-ups for clarity included. Reviewers: Dong Lin <lindong28@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Make sure that the transaction state is properly cleared when the `transactionalId-expiration` task fails. Operations on that transactional id would otherwise return a `CONCURRENT_TRANSACTIONS` error and appear "untouchable" to transaction state changes, preventing transactional producers from operating until a broker restart or transaction coordinator change. Unit tested by verifying that having the `transactionalId-expiration` task won't leave the transaction metadata in a pending state if the replica manager returns an error. Reviewers: Jason Gustafson <jason@confluent.io>
…e value (#5809) Reviewers: Guozhang Wang <guozhang@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, John Roesler <john@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Corrects an error in the system tests: ``` 07:55:45 [ERROR:2018-10-23 07:55:45,738]: Failed to import kafkatest.tests.connect.connect_test, which may indicate a broken test that cannot be loaded: NameError: name 'EXTERNAL_CONFIGS_FILE' is not defined ``` The constant is defined in the [services/connect.py](https://github.com/apache/kafka/blob/trunk/tests/kafkatest/services/connect.py#L43) file in the `ConnectServiceBase` class, but the problem is in the [tests/connect/connect_test.py](https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/connect/connect_test.py#L50) `ConnectStandaloneFileTest`, which does *not* extend the `ConnectServiceBase class`. Suggestions welcome to be able to reuse that variable without duplicating the literal (as in this PR). System test run with this PR: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2004/ If approved, this should be merged as far back as the `2.0` branch. Author: Randall Hauch <rhauch@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #5832 from rhauch/fix-connect-externals-tests
`ControllerIntegrationTest#waitUntilControllerEpoch` sometimes fails with the following error: ``` java.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:347) at scala.None$.get(Option.scala:345) at kafka.controller.ControllerIntegrationTest$$anonfun$waitUntilControllerEpoch$1.apply$mcZ$sp(ControllerIntegrationTest.scala:312) at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:779) at kafka.controller.ControllerIntegrationTest.waitUntilControllerEpoch(ControllerIntegrationTest.scala:312) at kafka.controller.ControllerIntegrationTest.testEmptyCluster(ControllerIntegrationTest.scala:51) ``` We should retry until the value is defined or it times out. Reviewers: Ismael Juma <ismael@juma.me.uk>
…-371) Author: Manikumar Reddy <manikumar.reddy@gmail.com> Reviewers: Sriharsha Chintalapani <sriharsha@apache.org> Closes #5684 from omkreddy/KAFKA-5462-SSL-Name
…el is read_committed FetchResponse should return the partitionData's lastStabeleOffset Author: lambdaliu <lambdaliu@tencent.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Dhruvil Shah <dhruvil@confluent.io>, Dong Lin <lindong28@gmail.com> Closes #5835 from lambdaliu/KAFKA-7535
Author: Viktor Somogyi <viktorsomogyi@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Andras Katona <41361962+akatona84@users.noreply.github.com>, Dong Lin <lindong28@gmail.com> Closes #5685 from viktorsomogyi/upgrade-notes-for-serializer-consolidation
Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Dong Lin <lindong28@gmail.com> Closes #5781 from ijuma/simplify-producer-constructor
This line prints out (when empty): ``` [2018-10-23 12:19:59,977] INFO [Controller id=0] Removed ArrayBuffer() from list of shutting down brokers. (kafka.controller.KafkaController) ``` Use `mkString` to eliminate `ArrayBuffer` and only log if not empty. Reviewers: Ismael Juma <ismael@juma.me.uk>
Dead code is confusing. Reviewers: Colin Patrick McCabe <colin@cmccabe.xyz>
…IP-368) (#5582) KIP-368 implementation to enable periodic re-authentication of SASL clients. Also adds a broker configuration option to terminate client connections that do not re-authenticate within the configured interval.
… from closing (#5833) Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>
…#5826) Set `StreamsConfig.STATED_DIR_CONFIG` in `SuppressScenarioTest`, as with `StreamsTestUtils`. I have deliberately avoided using `StreamsTestUtils` as this test sets bogus config parameters, but still fails if the default `STATE_DIR_CONFIG` does not exist. Reviewers: Colin Patrick McCabe <colin@cmccabe.xyz>, John Roesler <john@confluent.io>, Ismael Juma <ismael@juma.me.uk>
`testMarksPartitionsAsOfflineAndPopulatesUncleanableMetrics` sometimes fails because the 15 second timeout expires. Inspecting the error message from the build failure, we see that this timeout happens in the writeDups() calls which call roll(). ```text [2018-10-23 15:18:51,018] ERROR Error while flushing log for log-1 in dir /tmp/kafka-8190355063195903574 with offset 74 (kafka.server.LogDirFailureChannel:76) java.nio.channels.ClosedByInterruptException ... at kafka.log.Log.roll(Log.scala:1550) ... at kafka.log.AbstractLogCleanerIntegrationTest.writeDups(AbstractLogCleanerIntegrationTest.scala:132) ... ``` After investigating, I saw that this test would call Log#roll() around 60 times every run. Increasing the segmentSize config to `2048` reduces the number of Log#roll() calls while ensuring that there are multiple rolls still. I saw that most other LogCleaner tests also call roll() ~90 times, so I've changed the default to be `2048`. I've also made the one test which requires a smaller segmentSize to set it via the args. Reviewers: Ismael Juma <ismael@juma.me.uk>
Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>
In the `consume` method, the consumer subscribes the topic, so no need to do the same thing before the method call. Also include minor clean-up in `consume`. Reviewers: Ismael Juma <ismael@juma.me.uk>
KAFKA-7597: Add configurable transaction support to ProduceBenchWorker. In order to get support for serializing Optional<> types to JSON, add a new library: jackson-datatype-jdk8. Once Jackson 3 comes out, this library will not be needed. Reviewers: Colin McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>
Add the final batch of metrics from KIP-328 Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
This is a new system test testing for optimizing an existing topology. This test takes the following steps 1. Start a Kafka Streams application that uses a selectKey then performs 3 groupByKey() operations and 1 join creating four repartition topics 2. Verify all instances start and process data 3. Stop all instances and verify stopped 4. For each stopped instance update the config for TOPOLOGY_OPTIMIZATION to all then restart the instance and verify the instance has started successfully also verifying Kafka Streams reduced the number of repartition topics from 4 to 1 5. Verify that each instance is processing data from the aggregation, reduce, and join operation Stop all instances and verify the shut down is complete. 6. For testing I ran two passes of the system test with 25 repeats for a total of 50 test runs. All test runs passed Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
…they are needed (#5696) * KAFKA-7367: Ensure stateless topologies don't require disk access * KAFKA-7367: Streams should not create state store directories unless they are needed. * Addressed the review comments. * Addressed the review-2 comments. * Fixed FileAlreadyExistsException * Addressed the review-3 comments. * Resolved the conflicts.
…arnings (#5943) See spotbugs/spotbugs#756 for details on the false positives affecting try with resources. An example is: > RCN | Nullcheck of fc at line 629 of value previously dereferenced in > org.apache.kafka.common.utils.Utils.readFileAsString(String, Charset) Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
The restart logic for TTLs in `WorkerConfigTransformer` was broken when trying to make it toggle-able. Accessing the toggle through the `Herder` causes the same code to be called recursively. This fix just accesses the toggle by simply looking in the properties map that is passed to `WorkerConfigTransformer`. Author: Robert Yokota <rayokota@gmail.com> Reviewers: Magesh Nandakumar <magesh.n.kumar@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #5914 from rayokota/KAFKA-7620
My top 2 reasons for visiting the Kafka docs are to: - View configurations - View metrics This PR aims to improve the user experience for viewing metrics: - Add href links to the `Monitoring` section of the Table of Contents so users do not need to scroll or Ctrl/Cmd-F to find specific metric details (Monitoring section has grown large as more component & metrics are added) Author: lu.kevin@berkeley.edu <kelu@paypal.com> Reviewers: Viktor Somogyi <viktorsomogyi@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com> Closes #5511 from KevinLiLu/feature/minor-improve-docs
Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> Closes #5193 from vahidhashemian/KAFKA-7037
…ers (#5957) Reivewers: Colin McCabe <cmccabe@apache.org>
#5959) This PR fixes an issue reported from a user. When we join a KStream with a GlobalKTable we should not reset the repartition flag as the stream may have previously changed its key, and the resulting stream could be used in an aggregation operation or join with another stream which may require a repartition for correct results. I've added a test which fails without the fix. Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Some deserializer needs the topic name to be able to correctly deserialize the payload of the message. Console consumer works great with Deserializer<String> however it calls deserializer with topic set as null. This breaks the API and the topic information is available in the ConsumerRecord. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, Gardner Vickers <gardner@vickers.me>, Jun Rao <junrao@gmail.com>
Current code will fall into non-stop loop and send more message to broker, and Stat in PerfCallback method record will throw ArrayIndexOutOfBoundsException
In StreamsMetricsImpl, the parentSensors map was keeping references to Sensors after the sensors themselves had been removed. Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Gradle 5.0 was released on 26 November. See the release notes for enhancements and fixes: https://docs.gradle.org/5.0/release-notes.html. A notable bugfix ensures that Javadoc artifacts are cleaned between builds. The upgraded wrapper was spot-checked against the commands in the README and the README was updated not to use removed system property `-Dtest.single`. Reviewers: Ismael Juma <ismael@juma.me.uk>
A heap dump provided by Patrik Kleindl in https://issues.apache.org/jira/browse/KAFKA-7660 identifies the childrenSensors map in Metrics as keeping references to sensors alive after they have been removed. This PR fixes it and adds a test to be sure. Reviewers: Jason Gustafson <jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
- Remove ZKUtils usage from various tests Author: Manikumar Reddy <manikumar.reddy@gmail.com> Reviewers: Sriharsha Chintalapani <sriharsha@apache.org>, Ismael Juma <ismael@juma.me.uk>, Satish Duggana <satishd@apache.org>, Jun Rao <junrao@gmail.com>, Ryanne Dolan <ryannedolan@gmail.com> Closes #5480 from omkreddy/zkutils
EndToEndLatency tool produces a dummy record in case the topic does not exist. This behavior was introduced in this PR #5319 as part of updating the tool to use latest consumer API. However, if we run the tool with producer acks == 1, the high watermark may not be updated before we reset consumer offsets to latest. In rare cases when this happens, the tool will throw an exception in the for loop where the consumer will unexpectedly consume the dummy record. As a result, we occasionally see Benchmark.test_end_to_end_latency system test failures. This PR checks if topic exists, and creates the topic using AdminClient if it does not exist. Author: Anna Povzner <anna@confluent.io> Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #5950 from apovzner/fix-EndToEndLatency
Includes Update to ConnectRecord string representation to give visibility into schemas, useful in SMT tracing Author: Cyrus Vafadari <cyrus@confluent.io> Reviewers: Randall Hauch <rhauch@gmail.com>, Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #5860 from cyrusv/cyrus-logging
This is minor refactoring that brings in the creation of producer and consumer to the Worker. Currently, the consumer is created in the WorkerSinkTask. This should not affect any functionality and it just makes the code structure easier to understand. Author: Magesh Nandakumar <magesh.n.kumar@gmail.com> Reviewers: Ryanne Dolan <ryannedolan@gmail.com>, Randall Hauch <rhauch@gmail.com>, Robert Yokota <rayokota@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #5842 from mageshn/KAFKA-7551
This is a security page improvement that adds documentation about Kafka authorization primitives to the security page. Author: Viktor Somogyi-Vass <viktorsomogyi@gmail.com> Author: Viktor Somogyi <viktorsomogyi@gmail.com> Author: Manikumar Reddy <manikumar.reddy@gmail.com> Reviewers: Stanislav Kozlovski <stanislav_kozlovski@outlook.com>, Srinivas <srinivas96alluri@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Mickael Maison <mickael.maison@gmail.com> Closes #5906 from viktorsomogyi/security-page-improvement
Reviewers: Bill Bejeck <bill@confluent.io>, Kamal Chandraprakash (@kamalcph), Guozhang Wang <guozhang@confluent.io>
Reviewers: Andras Katona <41361962+akatona84@users.noreply.github.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
This change adds some basic system tests for delegation token based authentication: - basic delegation token creation - producing with a delegation token - consuming with a delegation token - expiring a delegation token - producing with an expired delegation token New files: - delegation_tokens.py: a wrapper around kafka-delegation-tokens.sh - executed in container where a secure Broker is running (taking advantage of automatic cleanup) - delegation_tokens_test.py: basic test to validate the lifecycle of a delegation token Changes were made in the following file to extend their functionality: - config_property was updated to be able to configure Kafka brokers with delegation token related settings - jaas.conf template because a broker needs to support multiple login modules when delegation tokens are used - consule-consumer and verifiable_producer to override KAFKA_OPTS (to specify custom jaas.conf) and the client properties (to authenticate with delegation token). Author: Attila Sasvari <asasvari@apache.org> Reviewers: Reviewers: Viktor Somogyi <viktorsomogyi@gmail.com>, Andras Katona <41361962+akatona84@users.noreply.github.com>, Manikumar Reddy <manikumar.reddy@gmail.com> Closes #5660 from asasvari/KAFKA-4544
…g broker generation (#5821) * KAFKA-7235: Detect outdated control requests and bounced brokers using broker generation * Add broker_epoch in controlled shutdown request * Move broker epoch check into controller for ControlledShutdownRequest * Refactor schema definition for controler requests/responses * Address comments * Address comments * Address comments * Send back STALE_BROKER_EPOCH error in ControlledShutdown response * Fix build issue * Address comments * Address comments * Address comments * Address comments * Fix tests after rebase * Address comments * Address comments
Fix test Comparators plus Java8 cleanup Reviewers: Guozhang Wang <wangguoz@gmail.com>
This is a system test for doing a rolling upgrade of a topology with a named repartition topic. 1. An initial Kafka Streams application is started on 3 nodes. The topology has one operation forcing a repartition and the repartition topic is explicitly named. 2. Each node is started and processing of data is validated 3. Then one node is stopped (full stop is verified) 4. A property is set signaling the node to add operations to the topology before the repartition node which forces a renumbering of all operators (except repartition node) 5. Restart the node and confirm processing records 6. Repeat the steps for the other 2 nodes completing the rolling upgrade I ran two runs of the system test with 25 repeats in each run for a total of 50 test runs. All test runs passed Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Changes made as part of this commit. - Improved error message for better readability at millis validation utility - Corrected java documentation on `AdvanceInterval` check. - Added caller specific prefix text to make error message more clear to developers/users. Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Jacek Laskowski <jacek@japila.pl>
…#5994) Reviewers: Jun Rao <junrao@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)