KAFKA-4914: Partition re-assignment tool should check types before pe… by nicktrav · Pull Request #2708 · apache/kafka

nicktrav · 2017-03-19T21:38:43Z

…rsisting state in ZooKeeper

Prior to this, there have been instances where invalid data was allowed to be persisted in ZooKeeper, which causes ClassCastExceptions when a broker is restarted and reads this type-unsafe data.

Adds basic structural and type validation for the reassignment JSON via introduction of Scala case classes that map to the expected JSON structure.

asfbot · 2017-03-20T00:40:27Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2269/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-03-20T00:41:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2266/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-03-20T01:10:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2266/
Test PASSed (JDK 8 and Scala 2.12).

nicktrav · 2017-03-20T17:35:13Z

Hey @ijuma - would you mind taking a look? I see you've worked on this previously.

nicktrav · 2017-04-14T19:45:07Z

@ewencp @ijuma - any chance of a review on this one?

asfbot · 2017-04-14T20:36:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2952/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-04-14T20:39:33Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2947/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-04-14T21:18:06Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2948/
Test PASSed (JDK 8 and Scala 2.12).

omkreddy · 2017-07-19T15:32:06Z

@nicktrav Thanks for the patch. LGTM. We also faced this issue in production. Can you rebase the PR?

asfgit · 2017-07-19T15:33:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6169/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · 2017-07-19T15:33:04Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6153/
Test FAILed (JDK 8 and Scala 2.12).

nicktrav · 2017-07-22T05:16:57Z

@omkreddy - thanks for checking this one out! I completely forgot that I had this open. I've rebased.

Would love to get it merged, as it was the source of a reasonable amount of (avoidable) pain for us a few months ago, and would love to save some others from that. Are you able to merge this in when green?

asfgit · 2017-07-22T06:10:48Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6281/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-07-22T06:35:37Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6265/
Test PASSed (JDK 8 and Scala 2.12).

omkreddy · 2017-07-24T11:07:22Z

LGTM.

Though we are deprecating "--zookeeper" option in KIP-179 (Change ReassignPartitionsCommand to use adminClient), It is good to have this check.

cc @ijuma pinging for review.

ijuma · 2017-08-04T11:55:35Z

Thanks for the PR. I think we should consider using Jackson to parse into classes with the expected structure. That will do the necessary validation in a more robust way. Thoughts?

asfgit · 2017-08-04T12:48:17Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6565/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · 2017-08-04T13:11:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6550/
Test FAILed (JDK 8 and Scala 2.12).

nicktrav · 2017-08-04T15:38:31Z

Sounds reasonable, @ijuma. I'll take another pass at it.

nicktrav · 2017-08-05T21:24:33Z

@ijuma - I've taken a different approach, as suggested, using case classes to hold the deserialized json. PTAL.

cc: @omkreddy

asfgit · 2017-08-05T22:23:06Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6593/
Test PASSed (JDK 7 and Scala 2.11).

asfgit · 2017-08-05T22:43:22Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6578/
Test PASSed (JDK 8 and Scala 2.12).

ijuma · 2017-08-08T22:25:42Z

Thanks, I'll take a look tomorrow.

nicktrav · 2017-08-22T18:17:09Z

@ijuma - sorry to keep pinging you on this. How is this looking? Happy to address any feedback you might have.

nicktrav · 2017-09-07T16:06:16Z

Ping @omkreddy

omkreddy · 2017-09-08T06:47:38Z

@nicktrav Thanks for your patience. committers are busy with 0.11.0.1 release and other tasks. I am sure this will get merged into 1.0.0/October release. Pls, rebase the PR when you get a chance.

cc @ijuma

nicktrav · 2017-09-18T02:33:14Z

Retest this please.

ijuma

Sorry for the delay @nicktrav. I left a few comments.

ijuma · 2018-01-05T10:39:32Z

The result type can be Either[JsonProcessingException, T].

ijuma · 2018-01-05T10:43:08Z

Field name should be partition

ijuma · 2018-01-05T10:44:01Z

Seems like we'd want to include the original exception as the cause. Also, why is it a ConfigException?

ijuma · 2018-01-05T10:44:46Z

We should be using this in ReassignPartitionsZNode as in my PR.

ijuma · 2018-01-05T10:45:15Z

We should use ReassignPartitionsZNode.decode

ijuma · 2018-01-05T10:46:56Z

We should rewrite these tests to use ReassignPartitionsZNode. Also, since they don't require ZooKeeper, they should be unit tests. Perhaps ReassignPartitionsZNodeTest.

nicktrav · 2018-01-05T18:34:11Z

Thanks for the review @ijuma. I pushed a new commit addressing your last round of comments. I'm sure some of it is not idiomatic Scala, so would appreciate some guidance, if any.

omkreddy · 2018-01-08T07:19:50Z

Can we use zkUtils.parsePartitionReassignmentData here
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala#L245
above change will solve KAFKA-6084. This can be done in another PR.

ijuma · 2018-01-08T09:33:11Z

@omkreddy Ideally, we should not be using ZkUtils in ReassignPartitionsCommand (i.e. we should use KafkaZkClient). Also, ReassignPartitionsCommand also parses a log_dirs field which is not handled in ZkUtils (or KafkaZkClient). Let's do that clean-up in a separate PR.

nicktrav · 2018-01-11T16:19:25Z

@omkreddy @ijuma - any action for me to take here based on the last two comments?

omkreddy · 2018-01-11T17:04:47Z

LGTM. +1

ijuma · 2018-01-18T15:07:55Z

Sorry, I have just moved countries and won't have time until next week.

nicktrav · 2018-01-18T16:40:10Z

No worries at all. Good luck with the move!

nicktrav · 2018-03-20T02:58:40Z

Any chance of an update on this @ijuma @omkreddy? (coincidentally, today is the 1 year birthday of the ticket! 🍰)

…rsisting state in ZooKeeper Prior to this, there have been instances where invalid data was allowed to be persisted in ZooKeeper, which causes ClassCastExceptions when a broker is restarted and reads this type-unsafe data. Adds basic structural and type validation for the reassignment JSON via introduction of Scala case classes that map to the expected JSON structure.

omkreddy · 2018-03-20T12:26:36Z

maybe @hachikuji can help to merge this patch.

viktorsomogyi

Although (as mentioned on my JIRA) this PR seems to aim bit different goals then mine, I reviewed this PR and made one minor comment/question.
All in all I think the PR looks good. Would be happy to see this merged.

viktorsomogyi · 2018-03-20T17:07:44Z

+   * Parse a JSON string into either a generic type T, or a JsonProcessingException in the case of
+    * exception.
+   */
+  def parseStringAs[T](input: String)(implicit tag: ClassTag[T]): Either[JsonProcessingException, T] = {


Hmm, just a very basic question: why are you using here Either/Left/Right instead of Try/Success/Failure?
As from my previous experiences we used Try when we wanted to propagate the exception.
And please not I don't mind using Either, just asking for reasons. :)

My (idiomatic) Scala isn't great, but this came at the suggestion of @ijuma (above):
#2708 (comment)

Try doesn't let you specify an exception subclass and behaves a bit different if an exception is thrown during map, etc.

ijuma

I pushed minor changes and it now LGTM. Will merge it after the tests pass. Thanks so much for your patience.

nicktrav · 2018-03-31T21:33:45Z

Thanks @ijuma!

nicktrav force-pushed the nickt.reassign-partitions-validation branch from a5bb980 to 9cb5002 Compare July 22, 2017 05:09

nicktrav force-pushed the nickt.reassign-partitions-validation branch from 9cb5002 to 0231adf Compare August 5, 2017 21:22

nicktrav force-pushed the nickt.reassign-partitions-validation branch from 0231adf to 17fc8cf Compare September 17, 2017 17:16

ijuma reviewed Jan 5, 2018

View reviewed changes

nicktrav force-pushed the nickt.reassign-partitions-validation branch from 2954d4c to b7b210e Compare February 8, 2018 17:22

Nicholas Travers and others added 11 commits March 19, 2018 20:26

[feedback] Pattern matching

f51e498

[feedback] Relocate POJOs and update documentation

7b05f21

[feedback] JsonTest companion object

13583dc

[feedback] parseAs

5aa7f4b

Use PartitionAssignment/ReplicaAssignment in ReassignPartitionsZNode

8a1c7f2

[rebase fixups]

2f36f0b

[feedback]

ec7881b

[fixup] Encode bytes as UTF-8

fae8341

[fixup] Add missing license

5b2c54c

[fixup] Revert unnecessary newline removal

ec7f36d

nicktrav force-pushed the nickt.reassign-partitions-validation branch from b7b210e to ec7f36d Compare March 20, 2018 03:33

viktorsomogyi approved these changes Mar 20, 2018

View reviewed changes

Minor fixes

019ab70

ijuma approved these changes Mar 31, 2018

View reviewed changes

ijuma merged commit 4106cb1 into apache:trunk Mar 31, 2018

Uh oh!

Conversation

nicktrav commented Mar 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asfbot commented Mar 20, 2017

Uh oh!

asfbot commented Mar 20, 2017

Uh oh!

asfbot commented Mar 20, 2017

Uh oh!

nicktrav commented Mar 20, 2017

Uh oh!

nicktrav commented Apr 14, 2017

Uh oh!

asfbot commented Apr 14, 2017

Uh oh!

asfbot commented Apr 14, 2017

Uh oh!

asfbot commented Apr 14, 2017

Uh oh!

omkreddy commented Jul 19, 2017

Uh oh!

asfgit commented Jul 19, 2017

Uh oh!

asfgit commented Jul 19, 2017

Uh oh!

nicktrav commented Jul 22, 2017

Uh oh!

asfgit commented Jul 22, 2017

Uh oh!

asfgit commented Jul 22, 2017

Uh oh!

omkreddy commented Jul 24, 2017

Uh oh!

ijuma commented Aug 4, 2017

Uh oh!

asfgit commented Aug 4, 2017

Uh oh!

asfgit commented Aug 4, 2017

Uh oh!

nicktrav commented Aug 4, 2017

Uh oh!

nicktrav commented Aug 5, 2017

Uh oh!

asfgit commented Aug 5, 2017

Uh oh!

asfgit commented Aug 5, 2017

Uh oh!

ijuma commented Aug 8, 2017

Uh oh!

nicktrav commented Aug 22, 2017

Uh oh!

nicktrav commented Sep 7, 2017

Uh oh!

omkreddy commented Sep 8, 2017

Uh oh!

nicktrav commented Sep 18, 2017

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicktrav commented Jan 5, 2018

Uh oh!

omkreddy commented Jan 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ijuma commented Jan 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

nicktrav commented Mar 19, 2017 •

edited

Loading

omkreddy commented Jan 8, 2018 •

edited

Loading

ijuma commented Jan 8, 2018 •

edited

Loading