MINOR: fix security_test for ZK case due to error change

[rondagostino]

The ZooKeeper version of this system test is failing because the producer is no longer seeing LEADER_NOT_AVAILABLE. When the broker sees a METADATA request for the test topic after it restarts the auto topic creation manager is determining that the topic needs to be created due to the TLS hostname verification failure on the inter-broker security protocol. It also thinks there aren't enough brokers available to meet the default topic replication factor (it sees 0 available due to the TLS issue), so it returnsINVALID_REPLICATION_FACTOR for that topic in the Metadata response. In other words, the flow has changed and the inability to produce is not manifesting as it was before, and the test is failing. This patch updates the test to check for INVALID_REPLICATION_FACTOR instead of LEADER_NOT_AVAILABLE.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[rondagostino]

@abbccdda, @cmccabe. Ported from #10199 to discuss separately. We used to see this error message in verifiable_producer.log when security_protocol='PLAINTEXT', interbroker_security_protocol='SSL':

WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {test_topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

The test does a grep LEADER_NOT_AVAILABLE on the log in this case, and it used to pass.

Now we are instead seeing this in the log file:

WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {test_topic=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)

And of course now the test fails.

The INVALID_REPLICATION_FACTOR is coming from the auto topic creation manager as I described above.

It is a simple matter to make the test pass -- I have confirmed that it passes if we grep for INVALID_REPLICATION_FACTOR in the log file instead of LEADER_NOT_AVAILABLE.

I think we just need to decide if this change in behavior is acceptable or not.

[hachikuji]

@rondagostino Thanks for identifying the issue. Returning INVALID_REPLICATION_FACTOR seems like a mistake to me if we were previously returning LEADER_NOT_AVAILABLE. I'd suggest we fix the code. Would you mind filing a JIRA for this so that we can mark it as a 2.8 blocker?

[rondagostino]

https://issues.apache.org/jira/browse/KAFKA-12381

[rondagostino]

This PR "fixes" the issue by changing the system test to check for the new error. We can close this PR in favor of another one if @abbccdda decides the behavior change is incorrect and can be corrected.

[rondagostino]

Closed in favor of #10240