KAFKA-15179: Add integration tests for the file sink and source connectors#14279
Conversation
C0urante
left a comment
C0urante
left a comment
There was a problem hiding this comment.
Thanks Yash! Looking pretty good, had a few thoughts. Hoping most should be fairly straightforward
…ime for green path
…urce and sink connectors
…one in the 'verifyLinesInFile' method
yashmayya
left a comment
yashmayya
left a comment
There was a problem hiding this comment.
Thanks for the review Chris!
| * @param numLines the expected number of lines in the file | ||
| * @param verifyLinearity true if the line contents are to be verified | ||
| */ | ||
| private void verifyLinesInFile(Path filePath, int numLines, boolean verifyLinearity) throws Exception { |
There was a problem hiding this comment.
Thanks @yashmayya - we could also check the number of times a message is expected to be present in the file.
There was a problem hiding this comment.
Hm, I intentionally kept the check looser to make it a little more generic and allow for covering all the scenarios. In the alter offsets case for instance, only the last message would be written twice and all the others would only be written once.
There was a problem hiding this comment.
I think the checks here, while fairly loose, are sufficient. We have pretty granular unit testing coverage for the sink and source task classes. It's not impossible to add the same level of granularity to these tests but it'd be fairly complex to do it right (i.e., handle offset resets and consumption from multiple topics). I'm fine with erring on the side of simplicity for now and, if necessary, hardening these tests in the future.
|
Thanks Chris! |
…ctors (apache#14279) Reviewers: Ashwin Pankaj <apankaj@confluent.io>, Chris Egerton <chrise@aiven.io>
3 participants
FileStreamSinkConnectorandFileStreamSourceConnectorto test some complex interactions between the Connect runtime and the connector / task implementations.Committer Checklist (excluded from commit message)