Refactor connector to match Confluent HDFS connect 4.1.0 #21
Conversation
blootsvoets
commented
Jul 4, 2018
•
blootsvoets
commented
- Intends to fix Re-processing of records on restart because of conflicting offsets ?? #20.
- Added integration test for single topic with multiple schemas
|
Will this work if we enable it with uncompressed data still there in hdfs? |
|
The codec used for writing does not influence readers: the codec name stored in the metadata part of the file, and the reader will use that metadata to do correct decoding. |
blootsvoets
changed the title
gradle/publishing.gradle
Outdated
| pom { | ||
| name = 'radar-hdfs-sink-connector' | ||
| description = 'A HDFS Sink connector for RADAR-CNS platform.' | ||
| url = 'https://github.com/RADAR-CNS/RADAR-HDFS-Sink-Connector.git' |
There was a problem hiding this comment.
'RADAR-base' ? or githubUrl
gradle/publishing.gradle
Outdated
| pkg { | ||
| repo = 'org.radarcns' | ||
| name = 'radar-hdfs-sink-connector' | ||
| userOrg = 'radar-cns' | ||
| licenses = ['Apache-2.0'] | ||
| vcsUrl = 'https://github.com/RADAR-CNS/RADAR-HDFS-Sink-Connector.git' | ||
| vcsUrl = githubUrl |
There was a problem hiding this comment.
does this need .git suffix ? Because some cases it does not work without suffix for example cloning in travis
There was a problem hiding this comment.
If the git version is recent enough, both with and without the .git suffix will work.
| System.out.println(filePaths); | ||
| break; | ||
| } | ||
| } while (true); |
There was a problem hiding this comment.
Can we have a max retries constraint here to prevent infinite loop because of any problems
There was a problem hiding this comment.
There is a timeout on the method limiting it to two minutes.
There was a problem hiding this comment.
Ah okay, missed that. Thanks
There was a problem hiding this comment.
LGTM. Will still need to test if #20 is indeed fixed with this.
