Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
f8dc2b0
InputFormat support for Iceberg
rdsr Mar 3, 2020
5f5ffbd
Address review comments
rdsr Mar 18, 2020
1d07d8c
Address review comments
rdsr Mar 18, 2020
b48fb17
added mapred inputformat
massdosage Mar 18, 2020
e38f88b
added mapred inputformat
massdosage Mar 18, 2020
3fef146
Merge remote-tracking branch 'rdsr/mr_generic' into mr_generic_with_hive
massdosage Mar 18, 2020
ab5a65f
Merge branch 'master' into mr_generic_with_hive
massdosage Mar 19, 2020
274adbb
move hive runner to version that matches Hive 2.3.6 used here
massdosage Mar 19, 2020
ad612ad
checkstyle fixes
massdosage Mar 19, 2020
1bb76bf
added test table data
massdosage Mar 19, 2020
03f375a
Shading modules needed for mapred api
cmathiesen Mar 25, 2020
d7dfb0e
Fix checkstyle issues
cmathiesen Mar 25, 2020
07307ec
backing up the crazy - need to find out how to get hive-exec:core back
massdosage Mar 26, 2020
7398753
only hive-exec core on mr classpath now
massdosage Mar 26, 2020
e3c26de
brutal attempt at overriding guava version
massdosage Mar 26, 2020
5d728e5
Shade all the guava
cmathiesen Mar 27, 2020
fff5e0e
Fix the guava
cmathiesen Mar 30, 2020
ee213ad
Nuke jackson dependencies
cmathiesen Mar 30, 2020
a0d07a7
remove test method
massdosage Mar 30, 2020
11296c3
tidy up checkstyle
massdosage Mar 30, 2020
69cef2b
remove classes from mapreduce inputformat branch
massdosage Mar 30, 2020
a08c087
revert baseline plugin version (no idea why it was failing earlier)
massdosage Mar 30, 2020
d4cff1c
re-enable error-prone plugin
massdosage Mar 30, 2020
72a04a8
tidy up dependency scopes
massdosage Mar 30, 2020
078a06d
merge master back in
massdosage Mar 30, 2020
28ba743
merge master
massdosage Apr 8, 2020
d57037a
fix build after merge
massdosage Apr 8, 2020
f135b78
trim down to non-hive related classes
massdosage Apr 9, 2020
190fb37
tidy up, add tests, incorporate some code from upstream
massdosage Apr 15, 2020
99a566a
Merge pull request #4 from ExpediaGroup/tidy-up-before-upstream-wip-pr
massdosage Apr 15, 2020
2989591
Merge branch 'master' into mapred-input-format
massdosage Apr 16, 2020
dc8f6a1
revert public access
massdosage Apr 16, 2020
ce4e88c
Fix mapred serialization bug (#6)
cmathiesen Apr 20, 2020
a013e34
remove test data
massdosage Apr 24, 2020
6d24d41
generate test data in tests
massdosage Apr 24, 2020
ce76c36
fix missed static call
massdosage Apr 24, 2020
d75d683
Merge branch 'master' into mapred-input-format
massdosage Apr 27, 2020
0d25ea6
- refactor tests common to both input formats
massdosage May 4, 2020
3695c08
removed UncheckedIOException try/catch
massdosage May 4, 2020
ab3082e
Merge branch 'master' into mapred-input-format
massdosage May 4, 2020
ad27b35
first cut at refactoring duplicate code between InputFormats
massdosage May 4, 2020
9250253
remove test data (tests now create their own)
massdosage May 5, 2020
f94401a
refactor (mostly common) findTable and tableScan code
massdosage May 5, 2020
1338676
put some of the generic templates back
massdosage May 5, 2020
b5a389c
orc tests appear to be working
massdosage May 6, 2020
632a4cc
Merge branch 'master' into mapred-input-format
massdosage Jun 3, 2020
e1b81dc
added a HiveRunner test for the mapred InputFormat
massdosage Jun 4, 2020
f6c5108
tidy up
massdosage Jun 4, 2020
a6c2b19
exclude pentaho
massdosage Jun 5, 2020
937e228
wip checkpoint
massdosage Jun 8, 2020
382b3bb
debugging failing tests
massdosage Jun 8, 2020
9843691
fix tests
massdosage Jun 8, 2020
7570d06
tidy up
massdosage Jun 8, 2020
2fce735
Add SnapshotIterable
cmathiesen Jun 10, 2020
52a6a17
Clean up FilterFactory
cmathiesen Jun 12, 2020
d2eee30
Convert Hive types
cmathiesen Jun 15, 2020
00dcf76
Remove system tables code
cmathiesen Jun 15, 2020
cb764b9
Add type conversion test
cmathiesen Jun 15, 2020
0f0b39d
Removing old todo comment
cmathiesen Jun 15, 2020
a6eb6c7
Timestamps in microseconds
cmathiesen Jun 16, 2020
81b933c
merge master
massdosage Jul 2, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ subprojects {
all {
exclude group: 'org.slf4j', module: 'slf4j-log4j12'
exclude group: 'org.mortbay.jetty'
exclude group: 'org.pentaho', module: 'pentaho-aggdesigner-algorithm'

resolutionStrategy {
force 'com.fasterxml.jackson.module:jackson-module-scala_2.11:2.10.2'
Expand Down Expand Up @@ -342,6 +343,44 @@ project(':iceberg-mr') {
}
compileOnly("org.apache.hive:hive-serde")

compileOnly("org.apache.hive:hive-exec::core") {
exclude group: 'org.apache.avro', module: 'avro'
exclude group: 'org.slf4j', module: 'slf4j-log4j12'
exclude group: 'org.pentaho' // missing dependency
exclude group: 'org.apache.hive', module: 'hive-llap-tez'
exclude group: 'org.apache.logging.log4j'
exclude group: 'com.google.protobuf', module: 'protobuf-java'
exclude group: 'org.apache.calcite'
exclude group: 'org.apache.calcite.avatica'
exclude group: 'com.google.code.findbugs', module: 'jsr305'
exclude group: 'com.google.guava'
}

compileOnly "org.apache.hive:hive-metastore"
compileOnly "org.apache.hive:hive-serde"

testCompile("com.klarna:hiverunner:5.2.1") {
exclude group: 'javax.jms', module: 'jms'
exclude group: 'org.apache.hive', module: 'hive-exec'
exclude group: 'org.codehaus.jettison', module: 'jettison'
exclude group: 'org.apache.calcite.avatica'
}

testCompile("org.apache.hive:hive-exec::core") {
exclude group: 'org.apache.avro', module: 'avro'
exclude group: 'org.slf4j', module: 'slf4j-log4j12'
exclude group: 'org.pentaho' // missing dependency
exclude group: 'org.apache.hive', module: 'hive-llap-tez'
exclude group: 'org.apache.logging.log4j'
exclude group: 'com.google.protobuf', module: 'protobuf-java'
exclude group: 'org.apache.calcite.avatica'
exclude group: 'com.google.code.findbugs', module: 'jsr305'
}

testCompile("org.apache.calcite:calcite-core")
testCompile("com.esotericsoftware.kryo:kryo:2.24.0")
testCompile("com.fasterxml.jackson.core:jackson-annotations:2.6.5")

testCompile project(path: ':iceberg-data', configuration: 'testArtifacts')
testCompile project(path: ':iceberg-api', configuration: 'testArtifacts')
testCompile project(path: ':iceberg-core', configuration: 'testArtifacts')
Expand Down
Loading