Skip to content
This repository was archived by the owner on Jun 15, 2021. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
f8dc2b0
InputFormat support for Iceberg
rdsr Mar 3, 2020
5f5ffbd
Address review comments
rdsr Mar 18, 2020
1d07d8c
Address review comments
rdsr Mar 18, 2020
b48fb17
added mapred inputformat
massdosage Mar 18, 2020
e38f88b
added mapred inputformat
massdosage Mar 18, 2020
3fef146
Merge remote-tracking branch 'rdsr/mr_generic' into mr_generic_with_hive
massdosage Mar 18, 2020
ab5a65f
Merge branch 'master' into mr_generic_with_hive
massdosage Mar 19, 2020
274adbb
move hive runner to version that matches Hive 2.3.6 used here
massdosage Mar 19, 2020
ad612ad
checkstyle fixes
massdosage Mar 19, 2020
1bb76bf
added test table data
massdosage Mar 19, 2020
03f375a
Shading modules needed for mapred api
cmathiesen Mar 25, 2020
d7dfb0e
Fix checkstyle issues
cmathiesen Mar 25, 2020
07307ec
backing up the crazy - need to find out how to get hive-exec:core back
massdosage Mar 26, 2020
7398753
only hive-exec core on mr classpath now
massdosage Mar 26, 2020
e3c26de
brutal attempt at overriding guava version
massdosage Mar 26, 2020
5d728e5
Shade all the guava
cmathiesen Mar 27, 2020
fff5e0e
Fix the guava
cmathiesen Mar 30, 2020
ee213ad
Nuke jackson dependencies
cmathiesen Mar 30, 2020
a0d07a7
remove test method
massdosage Mar 30, 2020
11296c3
tidy up checkstyle
massdosage Mar 30, 2020
69cef2b
remove classes from mapreduce inputformat branch
massdosage Mar 30, 2020
a08c087
revert baseline plugin version (no idea why it was failing earlier)
massdosage Mar 30, 2020
d4cff1c
re-enable error-prone plugin
massdosage Mar 30, 2020
72a04a8
tidy up dependency scopes
massdosage Mar 30, 2020
078a06d
merge master back in
massdosage Mar 30, 2020
28ba743
merge master
massdosage Apr 8, 2020
d57037a
fix build after merge
massdosage Apr 8, 2020
f135b78
trim down to non-hive related classes
massdosage Apr 9, 2020
190fb37
tidy up, add tests, incorporate some code from upstream
massdosage Apr 15, 2020
99a566a
Merge pull request #4 from ExpediaGroup/tidy-up-before-upstream-wip-pr
massdosage Apr 15, 2020
2989591
Merge branch 'master' into mapred-input-format
massdosage Apr 16, 2020
dc8f6a1
revert public access
massdosage Apr 16, 2020
ce4e88c
Fix mapred serialization bug (#6)
cmathiesen Apr 20, 2020
a013e34
remove test data
massdosage Apr 24, 2020
6d24d41
generate test data in tests
massdosage Apr 24, 2020
ce76c36
fix missed static call
massdosage Apr 24, 2020
d75d683
Merge branch 'master' into mapred-input-format
massdosage Apr 27, 2020
0d25ea6
- refactor tests common to both input formats
massdosage May 4, 2020
3695c08
removed UncheckedIOException try/catch
massdosage May 4, 2020
ab3082e
Merge branch 'master' into mapred-input-format
massdosage May 4, 2020
ad27b35
first cut at refactoring duplicate code between InputFormats
massdosage May 4, 2020
9250253
remove test data (tests now create their own)
massdosage May 5, 2020
f94401a
refactor (mostly common) findTable and tableScan code
massdosage May 5, 2020
1338676
put some of the generic templates back
massdosage May 5, 2020
b5a389c
orc tests appear to be working
massdosage May 6, 2020
632a4cc
Merge branch 'master' into mapred-input-format
massdosage Jun 3, 2020
e1b81dc
added a HiveRunner test for the mapred InputFormat
massdosage Jun 4, 2020
f6c5108
tidy up
massdosage Jun 4, 2020
a6c2b19
exclude pentaho
massdosage Jun 5, 2020
a96f950
Adding serde classes
cmathiesen Jun 8, 2020
937e228
wip checkpoint
massdosage Jun 8, 2020
382b3bb
debugging failing tests
massdosage Jun 8, 2020
8ec7a4b
added some required classes
massdosage Jun 8, 2020
90311cc
Add tests
cmathiesen Jun 8, 2020
9843691
fix tests
massdosage Jun 8, 2020
7570d06
tidy up
massdosage Jun 8, 2020
c0cec17
Remove try/catch
cmathiesen Jun 9, 2020
65fe819
Add StorageHandler
cmathiesen Jun 10, 2020
2fce735
Add SnapshotIterable
cmathiesen Jun 10, 2020
e4ccfd7
Fix date/time types, address format comments
cmathiesen Jun 11, 2020
eecb833
Remove HadoopCatalog, clean up SerDe and tests
cmathiesen Jun 12, 2020
52a6a17
Clean up FilterFactory
cmathiesen Jun 12, 2020
2350025
refactored code from IcebergInputFormat into InputFormatConfig
massdosage Jun 15, 2020
d2eee30
Convert Hive types
cmathiesen Jun 15, 2020
00dcf76
Remove system tables code
cmathiesen Jun 15, 2020
cb764b9
Add type conversion test
cmathiesen Jun 15, 2020
0f0b39d
Removing old todo comment
cmathiesen Jun 15, 2020
a6eb6c7
Timestamps in microseconds
cmathiesen Jun 16, 2020
abaafb5
Remove system tables
cmathiesen Jun 17, 2020
c5b6cd8
revert whitespace changes
massdosage Jun 17, 2020
8bbcf7b
Refactor IcebergObjectInspector and implement custom object inspectors
guilload Jun 20, 2020
5a587d7
merged storage handler
massdosage Jun 22, 2020
ec21a33
get build passing
massdosage Jun 22, 2020
85b6823
tidy up and test fixes/ignore
massdosage Jun 22, 2020
71bf913
fixing most tests
massdosage Jun 24, 2020
b7611fb
merge proposed changes from guilload--implement-object-inspectors
massdosage Jun 24, 2020
b88f1a8
Use field ids rather names to lookup fields
guilload Jun 25, 2020
44ce31d
Implement getStructFieldRef with ObjectInspectorUtils
guilload Jun 25, 2020
3b18a5a
Fix checkstyle
guilload Jun 25, 2020
ada1449
Merge remote-tracking branch 'origin/guilload--implement-object-inspe…
massdosage Jun 26, 2020
f7c5c39
Refactor IcebergObjectInspector and implement custom object inspector…
guilload Jun 26, 2020
6421ed6
Merge branch 'master' into iceberg-serde
massdosage Jun 26, 2020
38e4339
fix compiler errors
massdosage Jun 26, 2020
b3a0cfe
tidied up and fleshed out tests
massdosage Jun 29, 2020
e9cfdc9
merge master
massdosage Jun 29, 2020
75547a2
merge iceberg-serde branch
massdosage Jun 29, 2020
037c928
Add metadata reader method
cmathiesen Jun 29, 2020
4f83820
StaticRow to Record conversion
cmathiesen Jun 30, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ subprojects {
all {
exclude group: 'org.slf4j', module: 'slf4j-log4j12'
exclude group: 'org.mortbay.jetty'
exclude group: 'org.pentaho', module: 'pentaho-aggdesigner-algorithm'

resolutionStrategy {
force 'com.fasterxml.jackson.module:jackson-module-scala_2.11:2.10.2'
Expand Down Expand Up @@ -338,6 +339,34 @@ project(':iceberg-mr') {
compileOnly("org.apache.hadoop:hadoop-client") {
exclude group: 'org.apache.avro', module: 'avro'
}
compileOnly("org.apache.hive:hive-serde")

compileOnly("org.apache.hive:hive-exec::core") {
//exclude group: 'org.apache.avro', module: 'avro'
exclude group: 'org.slf4j', module: 'slf4j-log4j12'
exclude group: 'org.pentaho' // missing dependency
exclude group: 'org.apache.hive', module: 'hive-llap-tez'
exclude group: 'org.apache.logging.log4j'
exclude group: 'com.google.protobuf', module: 'protobuf-java'
exclude group: 'org.apache.calcite.avatica'
exclude group: 'com.google.code.findbugs', module: 'jsr305'
exclude group: 'com.google.guava'
}

compileOnly "org.apache.hive:hive-metastore"
compileOnly "org.apache.hive:hive-serde"

testCompile("com.klarna:hiverunner:5.2.1") {
exclude group: 'javax.jms', module: 'jms'
exclude group: 'org.apache.hive', module: 'hive-exec'
exclude group: 'org.codehaus.jettison', module: 'jettison'
exclude group: 'org.apache.calcite.avatica'
}

testCompile("org.apache.avro:avro:1.9.2")
testCompile("org.apache.calcite:calcite-core")
testCompile("com.esotericsoftware:kryo-shaded:4.0.2")
testCompile("com.fasterxml.jackson.core:jackson-annotations:2.6.5")

testCompile project(path: ':iceberg-data', configuration: 'testArtifacts')
testCompile project(path: ':iceberg-api', configuration: 'testArtifacts')
Expand Down
Loading