Skip to content

ORC column map fix #227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rdblue merged 26 commits into from
Jan 7, 2020
Merged

ORC column map fix #227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
e8e8619
Read iceberg to ORC mapping from ORC file if found
edgarRd Jun 11, 2019
c4d8017
Use the right ORC schema in SparkOrcReader
edgarRd Jun 15, 2019
9d7c4fd
Use ORC 1.6.0 TypeDescription attributes for column mapping
edgarRd Jun 25, 2019
a833102
Address some PR comments
edgarRd Oct 10, 2019
ec8bba5
Add example on ORC schema evolution handling
edgarRd Oct 10, 2019
3c84d0a
Add metadata for converting different types of binary fields
edgarRd Oct 10, 2019
e0bbee2
Verify roundtrip convertion of types
edgarRd Oct 10, 2019
028acc6
Add GenericOrcWriter implementation in iceberg-data
edgarRd Oct 27, 2019
7f86427
Update to ORC 1.6.1
edgarRd Nov 4, 2019
39b3b62
Save Game
edgarRd Nov 4, 2019
2fca5c1
Fix projection by computing last max iceberg id
edgarRd Nov 8, 2019
6d26fee
Fix duplicated classes in runtime
edgarRd Nov 8, 2019
407b2e8
Fix style check
edgarRd Nov 8, 2019
e392ffb
Make hadoop dependency compileOnly
edgarRd Nov 27, 2019
df64260
Remove unnecessary call to buildOrcProjection
edgarRd Nov 27, 2019
c7f9a80
Remove empty comment
edgarRd Nov 27, 2019
e332067
Fix typo on test column id
edgarRd Nov 27, 2019
f740509
Handle case when renaming a column and reusing previous column name
edgarRd Nov 28, 2019
dc30658
Avoid trying to project ORC columns if no Iceberg ID is found
edgarRd Nov 28, 2019
2c0687a
Revert white space changes in build.gradle
edgarRd Jan 3, 2020
0782063
Upgrade ORC to 1.6.2
edgarRd Jan 4, 2020
3e86d28
Fix typo
edgarRd Jan 4, 2020
eeb0a56
Use RuntimeIOException
edgarRd Jan 4, 2020
8378036
Using this for setting instance fields
edgarRd Jan 4, 2020
1f5668e
Add more complex schema roundtrip conversion tests
edgarRd Jan 4, 2020
37f7c8a
Split ORC schema tests for build projection tests
edgarRd Jan 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,12 @@ project(':iceberg-data') {
compile project(':iceberg-api')
compile project(':iceberg-core')
compileOnly project(':iceberg-parquet')
compileOnly project(':iceberg-orc')
compileOnly("org.apache.hadoop:hadoop-common") {
Copy link
Contributor

@rdblue rdblue Dec 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was it necessary to add this hadoop-common dependency?

Copy link
Contributor Author

@edgarRd edgarRd Jan 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since org.apache.hadoop:hadoop-common is a non-transitive dependency used at compilation time on the other projects we need to add it here as ORC requires it for using org.apache.hadoop.io.WritableComparable on HiveDecimalWritable.

Copy link
Contributor

@rdblue rdblue Jan 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just thought that hadoop-client would pull it in transitively.

exclude group: 'commons-beanutils'
exclude group: 'org.apache.avro', module: 'avro'
exclude group: 'org.slf4j', module: 'slf4j-log4j12'
}

testCompile("org.apache.hadoop:hadoop-client") {
exclude group: 'org.apache.avro', module: 'avro'
Expand Down Expand Up @@ -200,10 +206,16 @@ project(':iceberg-orc') {
exclude group: 'org.apache.hive', module: 'hive-storage-api'
}

compileOnly("org.apache.hadoop:hadoop-common") {
exclude group: 'commons-beanutils'
exclude group: 'org.apache.avro', module: 'avro'
exclude group: 'org.slf4j', module: 'slf4j-log4j12'
}
compileOnly("org.apache.hadoop:hadoop-client") {
exclude group: 'org.apache.avro', module: 'avro'
}

testCompile project(path: ':iceberg-api', configuration: 'testArtifacts')
}
}

Expand Down
Loading