Add system tables functionality #14

cmathiesen · 2020-07-01T10:48:41Z

This PR is based on feedback from the IF PR.

Added a generic "MetadataReader" that can be used to create an Iterable of Records for any of the metadata table types (snapshots, history, partitions)
^This uses the table schema to convert StaticRows from the metadata table into Record's that the InputFormat RecordReader can interpret correctly using the new ObjectInspectors
^This means we can get rid of the SnapshotIterable class entirely
I've added the tests for the Snapshots table and the History table for now

(Also apologies about the formatting changes, I'll revert those)

# Conflicts: # mr/src/main/java/org/apache/iceberg/mr/IcebergInputFormat.java # mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormat.java

on CP

…stream-wip-pr Prepare for upstream WIP PR

…ctors' into if-all-the-things

ExpediaGroup#12)

massdosage

I think this looks pretty good

massdosage · 2020-07-02T11:05:08Z

mr/src/main/java/org/apache/iceberg/mr/IcebergRecordReader.java

-import org.apache.iceberg.FileScanTask;
-import org.apache.iceberg.Schema;
-import org.apache.iceberg.StructLike;
+import org.apache.iceberg.*;


You need to run the whole build as I'm sure the checkstyle plugin will moan about this.

massdosage · 2020-07-02T11:06:52Z

mr/src/main/java/org/apache/iceberg/mr/IcebergRecordReader.java

      default:
        throw new UnsupportedOperationException(
-            String.format("Cannot read %s file: %s", file.format().name(), file.path()));
+                String.format("Cannot read %s file: %s", file.format().name(), file.path()));


The previous indentation here is more what Iceberg uses.

massdosage · 2020-07-02T11:07:05Z

mr/src/main/java/org/apache/iceberg/mr/IcebergRecordReader.java

-        .filter(task.residual())
-        .caseSensitive(caseSensitive)
-        .split(task.start(), task.length());
+            .read(inputFile)


Ditto for all the indentation changes here and below.

massdosage · 2020-07-02T11:10:19Z

mr/src/test/java/org/apache/iceberg/mr/mapred/TestReadMetadataTables.java

+    List<Object[]> result = shell.executeStatement("SELECT * FROM source_db.table_a");

    assertEquals(3, result.size());
  }


Can we remove the @ignore annotations from the tests below?

massdosage · 2020-07-02T11:11:37Z

mr/src/test/java/org/apache/iceberg/mr/mapred/TestReadMetadataTables.java

+            .toString());

-    List<Object[]> result = shell.executeStatement("SELECT * FROM source_db.table_a__snapshots");
+    List<Object[]> result = shell.executeStatement("SELECT * FROM source_db.table_a");


Is there anything we can verify to check that it actually read a history table rather than just a normal table? Are there some fields that are unique to the history who's values we can check?

massdosage · 2020-07-02T11:12:14Z

mr/src/test/java/org/apache/iceberg/mr/mapred/TestReadMetadataTables.java

+
+    List<Object[]> result = shell.executeStatement("SELECT * FROM source_db.table_a");
+
+    assertEquals(3, result.size());


Is there anything we can verify to check that it actually read a snapshot table rather than just a normal table? Are there some fields that are unique to the snapshot who's values we can check?

massdosage

Merging this branch into an EG branch so it can be worked on from within this org

rdsr and others added 30 commits March 15, 2020 17:27

InputFormat support for Iceberg

f8dc2b0

Address review comments

5f5ffbd

Address review comments

1d07d8c

added mapred inputformat

b48fb17

added mapred inputformat

e38f88b

Merge remote-tracking branch 'rdsr/mr_generic' into mr_generic_with_hive

3fef146

# Conflicts: # mr/src/main/java/org/apache/iceberg/mr/IcebergInputFormat.java # mr/src/test/java/org/apache/iceberg/mr/TestIcebergInputFormat.java

Merge branch 'master' into mr_generic_with_hive

ab5a65f

move hive runner to version that matches Hive 2.3.6 used here

274adbb

checkstyle fixes

ad612ad

added test table data

1bb76bf

Shading modules needed for mapred api

03f375a

Fix checkstyle issues

d7dfb0e

backing up the crazy - need to find out how to get hive-exec:core back

07307ec

on CP

only hive-exec core on mr classpath now

7398753

brutal attempt at overriding guava version

e3c26de

Shade all the guava

5d728e5

Fix the guava

fff5e0e

Nuke jackson dependencies

ee213ad

remove test method

a0d07a7

tidy up checkstyle

11296c3

remove classes from mapreduce inputformat branch

69cef2b

revert baseline plugin version (no idea why it was failing earlier)

a08c087

re-enable error-prone plugin

d4cff1c

tidy up dependency scopes

72a04a8

merge master back in

078a06d

merge master

28ba743

fix build after merge

d57037a

trim down to non-hive related classes

f135b78

tidy up, add tests, incorporate some code from upstream

190fb37

Merge pull request ExpediaGroup#4 from ExpediaGroup/tidy-up-before-up…

99a566a

…stream-wip-pr Prepare for upstream WIP PR

cmathiesen and others added 24 commits June 15, 2020 15:20

Remove system tables code

00dcf76

Add type conversion test

cb764b9

Removing old todo comment

0f0b39d

Timestamps in microseconds

a6eb6c7

Remove system tables

abaafb5

revert whitespace changes

c5b6cd8

Refactor IcebergObjectInspector and implement custom object inspectors

8bbcf7b

merged storage handler

5a587d7

get build passing

ec21a33

tidy up and test fixes/ignore

85b6823

fixing most tests

71bf913

merge proposed changes from guilload--implement-object-inspectors

b7611fb

Use field ids rather names to lookup fields

b88f1a8

Implement getStructFieldRef with ObjectInspectorUtils

44ce31d

Fix checkstyle

3b18a5a

Merge remote-tracking branch 'origin/guilload--implement-object-inspe…

ada1449

…ctors' into if-all-the-things

Refactor IcebergObjectInspector and implement custom object inspectors (

f7c5c39

ExpediaGroup#12)

Merge branch 'master' into iceberg-serde

6421ed6

fix compiler errors

38e4339

tidied up and fleshed out tests

b3a0cfe

merge master

e9cfdc9

merge iceberg-serde branch

75547a2

Add metadata reader method

037c928

StaticRow to Record conversion

4f83820

cmathiesen requested review from massdosage and teabot July 1, 2020 10:48

massdosage suggested changes Jul 2, 2020

View reviewed changes

massdosage changed the base branch from if-all-the-things to system-tables November 19, 2020 15:52

massdosage approved these changes Nov 19, 2020

View reviewed changes

massdosage merged commit 66b44f4 into ExpediaGroup:system-tables Nov 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add system tables functionality #14

Add system tables functionality #14

Uh oh!

cmathiesen commented Jul 1, 2020

Uh oh!

massdosage left a comment

Uh oh!

massdosage Jul 2, 2020

Uh oh!

massdosage Jul 2, 2020

Uh oh!

massdosage Jul 2, 2020

Uh oh!

massdosage Jul 2, 2020

Uh oh!

massdosage Jul 2, 2020

Uh oh!

massdosage Jul 2, 2020

Uh oh!

massdosage left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		List<Object[]> result = shell.executeStatement("SELECT * FROM source_db.table_a");

		assertEquals(3, result.size());

Add system tables functionality #14

Add system tables functionality #14

Uh oh!

Conversation

cmathiesen commented Jul 1, 2020

Uh oh!

massdosage left a comment

Choose a reason for hiding this comment

Uh oh!

massdosage Jul 2, 2020

Choose a reason for hiding this comment

Uh oh!

massdosage Jul 2, 2020

Choose a reason for hiding this comment

Uh oh!

massdosage Jul 2, 2020

Choose a reason for hiding this comment

Uh oh!

massdosage Jul 2, 2020

Choose a reason for hiding this comment

Uh oh!

massdosage Jul 2, 2020

Choose a reason for hiding this comment

Uh oh!

massdosage Jul 2, 2020

Choose a reason for hiding this comment

Uh oh!

massdosage left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants