Skip to content

Conversation

@nsivabalan
Copy link
Contributor

@nsivabalan nsivabalan commented Dec 25, 2020

What is the purpose of the pull request

Fixing schema compatibility check. Reader and writer schema were wrongly sent. INT to LONG schema evolution for the same field fails if not for the patch.

Check #2063 for issue raised by customer.
Reran the customer script and ensured the patch fixed the issue.

Brief change log

  • Fixed correct args for schema compatibility check.

Verify this pull request

This change added tests and can be verified as follows:

  • Added tests to TestTableSchemaEvolution to verify the change.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@nsivabalan
Copy link
Contributor Author

@n3nash @bvaradar : Can you confirm that this is a bug in our schema compatibility check. Also, please review when you can.

@codecov-io
Copy link

codecov-io commented Dec 25, 2020

Codecov Report

Merging #2381 (cd4ab14) into master (286055c) will decrease coverage by 42.16%.
The diff coverage is n/a.

Impacted file tree graph

@@              Coverage Diff              @@
##             master    #2381       +/-   ##
=============================================
- Coverage     52.20%   10.04%   -42.17%     
+ Complexity     2659       48     -2611     
=============================================
  Files           335       52      -283     
  Lines         14981     1852    -13129     
  Branches       1505      223     -1282     
=============================================
- Hits           7821      186     -7635     
+ Misses         6535     1653     -4882     
+ Partials        625       13      -612     
Flag Coverage Δ Complexity Δ
hudicli ? ?
hudiclient 100.00% <ø> (ø) 0.00 <ø> (ø)
hudicommon ? ?
hudihadoopmr ? ?
huditimelineservice ? ?
hudiutilities 10.04% <ø> (-59.62%) 0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ Complexity Δ
...va/org/apache/hudi/utilities/IdentitySplitter.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-2.00%)
...va/org/apache/hudi/utilities/schema/SchemaSet.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-3.00%)
...a/org/apache/hudi/utilities/sources/RowSource.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-4.00%)
.../org/apache/hudi/utilities/sources/AvroSource.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-1.00%)
.../org/apache/hudi/utilities/sources/JsonSource.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-1.00%)
...rg/apache/hudi/utilities/sources/CsvDFSSource.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-10.00%)
...g/apache/hudi/utilities/sources/JsonDFSSource.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-4.00%)
...apache/hudi/utilities/sources/JsonKafkaSource.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-6.00%)
...pache/hudi/utilities/sources/ParquetDFSSource.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-5.00%)
...lities/schema/SchemaProviderWithPostProcessor.java 0.00% <0.00%> (-100.00%) 0.00% <0.00%> (-4.00%)
... and 311 more

@nsivabalan nsivabalan added the priority:blocker Production down; release blocker label Dec 26, 2020
@nsivabalan
Copy link
Contributor Author

Closing in favor of #2350

@nsivabalan nsivabalan closed this Dec 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority:blocker Production down; release blocker

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants