HUDI-1283 Fill missing columns with default value when spark dataframe save to hudi table

[ivorzhou]

Fill missing normal column (except key) with schema default value when data write is upsert and
payload is OverwriteNonDefaultsWithLatestAvroPayload

Given (id PK ts precombied KEY dt partition KEY)

id  ,    name,    age,     ts,         dt
1    ,  Jack    ,   10   ,     1  ,          20191212
2    ,  Tom    ,    11   ,     1   ,         20191213
3    ,    Bill    ,    12    ,    1    ,        20191212

Upsert with following data (name column value missing)

id   ,    age,     ts,         dt
1     ,    22  ,      2   ,        20191212

Result

id  ,    name,    age,     ts,         dt
1   ,    Jack  ,     22 ,       2,            20191212
2   ,    Tom  ,      11 ,       1 ,           20191213
3   ,     Bill   ,      12 ,       1 ,           20191212

[umehrot2]

Thank you for creating this PR. At this point, I am not fully convinced if we really need this logic. A missing column in the DataFrame could also mean that column has been dropped, although Hudi schema evolution does not really support dropping of fields at this point of time. But if in future, if we are planning to support something like that then this would contradict with it.

While the logic LGTM, lets get a second opinion. @vinothchandar @bvaradar thoughts on this use-case ?

[ivorzhou]

Thank you for creating this PR. At this point, I am not fully convinced if we really need this logic. A missing column in the DataFrame could also mean that column has been dropped, although Hudi schema evolution does not really support dropping of fields at this point of time. But if in future, if we are planning to support something like that then this would contradict with it.

While the logic LGTM, lets get a second opinion. @vinothchandar @bvaradar thoughts on this use-case ?

Thank you for your reviewing, I think sometimes a missing column does not mean dropped ,but ignored when update specified columns. Like update SQL, sometimes I only want to update one column , keep others unchanged.

[codecov-io]

Codecov Report

Merging #2091 (1ba5f9c) into master (78fd122) will increase coverage by 0.07%.
The diff coverage is 81.48%.

@@             Coverage Diff              @@
##             master    #2091      +/-   ##
============================================
+ Coverage     53.45%   53.52%   +0.07%     
- Complexity     2781     2782       +1     
============================================
  Files           354      354              
  Lines         16158    16180      +22     
  Branches       1648     1653       +5     
============================================
+ Hits           8637     8661      +24     
+ Misses         6821     6817       -4     
- Partials        700      702       +2     

Flag	Coverage Δ	Complexity Δ
hudicli	38.50% <ø> (ø)	0.00 <ø> (ø)
hudiclient	100.00% <ø> (ø)	0.00 <ø> (ø)
hudicommon	55.13% <ø> (+0.05%)	0.00 <ø> (ø)
hudihadoopmr	32.94% <ø> (ø)	0.00 <ø> (ø)
hudispark	66.04% <81.48%> (+0.31%)	0.00 <0.00> (ø)
huditimelineservice	65.30% <ø> (ø)	0.00 <ø> (ø)
hudiutilities	70.06% <ø> (ø)	0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ	Complexity Δ
...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala	53.97% <81.48%> (+2.66%)	0.00 <0.00> (ø)
...e/hudi/common/table/log/HoodieLogFormatWriter.java	78.81% <0.00%> (+1.69%)	23.00% <0.00%> (ø%)
...ache/hudi/common/fs/inline/InMemoryFileSystem.java	89.65% <0.00%> (+10.34%)	16.00% <0.00%> (+1.00%)

[umehrot2]

Thank you for creating this PR. At this point, I am not fully convinced if we really need this logic. A missing column in the DataFrame could also mean that column has been dropped, although Hudi schema evolution does not really support dropping of fields at this point of time. But if in future, if we are planning to support something like that then this would contradict with it.
While the logic LGTM, lets get a second opinion. @vinothchandar @bvaradar thoughts on this use-case ?

Thank you for your reviewing, I think sometimes a missing column does not mean dropped ,but ignored when update specified columns. Like update SQL, sometimes I only want to update one column , keep others unchanged.

@vinothchandar @bvaradar your thoughts here ?

[nsivabalan]

@ivorzhou : is the requirement to set default value or value from previous version of the record? if previous version of the record, then guess we already have another PR for this #2106

[ivorzhou]

@ivorzhou : is the requirement to set default value or value from previous version of the record? if previous version of the record, then guess we already have another PR for this #2106

I want to update record with specified field like SQL update.
UPDATE col='value' where id=1

[nsivabalan]

May be we can guard it by a config. So only interested users can leverage it and default behavior is untouched.
@umehrot2 @vinothchandar @bvaradar

[nsivabalan]

@vinothchandar @n3nash : any inputs appreciated. this is just waiting on your pointers.
btw @ivorzhou : In the mean time, a high level comment. not sure if you have handled this case. what if some of the missing columns don't have any default value set?

[nsivabalan]

@ivorzhou : can you please clarify if the requirement is to fill w/ default values for every column or use values from previous version if not present in incoming record?

[nsivabalan]

@ivorzhou : if your requirement is to fetch value from previous version of the record and inject into incoming record, we already have another PR open #2666. Feel to clarify and close this PR out if the requirement is same.

[nsivabalan]

#2927 puts in a fix, where in if you incoming data has subset of columns compared to table schema, it will populate default vals and proceed. Closing this PR in favor of the other. Please feel free to reopen if requirement is different or create a new one.