[HUDI-9705] Fix bugs in spark and avro reader contexts for type promotion and field renaming #13714

the-other-tim-brown · 2025-08-12T21:23:14Z

Change Logs

This PR fixes a few issues discovered while trying to move the Copy-on-Write path to use the FileGroupReader for reading base files and merging with incoming records. The issues mainly stem from schema evolution cases.

Cases fixed:

Spark reader was not properly rewriting the record in some type promotion scenarios so the validations are updated
The avro reader was not forcing a rewrite for some cases, requiring the validation to be updated to account for these evolutions
The avro reader was not passing in the renamed columns to the transform

Impact

Unblocks moving the writer path to reuse the same reader paths we use elsewhere in the code

Risk level (write none, low medium or high below)

Low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

...scala/org/apache/spark/sql/execution/datasources/parquet/HoodieParquetFileFormatHelper.scala

yihua · 2025-08-12T21:55:50Z

hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroReaderContext.java

+      fileOutputSchema = dataSchema;
+      renamedColumns = Collections.emptyMap();


Note to myself: the FileGroupRecordBuffer handles the schema-on-read evolution with composeEvolvedSchemaTransformer for log blocks. Only parquet log blocks requires calling readerContext.getFileRecordIterator before schema-on-read evolution is applied in FileGroupRecordBuffer, thus no need to handle schema-on-read in this case.

hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroReaderContext.java

hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java

hudi-common/src/test/java/org/apache/hudi/common/table/read/SchemaHandlerTestBase.java

hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieAvroFileReader.java

yihua

LGTM overall

danny0405 · 2025-08-13T00:58:18Z

hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java

      case DOUBLE:
        // To maintain precision, you need to convert Float -> String -> Double
-        return writerSchema.getType().equals(Schema.Type.FLOAT);
+        return writerSchema.getType().equals(Schema.Type.FLOAT) && !writerSchema.getType().equals(Schema.Type.STRING);


how could a type equals FLOAT and also STRING ?

If we can get this PR in, I think the areSchemasProjectionEquivalent is going to fit the needs of this and has some better testing. I will wait to see if this can be brought into a mergable shape.

yeah, that PR is landed.

danny0405 · 2025-08-13T01:09:49Z

hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupReaderSchemaHandler.java

+      return Pair.of(requiredSchema, Collections.emptyMap());
+    }
+    long commitInstantTime = Long.parseLong(FSUtils.getCommitTime(path.getName()));
+    InternalSchema fileSchema = InternalSchemaCache.searchSchemaAndCache(commitInstantTime, metaClient);


seems not right, the search happens in file split level, this would trigger the metaClient metadata file listing for every file slice read. Can we reuse the cache somewhere and shared by all the readers?

Is there any example of how to do this? I noticed that this is how it is currently done in the merge path. This path will at least cache per JVM. There are some other cases where I see calls to InternalSchemaCache.getInternalSchemaByVersionId but that skips the cache entirely so the commit metadata is parsed per file.

Looks like we already did this in FileGroupRecordBuffer.composeEvolvedSchemaTransformer, and we have optimized the logic in #13525 to get rid of the timeline listing, should be good now.

Yes, the existing logic of schema evolution on read in other places follows the same code logic, so this is OK in the sense that it brings feature parity and does not introduce regression.

I think what makes more sense is to have a schema history (schemas for range of completion/instant time, e.g., schema1: ts1-ts100, schema2: ts101-ts1000, etc.) constructed on driver and distribute that to executors. This schema history can be stored under .hoodie so one file read gets the whole schema history and executor does not pay cost of scanning commit metadata or reading schema from file (assuming that the file schema is based on the writer/table schema of the commit). This essentially needs a new schema system / abstraction, which is under the scope of RFC-88 @danny0405

yes, we have a plan to re-impl the schema evolution based on new schema abstraction in 1.2 release.

hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupReaderSchemaHandler.java

yihua

LGTM

yihua · 2025-08-13T06:46:55Z

@the-other-tim-brown you can decide whether the schema utils newly available on master can be reused before merging this PR.

This reverts commit 145cb8e.

hudi-bot · 2025-08-13T20:46:19Z

CI report:

4cb2fe5 UNKNOWN
377ee69 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

yihua

LGTM

…tion and field renaming (apache#13714)

github-actions bot added the size:M PR with lines of changes in (100, 300] label Aug 12, 2025