Skip to content

[HUDI-4113] Fix cannot parse <null> schema when use spark delete sql #5610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
codope merged 2 commits into from
Dec 13, 2022
Merged

[HUDI-4113] Fix cannot parse <null> schema when use spark delete sql #5610

codope merged 2 commits into from
Dec 13, 2022

Conversation

@KnightChess
Copy link
Contributor

@KnightChess KnightChess commented May 17, 2022 •
edited by codope
Loading

Change Logs

after alter table drop partition, will not add schema to Instance.

now, if use delete sql, will get latest instant to get schema, it's ""

before: replacecommit instant extraMetadata

"extraMetadata" : {
    "schema" : ""
  }

after: replacecommit instant extraMetadata

"extraMetadata" : {
    "schema" : "{\"type\":\"record\",\"name\":\"delete_error_test_record\",\"namespace\":\"hoodie.delete_error_test\",\"fields\":[{\"name\":\"_hoodie_commit_time\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_commit_seqno\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_record_key\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_partition_path\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"_hoodie_file_name\",\"type\":[\"null\",\"string\"],\"doc\":\"\",\"default\":null},{\"name\":\"id\",\"type\":[\"null\",\"int\"],\"default\":null},{\"name\":\"age\",\"type\":[\"null\",\"int\"],\"default\":null},{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"dt\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
  }

Impact

Fix cannot parse schema when use spark delete sql.

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
    ticket number here and follow the instruction to make
    changes to the website.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

nsivabalan
nsivabalan requested changes Jun 23, 2022
View reviewed changes
Copy link
Contributor

@nsivabalan nsivabalan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we write a test to validate the fix please.

...spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala
// Create a HoodieWriteClient & issue the delete.
val tableMetaClient = HoodieTableMetaClient.builder
.setConf(sparkContext.hadoopConfiguration).setBasePath(basePath.toString).build()
val schemaStr = new TableSchemaResolver(tableMetaClient).getTableAvroSchema.toString
Copy link
Contributor

@nsivabalan nsivabalan Jun 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we also fix L197.

Copy link
Contributor Author

@KnightChess KnightChess Jun 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for delete op, there has special logic in https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java#L1437

will use last instant schema

@nsivabalan
Copy link
Contributor

nsivabalan commented Jun 23, 2022

thanks for the fix!

@nsivabalan nsivabalan added priority:high Significant impact; potential bugs writer-core labels Jun 23, 2022
@nsivabalan nsivabalan added the release-0.12.2 Patches targetted for 0.12.2 label Dec 6, 2022
@codope codope changed the title [HUDI-4113] fix cannot parse <null> schema when use spark delete sql [HUDI-4113] Fix cannot parse <null> schema when use spark delete sql Dec 7, 2022
@codope codope force-pushed the fix-delete-sql-issue branch from 69987db to 5a591f8 Compare December 7, 2022 12:23
codope
codope approved these changes Dec 7, 2022
View reviewed changes
Copy link
Member

@codope codope left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. We can land once the CI is green.

@hudi-bot
Copy link
Collaborator

hudi-bot commented Dec 7, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@codope codope merged commit 5faa36f into apache:master Dec 13, 2022
nsivabalan pushed a commit that referenced this pull request Dec 14, 2022
@KnightChess @codope @nsivabalan
[HUDI-4113] Fix cannot parse <null> schema when use spark delete sql (#…
a56c00e 
…5610)

- Alter table drop partition will not add schema to instant. 
  If using delete sql, will get latest instant to get schema,
  which is "". This PR fixes the parsing of null or empty schema. 

Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
nsivabalan pushed a commit to nsivabalan/hudi that referenced this pull request Mar 22, 2023
@KnightChess @codope @nsivabalan
[HUDI-4113] Fix cannot parse <null> schema when use spark delete sql (a…
5744adb 
…pache#5610)

- Alter table drop partition will not add schema to instant. 
  If using delete sql, will get latest instant to get schema,
  which is "". This PR fixes the parsing of null or empty schema. 

Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Apr 5, 2023
@KnightChess @codope
[HUDI-4113] Fix cannot parse <null> schema when use spark delete sql (a…
781cb9d 
…pache#5610)

- Alter table drop partition will not add schema to instant. 
  If using delete sql, will get latest instant to get schema,
  which is "". This PR fixes the parsing of null or empty schema. 

Co-authored-by: Sagar Sumit <sagarsumit09@gmail.com>
@lvyanquan lvyanquan mentioned this pull request Jun 14, 2023
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nsivabalan nsivabalan nsivabalan approved these changes

@codope codope codope approved these changes

Assignees

@nsivabalan nsivabalan

Labels

priority:high Significant impact; potential bugs release-0.12.2 Patches targetted for 0.12.2

Projects

Hudi Issue Support
Status: 🚧 Needs Repro
Hudi PR Support
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@KnightChess @nsivabalan @hudi-bot @codope