Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

datafusion.catalog.has_headers default value set to true #11919

Merged
merged 1 commit into from
Aug 17, 2024

Conversation

korowa
Copy link
Contributor

@korowa korowa commented Aug 9, 2024

Which issue does this PR close?

Closes #11848.
Closes #11936.

Rationale for this change

Making datafusion.catalog.has_headers consistent with default CsvReadOptions / WriterBuilder has_header values.

What changes are included in this PR?

  • Default value for datafusion.catalog.has_header changed to True
  • Fixed catalog.has_header value inheritance in case of writing to csv files (e.g. insert statements)

Are these changes tested?

Covered by existing tests

Are there any user-facing changes?

Statements involving external CSV tables now write / try to read header row as a default behavior.

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Aug 9, 2024
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @korowa

Something we could consider is changing the default "has header" value in the configuration setting to be true as this is what I think most people would expect. I filed a ticket to propose doing so here #11936

@korowa korowa marked this pull request as draft August 12, 2024 06:13
@korowa
Copy link
Contributor Author

korowa commented Aug 12, 2024

It's in draft since I'm going to check if changing default value of datafusion.catalog.has_header will go smooth enough, and this PR can be replaced with default value change.

@github-actions github-actions bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Aug 15, 2024
@korowa korowa changed the title docs: fix csv headers in cli example datafusion.catalog.has_headers default value set to true Aug 16, 2024
@korowa korowa marked this pull request as ready for review August 16, 2024 04:35
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @korowa - I think this is a great step forward in usability personally.

Perhaps you can try to merge this PR yourself as a test of having the committer configuration setup correctly 🤔

@alamb alamb added the api change Changes the API exposed to users of the crate label Aug 16, 2024
@alamb
Copy link
Contributor

alamb commented Aug 16, 2024

Marking the PR as API change to get it some addtional visibility in documentation

@korowa korowa merged commit b06e8b0 into apache:main Aug 17, 2024
27 checks passed
@alamb
Copy link
Contributor

alamb commented Aug 17, 2024

🚀

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api change Changes the API exposed to users of the crate common Related to common crate core Core DataFusion crate documentation Improvements or additions to documentation sqllogictest SQL Logic Tests (.slt)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Change default value of datafusion.catalog.has_header to true CSV headers not recognized
2 participants