-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for newlines_in_values
to CsvOptions
#11472
Comments
That is correct, DataFusion parallel CSV scan does:
So I think the first step will be to add arrow support |
I see, so an approach could be to:
|
Turns out For example, if I add this to my code before I run ctx.state_weak_ref()
.upgrade()
.unwrap()
.write_arc()
.config_mut()
.options_mut()
.execution
.target_partitions = 1; From a UX perspective, this setting is quite disconnected from the intent. It also required changing the session-level |
Thank you for the investigation @connec , if that's the case, setting |
Thanks for the pointer! That would certainly be better than disabling parallelization throughout. Do you think it would still be desirable to be able to turn off file-level partioning for CSVs only (and/or to vary it on a per-type basis)? I'm not familiar enough with Parquet to understand if there would be a functional reason to disable it for that format? |
Furthermore, I think there could be value in controlling this on a per-file basis. E.g. if you're working with multiple CSVs, one of which you know contains newlines in values and others that you know do not (and so can benefit from parallelisation). |
Yes, using the global option is only a quick fix. I think it's a good idea to add an extra option field for individual files, and later the planner can make better decisions based on per file's And it can be CSV-specific, parquet seems not relevant here. |
I've opened a PR with the most straightforward implementation of this I could think of. I'd be glad to receive any feedback on that approach. |
Is your feature request related to a problem or challenge?
I'm trying to read CSVs that include newlines in (quoted) values.
Describe the solution you'd like
Some googling revealed that this isn't supported currently by the
arrow-csv
crate, whereas that functionality does exist in the C++ (ParseOptions::newlines_in_values
) and Python (ParseOptions.newlines_in_values
) implementations.Ideally, a
newlines_in_values
field could be added todatafusion::common::config::CsvOptions
to support this functionality.Note that the Python docs call out the performance implications of this:
I haven't dug into the implementation, but I imagine it becomes harder to find the right split point for multi-threaded reading (though, it seems not dissimilar to finding the prev/next linebreak, so perhaps not insurmountable...).
Describe alternatives you've considered
The only alternative I can see would be to preprocess the CSV before feeding it into DF. I haven't explored this option as I imagine it would take a lot of DF plumbing, and it seems valuable to have parity with other arrow CSV packages (C++ and Python, at least).
Additional context
I was originally planning to report this against the
arrow-rs
repository, but since my use-case is withdatafusion
I decided to report it here. Let me know if this issue would be more appropriate there and I can move/copy it.The text was updated successfully, but these errors were encountered: