Skip to content

add parquet-fromcsv (#1) #1798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tustvold merged 14 commits into from
Jun 10, 2022
+706 −1
Merged

add parquet-fromcsv (#1) #1798

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
cdf3657
add parquet-fromcsv (#1)
kazuk Jun 6, 2022
414c7b7
add `text` for non-rust documentation text
kazuk Jun 7, 2022
7338cc2
Update parquet/src/bin/parquet-fromcsv.rs
kazuk Jun 8, 2022
7174b2d
Update parquet/src/bin/parquet-fromcsv.rs
kazuk Jun 8, 2022
4b406be
Update parquet/src/bin/parquet-fromcsv.rs
kazuk Jun 8, 2022
6dbd7ca
Update parquet/src/bin/parquet-fromcsv.rs
kazuk Jun 8, 2022
3ccf705
automate update help text
kazuk Jun 8, 2022
d91bfde
remove anyhow
kazuk Jun 9, 2022
71d0f8f
add rat_exclude_files
kazuk Jun 10, 2022
7a1e80f
update test_command_help
kazuk Jun 10, 2022
ddeeff2
fix clippy warnings
kazuk Jun 10, 2022
ff77901
Merge branch 'master' of https://github.com/apache/arrow-rs into pr/k…
kazuk Jun 10, 2022
ce3cefd
add writer-version, max-row-group-size arg
kazuk Jun 10, 2022
312a596
fix cargo fmt lint
kazuk Jun 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion parquet/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ all-features = true

[features]
default = ["arrow", "snap", "brotli", "flate2", "lz4", "zstd", "base64"]
cli = ["serde_json", "base64", "clap"]
cli = ["serde_json", "base64", "clap", "arrow/csv"]
test_common = []
# Experimental, unstable functionality primarily used for testing
experimental = []
Expand All @@ -85,6 +85,10 @@ required-features = ["cli"]
name = "parquet-rowcount"
required-features = ["cli"]

[[bin]]
name = "parquet-fromcsv"
required-features = ["cli"]

[[bench]]
name = "arrow_writer"
harness = false
Expand Down
61 changes: 61 additions & 0 deletions parquet/src/bin/parquet-fromcsv-help.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
parquet 15.0.0
Comment thread
kazuk marked this conversation as resolved.
Apache Arrow <dev@arrow.apache.org>
Binary to convert csv to Parquet

USAGE:
parquet [OPTIONS] --schema <SCHEMA> --input-file <INPUT_FILE> --output-file <OUTPUT_FILE>

OPTIONS:
-b, --batch-size <BATCH_SIZE>
batch size

[env: PARQUET_FROM_CSV_BATCHSIZE=]
[default: 1000]

-c, --parquet-compression <PARQUET_COMPRESSION>
compression mode

[default: SNAPPY]

-d, --delimiter <DELIMITER>
field delimiter

default value: when input_format==CSV: ',' when input_format==TSV: 'TAB'

-D, --double-quote <DOUBLE_QUOTE>
double quote

-e, --escape-char <ESCAPE_CHAR>
escape charactor

-f, --input-format <INPUT_FORMAT>
input file format

[default: csv]
[possible values: csv, tsv]

-h, --has-header
has header

--help
Print help information

-i, --input-file <INPUT_FILE>
input CSV file

-o, --output-file <OUTPUT_FILE>
output Parquet file

-q, --quote-char <QUOTE_CHAR>
quate charactor

-r, --record-terminator <RECORD_TERMINATOR>
record terminator

[possible values: lf, crlf, cr]

-s, --schema <SCHEMA>
message schema for output Parquet

-V, --version
Print version information
Loading