Skip to content

Commit 6dac8aa

Browse files
committed
Allow --filename={}/file.csv in addition to {}.csv
This is a minor tweak to handle some problems which I've encountered several times now. The only real excuse for this change is that it's unobstrusive and has natural semantics. This patch modifies --filename to support two use cases: xsv partition --filename {}/cities.csv state . all-cities.csv xsv partition --filename {}/monuments.csv state . all-monuments.csv xsv partition --filename {}/parks.csv state . all-parks.csv Above, we want to partition our records by state into separate directories, but we have multiple kinds of data. xsv partition --filename {}/$(uuidgen).csv . input1.csv xsv partition --filename {}/$(uuidgen).csv . input2.csv Above, we're running multiple (possibly parallel) copies of xsv and we want to parition the data into multiple directories without a filename clash. There's one limitation to the implementation: We might theoretically hit the `create_dir_all` race condition recently fixed by rust-lang/rust#39799. I'm planning to supply a final patch which works around this race condition in stable Rust.
1 parent 074f14c commit 6dac8aa

File tree

2 files changed

+21
-0
lines changed

2 files changed

+21
-0
lines changed

src/util.rs

+5
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,11 @@ impl FilenameTemplate {
176176
{
177177
let filename = self.filename(unique_value);
178178
let full_path = path.as_ref().join(filename);
179+
if let Some(parent) = full_path.parent() {
180+
// If you're likely to call this concurrently, see
181+
// https://github.com/rust-lang/rust/pull/39799
182+
fs::create_dir_all(parent)?;
183+
}
179184
let spath = Some(full_path.display().to_string());
180185
Config::new(&spath).writer()
181186
}

tests/test_partition.rs

+16
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,22 @@ fn partition_custom_filename() {
9595
assert!(wrk.path("state-NY-partition.csv").exists());
9696
}
9797

98+
#[test]
99+
fn partition_custom_filename_with_directory() {
100+
let wrk = Workdir::new("partition_custom_filename_with_directory");
101+
wrk.create("in.csv", data(true));
102+
103+
let mut cmd = wrk.command("partition");
104+
cmd.args(&["--filename", "{}/cities.csv"])
105+
.arg("state")
106+
.arg(&wrk.path("."))
107+
.arg("in.csv");
108+
wrk.run(&mut cmd);
109+
110+
// This variation also helps with parallel partition jobs.
111+
assert!(wrk.path("NY/cities.csv").exists());
112+
}
113+
98114
#[test]
99115
fn partition_invalid_filename() {
100116
let wrk = Workdir::new("partition_invalid_filename");

0 commit comments

Comments
 (0)