Skip to content

wc: improve wc -c perf with unextended pipe input (Linux only)#11591

Merged
sylvestre merged 1 commit intouutils:mainfrom
oech3:wc-pipe-size
Apr 3, 2026
Merged

wc: improve wc -c perf with unextended pipe input (Linux only)#11591
sylvestre merged 1 commit intouutils:mainfrom
oech3:wc-pipe-size

Conversation

@oech3
Copy link
Copy Markdown
Contributor

@oech3 oech3 commented Apr 2, 2026

$ truncate -s 32GB huge
$ time cat huge|wc -c
Executed in    1.31 secs    fish           external
   usr time    0.64 secs    0.53 millis    0.63 secs
   sys time    1.53 secs    2.95 millis    1.52 secs
$ time cat huge|target/release/wc -c
Executed in  388.34 millis    fish           external
   usr time   39.03 millis    0.64 millis   38.39 millis
   sys time  657.96 millis    2.98 millis  654.98 millis

Depends on #11551 for smaller diff.

@oech3 oech3 changed the title wc: improve wc -c perf wc: improve wc -c perf (Linux only) Apr 2, 2026
@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented Apr 2, 2026

Is splice() unsupported at CodSpeed?

@sylvestre
Copy link
Copy Markdown
Contributor

please run your benchmark with hyperfine
time isn't great compared to hyperfine for this

@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented Apr 2, 2026

using stdin at hyperfine just fails. I was using -N .

@sylvestre
Copy link
Copy Markdown
Contributor

are you sure ?
hyperfine --warmup 1 'cat /tmp/huge | wc -c' 'cat /tmp/huge | target/release/wc -c' 'wc -c < /tmp/huge' 'target/release/wc -c < /tmp/huge'
worked for me

@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented Apr 2, 2026

Also we cannot use our cat from main since #11551 has same effect for input.

We lost the utility to benchmark this at #11551.

@oech3 oech3 changed the title wc: improve wc -c perf (Linux only) wc: improve wc -c perf with unextended pipe input (Linux only) Apr 2, 2026
@oech3

This comment was marked as low quality.

@sylvestre
Copy link
Copy Markdown
Contributor

and compared with GNU ? :)

@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented Apr 2, 2026

I did serious typo at prev comment (missing -c)
(and we actuall should use cat from main)

$ truncate -s 1GB huge
$ hyperfine "cat huge |wc-main -c" "cat huge |wc-fcntl-input -c" "cat huge|/hom
e/a/bin/gnu58d88d243/wc -c"
Benchmark 1: cat huge |wc-main -c
  Time (mean ± σ):      29.5 ms ±   1.8 ms    [User: 11.2 ms, System: 40.7 ms]
  Range (min … max):    27.0 ms …  35.8 ms    98 runs
 
Benchmark 2: cat huge |wc-fcntl-input -c
  Time (mean ± σ):      17.7 ms ±   1.1 ms    [User: 2.3 ms, System: 28.7 ms]
  Range (min … max):    16.0 ms …  22.3 ms    143 runs
 
Benchmark 3: cat huge|/home/a/bin/gnu58d88d243/wc -c
  Time (mean ± σ):      72.8 ms ±   2.2 ms    [User: 5.3 ms, System: 136.6 ms]
  Range (min … max):    69.2 ms …  78.9 ms    40 runs
 
Summary
  cat huge |wc-fcntl-input -c ran
    1.67 ± 0.14 times faster than cat huge |wc-main -c
    4.12 ± 0.28 times faster than cat huge|/home/a/bin/gnu58d88d243/wc -c

GNU does not use splice() even at master branch yet.

@oech3 oech3 marked this pull request as ready for review April 2, 2026 08:59
@sylvestre sylvestre merged commit bd69fe9 into uutils:main Apr 3, 2026
165 checks passed
@oech3 oech3 deleted the wc-pipe-size branch April 3, 2026 07:17
kevinburkesegment pushed a commit to kevinburkesegment/coreutils that referenced this pull request Apr 6, 2026
@oech3
Copy link
Copy Markdown
Contributor Author

oech3 commented Apr 7, 2026

I noticed that we don't need to increase pipe size of input fd since

  • if sender supports splice(), it probably extended the pipe size of input fd.
  • if sender uses write(), it is bottleneck for our splice wc -c.
    Exception is sender utility with splice() support which forgot to increase pipe size or sender is modest.

(this PR's splice()'s length is still meaningful)

I'll remove fcntl at #11636 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants