bench: measure both sides of every row on stdout+stderr

[KuSh]

The raw side of each benchmark row is measured with 2>&1 and the rtk side with 2>/dev/null, so the two columns are not the same measurement.

rtk forwards the stderr it does not filter. Dropping that stream from the rtk column books a passed-through stream as if the filter had removed it, and it hides the failure mode the rows exist to catch: a filter that goes silent while the tool reports on stderr scores as a saving. The golangci-lint row reads 157 -> - and fails the run for producing nothing, with no way to see that the diagnostic went to the other stream all along.

Why this cannot make rtk look worse

For an identical measurement rtk is better or equal by construction. stderr is common to both sides, so never-worse reduces to filtered <= stdout.

rg in CI

Measuring stderr also shows when the tool under test is missing. The CI runner has no rg, so the rg rows compared the shell's rg: command not found with rtk's own error and went negative. The benchmark job now installs ripgrep next to tree, and the rg section gets the same command -v guard as tree, so it is skipped when rg is absent.

The run reports 0 negative and 0 fail both ways:

rg	rg rows
installed (CI)	rg fn 125431 → 4028, rg struct 3434 → 2599, rg -l 800 → 675, rg -c 895 → 895
absent (local, rg hidden from PATH)	section skipped

The good/warn split moves by a row or two between machines and with what is installed locally, so it is not a property of this change and I have deliberately not quoted it as one.

Verified in golang:1.27 that the go rows read 220 -> 35 GOOD on golangci-lint v2.13.2, and 157 -> 157 — a tie, not a red run — if the v2 install were ever reverted.

Independent of the other benchmark work

This does not depend on #3773 and does not change its numbers (220 -> 35 either way). It is a measurement-methodology change that touches every row, which is why it is separated out rather than folded in.

It does read best after #3772: until stdout-only filters forward stderr at all, the golangci-lint row stays FAIL regardless of how it is measured.

[aeppling]

Hey @KuSh , thanks for this correctness fix in benchmark, LGTM

Could you also install rg in the ci runner so benchmark pass ? all results look same length so I suspect a problem with rg execution in this environment.

"tree" command has a guard for this, we could add an install step and the guard as well

[KuSh]

Thanks! Done in 0f6b844: the benchmark job now installs ripgrep next to tree, and the rg section gets the same command -v guard as tree, so it is skipped with "⏭️ rg (not installed, skipped)" when rg is not there.

You were right about the cause. With stderr now measured on both sides, the rg rows were comparing the shell's rg: command not found (14 tokens) with rtk's own error (22 tokens), hence the identical lengths. The benchmark job is green on this head: with rg installed the four rows read 125431 → 4028, 3434 → 2599, 800 → 675 and 895 → 895, and the run reports 0 negative, 0 fail. Locally with rg hidden from PATH the section is skipped and the run is clean as well.

I also rebased onto the current develop.