cmd/test2json: support emitting JSON directly from the test binary

[petrhosek]

It is now possible to emit JSON from a test binary (#2981, #22789), but this still requires the go tool:

go tool test2json ./foo.test -test.v

This doesn't work in -c case which is really useful when cross-platform running tests, i.e. we build the tests binaries on one platform but run them on a different one as plain binaries without having to use the go tool. Would it be possible to support that scenario as well, e.g. as -test.json file flag?

[ianlancetaylor]

I believe you can capture the test output on one machine and run it through test2json on a different machine.

[petrhosek]

It's certainly doable but it makes the CI pipeline more complicated because it requires additional post-processing.

[rsc]

We considered whether to do this originally, but it's very difficult to do completely. The test binary cannot reliably ensure that all its output goes into test2json. Maybe it crashes during init before test2json even starts up. Maybe it manages to print from a func init before replacing os.Stdout/os.Stderr with pipes into test2json. The binary also cannot reliably ensure that all output is processed by test2json. Maybe it crashes, in which case it could exit before test2json, so that the parent might not observe the final output processed by test2json. The output processing problem could be fixed by making the test binary exec test2json and have test2json then reexec the test binary as a subprocess, but then the test binary starts up twice, which may break other things.

Note that you don't need the whole go command for your CI system. You can do a cross-platform build of cmd/test2json once, install it onto your target system, and then run

path/to/test2json ./foo.test -test.v

That should be much more robust than anything we could do in the test binary itself. Please use that.