Hard to Tell if Test is Skipped

[cwfitzgerald]

We have a rather complex set of situations where a test is either skipped or expected to fail. As such we can't use the standard #[should_panic] or #[ignored] as we need to make that decision at runtime based on the device we're running on.

This can cause problems where the developer is trying to fix a particular problem, runs the tests, sees that they all pass, not realizing that the test they're trying to measure against isn't actually running or has an expected fail on it.

We should find some way to express if something is skipped.

One possibility is to use libtest-mimic to allow us to add a "kind" to tests which is then visible to cargo-nextest or cargo test. We likely would need our own macro for tests in that case and would have interesting ramifications for wasm testing via wasm-pack.

[cwfitzgerald]

Going to copy here some thinking I had regarding this issue:

Specifically that we actually can't use wasm-pack test to do what we really want - we do image comparison testing, which requires sending images back and forth from the filesystem and on the web that requires at minimum a webserver. I think the solution is to lean into xtask, make a test harness for the web that communicates back with the webserver.

for native when something like cargo xtask test is run:

• our xtask tool will enumerate all gpus on the system and get all of their information and serialize this to a file
• the test harness (using libtest-mimic) will read this file and enumerate which tests are skipped, expected fail, etc. This also gives us a chance to run permutations on tests and make sure we test all devices on the system.
• as all these decisions are made from pre-prepared data, it can be done extremely quickly, keeping the harness runtime short