Highlight: this is how we skip tests. Note the regexp similar to golang's matching

It's the same as the following, right?

Regex support is pretty recent, i think? https://tip.golang.org/doc/go1.20 states:

The go test command now accepts -skip to skip tests, subtests, or examples matching .

go help testflag conforms we now have parity for -run and -skip which is really nice!:

-run regexp
	    Run only those tests, examples, and fuzz tests matching the regular
	    expression. For tests, the regular expression is split by unbracketed
	    slash (/) characters into a sequence of regular expressions, and each
	    part of a test's identifier must match the corresponding element in
	    the sequence, if any. Note that possible parents of matches are
	    run too, so that -run=X/Y matches and runs and reports the result
	    of all tests matching X, even those without sub-tests matching Y,
	    because it must run them to look for those sub-tests.
	    See also -skip.
[..]
	-skip regexp
	    Run only those tests, examples, fuzz tests, and benchmarks that
	    do not match the regular expression. Like for -run and -bench,
	    for tests and benchmarks, the regular expression is split by unbracketed
	    slash (/) characters into a sequence of regular expressions, and each
	    part of a test's identifier must match the corresponding element in
	    the sequence, if any.

Both skip and run are useful. run allows more fine-grained control than our predefined presets, which saves time when someone develops a fix and has to re-run specific subset of tests without running the entire suite (not a problem today, but the suite will grow).

@laurentsenta do we need to have dedicated input in github action?
If we could keep this under args we would not have to do anything special to support -run here, right?

I'm also wondering if we even need dedicated run/skip gateway conformance cli flags if we can accomplish the same using the go flags.

If we do, e.g. as nicer to use sugar, I think it might be worth considering converting the custom flag values into regex and passing it to go to avoid having to reimplement run/skip logic.

👍 Thanks for the reviews and comments,
That work might be a dead end, Sorry for the noise 🤦

I ran more experiments, and not everything makes sense to me, but regular skips should be enough.

@galargh your suggestion would need a different parenthesizing:

(TestTrustlessCarPathing|TestNativeDag/GET_plain_JSON_codec_from_.*) # does not work
(TestTrustlessCarPathing)|(TestNativeDag/GET_plain_JSON_codec_from_.*) # would work

@lidel your issue should be solved with an | and parenthesis:

go test ... -skip '(TestGatewayCache/GET_for_/ipfs/_with_only-if-cached_succeeds_when_in_local_datastore)|(TestGatewayCache/HEAD_for_/ipfs/_with_only-if-cached_succeeds_when_in_local_datastore)'

Probably also:

go test ... -skip 'TestGatewayCache/(GET|HEAD)_for_/ipfs/_with_only-if-cached_succeeds_when_in_local_datastore'

We can drop that PR.

I think the easiest way to think about golang -skip is:

• create groups with (g1)|(g2) at the top level,
• then each group is split by / and each part is matched with the go test name.
  • willSkip("regexp1/regexp2", "testX/testY") = match(regexp1, testX) && match(regexp2, testY)

I'm still confused by some aspects of the matching... if you want to dig deeper: https://gist.github.com/laurentsenta/5ed9fe9954605eefc1ed9ee2db7f66a9

But I'll close this PR for now and continue the discussion in #141.

Thank you for looking into it 🙇 That's a nice write-up for a tricky feature 👏

highlight: use in CLI

Could we add ### Run only a specific test with example that uses go's go test -run <pattern> and use the same pattern as in the skip example?

Include this note:

The -run argument uses the same syntax as go test -run and gives more fine-grained control than predefined presets passed via --specs. It aims to save time when developing a fix and having to re-run specific subset of tests without running the entire suite.

Regex support is pretty recent, i think? https://tip.golang.org/doc/go1.20 states:

The go test command now accepts -skip to skip tests, subtests, or examples matching .

go help testflag conforms we now have parity for -run and -skip which is really nice!:

-run regexp
	    Run only those tests, examples, and fuzz tests matching the regular
	    expression. For tests, the regular expression is split by unbracketed
	    slash (/) characters into a sequence of regular expressions, and each
	    part of a test's identifier must match the corresponding element in
	    the sequence, if any. Note that possible parents of matches are
	    run too, so that -run=X/Y matches and runs and reports the result
	    of all tests matching X, even those without sub-tests matching Y,
	    because it must run them to look for those sub-tests.
	    See also -skip.
[..]
	-skip regexp
	    Run only those tests, examples, fuzz tests, and benchmarks that
	    do not match the regular expression. Like for -run and -bench,
	    for tests and benchmarks, the regular expression is split by unbracketed
	    slash (/) characters into a sequence of regular expressions, and each
	    part of a test's identifier must match the corresponding element in
	    the sequence, if any.

Both skip and run are useful. run allows more fine-grained control than our predefined presets, which saves time when someone develops a fix and has to re-run specific subset of tests without running the entire suite (not a problem today, but the suite will grow).

@laurentsenta do we need to have dedicated input in github action?
If we could keep this under args we would not have to do anything special to support -run here, right?

Parity with go-test -skip is 👍
Could we include the same for go-test -run in this PR?
It will make things feature-complete, and natural for anyone familiar with go, and at least remove surprises when someone tries -skip and -run – these should work the same, and if we support passing -skip multiple times, we should also support that for -run 🙏