test(cli): skip CLI tests if there is an override

[matzew]

When doing toolset override (E.g. to core, helm) or changing to read-Only: true, some tests fail.

One of those is the one on the CLI tests. In #517 some tests were skipped (due to downstream) if there are some overrides.

I think this works to some degree. IMO regardless of what the actual downstream (compile time) override of defaults is, is that test for "disabled" parts should be still tested, since they can be enabled by users

[matzew]

One options is on things like common_test.go do some assumption, like:

	s.Cfg.ReadOnly = false
	s.Cfg.Toolsets = []string{"core", "config", "helm"}

(in this particular example)

[Cali0707]

One options is on things like common_test.go do some assumption, like:

	s.Cfg.ReadOnly = false
	s.Cfg.Toolsets = []string{"core", "config", "helm"}

(in this particular example)

IMO this is a good solution for the unit tests - I think that relying on defaults (which may be overriden in forks) is going to cause pain. Better to be explicit with what we are testing.

Perhaps if we want to have unit tests verifying the "default" behaviour, we can separate those out into a separate file with the same skip that you provided here, along with a test_default_overrides.go that could contain downstream overriden default tests

[manusa]

IMO it's fine and good having tests that rely on defaults (or actually verify them).

These are tests that sometimes are there on purpose to ensure that the MCP server has a default behavior.

I also like to have them in the same test suite as the other, since they are grouped organically.

The skip logic (which is used in other tests too), is effective, but it's true that is becoming cumbersome.
We can try to find a simpler way to do this.
Options I can think of (Besides what Calum proposes):

• extract test cases to a separate file xxx_defaults_test.go with a build tag
• create a file with the constant assertions that can have overrides (similar to what the production code has but for tests and assertions) -Not sure if this is what Calum propses-
• Base Suite struct that exposes a skip method

For now I guess we should merge this PR and proceed with the agreed refactor later.

[manusa]

LGTM, thx!

[matzew]

The other option would be to have some (config) bundles and pass them down to the test sets., e. like the xxx_defaults_test.go - Options:

• default (as in upstream)
• custom (e.g. what downstream might want), to be maintained by downstream