Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testutil: Better registry state assertion framework/API #1639

Open
bwplotka opened this issue Oct 8, 2024 · 4 comments
Open

testutil: Better registry state assertion framework/API #1639

bwplotka opened this issue Oct 8, 2024 · 4 comments

Comments

@bwplotka
Copy link
Member

bwplotka commented Oct 8, 2024

Context

With our breaking (for a good reason) testutil (Gather|Scrape)AndCompare function change a good discussions e.g. here emerged where we chatted about what broke and how and what those downstream users (e.g. Mimir, Kubernetes and others) needs in terms of registry testing capabilities.

Historically, testutil was meant to be an experimental, internal utils for testing, mostly for client_golang use. Then Prometheus started to use it then everywhere else. Funny enough, we even did some tech talks with @kakkoyun around why it's useful. Now here we are -- many downstream project use it, primarily for:

  • Close-box testing (asserting state of a component/structure/binary by polling certain metrics), both on code and process/container level.
  • Instrumentation testing (are my metrics correctly instrumented).

Notably, when chatting with @dashpole on Kubernetes changes there are two cases currently impossible with client_golang functions:

  • Asserting certain metrics is NOT existing in registry at the given moment.
  • Mix of existing with certain value or not existing.

One could implement everything on their own using ToFloat64 etc, but maybe it's time to redesign testutil, perhaps as another package or even module?

Let's discuss some ideas here, help wanted to propose some design.

Ideally we verify the design on some downstream user e.g. https://github.com/kubernetes/component-base before releasing (:

Yolo Ideas

I guess we could create a redesigned package with a stable API and ways to:

  • Get float/histogram values for testing purposes (similar to ToFloat, just more flexible, more robust)
  • Use text format as an expectation with a certain configurable semantics around filtering, each metric values or custom conditions?
  • Potentially, move functional/variadic API similar to what we built in the e2e Go framework.

For example (yolo naming and structure):

err := testutilv2.For(registry).
   MetricsMatching(`kubernetes_some_counter_total{network="a"}`, testutilv2.Equals(2)).
   MetricsMatching(`kubernetes_some_counter_total{network=~"b|c"}`, testutilv2.LessThan(10)),
   MetricsMatching(`kubernetes_some_counter_total{network="yolo"}`, testutilv2.NotExists()).
   WaitExpect(5 * time.Second)

err := testutilv2.For(registry).MatchTextExpectation(`HELP kubernetes_some_counter_total "sdfsdfsfsfs"`, <opts>).Expect()
@bwplotka
Copy link
Member Author

This is now higher priority given the revert in https://github.com/prometheus/client_golang/releases/tag/v1.20.5

@dashpole
Copy link

One idea: If I can filter a registry/gatherer by metric names, then I would have been more easily assert that a metric is not present (after the breaking change)

@bwplotka
Copy link
Member Author

@machine424
Copy link

Thanks @bwplotka! The first PR totally slipped past me (not that I would have necessarily spotted anything ;)). I'll keep an eye on this second attempt and try to help, since there’s room for improvement as you explained.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants