adt: split interval tree by right endpoint on matched left endpoints by redwrasse · Pull Request #19768 · etcd-io/etcd

redwrasse · 2025-04-20T15:52:54Z

Splits interval trees by right endpoint on matched left endpoints, to improve exact interval matching performance.
This addresses the issue: #19769

Additionally I've attached below code snippets and outputs for non-rigorous (Go benchmarks on before/after performance of Find() and Insert() on randomly built interval trees with multiple occurrences of given left/right endpoint values, which is the regime where I think a performance difference may appear. Run on my laptop, there appears to be no change to the Insert() op performance, but a roughly 88% at p-value of 0 (according to benchstat output below) performance increase in Find(), on such interval trees.

// Benchmark of the Insert op on random interval trees of up to 128 nodes with endpoint values in 0..9
func BenchmarkIntervalTreeRandomInsert(b *testing.B) {
	ivs := make(map[xy]struct{})
	ivt := NewIntervalTree()
	maxv := 128

	xmax := 10
	ymax := 10

	for b.Loop() {
		// Setup random tree.
		for i := rand.Intn(maxv) + 1; i != 0; i-- {
			x, y := int64(rand.Intn(xmax)), int64(rand.Intn(ymax))
			if x > y {
				t := x
				x = y
				y = t
			} else if x == y {
				y++
			}
			iv := xy{x, y}
			if _, ok := ivs[iv]; ok {
				// don't double insert
				continue
			}
			ivt.Insert(NewInt64Interval(x, y), 123)
			ivs[iv] = struct{}{}
		}
	}

}

// Benchmark of the Find op on random interval trees of up to 128 nodes with endpoint values in 0..9
func BenchmarkIntervalTreeRandomFind(b *testing.B) {
	ivs := make(map[xy]struct{})
	ivt := NewIntervalTree()
	maxv := 128

	xmax := 10
	ymax := 10

	// Setup random tree.
	for i := rand.Intn(maxv) + 1; i != 0; i-- {
		x, y := int64(rand.Intn(xmax)), int64(rand.Intn(ymax))
		if x > y {
			t := x
			x = y
			y = t
		} else if x == y {
			y++
		}
		iv := xy{x, y}
		if _, ok := ivs[iv]; ok {
			// don't double insert
			continue
		}
		ivt.Insert(NewInt64Interval(x, y), 123)
		ivs[iv] = struct{}{}
	}

	// Benchmark Find() operation time.
	for b.Loop() {
		for ab := range ivs {
			_ = ivt.Find(NewInt64Interval(ab.x, ab.y))
		}
	}
}

Benchmark results for before (count of 10 in each case)


goos: darwin
goarch: arm64
pkg: go.etcd.io/etcd/pkg/v3/adt
cpu: Apple M1 Pro
BenchmarkIntervalTreeRandomFind-10    	   79048	     15180 ns/op
BenchmarkIntervalTreeRandomFind-10    	   67766	     17690 ns/op
BenchmarkIntervalTreeRandomFind-10    	   94506	     12688 ns/op
BenchmarkIntervalTreeRandomFind-10    	   77904	     15427 ns/op
BenchmarkIntervalTreeRandomFind-10    	  157444	      7627 ns/op
BenchmarkIntervalTreeRandomFind-10    	  247746	      4840 ns/op
BenchmarkIntervalTreeRandomFind-10    	   72440	     16566 ns/op
BenchmarkIntervalTreeRandomFind-10    	   98607	     12179 ns/op
BenchmarkIntervalTreeRandomFind-10    	   76416	     15652 ns/op
BenchmarkIntervalTreeRandomFind-10    	   67321	     17829 ns/op
PASS
ok  	go.etcd.io/etcd/pkg/v3/adt	12.270s


goos: darwin
goarch: arm64
pkg: go.etcd.io/etcd/pkg/v3/adt
cpu: Apple M1 Pro
BenchmarkIntervalTreeRandomInsert-10    	  531319	      2243 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  555193	      2137 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  555756	      2158 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  554901	      2136 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  542766	      2194 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  546540	      2212 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  555234	      2195 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  549458	      2154 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  543097	      2240 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  535682	      2254 ns/op
PASS
ok  	go.etcd.io/etcd/pkg/v3/adt	12.278s

Benchmark results for after:

goos: darwin
goarch: arm64
pkg: go.etcd.io/etcd/pkg/v3/adt
cpu: Apple M1 Pro
BenchmarkIntervalTreeRandomFind-10    	  698035	      1720 ns/op
BenchmarkIntervalTreeRandomFind-10    	 2167977	       553.4 ns/op
BenchmarkIntervalTreeRandomFind-10    	  692677	      1734 ns/op
BenchmarkIntervalTreeRandomFind-10    	 1000000	      1142 ns/op
BenchmarkIntervalTreeRandomFind-10    	  667176	      1797 ns/op
BenchmarkIntervalTreeRandomFind-10    	  714930	      1679 ns/op
BenchmarkIntervalTreeRandomFind-10    	  778672	      1543 ns/op
BenchmarkIntervalTreeRandomFind-10    	  687740	      1745 ns/op
BenchmarkIntervalTreeRandomFind-10    	  659038	      1823 ns/op
BenchmarkIntervalTreeRandomFind-10    	  643330	      1859 ns/op
PASS
ok  	go.etcd.io/etcd/pkg/v3/adt	12.244s

goos: darwin
goarch: arm64
pkg: go.etcd.io/etcd/pkg/v3/adt
cpu: Apple M1 Pro
BenchmarkIntervalTreeRandomInsert-10    	  553756	      2149 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  540813	      2224 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  546811	      2181 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  525974	      2237 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  542532	      2218 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  514786	      2328 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  538801	      2233 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  532995	      2262 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  526350	      2265 ns/op
BenchmarkIntervalTreeRandomInsert-10    	  559215	      2183 ns/op
PASS
ok  	go.etcd.io/etcd/pkg/v3/adt	12.296s

Benchstat (https://pkg.go.dev/golang.org/x/perf/cmd/benchstat#hdr-Example) summary:

goos: darwin
goarch: arm64
pkg: go.etcd.io/etcd/pkg/v3/adt
cpu: Apple M1 Pro
                          │ old_find.txt  │             new_find.txt             │
                          │    sec/op     │    sec/op     vs base                │
IntervalTreeRandomFind-10   15.304µ ± 50%   1.727µ ± 34%  -88.71% (p=0.000 n=10)



goos: darwin
goarch: arm64
pkg: go.etcd.io/etcd/pkg/v3/adt
cpu: Apple M1 Pro
                            │ old_insert.txt │        new_insert.txt         │
                            │     sec/op     │   sec/op     vs base          │
IntervalTreeRandomInsert-10      2.195µ ± 3%   2.228µ ± 2%  ~ (p=0.190 n=10)

which I believe says, for the above benchmark cases, 88% speedup with p-value of 0 for Find, and no noticeable difference for Insert.

…dpoints, to improve find() performance Signed-off-by: redwrasse <mail@redwrasse.io>

k8s-ci-robot · 2025-04-20T15:53:04Z

Hi @redwrasse. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pkg/adt/interval_tree.go

serathius · 2025-04-22T10:46:33Z

Please include benchmark from #19769 in the PR.

redwrasse · 2025-04-23T04:54:28Z

Please include benchmark from #19769 in the PR.

Will do, I'll investigate using benchstats.

redwrasse · 2025-05-05T18:14:45Z

Please include benchmark from #19769 in the PR.

Will do, I'll investigate using benchstats.

Will try to get benchstats added in the next few days.

redwrasse · 2025-05-12T23:15:09Z

Please include benchmark from #19769 in the PR.

Will do, I'll investigate using benchstats.

Will try to get benchstats added in the next few days.

Added benchmarks to this PR description, using benchstat. cc @serathius

serathius · 2025-05-13T07:20:45Z

Can you add the benchmarks to the PR so we can use them to evaluate performance in the future?
Also we should review them as any other code to make sure they are accurate.

We could also consider merging them first in separate PR.

serathius · 2025-05-13T07:25:06Z

If there is no performance change for Insert maybe let's drop the changes there from the PR. Reducing the scope will make the PR easier to review. Complicating Insert code without improving performance doesn't seem like a good tradeoff.

redwrasse · 2025-05-13T17:48:15Z

If I'm understanding , are you suggesting removing the code changes to Insert but keeping the ones in Find? I'm not sure how that would work- the interval tree structure changes need to be reflected in both methods I believe.

I can definitely add in the benchmarks from the description, if we feel they are appropriate.

serathius · 2025-05-14T09:00:44Z

If I'm understanding , are you suggesting removing the code changes to Insert but keeping the ones in Find? I'm not sure how that would work- the interval tree structure changes need to be reflected in both methods I believe.

Sorry, might have misunderstood. Didn't noticed that you changed tree structure.

serathius · 2025-05-14T12:13:40Z

cc @ahrtr

redwrasse · 2025-05-16T18:55:35Z

Cormen "Introduction to Algorithms", Chapter 14 Exercise 14.3.5 is I think relevant:

14.3-5
Suggest modifications to the interval-tree procedures to support the new operation INTERVAL-SEARCH-EXACTLY(T,i), where T is an interval tree and i is an interval. The operation should return a pointer to a node x in T such that x.int.low = i.low and x.int.high = i.high, or T.nil if T contains no such node. All operations, including INTERVAL-SEARCH-EXACTLY, should run in O(lg n) time on an n-node interval tree.

serathius · 2025-05-17T12:05:34Z

/ok-to-test

serathius

Change makes sense to me, but I'm not an expert on this. Will ask for other maintainers to review. cc @ahrtr @fuweid

One additional think I would want to confirm is whether new tests properly cover this case #19801 by checking function coverage.

serathius · 2025-05-17T12:07:13Z

pkg/adt/interval_tree.go

 	for x != ivt.sentinel {
 		y = x
-		if z.iv.Ivl.Begin.Compare(x.iv.Ivl.Begin) < 0 {
+		// Split on left endpoint. If left endpoints match, instead split on right endpoint.


Cormen "Introduction to Algorithms", Chapter 14 Exercise 14.3.5 is I think relevant:

14.3-5
Suggest modifications to the interval-tree procedures to support the new operation INTERVAL-SEARCH-EXACTLY(T,i), where T is an interval tree and i is an interval. The operation should return a pointer to a node x in T such that x.int.low = i.low and x.int.high = i.high, or T.nil if T contains no such node. All operations, including INTERVAL-SEARCH-EXACTLY, should run in O(lg n) time on an n-node interval tree.

Can you update the function docstring?

cc @redwrasse @ahrtr
Can we address this in followup?

Sorry, missed this comment. The comment for the Insert method isn't accurate anymore, we need to update it. We also need to add comment for the find method. @redwrasse

etcd/pkg/adt/interval_tree.go

Lines 437 to 468 in c849507

// TODO: make this consistent with textbook implementation

//

// "Introduction to Algorithms" (Cormen et al, 3rd ed.), chapter 13.3, p315

//

// RB-INSERT(T, z)

//

// y = T.nil

// x = T.root

//

// while x ≠ T.nil

// y = x

// if z.key < x.key

// x = x.left

// else

// x = x.right

//

// z.p = y

//

// if y == T.nil

// T.root = z

// else if z.key < y.key

// y.left = z

// else

// y.right = z

//

// z.left = T.nil

// z.right = T.nil

// z.color = RED

//

// RB-INSERT-FIXUP(T, z)

// Insert adds a node with the given interval into the tree.

@serathius @ahrtr I'll open an MR for updating the comments.

Created a pull request with docstring updates: #20015

codecov · 2025-05-17T12:27:39Z

Codecov Report

Attention: Patch coverage is 90.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 68.87%. Comparing base (aa2a025) to head (21515e1).
Report is 246 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/adt/interval_tree.go	90.00%	1 Missing and 2 partials ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
pkg/adt/interval_tree.go	`86.87% <90.00%> (-1.85%)`	⬇️

... and 38 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #19768      +/-   ##
==========================================
+ Coverage   68.78%   68.87%   +0.09%     
==========================================
  Files         421      424       +3     
  Lines       35857    35880      +23     
==========================================
+ Hits        24665    24714      +49     
+ Misses       9759     9739      -20     
+ Partials     1433     1427       -6

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa2a025...21515e1. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

serathius · 2025-05-17T12:37:39Z

Coverage looks pretty good https://app.codecov.io/gh/etcd-io/etcd/pull/19768?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=etcd-io

Just one line missing in find

serathius · 2025-05-17T12:42:10Z

/retest

ahrtr · 2025-05-19T12:47:08Z

pkg/adt/interval_tree.go

+		if beginCompare < 0 {
+			x = x.left
+		} else if beginCompare == 0 {
+			if endCompare < 0 {
+				x = x.left
+			} else {
+				x = x.right
+			}
+		} else {
+			x = x.right


Pls revert the change to the Insert method

Suggested change

if beginCompare < 0 {

x = x.left

} else if beginCompare == 0 {

if endCompare < 0 {

x = x.left

} else {

x = x.right

}

} else {

x = x.right

if beginCompare < 0 {

x = x.left

} else {

x = x.right

ahrtr · 2025-05-19T12:54:08Z

pkg/adt/interval_tree.go

+		} else if beginCompare == 0 {
+			if z.iv.Ivl.End.Compare(x.iv.Ivl.End) < 0 {
+				x = x.left
+			} else {
+				x = x.right
+			}


You changed the behaviour, so it's a breaking change.

Previously it only checks the interval's Begin; an intervalTree inserts a new node to the left if newNode.Ivl.Begin < x.Ivl.Begin, otherwise inserts it to the right.

Now you not only checks the Begin, but also checks the End, and inserts the new node to the left if Begin matches and newNode's End is less.

This PR's purpose is to optimize the exact search (find), I don't see a reason why update the Insert. So please consider to revert the change. Pls also see comment for the find method.

@ahrtr thanks very much for reviewing.

So now I'm confused. The goal I had with this MR / issue is indeed to speed up Find to logarithmic time (as, for example, the Cormen 14.3-5 excercise addresses), instead of the existing visitor implementation. To do this, in say a textbook implementation, I thought requires updating both Find and Insert operations to further split on right endpoint if the left endpoints are matched?

to further split on right endpoint if the left endpoints are matched

As mentioned above,

You can optimize the find, no matter you update the Insert. Please let me know if it isn't true.

Changing the Insert changes the behaviour. So I suggest not to change it.

I think you're proposing: keep the existing interval tree structure of splitting left if less than left endpoint, else split right. Update the Find implementation to follow this split logic, hence an optimization in the sense that Find will never be checking both left and right subtrees.

With this logic I see the existing etcd interval tree tests fail. The issue is I think this approach doesn't preserve red-black tree rotation invariance.

(code changes for below snippets on this branch 8412021)

Here is updated find, using the proposed logic of split left if less than left endpoint, else split right:

// find the exact node for a given interval func (ivt *intervalTree) find(ivl Interval) *intervalNode { x := ivt.root // Search until hit sentinel or exact match. for x != ivt.sentinel { beginCompare := ivl.Begin.Compare(x.iv.Ivl.Begin) endCompare := ivl.End.Compare(x.iv.Ivl.End) if beginCompare == 0 && endCompare == 0 { // Found a match. return x } // Split on left endpoint: if less than, go left, else go right. if beginCompare < 0 { x = x.left } else { x = x.right } } return x }

and a corresponding unit test (which fails) I think illustrating the point about rotations:

func TestProposedFindExample(t *testing.T) { //won't work because won't satisfy tree rotation invariance property required of red-black trees, eg. if [2,7] is written then can't asssme all subsequently written [2,*] entities will remain in right subtree of [2,7]- won't, because tree rotations are possible. // OTOH with the proposed approach, all subsequent [2, x] writes with x > 7 will land in right/after of [2,7], and those with x < 7 will land to left /before. Under tree rotation ordering is preserved. ivt := NewIntervalTree() lEndp := int64(2) rEndps := []int64{7, 3, 9} val := 123 for _, re := range rEndps { ivt.Insert(NewInt64Interval(lEndp, re), val) } // What we would expect from 'insert left if less than left endpoint, else insert right' without tree rotations: // (we can generate this tree by commenting out the `ivt.insertFixup(z)` line in the `Insert` op) // Insert [2, 7), then Insert [2, 3), then Insert [2, 9) becomes: // [2, 7) // \ // [2, 3) // \ // [2, 9) // Instead, due to rotations (rb-fixup), we get: // [2, 3) // / \ // [2, 7) [2, 9) // Find fails because it assumes the former tree structure, eg. thinks it can always search right in the above example: for _, re := range rEndps { ivl := NewInt64Interval(lEndp, re) assert.NotNil(t, ivt.Find(ivl)) assert.Equal(t, ivl, ivt.Find(ivl).Ivl) } }

I see. I was thinking the find should work as long as it matches logic (search path) as Insert, but actually it might not match due to insertFixup.

The question for now is will insertFixup cause the same problem for this PR?

In this PR, you updated both find and Insert, and follow the same logic (search path): splitting both left(Begin) and right(End) endpoints. Due to insertFixup, is it possible that it may also cause the find fail?

I'm not an expert on these algorithms, but my working assumption has been that the total ordering introduced by secondary split on right endpoints allows satisfying the tree rotation invariance needed of red-black tree structure. I think this is the textbook approach (eg. the Cormen exercise referenced earlier.)

Any thoughts for how to further test/guarantee correctness? During development I relied on the TestIntervalTreeRandom, which is parameterized by # of nodes maxv, set to 128. Increasing that by an order of magnitude, and rerunning, I didn't encounter any test failures.

Any thoughts for how to further test/guarantee correctness?

Good question.

I think both rotateLeft and rotateRight always keep the invariable property: x.left < x < x.right

rotateLeft (a):

a b \ / \ b ---> a d / \ \ c d c

rotateRight (a):

a b / / \ b ---> c a / \ / c d d

I ran go test -run TestIntervalTreeRandom -v -count 200 -failfast multiple times, and always passed.

As mentioned in #19768 (comment), it changes the behaviour, but I think we are good as long as we don't break the IntervalTree API

Sounds good, thanks for reviewing and investigating @ahrtr!

ahrtr

LGTM

Thanks for the optimisation.

k8s-ci-robot · 2025-05-21T12:06:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, redwrasse, serathius

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahrtr,serathius]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ahrtr · 2025-05-21T12:07:10Z

We should add a performance item in 3.7 changelog, let me take care of together with another change.

adt: further split interval tree by right endpoint on matched left en…

21515e1

…dpoints, to improve find() performance Signed-off-by: redwrasse <mail@redwrasse.io>

k8s-ci-robot added the needs-ok-to-test label Apr 20, 2025

k8s-ci-robot added the size/M label Apr 20, 2025

redwrasse mentioned this pull request Apr 20, 2025

adt: split interval tree on right endpoints for matched left endpoints #19769

Closed

redwrasse changed the title ~~[DRAFT] adt: split interval tree by right endpoint on matched left endpoints~~ adt: split interval tree by right endpoint on matched left endpoints Apr 20, 2025

serathius reviewed Apr 22, 2025

View reviewed changes

pkg/adt/interval_tree.go Show resolved Hide resolved

redwrasse mentioned this pull request Apr 25, 2025

adt: add tests for IntervalTree.Find() #19801

Merged

ahrtr self-requested a review May 16, 2025 18:58

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels May 17, 2025

serathius approved these changes May 17, 2025

View reviewed changes

k8s-ci-robot added the approved label May 17, 2025

serathius reviewed May 17, 2025

View reviewed changes

ahrtr reviewed May 19, 2025

View reviewed changes

ahrtr approved these changes May 21, 2025

View reviewed changes

ahrtr added the area/performance label May 21, 2025

ahrtr merged commit a119975 into etcd-io:main May 21, 2025
34 checks passed

ahrtr mentioned this pull request May 21, 2025

Update 3.7 changelog to cover two changes #20000

Merged

redwrasse mentioned this pull request May 23, 2025

adt: update docstrings for IntervalTree.Insert and IntervalTree.find #20015

Merged

	// TODO: make this consistent with textbook implementation
	//
	// "Introduction to Algorithms" (Cormen et al, 3rd ed.), chapter 13.3, p315
	//
	// RB-INSERT(T, z)
	//
	// y = T.nil
	// x = T.root
	//
	// while x ≠ T.nil
	// y = x
	// if z.key < x.key
	// x = x.left
	// else
	// x = x.right
	//
	// z.p = y
	//
	// if y == T.nil
	// T.root = z
	// else if z.key < y.key
	// y.left = z
	// else
	// y.right = z
	//
	// z.left = T.nil
	// z.right = T.nil
	// z.color = RED
	//
	// RB-INSERT-FIXUP(T, z)

	// Insert adds a node with the given interval into the tree.

Conversation

redwrasse commented Apr 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 20, 2025

Uh oh!

Uh oh!

serathius commented Apr 22, 2025

Uh oh!

redwrasse commented Apr 23, 2025

Uh oh!

redwrasse commented May 5, 2025

Uh oh!

redwrasse commented May 12, 2025

Uh oh!

serathius commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serathius commented May 13, 2025

Uh oh!

redwrasse commented May 13, 2025

Uh oh!

serathius commented May 14, 2025

Uh oh!

serathius commented May 14, 2025

Uh oh!

redwrasse commented May 16, 2025

Uh oh!

serathius commented May 17, 2025

Uh oh!

serathius left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serathius May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

serathius commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serathius commented May 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahrtr left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented May 21, 2025

Uh oh!

redwrasse commented Apr 20, 2025 •

edited

Loading

serathius commented May 13, 2025 •

edited

Loading

serathius left a comment •

edited

Loading

serathius May 17, 2025 •

edited

Loading

codecov bot commented May 17, 2025 •

edited

Loading

serathius commented May 17, 2025 •

edited

Loading