Skip to content

Update client_golang#42527

Closed
ywwg wants to merge 0 commit into
open-telemetry:mainfrom
ywwg:owilliams/update-otel
Closed

Update client_golang#42527
ywwg wants to merge 0 commit into
open-telemetry:mainfrom
ywwg:owilliams/update-otel

Conversation

@ywwg
Copy link
Copy Markdown
Contributor

@ywwg ywwg commented Sep 5, 2025

Fixes unset TextParser issues

JobName: "job1",
JobName: "job1",
// XXXXXXXXXXXXXXXx bad
MetricRelabelRegex: relabel.MustNewRegexp("regex2"),
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the test is returning regex1, indicating that the relabel config is not getting updated properly. I cannot figure out how this change would have done that. race condition?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think this is a race condition. I think the test is being run before the second scrapeconfig call is made.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The allocator waitgroup returns after only one response has been given, which is weirdly by design.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that we call Refresh and then immediately check for the result, but the updated config takes a while to be set. So we need to find a way to wait until we know the update has been done.

Comment on lines +778 to +799
found := false
var lastGot string
// The manager may not be done processing the Refresh call by the
// time we check the value of the ScrapeConfig, so retry a couple
// times to see if it fixes itself.
RETRY:
for j := 0; j < 3; j++ {
for _, sc := range manager.promCfg.ScrapeConfigs {
if sc.JobName == s.MetricRelabelConfig.JobName {
for _, mc := range sc.MetricRelabelConfigs {
lastGot = mc.Regex.String()
if s.MetricRelabelConfig.MetricRelabelRegex.String() == mc.Regex.String() {
found = true
break RETRY
}
}
}
}
time.Sleep(2 * time.Second)
}
if !found {
t.Errorf("MetricRelabelConfig did not match expectation, want %v got %v", s.MetricRelabelConfig.MetricRelabelRegex.String(), lastGot)
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will heartily welcome a better way to do this. I don't think we can use a waitgroup because the thing we are waiting for is the sync() call in the manager, and there is no way I can see for a test to tie in to that logic to know when a sync has finished.

Copy link
Copy Markdown
Member

@songy23 songy23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the target branch open-telemetry:otelbot/update-otel-1757061301 is already stale, you may want to recreate the PR to mainline


baseCfg := promconfig.Config{GlobalConfig: promconfig.DefaultGlobalConfig}
manager := NewManager(receivertest.NewNopSettings(metadata.Type), tc.cfg, &baseCfg, false)
baseCfg, err := promconfig.Load("", nil)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're changing from "DefaultGlobalConfig" to an empty string... totally shooting in the dark here, could this change influence the flakiness?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Load pre-initializes with DefaultConfig (which contains DefaultGlobalConfig) and then merges the string into it. I compared the old init with the new, and the only diffs are the added Runtime and OtlpConfig items that were unset in the previous code.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ywwg
Copy link
Copy Markdown
Contributor Author

ywwg commented Sep 8, 2025

marking as draft because this is a discussion piece, not intended for direct merging

@ywwg ywwg marked this pull request as draft September 8, 2025 18:17
@ywwg ywwg changed the base branch from otelbot/update-otel-1757061301 to main September 8, 2025 20:23
@ywwg ywwg force-pushed the owilliams/update-otel branch from 70a2623 to af8b7c3 Compare September 8, 2025 20:25
@ywwg
Copy link
Copy Markdown
Contributor Author

ywwg commented Sep 8, 2025

well I tried to rebase and it was a gigantic disaster

@ywwg ywwg closed this Sep 8, 2025
@ywwg ywwg force-pushed the owilliams/update-otel branch from af8b7c3 to ab268ae Compare September 8, 2025 20:26
@ywwg
Copy link
Copy Markdown
Contributor Author

ywwg commented Sep 8, 2025

github automatically closed this and I can't reopen. new PR at: #42568

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants