OCPBUGS-56594: UPSTREAM: <carry>: backporting fix for concurrent map iteration and write

[ricardomaraschini]

Warning

Reviewer: the target branch of this PR is master but the changes introduced here are already present in upstream master and release-1.34 branches. We may need to use the release-4.20 target branch here instead (we won't need to carry this over versions >= 1.34).

Note

Reviewer: The back port of these upstream patches broke one of our unit tests. The broken test seems to cover features that are not available upstream (with_early_late_annotations_test.go).

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR contains upstream commits that are currently available on Kubernetes v1.34 and for which no back port plans exist. We need these to fix the bug we are currently seeing on the field.

The original PRs:

• Fix API server crash on concurrent map iteration and write kubernetes/kubernetes#129472
• Avoid encoding in LogResponseObject when we are not going to use it kubernetes/kubernetes#131725
• Eliminate AuditContext`s SetEventLevel kubernetes/kubernetes#131694

Which issue(s) this PR fixes:

Fixes OCPBUGS-56594

Does this PR introduce a user-facing change?

NONE

[openshift-ci-robot]

@ricardomaraschini: This pull request references Jira Issue OCPBUGS-56594, which is invalid:

• expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

[!Warning]
Reviewer: the target branch of this PR is master but the changes introduced here are already present in upstream master and release-1.34 branches. We may prefer to use the release-4.20 target branch here instead.

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR contains two fixes that are currently available on Kubernetes v1.34 and for which no back port plans exist. We need these to fix the bug we are currently seeing on the field.

There are two commits that were manually cherry picked: one fixes the bug but and introduced a performance regression, the second one address the performance regression.

The original PRs are kubernetes#129472 and kubernetes#131725.

Which issue(s) this PR fixes:

Fixes OCPBUGS-56594

Does this PR introduce a user-facing change?

NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@ricardomaraschini: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 9f07788|OCPBUGS-56594: UPSTREAM: : Fix API server crash on concurrent map iteration and write: does not specify an upstream backport in the commit message
• a8a5957|OCPBUGS-56594: UPSTREAM: : Avoid encoding in LogResponseObject when we are not going to use it: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci-robot]

@ricardomaraschini: This pull request references Jira Issue OCPBUGS-56594, which is invalid:

• expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

[!Warning]
Reviewer: the target branch of this PR is master but the changes introduced here are already present in upstream master and release-1.34 branches. We may need to use the release-4.20 target branch here instead (we won't need to carry this over versions >= 1.34).

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR contains two fixes that are currently available on Kubernetes v1.34 and for which no back port plans exist. We need these to fix the bug we are currently seeing on the field.

There are two commits that were manually cherry picked: one fixes the bug but and introduced a performance regression, the second one address the performance regression.

The original PRs are kubernetes#129472 and kubernetes#131725.

Which issue(s) this PR fixes:

Fixes OCPBUGS-56594

Does this PR introduce a user-facing change?

NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@ricardomaraschini: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 820ff3c|UPSTREAM: : Fix API server crash on concurrent map iteration and write: does not specify an upstream backport in the commit message
• 960aa55|UPSTREAM: : Avoid encoding in LogResponseObject when we are not going to use it: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[ricardomaraschini]

/test verify-commits

[openshift-ci-robot]

@ricardomaraschini: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 820ff3c|UPSTREAM: : Fix API server crash on concurrent map iteration and write: does not specify an upstream backport in the commit message
• 960aa55|UPSTREAM: : Avoid encoding in LogResponseObject when we are not going to use it: does not specify an upstream backport in the commit message
• bdddf28|UPSTREAM: : Eliminate AuditContext`s SetEventLevel: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci-robot]

@ricardomaraschini: This pull request references Jira Issue OCPBUGS-56594, which is invalid:

• expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

[!Warning]
Reviewer: the target branch of this PR is master but the changes introduced here are already present in upstream master and release-1.34 branches. We may need to use the release-4.20 target branch here instead (we won't need to carry this over versions >= 1.34).

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR contains three upstream commits that are currently available on Kubernetes v1.34 and for which no back port plans exist. We need these to fix the bug we are currently seeing on the field.

The original PRs:

• Fix API server crash on concurrent map iteration and write kubernetes/kubernetes#129472
• Avoid encoding in LogResponseObject when we are not going to use it kubernetes/kubernetes#131725
• Eliminate AuditContext`s SetEventLevel kubernetes/kubernetes#131694

Which issue(s) this PR fixes:

Fixes OCPBUGS-56594

Does this PR introduce a user-facing change?

NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[benluddy]

You'll want to replace with the upstream PR number on each picked commit. The automation will be looking to correlate those, and that's also how we will know to drop these commits as part of the 1.34 rebase.

[openshift-ci-robot]

@ricardomaraschini: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

• 820ff3c|UPSTREAM: : Fix API server crash on concurrent map iteration and write: does not specify an upstream backport in the commit message
• 960aa55|UPSTREAM: : Avoid encoding in LogResponseObject when we are not going to use it: does not specify an upstream backport in the commit message
• bdddf28|UPSTREAM: : Eliminate AuditContext`s SetEventLevel: does not specify an upstream backport in the commit message
• c9c272b|UPSTREAM: : Fix tests after backport: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[openshift-ci-robot]

@ricardomaraschini: This pull request references Jira Issue OCPBUGS-56594, which is invalid:

• expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

[!Warning]
Reviewer: the target branch of this PR is master but the changes introduced here are already present in upstream master and release-1.34 branches. We may need to use the release-4.20 target branch here instead (we won't need to carry this over versions >= 1.34).

[!Note]
Reviewer: The back port of these upstream patches broke one of our unit tests. The broken test seems to cover features that are not available upstream (with_early_late_annotations_test.go).

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR contains upstream commits that are currently available on Kubernetes v1.34 and for which no back port plans exist. We need these to fix the bug we are currently seeing on the field.

The original PRs:

• Fix API server crash on concurrent map iteration and write kubernetes/kubernetes#129472
• Avoid encoding in LogResponseObject when we are not going to use it kubernetes/kubernetes#131725
• Eliminate AuditContext`s SetEventLevel kubernetes/kubernetes#131694

Which issue(s) this PR fixes:

Fixes OCPBUGS-56594

Does this PR introduce a user-facing change?

NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[benluddy]

For the test in the other patch, I think you want to introduce another commit to fix it, indicating in its commit message body the name of the commit that introduced "with_early_late_annotations_test.go" with instructions that it should be squashed on the next rebase. @bertinatto please correct me if I am wrong or missed anything about how we track changes to existing carries.

[openshift-ci-robot]

@ricardomaraschini: the contents of this pull request could not be automatically validated.

The following commits are valid:

• 1be80eb|UPSTREAM: 131725: Avoid encoding in LogResponseObject when we are not going to use it: the upstream PR kubernetes/kubernetes#131725 has merged
• 76f5a22|UPSTREAM: 129472: Fix API server crash on concurrent map iteration and write: the upstream PR kubernetes/kubernetes#129472 has merged
• bc53e37|UPSTREAM: 131694: Eliminate AuditContext`s SetEventLevel: the upstream PR kubernetes/kubernetes#131694 has merged

The following commits could not be validated and must be approved by a top-level approver:

• f96a151|UPSTREAM: : Fix tests after backport: does not specify an upstream backport in the commit message

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

[ricardomaraschini]

@benluddy I think I have done all the adjustments you have asked for. Can you please take a look ?

[benluddy]

/lgtm
/assign @bertinatto

Is f96a151 doing what we expect to get picked up and squashed during the next rebase?

[benluddy]

/jira refresh

[ricardomaraschini]

/retest

[ricardomaraschini]

/label acknowledge-critical-fixes-only

[ricardomaraschini]

OK, I commented in the wrong PR. I did not intend to add the acknowledge-critical-fixes-only label here. Removing.

[ricardomaraschini]

/remove-label acknowledge-critical-fixes-only

[wangke19]

/verified later @wangke19

[openshift-ci-robot]

@wangke19: This PR has been marked to be verified later by @wangke19.

Details

In response to this:

/verified later @wangke19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ricardomaraschini]

/test e2e-aws-ovn-fips

[ricardomaraschini]

This metal IPI on IPv6 test is consistently timing out, judging by the job history this seems to be the reality for multiple PRs.

[bertinatto]

/retest-required

[bertinatto]

/lgtm /assign @bertinatto

Is f96a151 doing what we expect to get picked up and squashed during the next rebase?

I prefer the commit title to match the one it's fixing, i.e., UPSTREAM: 115328: annotate early and late requests. That makes it obvious that it should be squashed. During rebases, we rarely have time to create the commit description.

So if you need to force-push more changes to this PR, please amend the commit title. Otherwise, leave it as is and we'll squash it during the 1.34 rebase.

tl;dr for @jacobsee: please squash f96a151 into UPSTREAM: 115328: annotate early and late requests.

[wangke19]

Maybe we can overwrite the e2e-metal-ipi-ovn-ipv6 test.

[ricardomaraschini]

Maybe we can overwrite the e2e-metal-ipi-ovn-ipv6 test.

This one has a recent history of failures. @benluddy What do you think ?

[benluddy]

INFO[2025-09-04T12:13:11Z] ci-operator version v20250903-744c3630c      
INFO[2025-09-04T13:41:44Z] Snapshot integration stream into release 4.21.0-0.ci-2025-09-04-133938-test-ci-op-rlrxzqm0-latest to tag release:latest  
INFO[2025-09-04T15:08:33Z] Step e2e-metal-ipi-ovn-ipv6-baremetalds-devscripts-setup succeeded after 1h24m59s. 
INFO[2025-09-04T15:08:33Z] Running step e2e-metal-ipi-ovn-ipv6-baremetalds-e2e-test. 
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 4h0m0s timeout","severity":"error","time":"2025-09-04T16:13:11Z"}
INFO[2025-09-04T16:13:11Z] Received signal.                              signal=interrupt
INFO[2025-09-04T16:13:11Z] error: Process interrupted with signal interrupt, cancelling execution... 
INFO[2025-09-04T16:13:11Z] cleanup: Deleting release pod release-latest 
INFO[2025-09-04T16:13:11Z] Step e2e-metal-ipi-ovn-ipv6-baremetalds-e2e-test failed after 1h4m37s. 

Hm, so it takes ~90 minutes to build and another ~90 minutes to do the cluster installation, leaving only 1hr for the actual test run and teardown? The job history goes back to March with apparently the same problem the entire time.

Could you open a PR to give this job a 5h timeout? I don't think we have so much urgency behind this PR that we can't take the time to fix it for everyone now.

[benluddy]

/remove-label backports/unvalidated-commits

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 1b675e1 and 2 for PR HEAD f96a151 in total

[openshift-ci]

@ricardomaraschini: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@ricardomaraschini: Jira Issue OCPBUGS-56594: All pull requests linked via external trackers have merged:

• openshift/kubernetes#2443

This pull request has the verified-later tag and will need to be manually moved to VERIFIED after testing. Jira Issue OCPBUGS-56594 has been moved to the MODIFIED state.

Details

In response to this:

[!Warning]
Reviewer: the target branch of this PR is master but the changes introduced here are already present in upstream master and release-1.34 branches. We may need to use the release-4.20 target branch here instead (we won't need to carry this over versions >= 1.34).

[!Note]
Reviewer: The back port of these upstream patches broke one of our unit tests. The broken test seems to cover features that are not available upstream (with_early_late_annotations_test.go).

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR contains upstream commits that are currently available on Kubernetes v1.34 and for which no back port plans exist. We need these to fix the bug we are currently seeing on the field.

The original PRs:

• Fix API server crash on concurrent map iteration and write kubernetes/kubernetes#129472
• Avoid encoding in LogResponseObject when we are not going to use it kubernetes/kubernetes#131725
• Eliminate AuditContext`s SetEventLevel kubernetes/kubernetes#131694

Which issue(s) this PR fixes:

Fixes OCPBUGS-56594

Does this PR introduce a user-facing change?

NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ricardomaraschini]

@benluddy It seems like someone has already bumped the timeout to 5h30m: openshift/release#68890

[ricardomaraschini]

/cherry-pick release-4.20

[openshift-cherrypick-robot]

@ricardomaraschini: new pull request created: #2450

Details

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.