OCPBUGS-37300: Fix forced reinit issue

[Vincent056]

This commit resolves issues in GetRemovedNodeHoldoffAnnotation and GetAddedNodeHoldoffAnnotation related to managing the holdoff annotation for FileIntegrity Object. Previously, node removal could fail in cases where the node to be removed was at the beginning, middle, or end of a comma separated list due to incorrect string manipulation.

[openshift-ci-robot]

@Vincent056: This pull request references Jira Issue OCPBUGS-37300, which is invalid:

• expected the bug to target the "4.19.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This commit resolves issues in GetRemovedNodeHoldoffAnnotation and GetAddedNodeHoldoffAnnotation related to managing the holdoff annotation for FileIntegrity Object. Previously, node removal could fail in cases where the node to be removed was at the beginning, middle, or end of a comma separated list due to incorrect string manipulation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Vincent056

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Vincent056]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@Vincent056: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-rosa	b386997	link	false	/test e2e-rosa

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[rhmdnd]

Looks like a reasonable change. Is there a way to force the operator into a state where it needs to exercise this code in an e2e test?

[rhmdnd]

nit: We could adjust the visibility of this since it's only used in this function.

[rhmdnd]

Is this where the main issue was? Otherwise, most of the logic looks the same, just in a different order, but here we changed the comparison from using strings.Contains to iterating the entire list of nodes and comparing them.

[Vincent056]

the main issue was because https://github.com/openshift/file-integrity-operator/pull/619/files#diff-a3206732c69662d901d8e5ccf08d084332c8dbfb58c67996b5bb9ec5b6ed4425L164, the replace is not correct

[xiaojiey]

@Vincent056 I deployed the fio operator from this PR #619. Two test scenarios tested. Both passed.
Scenario 1: Add annotations manually
I added the annotation file-integrity.openshift.io/holdoff:ip-10-0-50-40.us-east-2.compute.internal,ip-10-0-74-36.us-east-2.compute.internal,node-not-exist for fileintegrity manually. The annotations for the two nodes in ready status will be cleared in about 2-3 minutes.

% oc get fileintegrity example-fileintegrity -o custom-columns=NAME:metadata.name,ANNOTATIONS:metadata.annotations -n openshift-file-integrity -w
NAME                    ANNOTATIONS
example-fileintegrity   map[file-integrity.openshift.io/holdoff:ip-10-0-50-40.us-east-2.compute.internal,ip-10-0-74-36.us-east-2.compute.internal,node-not-exist kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
]
example-fileintegrity   map[file-integrity.openshift.io/holdoff:ip-10-0-50-40.us-east-2.compute.internal,node-not-exist file-integrity.openshift.io/re-init:ip-10-0-74-36.us-east-2.compute.internal kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
]
example-fileintegrity   map[file-integrity.openshift.io/holdoff:ip-10-0-50-40.us-east-2.compute.internal,node-not-exist file-integrity.openshift.io/re-init:ip-10-0-74-36.us-east-2.compute.internal kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
]
example-fileintegrity   map[file-integrity.openshift.io/holdoff:ip-10-0-50-40.us-east-2.compute.internal,node-not-exist kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
]
example-fileintegrity   map[file-integrity.openshift.io/holdoff:ip-10-0-50-40.us-east-2.compute.internal,node-not-exist kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
]
example-fileintegrity   map[file-integrity.openshift.io/holdoff:node-not-exist file-integrity.openshift.io/re-init:ip-10-0-50-40.us-east-2.compute.internal kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}

Scenario 2:

1. Scale up machineset add add more workers.
2. oc patch mcp worker --type='merge' -p '{"spec": {"maxUnavailable": 2}}'
3. Create a mc to trigger worker mcp restart.
4. Watch the annotations on fileintegrity:

% oc get fileintegrity example-fileintegrity -o custom-columns=NAME:metadata.name,ANNOTATIONS:metadata.annotations -n openshift-file-integrity -w
NAME                    ANNOTATIONS
example-fileintegrity   map[file-integrity.openshift.io/holdoff:node-not-exist,ip-10-0-1-108.us-east-2.compute.internal,ip-10-0-50-40.us-east-2.compute.internal kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
]
example-fileintegrity   map[file-integrity.openshift.io/holdoff:node-not-exist,ip-10-0-1-108.us-east-2.compute.internal,ip-10-0-50-40.us-east-2.compute.internal kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
]
example-fileintegrity   map[file-integrity.openshift.io/holdoff:node-not-exist,ip-10-0-1-108.us-east-2.compute.internal,ip-10-0-50-40.us-east-2.compute.internal kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
]
example-fileintegrity   map[file-integrity.openshift.io/holdoff:node-not-exist,ip-10-0-50-40.us-east-2.compute.internal file-integrity.openshift.io/re-init:ip-10-0-1-108.us-east-2.compute.internal kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
...
% oc get fileintegrity example-fileintegrity -o custom-columns=NAME:metadata.name,ANNOTATIONS:metadata.annotations -n openshift-file-integrity -w
NAME                    ANNOTATIONS
example-fileintegrity   map[file-integrity.openshift.io/holdoff:node-not-exist kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"fileintegrity.openshift.io/v1alpha1","kind":"FileIntegrity","metadata":{"annotations":{},"name":"example-fileintegrity","namespace":"openshift-file-integrity"},"spec":{"config":{"gracePeriod":60,"maxBackups":5},"debug":true}}
% oc get fileintegrity example-fileintegrity -o=jsonpath={.status.phase}
Active% 

[xiaojiey]

/label qe-approved

[xiaojiey]

/jira refresh

[openshift-ci-robot]

@xiaojiey: This pull request references Jira Issue OCPBUGS-37300, which is invalid:

• expected the bug to target the "4.19.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[xiaojiey]

/jira refresh

[openshift-ci-robot]

@xiaojiey: This pull request references Jira Issue OCPBUGS-37300, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @xiaojiey

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.