Skip to content

feat: integrate with cluster TLS security profile - #146

Closed
ugiordan wants to merge 1 commit into
opendatahub-io:masterfrom
ugiordan:RHOAIENG-61072-tls-profile
Closed

ugiordan wants to merge 1 commit into
opendatahub-io:masterfrom
ugiordan:RHOAIENG-61072-tls-profile

Conversation

@ugiordan

@ugiordan ugiordan commented Jun 10, 2026

Copy link
Copy Markdown
Member

Summary

  • Integrate feast-operator with the cluster-wide TLS security profile from apiservers.config.openshift.io/cluster
  • Uses controller-runtime-common/pkg/tls to fetch the profile at startup and apply it to webhook and metrics server TLSOpts
  • Registers SecurityProfileWatcher to restart on profile or adherence policy changes
  • Adds RBAC for config.openshift.io/apiservers (get/list/watch)
  • Fails closed on OpenShift (abort on 403/transport errors), gracefully falls back in non-OpenShift environments
  • Sets NextProtos explicitly for ALPN compliance
  • Removed the contradictory `!enableHTTP2` conditional that was setting h2 in NextProtos despite the flag name (always set NextProtos now)

Motivation

OCP 5.0 (GA October 2026) requires all components to honor the centralized TLS profile (OCPSTRAT-2611).

Reference: openshift/cluster-machine-approver #286

Changes

  • infra/feast-operator/cmd/main.go: TLS profile integration (fetch profile, build tls.Config, apply TLSOpts, register watcher)
  • infra/feast-operator/internal/controller/featurestore_controller.go: RBAC marker for apiservers
  • infra/feast-operator/config/rbac/role.yaml: regenerated with apiservers permission
  • infra/feast-operator/go.mod/go.sum: added controller-runtime-common, controller-runtime upgraded to v0.23.3

Test plan

  • go build ./... passes
  • Existing unit tests pass (no regressions)
  • Deploy on OpenShift cluster with Intermediate profile, verify TLS endpoints
  • Change profile to Modern, verify pod restarts and negotiates TLS 1.3
  • Verify NextProtos with openssl s_client -alpn

Ref: RHOAIENG-67682

Summary by CodeRabbit

  • New Features
    • Automatically detects and applies OpenShift API server TLS profile and adherence policy, using defaults when unavailable.
    • Watches for TLS and related policy changes and updates the controller runtime accordingly (enabling HTTP/2 negotiation).
  • Security / RBAC
    • Expanded controller permissions with read-only access to OpenShift apiserver configuration.
  • Chores
    • Updated the Go toolchain to 1.25.0 and refreshed project dependencies.

@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds OpenShift TLS profile bootstrap and dynamic reload to the feast operator. The operator now imports OpenShift config/v1 types, registers them in the controller-runtime scheme, fetches TLS profile and adherence policy at startup via a timeout-scoped bootstrap client (falling back on NotFound/NoMatch errors, failing fast on others), builds TLS options from profiles, registers a SecurityProfileWatcher to cancel a derived context when profiles change, and starts the manager with that context. RBAC rules grant access to config.openshift.io/apiservers with get, list, and watch verbs. Dependencies upgrade to Go 1.25.0 with bumped Kubernetes, controller-runtime, OpenShift API, and Prometheus libraries; indirect dependencies see removals (github.com/gogo/protobuf, github.com/pkg/errors) and replacements (sigs.k8s.io/structured-merge-diff/v4 → v6).

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 8 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description provides comprehensive details on motivation, changes, and test plan. However, it deviates significantly from the template structure by omitting required sections like 'Which issue(s) this PR fixes', explicit 'Checks' checkboxes, and 'Testing Strategy' selections. Add missing template sections: fill 'Which issue(s) this PR fixes' with issue links (OCPSTRAT-2611, RHOAIENG-61072), check applicable 'Testing Strategy' boxes, and confirm all 'Checks' items (tests passing, commits signed, conventional commits).
No Sensitive Data In Logs ⚠️ Warning Lines 143, 153, 171 in cmd/main.go log unredacted error objects from Kubernetes API calls (client.New, FetchAPIServerTLSProfile, FetchAPIServerTLSAdherencePolicy) that can contain bearer tokens/cre... Redact error objects before logging: extract only error.Reason() or sanitized error message, excluding error details that could contain bearer tokens or certificates.
✅ Passed checks (8 passed)
Check name Status Explanation
Title check ✅ Passed The title 'feat: integrate with cluster TLS security profile' clearly and concisely describes the main change—integrating TLS security profile support into the feast-operator.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Contribution Quality And Spam Detection ✅ Passed PR implements legitimate OCP 5.0 compliance requirement (OCPSTRAT-2611) with proper error handling; multi-file changes, no security theater signals, clean code quality, and legitimate Red Hat contr...
No Hardcoded Secrets ✅ Passed No hardcoded secrets found. Reviewed cmd/main.go (TLS integration), config/rbac/role.yaml (RBAC rules), featurestore_controller.go (RBAC marker), and go.mod/go.sum (dependencies). Verified no API k...
No Weak Cryptography ✅ Passed PR introduces no weak cryptography: no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB found; uses official OpenShift controller-runtime-common library for TLS; no custom crypto or non-constant-time comparisons...
No Injection Vectors ✅ Passed No injection vectors found. Code uses only hardcoded values: notebookGVK (lines 279-283) provides hardcoded "Notebook"/"kubeflow.org"; string concatenation (lines 368-369) derives CRD name from har...
No Privileged Containers ✅ Passed PR does not introduce privileged containers. Dockerfiles run non-root (USER 65532:65532), manifests enforce runAsNonRoot, allowPrivilegeEscalation: false, capabilities dropped. RBAC additions are r...

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands.

@ugiordan
ugiordan force-pushed the RHOAIENG-61072-tls-profile branch from 6eeb2b5 to e6fe91f Compare June 10, 2026 10:24
@ugiordan ugiordan changed the title feat(operator): integrate cluster TLS profile for OCP 5.0 compliance feat: integrate cluster TLS profile for OCP 5.0 compliance Jun 10, 2026
@ugiordan ugiordan changed the title feat: integrate cluster TLS profile for OCP 5.0 compliance feat: integrate with cluster TLS security profile Jun 11, 2026
@ugiordan
ugiordan force-pushed the RHOAIENG-61072-tls-profile branch from e6fe91f to 29495ec Compare June 16, 2026 11:05
@ugiordan
ugiordan requested review from jyejare and ntkathole and removed request for accorvin June 17, 2026 10:37

@jyejare jyejare left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds cluster TLS security profile integration to the Feast operator, enabling it to fetch OpenShift's cluster TLS configuration and adapt accordingly. The implementation follows OpenShift security best practices with proper fallback handling and includes extensive CRD updates. However, there are some concerns around HTTP/2 configuration and error handling that should be addressed.

Comment thread infra/feast-operator/cmd/main.go Outdated
Comment thread infra/feast-operator/cmd/main.go Outdated
Comment on lines +323 to 349
// Register SecurityProfileWatcher to restart on TLS profile changes
ctx, cancel := context.WithCancel(ctrl.SetupSignalHandler())
defer cancel()

if tlsProfileFetched {
watcher := &tlspkg.SecurityProfileWatcher{
Client: mgr.GetClient(),
InitialTLSProfileSpec: tlsProfile,
OnProfileChange: func(_ context.Context, _, _ configv1.TLSProfileSpec) {
setupLog.Info("TLS profile changed, initiating shutdown to reload")
cancel()
},
}
if tlsAdherenceFetched {
watcher.InitialTLSAdherencePolicy = tlsAdherence
watcher.OnAdherencePolicyChange = func(_ context.Context, _, _ configv1.TLSAdherencePolicy) {
setupLog.Info("TLS adherence policy changed, initiating shutdown to reload")
cancel()
}
}
if err := watcher.SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to set up TLS profile watcher")
os.Exit(1)
}
}

if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] Consider more graceful shutdown handling

The TLS profile watcher immediately cancels the context when profiles change, which triggers operator shutdown. While this ensures the new TLS config is applied, it might be disruptive in environments with frequent TLS profile changes. Consider adding a delay or debouncing mechanism.

@ugiordan ugiordan Jun 18, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SecurityProfileWatcher from controller-runtime-common invokes a caller-provided OnProfileChange callback when the profile changes. In our implementation (same as the OCP reference), that callback calls cancel() on the manager context, which triggers a graceful shutdown. Kubelet then restarts the pod with the new TLS config.

This is the standard pattern from openshift/cluster-machine-approver, and it's built into the SecurityProfileWatcher API design.

Debouncing isn't needed because TLS profile changes are rare cluster-level operations, and kubelet's restart backoff naturally handles the case where multiple changes happen in quick succession.

@ugiordan
ugiordan force-pushed the RHOAIENG-61072-tls-profile branch from 29495ec to e1a1952 Compare June 18, 2026 09:26
@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ugiordan
Once this PR has been reviewed and has the lgtm label, please ask for approval from jyejare. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ugiordan
ugiordan force-pushed the RHOAIENG-61072-tls-profile branch from e1a1952 to bc90065 Compare June 18, 2026 11:47

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
infra/feast-operator/cmd/main.go (1)

112-112: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Dead code: enableHTTP2 flag and stale comment block

The enableHTTP2 flag (line 112, 123-124) is declared and parsed but never read—NextProtos is now unconditionally set at lines 181-183. The comment block at lines 136-141 references the old HTTP/2 disabling behavior that no longer exists.

Remove to avoid confusion:

Proposed cleanup
 	var secureMetrics bool
-	var enableHTTP2 bool
 	var featureStoreMetrics bool
-	flag.BoolVar(&enableHTTP2, "enable-http2", false,
-		"If set, HTTP/2 will be enabled for the metrics and webhook servers")
-	// if the enable-http2 flag is false (the default), http/2 should be disabled
-	// due to its vulnerabilities. More specifically, disabling http/2 will
-	// prevent from being vulnerable to the HTTP/2 Stream Cancellation and
-	// Rapid Reset CVEs. For more information see:
-	// - https://github.com/advisories/GHSA-qppj-fm5r-hxr3
-	// - https://github.com/advisories/GHSA-4374-p667-p6c8
 	// Fetch cluster TLS profile from apiservers.config.openshift.io/cluster

Also applies to: 123-124, 136-141

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@infra/feast-operator/cmd/main.go` at line 112, The `enableHTTP2` variable
declaration and its associated flag parsing are dead code that is never used
since `NextProtos` is now set unconditionally elsewhere in the code. Remove the
`enableHTTP2` variable declaration, remove the flag parsing code that registers
this flag, and remove the stale comment block that references the old HTTP/2
disabling behavior that no longer exists. These removals will clean up confusion
and eliminate unused code.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@infra/feast-operator/cmd/main.go`:
- Around line 150-166: The bootstrap API calls use context.Background() without
any deadline, which can cause indefinite blocking and operator startup hangs.
Replace context.Background() with a context created using context.WithTimeout or
context.WithDeadline to add a reasonable timeout for the
FetchAPIServerTLSProfile call around line 151, and apply the same timeout
context to the other bootstrap API call mentioned at line 169. This ensures the
operator fails fast if the API server is slow or unreachable rather than hanging
indefinitely.
- Around line 181-183: The tlsOpts append operation unconditionally enables
HTTP/2 by setting NextProtos to include "h2", but this should only happen when
the enableHTTP2 flag is true. Wrap the tlsOpts append call that sets
c.NextProtos with a conditional check for the enableHTTP2 flag, so that HTTP/2
is only added to the TLS configuration when the flag is explicitly enabled. If
enableHTTP2 is false, either skip appending this TLS option entirely or
explicitly set NextProtos to only include "http/1.1".

---

Outside diff comments:
In `@infra/feast-operator/cmd/main.go`:
- Line 112: The `enableHTTP2` variable declaration and its associated flag
parsing are dead code that is never used since `NextProtos` is now set
unconditionally elsewhere in the code. Remove the `enableHTTP2` variable
declaration, remove the flag parsing code that registers this flag, and remove
the stale comment block that references the old HTTP/2 disabling behavior that
no longer exists. These removals will clean up confusion and eliminate unused
code.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: dce0ab20-8128-4620-aa0f-88e95a10d4f2

📥 Commits

Reviewing files that changed from the base of the PR and between 29495ec and bc90065.

⛔ Files ignored due to path filters (2)
  • infra/feast-operator/dist/install.yaml is excluded by !**/dist/**
  • infra/feast-operator/go.sum is excluded by !**/*.sum, !**/*.sum
📒 Files selected for processing (5)
  • infra/feast-operator/cmd/main.go
  • infra/feast-operator/config/crd/bases/feast.dev_featurestores.yaml
  • infra/feast-operator/config/rbac/role.yaml
  • infra/feast-operator/go.mod
  • infra/feast-operator/internal/controller/featurestore_controller.go
🚧 Files skipped from review as they are similar to previous changes (3)
  • infra/feast-operator/internal/controller/featurestore_controller.go
  • infra/feast-operator/config/rbac/role.yaml
  • infra/feast-operator/go.mod

Comment thread infra/feast-operator/cmd/main.go
Comment thread infra/feast-operator/cmd/main.go
@ugiordan
ugiordan force-pushed the RHOAIENG-61072-tls-profile branch 2 times, most recently from 33881a8 to c92fcdf Compare June 18, 2026 12:38
@ugiordan
ugiordan requested a review from jyejare June 18, 2026 14:34
@ugiordan

Copy link
Copy Markdown
Member Author

/retest

@jyejare

jyejare commented Jun 18, 2026

Copy link
Copy Markdown

@ugiordan Should the changes be applied at upstream itself?

@jyejare jyejare added ok-to-test sync-to-stable Sync the PR to stable automatically, after authors approval and CI checks. labels Jun 18, 2026
@ugiordan

Copy link
Copy Markdown
Member Author

@ugiordan Should the changes be applied at upstream itself?

Good question. The feast-operator already depends on openshift/api upstream (https://github.com/feast-dev/feast/blob/master/infra/feast-operator/go.mod). The TLS code gracefully falls back to hardened defaults on non-OpenShift clusters (IsNoMatchError/IsNotFound handling), so it wouldn't break anything for non-OpenShift users. The only new dependency would be controller-runtime-common. Happy to open an upstream PR if the maintainers are open to it.

@ugiordan
ugiordan force-pushed the RHOAIENG-61072-tls-profile branch from c92fcdf to e82c4b5 Compare June 22, 2026 13:48
@ugiordan

Copy link
Copy Markdown
Member Author

@jyejare any thoughts?

Honor the cluster-wide TLS security profile from
apiservers.config.openshift.io/cluster instead of hardcoding TLS
settings. Uses controller-runtime-common/pkg/tls to fetch the profile
at startup, apply it to webhook and metrics server TLSOpts, and watch
for profile changes via SecurityProfileWatcher.

On profile or adherence policy change, the manager context is cancelled
so the pod restarts with the new configuration.

Fails closed on OpenShift (abort on 403/transport errors), gracefully
falls back to defaults if the APIServer resource is not available
(non-OpenShift environments).

RHOAIENG-61072

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Ugo Giordano <ugiordan@redhat.com>
@ugiordan
ugiordan force-pushed the RHOAIENG-61072-tls-profile branch from e82c4b5 to 817ef09 Compare June 23, 2026 11:15
@ugiordan

Copy link
Copy Markdown
Member Author

/retest

@openshift-ci

openshift-ci Bot commented Jun 26, 2026

Copy link
Copy Markdown

@ugiordan: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-in-cluster-presubmit 817ef09 link true /test e2e-in-cluster-presubmit

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@rhods-ci-bot

Copy link
Copy Markdown

@ugiordan: The following test has Succeeded:

OCI Artifact Browser URL

View in Artifact Browser

Inspecting Test Artifacts Manually

To inspect your test artifacts manually, follow these steps:

  1. Install ORAS (see the ORAS installation guide).
  2. Download artifacts with the following commands:
mkdir -p oras-artifacts
cd oras-artifacts
oras pull quay.io/opendatahub/odh-ci-artifacts:feast-group-test-vsklj

@ntkathole

Copy link
Copy Markdown
Member

@ugiordan Should the changes be applied at upstream itself?

Good question. The feast-operator already depends on openshift/api upstream (https://github.com/feast-dev/feast/blob/master/infra/feast-operator/go.mod). The TLS code gracefully falls back to hardened defaults on non-OpenShift clusters (IsNoMatchError/IsNotFound handling), so it wouldn't break anything for non-OpenShift users. The only new dependency would be controller-runtime-common. Happy to open an upstream PR if the maintainers are open to it.

@ugiordan Yes, I think it would be great if this change goes upstream even though it;s openshift specific. It would cause less conflicts while syncing code in future.

@ugiordan

Copy link
Copy Markdown
Member Author

@ugiordan Should the changes be applied at upstream itself?

Good question. The feast-operator already depends on openshift/api upstream (https://github.com/feast-dev/feast/blob/master/infra/feast-operator/go.mod). The TLS code gracefully falls back to hardened defaults on non-OpenShift clusters (IsNoMatchError/IsNotFound handling), so it wouldn't break anything for non-OpenShift users. The only new dependency would be controller-runtime-common. Happy to open an upstream PR if the maintainers are open to it.

@ugiordan Yes, I think it would be great if this change goes upstream even though it;s openshift specific. It would cause less conflicts while syncing code in future.

Here it is feast-dev#6567

@ugiordan

ugiordan commented Jun 29, 2026

Copy link
Copy Markdown
Member Author

Closing this midstream PR. The upstream PR (feast-dev#6567) has been merged. The changes will come into the midstream via the regular upstream sync.

@ntkathole, @jyejare, once the sync cherry-pick PR is created, please link it to RHOAIENG-67682.

@ugiordan ugiordan closed this Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test sync-to-stable Sync the PR to stable automatically, after authors approval and CI checks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants