From adeb24765b6d6199bb374fbc6683d3804d5c1632 Mon Sep 17 00:00:00 2001 From: EladT <3798998+tzvatot@users.noreply.github.com> Date: Tue, 2 Jun 2026 15:49:17 +0300 Subject: [PATCH 001/450] OSAC-1002: align catalog items user stories with OSAC personas (#47) Replace user stories to match the defined OSAC personas (Cloud Provider Admin, Tenant Admin, Tenant User). The previous stories focused on implementation details (pre-define fields, prevent overrides) rather than the persona-driven workflows (create, publish, browse, provision). Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Elad Tabak --- enhancements/catalog-items/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/enhancements/catalog-items/README.md b/enhancements/catalog-items/README.md index 9c9036fed..c1ac1d215 100644 --- a/enhancements/catalog-items/README.md +++ b/enhancements/catalog-items/README.md @@ -39,13 +39,14 @@ don't have the ability to add or modify ansible roles. ### User Stories -* As a Cloud Provider Admin, I want to publish multiple catalog item that are similar to each other. -* As a Cloud Provider Admin, I want to publish multiple catalog item without having to create a new ansible role. -* As a Cloud Provider Admin, I want to offer catalog item to my users that pre-define the values for certain fields. -* As a Cloud Provider Admin, I want to offer catalog item to my users that prevent them from setting certain fields. +* As a Cloud Provider Admin, I need to create global catalog items that reference templates and define which fields are pre-set vs. editable, so I can offer curated options to all tenants. +* As a Cloud Provider Admin, I need to publish/unpublish catalog items to control visibility to Tenant Users. -* As a Tenant Admin, I want to offer catalog item to my users that pre-define the values for certain fields. -* As a Tenant Admin, I want to offer catalog item to my users that prevent them from setting certain fields. +* As a Tenant Admin, I need to create organization-specific catalog items from available templates, so I can offer tenant-specific configurations. +* As a Tenant Admin, I need to control which global catalog items are visible to users in my organization. + +* As a Tenant User, I need to list available catalog items via API and provision resources through them. +* As a Tenant User, I need to browse catalog items to find infrastructure automation options. ### Goals From d444982eda8e731c8e85ce6d7249f39cf32bd60a Mon Sep 17 00:00:00 2001 From: Avishay Traeger Date: Tue, 2 Jun 2026 17:36:37 +0300 Subject: [PATCH 002/450] OSAC-46: Add VM instance types enhancement proposal (#39) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * OSAC-46: Add VM instance types enhancement proposal - Phase 1 MVP: Provider-controlled global instance types - Strict mode: instance_type field required, cores/memory_gib removed - Enables/disables control visibility to org users - VMaaS only (ComputeInstance), CaaS uses HostType for bare metal - GPU support deferred to Phase 2 * Replace boolean enabled field with InstanceTypeState enum Changes based on PR discussion with mhrivnak and ygalblum: State management: - Replace `enabled` boolean with `InstanceTypeState` enum (ACTIVE, DEPRECATED, OBSOLETE) - Follow GCP deprecation model for lifecycle management - ACTIVE: Fully available for new VM creation - DEPRECATED: Available with warnings, includes optional replacement field - OBSOLETE: Cannot create new VMs (409 Conflict), still visible via GetInstanceType API behavior: - ListInstanceTypes: Returns ACTIVE and DEPRECATED by default, supports filter for OBSOLETE - GetInstanceType: Returns any state (for viewing details of existing VMs) - CreateComputeInstance: ACTIVE succeeds, DEPRECATED succeeds with warning, OBSOLETE rejected with 409 Summary updates: - Replace long "In the initial implementation..." with "All instance types are global" - Highlight that instance types exist only at gRPC API layer, not CRD layer Other changes: - Use osac-admin CLI for instance type management operations - Simplify Proposal section (just mention deprecation mechanism) - Remove parenthetical visibility qualifications - Add filter parameter support to ListInstanceTypes - Update all examples, validation rules, and test cases Addresses all review feedback from mhrivnak and ygalblum. Signed-off-by: Avishay Traeger Assisted-by: Claude Code * clarify name as primary identifier and uniqueness model Address review feedback from ygalblum about ID vs name usage: - Standardize on 'name' as the primary identifier for InstanceTypes - Clarify that instance type names are globally unique (Phase 1) - Update database schema to include UNIQUE constraint on name column - Update proto definitions to use 'name' field instead of 'id' - Update API paths to use {name} instead of {id} - Add new section explaining naming and uniqueness guarantees - Document future Phase 2 approach for tenant-specific types (prefixed names) - Update all validation rules to reference "name" consistently - Clarify that id field is internal database primary key (UUID) The name field serves as both the user-visible identifier and the reference used in ComputeInstance.spec.instance_type (e.g., "standard-4-16"). For future tenant-specific types, the proposal recommends a prefixed naming convention (e.g., "org-name/type-name") to maintain global uniqueness while supporting organizational customization. Assisted-by: Claude Code Signed-off-by: Avishay Traeger * docs(instance-types): fix troubleshooting terminology to use OBSOLETE and 409 Update troubleshooting section to use consistent lifecycle terminology: - Change "instance type disabled" to "instance type is obsolete" - Update HTTP status from 400 Bad Request to 409 Conflict - Use OBSOLETE/DEPRECATED/ACTIVE terminology consistently - Update resolution guidance to reference state transitions - Clarify deletion protection references instance type by name Addresses CodeRabbit review feedback. Assisted-by: Claude Code Signed-off-by: Avishay Traeger * docs(instance-types): add GCP-style deprecation metadata with timestamps Add comprehensive deprecation metadata following GCP's machine type deprecation model to support transparent migration timelines: Proto Changes: - Add InstanceTypeDeprecation message with state, replacement, deprecated, and obsolete timestamps - Replace standalone replacement field with deprecation sub-message - Auto-populate timestamps on state transitions API Changes: - Support --obsolete-at flag when deprecating instance types - Include deprecation metadata in List/Get responses - Warning messages include obsolete date when creating VMs with DEPRECATED types - Validate obsolete timestamp is in the future when transitioning to DEPRECATED Database Changes: - Store full deprecation metadata in JSONB data field - Add examples for both ACTIVE and DEPRECATED instance types Workflow Changes: - Admin can specify future obsolete date when deprecating - Users see migration timeline in warnings and list responses - Deprecation timestamps provide audit trail for lifecycle changes Testing: - Validate deprecation metadata in unit and integration tests - Test warning messages include obsolete dates - Verify auto-population of timestamps on state transitions Non-Goals (Phase 1): - Automatic state transitions based on timestamps (scheduled job deferred) Reference: https://cloud.google.com/compute/docs/reference/rest/v1/machineTypes Assisted-by: Claude Code Signed-off-by: Avishay Traeger * docs(instance-types): clarify name uniqueness semantics (never reusable) Fix contradictory soft-delete uniqueness policy in database schema: - Remove redundant partial unique index - Clarify that PRIMARY KEY on name prevents all duplicates - Document that instance type names are never reusable, even after soft deletion The PRIMARY KEY constraint already enforces global uniqueness across all rows, making the conditional unique index redundant and confusing. Addresses CodeRabbit review feedback. Assisted-by: Claude Code Signed-off-by: Avishay Traeger * docs(instance-types): clarify deprecation timestamp validation rules Address ygalblum's question about timestamp enforcement: Deprecation Timestamp Validation: - deprecated timestamp: Can be past or future (records when deprecation was/will be announced) - obsolete timestamp: Can be past or future for OBSOLETE state, must be future for DEPRECATED state - API behavior: Based on state field, not timestamps (future timestamps don't affect current behavior) Auto-population: - deprecated: Set to current time when transitioning to DEPRECATED (if not provided) - obsolete: Set to current time when transitioning to OBSOLETE (if not provided) Key Design Decision: State transitions are manual (admin-initiated), not automatic based on timestamps. Timestamps are for communication and planning, not enforcement. This allows admins to: - Backdate deprecated timestamp when recording historical decisions - Set future obsolete dates for migration planning - Manually control all state transitions Addresses review comment on line 288. Assisted-by: Claude Code Signed-off-by: Avishay Traeger * docs(instance-types): clarify manual state transitions in Phase 1 Fix workflow contradiction - remove misleading "(Future)" note that suggested automatic transitions might happen: - Change workflow step 6 from "(Future) Scheduled job automatically transitions" to explicit note that Phase 1 requires manual transitions - Keep Non-Goals bullet consistent: automatic transitions are out of scope - Clarify that Cloud Provider Admin must manually transition to OBSOLETE Phase 1 Behavior: - Timestamps communicate migration timeline to users - Admins manually trigger DEPRECATED → OBSOLETE transitions - No scheduled jobs or automatic state changes Future Phase: - Scheduled job could auto-transition based on obsolete timestamp - Remains explicitly out of scope for Phase 1 Addresses CodeRabbit review feedback about contradictory statements. Assisted-by: Claude Code Signed-off-by: Avishay Traeger * docs(instance-types): add warnings field to CreateComputeInstanceResponse Address CodeRabbit review feedback - specify API contract for DEPRECATED warnings: Proto Changes: - Add CreateComputeInstanceResponse message with warnings field - warnings: repeated string field containing deprecation messages - Example: "Instance type 'standard-2-4' is deprecated and will become obsolete on 2026-12-31. Consider migrating to 'standard-2-8'." Updated Documentation: - Workflow: Show CreateComputeInstanceResponse with warnings array - Validation: Reference CreateComputeInstanceResponse.warnings field explicitly - API behavior: Specify DEPRECATED returns warning in response.warnings field Warning Format: - Empty array for ACTIVE instance types - Contains deprecation message with replacement and obsolete timestamp for DEPRECATED types This provides a clear API contract for returning warnings consistently across all CreateComputeInstance operations. Addresses review feedback on lines 202-203. Assisted-by: Claude Code Signed-off-by: Avishay Traeger * docs(instance-types): update last-updated date to 2026-05-17 Signed-off-by: Avishay Traeger * docs(instance-types): address reviewer feedback - Add update_timestamp field for auditing (ygalblum) - Clarify soft-delete pattern used by DeleteInstanceType (ygalblum) - Change deletion error to not leak VM details, just indicate in-use (ygalblum, mhrivnak) - Add Private API List/Get operations for completeness (ygalblum) - Document labels/annotations reserved for future extensibility (ygalblum) - Explain JSONB usage follows OSAC standard pattern (ygalblum) - Add note about bidirectional state transitions (ygalblum) Co-Authored-By: Claude Sonnet 4.5 Signed-off-by: Avishay Traeger * docs(instance-types): document bidirectional state transitions Update lifecycle descriptions to show ACTIVE ↔ DEPRECATED ↔ OBSOLETE with bidirectional arrows and add Reactivating workflow section. Co-Authored-By: Claude Sonnet 4.5 Signed-off-by: Avishay Traeger --------- Signed-off-by: Avishay Traeger Co-authored-by: Claude Sonnet 4.5 --- enhancements/vm-instance-types/README.md | 730 +++++++++++++++++++++++ 1 file changed, 730 insertions(+) create mode 100644 enhancements/vm-instance-types/README.md diff --git a/enhancements/vm-instance-types/README.md b/enhancements/vm-instance-types/README.md new file mode 100644 index 000000000..eecb07a72 --- /dev/null +++ b/enhancements/vm-instance-types/README.md @@ -0,0 +1,730 @@ +--- +title: vm-instance-types +authors: + - atraeger@redhat.com +creation-date: 2026-05-06 +last-updated: 2026-06-01 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-46 +see-also: + - "/enhancements/vmaas" +replaces: +superseded-by: +--- + +# VM Instance Types + +## Summary + +This enhancement introduces VM instance types as pre-defined compute resource bundles (cores, memory) that can be referenced by name when creating virtual machines. Instance types simplify VM creation by providing a standardized set of compute configurations, following cloud-native patterns similar to AWS EC2 instance types, Azure VM sizes, and GCP machine types. All instance types are global. The instance type concept exists only at the gRPC API layer (fulfillment-service); at the Kubernetes CR level, instance types are expanded to cores/memory_gib fields. Users select an instance type by name rather than specifying individual cores and memory values. This proposal focuses on Virtual Machine-as-a-Service (VMaaS) compute instances only; Cluster-as-a-Service (CaaS) uses bare metal provisioning with a separate resource classification system (`HostType`) and is not covered by this enhancement. + +## Motivation + +The current ComputeInstance API allows users to specify compute resources (cores, memory) as individual numeric fields. While flexible, this approach creates several challenges: it complicates capacity planning and resource optimization, makes it harder to standardize VM offerings across tenants, and increases the cognitive load for users who must understand appropriate core-to-memory ratios. Instance types address these problems by providing a curated set of pre-validated compute configurations that align with infrastructure capabilities and organizational standards. This also lays the groundwork for future enhancements such as usage-based metering, capacity reservations, and custom instance type definitions (Phase 2). + +### User Stories + +* As a Organization User, I want to create a VM by selecting an instance type (e.g., 'standard-4-16') that defines the compute resources, so that I can quickly provision VMs without needing to understand appropriate core-to-memory ratios. +* As a Organization User, I want to list available instance types with their compute specifications, so that I can choose the right size for my workload. +* As a Cloud Provider Admin, I want to define global instance types including their compute specifications, so that I can standardize VM offerings and align them with available infrastructure capacity. +* As a Cloud Provider Admin, I want to enforce governance by requiring instance types for all VM creation, so that I can prevent resource fragmentation and ensure efficient capacity utilization. +* As a Cloud Provider Admin, I want to control instance type availability by managing lifecycle states (ACTIVE/DEPRECATED/OBSOLETE), so that I can manage which VM sizes are available to tenants without deleting instance type definitions. + +### Goals + +* Provide a self-service API for Organization Users to list available instance types and create VMs using instance type names +* Require all ComputeInstance creation to use instance types (strict mode), removing the ability to specify cores and memory individually +* Support globally-scoped instance types defined and managed by Cloud Provider Admins only +* Support instance type lifecycle states (ACTIVE ↔ DEPRECATED ↔ OBSOLETE) with bidirectional transitions, allowing graceful deprecation and reactivation +* Provide deprecation metadata (timestamps, replacement suggestions) to communicate migration timelines to users +* Ensure instance type compute specifications (cores, memory, name) are immutable after creation to prevent inconsistencies + +### Non-Goals + +The following are explicitly out of scope for Phase 1: + +* Organization-specific instance types (Organization Admin-defined types scoped to a single organization) +* Instance type "Series" with technical constraint ranges (min/max RAM-to-CPU ratios, supported architectures) +* Flexible mode allowing users to specify custom cores/memory values within series constraints +* GPU support (gpu_count, gpu_type fields) - deferred to Phase 2 +* Storage specifications as part of instance types (boot disk size remains separate, following AWS/Azure/GCP pattern) +* Quota enforcement and capacity planning based on instance types (quota system not yet defined in OSAC) +* Pricing or metering metadata associated with instance types +* Network bandwidth constraints or network performance tiers +* Per-organization instance type restrictions +* Automatic state transitions based on deprecation timestamps (scheduled job to transition DEPRECATED → OBSOLETE) + +## Proposal + +This proposal introduces a new `InstanceType` resource that defines pre-configured compute bundles with immutable core and memory specifications. The Cloud Service Provider creates and manages all instance types, which are globally visible to all organizations. The `ComputeInstance` gRPC API is modified to require an `instance_type` field and remove the existing `cores` and `memory_gib` fields (the Kubernetes CR retains these fields, populated by fulfillment-service). Disk specifications remain separate from instance types, as storage and compute are decoupled following industry-standard cloud patterns. Instance types include a deprecation mechanism to manage lifecycle transitions. + +### Key Resources + +**InstanceType** - Provider-defined compute specification +* Globally scoped, visible to all organizations +* Identified by name (globally unique, user-specified, e.g., "standard-4-16") +* Immutable compute specs (cores, memory_gib, name) +* Mutable metadata (description, state, deprecation) +* State lifecycle: ACTIVE ↔ DEPRECATED ↔ OBSOLETE (bidirectional transitions supported) +* Deprecation metadata includes timestamps and replacement suggestions for migration planning +* Cloud Provider Admin-only creation and management + +**ComputeInstance (modified)** - Virtual machine resource +* Now requires `instance_type` field (string reference to InstanceType name) +* Removes `cores` and `memory_gib` fields from ComputeInstanceSpec (API layer only) +* Retains `boot_disk` and `storage_class` as separate specifications + +### Scoping and Visibility + +All instance types in Phase 1 are globally scoped: +* **Created by:** Cloud Provider Admin only +* **Lifecycle states:** + - **ACTIVE**: Fully available for new VM creation, no warnings + - **DEPRECATED**: Available with warnings on VM creation (e.g., "Instance type 'standard-2-4' is deprecated. Consider using 'standard-2-8' instead.") + - **OBSOLETE**: Not available for new VM creation (rejected), still visible for lookups + +**API behavior by state:** +* ListInstanceTypes: Returns ACTIVE and DEPRECATED by default; supports filter parameter to include OBSOLETE (e.g., `filter: "state IN (ACTIVE, DEPRECATED, OBSOLETE)"`) +* GetInstanceType: Returns any instance type regardless of state (for viewing details by name) +* CreateComputeInstance: ACTIVE succeeds, DEPRECATED succeeds with warning in `CreateComputeInstanceResponse.warnings` field, OBSOLETE is rejected with 409 Conflict + +### Naming and Uniqueness + +* Instance type names are globally unique across the entire platform +* Name is the primary identifier for referencing instance types (not an opaque ID) +* ComputeInstance.spec.instance_type holds the name as a string (e.g., "standard-4-16") +* Names are immutable after creation to prevent reference breakage +* Database uses `name` as the primary key + +### Workflow Description + +#### Instance Type Management (Cloud Provider Admin) + +**Creating an Instance Type** + +1. Cloud Provider Admin uses the OSAC CLI to create a new global instance type: + ```bash + osac-admin create instance-type standard-4-16 \ + --cores 4 \ + --memory-gib 16 \ + --description "Balanced compute: 4 cores, 16 GiB RAM" + ``` +2. The Fulfillment Service validates the request and creates the InstanceType resource +3. The instance type is immediately available (state=ACTIVE by default) to all Organization Users + +**Deprecating an Instance Type** + +1. Cloud Provider Admin marks an instance type as deprecated with optional transition timeline: + ```bash + osac-admin update instance-type standard-2-4 \ + --state DEPRECATED \ + --replacement standard-2-8 \ + --obsolete-at 2026-12-31T23:59:59Z + ``` +2. The Fulfillment Service updates the InstanceType state and records deprecation metadata (current timestamp as `deprecated`, future timestamp as `obsolete`) +3. The instance type remains visible in ListInstanceTypes for Organization Users +4. New VM creation requests succeed but return a warning: "Instance type 'standard-2-4' is deprecated and will become obsolete on 2026-12-31. Consider migrating to 'standard-2-8'." +5. Existing VMs continue to run unchanged +6. **Note:** Phase 1 does not include automatic state transitions - Cloud Provider Admin must manually transition to OBSOLETE on or after the obsolete date + +**Obsoleting an Instance Type** + +1. Cloud Provider Admin marks an instance type as obsolete to prevent new VM creation: + ```bash + osac-admin update instance-type standard-2-4 --state OBSOLETE + ``` +2. The Fulfillment Service updates the InstanceType state and records obsolete timestamp +3. The instance type is hidden from ListInstanceTypes for Organization Users (unless explicitly filtered) +4. Existing VMs using this instance type continue to run unchanged +5. New VM creation requests with this instance type are rejected with 409 Conflict + +**Reactivating an Instance Type** + +1. Cloud Provider Admin restores a deprecated or obsolete instance type: + ```bash + osac-admin update instance-type standard-2-4 --state ACTIVE + ``` +2. The Fulfillment Service updates the InstanceType state to ACTIVE +3. The instance type becomes available for new VM creation again without warnings +4. Deprecation metadata (timestamps, replacement) is retained for historical reference + +**Deleting an Instance Type** + +1. Cloud Provider Admin attempts to delete an instance type: + ```bash + osac-admin delete instance-type standard-4-16 + ``` +2. The Fulfillment Service checks if any VMs reference this instance type +3. If in use: deletion is rejected with error "instance type is in use by at least one VM" +4. If not in use: deletion succeeds + +#### VM Creation (Organization User) + +**Listing Available Instance Types** + +1. Organization User lists available instance types: + ```bash + osac list instance-types + ``` +2. The Fulfillment Service returns ACTIVE and DEPRECATED instance types (OBSOLETE types can be included via filter parameter): + ```json + { + "items": [ + { + "name": "standard-2-4", + "metadata": { "name": "standard-2-4" }, + "cores": 2, + "memory_gib": 4, + "description": "Small balanced compute", + "state": "DEPRECATED", + "deprecation": { + "state": "DEPRECATED", + "replacement": "standard-2-8", + "deprecated": "2026-03-15T10:00:00Z", + "obsolete": "2026-12-31T23:59:59Z" + } + }, + { + "name": "standard-4-16", + "metadata": { "name": "standard-4-16" }, + "cores": 4, + "memory_gib": 16, + "description": "Medium balanced compute", + "state": "ACTIVE" + } + ] + } + ``` + +**Creating a VM with Instance Type** + +1. Organization User creates a VM specifying an instance type: + ```bash + osac create compute-instance my-vm \ + --instance-type standard-4-16 \ + --boot-disk-gib 50 \ + --image quay.io/fedora/fedora:40 + ``` +2. The Fulfillment Service validates and resolves: + - Instance type "standard-4-16" exists and state is ACTIVE or DEPRECATED + - User has appropriate permissions + - Resolves instance type to cores=4, memory_gib=16 + - If state is DEPRECATED, includes warning in `CreateComputeInstanceResponse.warnings` field +3. The API returns `CreateComputeInstanceResponse`: + ```json + { + "compute_instance": { /* ComputeInstance resource */ }, + "warnings": [] // Empty for ACTIVE instance types; contains deprecation message for DEPRECATED types + } + ``` +4. The Fulfillment Service creates the ComputeInstance CR with expanded values: + ```yaml + apiVersion: osac.openshift.io/v1alpha1 + kind: ComputeInstance + metadata: + name: my-vm + labels: + osac.io/instance-type-name: "standard-4-16" # Audit trail + spec: + cores: 4 # Expanded from instance type + memory_gib: 16 # Expanded from instance type + template: "default-vm-template" + boot_disk: + size_gib: 50 + image: + source_type: "registry" + source_ref: "quay.io/fedora/fedora:40" + ``` +5. The osac-operator reconciles the ComputeInstance, reading cores and memory_gib from spec +6. The VM is provisioned via KubeVirt with 4 cores and 16 GiB RAM + +**Error Cases** + +**Non-existent instance type:** +```bash +$ osac create compute-instance my-vm --instance-type nonexistent +Error: instance type "nonexistent" not found +``` + +**OBSOLETE instance type:** +```bash +$ osac create compute-instance my-vm --instance-type old-type +Error: instance type "old-type" is obsolete and cannot be used for new VMs +``` + +**Missing instance type:** +```bash +$ osac create compute-instance my-vm --boot-disk-gib 50 +Error: instance_type field is required +``` + +### API Extensions + +This enhancement modifies existing OSAC APIs and introduces new resources: + +#### New Resources + +**InstanceType** (proto/public/osac/public/v1/instance_type_type.proto) +- Public API: Read-only List and Get operations for Organization Users +- Private API: Full CRUD operations for Cloud Provider Admins + +**InstanceTypes Service** (proto/public/osac/public/v1/instance_types_service.proto) +- Public API: ListInstanceTypes, GetInstanceType +- Private API: ListInstanceTypes, GetInstanceType, CreateInstanceType, UpdateInstanceType, DeleteInstanceType + +#### Modified Resources + +**ComputeInstanceSpec** (proto/public/osac/public/v1/compute_instance_type.proto - API layer only) +- Add: `instance_type` field (string, required) - reference to InstanceType name +- Remove: `cores` field (optional int32) - replaced by instance_type in API +- Remove: `memory_gib` field (optional int32) - replaced by instance_type in API +- Note: fulfillment-service resolves instance_type name to cores/memory_gib when creating the CR +- Retain: `boot_disk`, `storage_class`, `template`, `image`, `user_data`, `ssh_key`, etc. + +Note: The Kubernetes CR schema retains cores/memory_gib fields. The fulfillment-service expands instance_type to these fields when creating the CR. See "Instance Type Resolution Strategy" for details. + +#### Validation + +**Public API (Organization Users):** +- CreateComputeInstance: Require instance_type field (string name), reject if missing +- CreateComputeInstance: Validate instance_type name exists and state is ACTIVE or DEPRECATED +- CreateComputeInstance: If state is DEPRECATED, include warning in `CreateComputeInstanceResponse.warnings` field with replacement suggestion (if set) and obsolete timestamp (if set) +- CreateComputeInstance: If state is OBSOLETE, reject with HTTP 409 Conflict ("instance type is obsolete") +- CreateComputeInstance: If instance_type name not found, reject with HTTP 404 Not Found +- CreateComputeInstance: Expand instance_type name to cores/memory_gib before creating CR +- CreateComputeInstance: Add osac.io/instance-type-name label to CR +- UpdateComputeInstance: Reject changes to instance_type field (immutable in Phase 1) + +**Private API (Cloud Provider Admin):** +- CreateInstanceType: Require name, cores, and memory_gib fields +- CreateInstanceType: Validate name uniqueness (globally unique across all instance types) +- CreateInstanceType: Validate cores > 0 and memory_gib > 0 +- CreateInstanceType: Default state to ACTIVE if not specified +- UpdateInstanceType: Reject changes to name, cores, or memory_gib (immutable) +- UpdateInstanceType: Allow changes to description, state, and deprecation fields +- UpdateInstanceType: State transitions are bidirectional (e.g., DEPRECATED → ACTIVE is allowed to "resurrect" an instance type if needed) +- UpdateInstanceType: When transitioning to DEPRECATED, auto-populate deprecation.deprecated timestamp (set to current time) if not provided +- UpdateInstanceType: When transitioning to OBSOLETE, auto-populate deprecation.obsolete timestamp (set to current time) if not provided +- UpdateInstanceType: If deprecation.deprecated is provided explicitly, allow any timestamp (past or future) - represents when deprecation was/will be announced +- UpdateInstanceType: If deprecation.obsolete is provided explicitly when transitioning to DEPRECATED, validate it is in the future (cannot set future DEPRECATED type to already-obsolete) +- UpdateInstanceType: If deprecation.obsolete is provided explicitly when transitioning to OBSOLETE, allow any timestamp (past or future) - represents when it became/will become obsolete +- UpdateInstanceType: API behavior is based on state field, not timestamps (future deprecation timestamps do not affect current behavior) +- DeleteInstanceType: Reject if any ComputeInstances reference this instance type by name +- DeleteInstanceType: Uses soft-delete pattern (sets deletion_timestamp), consistent with OSAC resource lifecycle management + +#### Deletion Protection + +**InstanceType deletion checks:** +- fulfillment-service queries database for ComputeInstances using this instance type name (via osac.io/instance-type-name label) +- If any references exist, deletion is rejected with 409 Conflict +- Error message indicates the instance type is in use (e.g., "Cannot delete instance type 'standard-4-16': instance type is in use") +- Deletion succeeds only when no ComputeInstances reference the instance type name + +### Implementation Details/Notes/Constraints + +#### Database Schema + +**instance_types table:** +```sql +CREATE TABLE instance_types ( + name TEXT PRIMARY KEY, -- User-specified name (globally unique, immutable, never reusable, e.g., "standard-4-16") + creation_timestamp TIMESTAMPTZ NOT NULL, + update_timestamp TIMESTAMPTZ NOT NULL, -- Last modification time for auditing + deletion_timestamp TIMESTAMPTZ, + finalizers TEXT[], + creators TEXT[], + tenants TEXT[], -- Empty for global instance types in Phase 1 + labels JSONB, -- Reserved for future extensibility (not exposed in Phase 1 Create/List APIs) + annotations JSONB, -- Reserved for future extensibility (not exposed in Phase 1 Create/List APIs) + data JSONB NOT NULL -- Serialized InstanceType protobuf +); + +-- No additional unique index needed: PRIMARY KEY on name enforces uniqueness for all rows (active and soft-deleted) +-- Instance type names are never reusable, even after soft deletion +``` + +The `data` JSONB column contains the serialized InstanceType protobuf. This follows OSAC's standard database pattern of storing proto messages as JSONB for schema flexibility and consistency across resources. + +Example of an ACTIVE instance type: +```json +{ + "name": "standard-4-16", + "metadata": { + "name": "standard-4-16" + }, + "cores": 4, + "memory_gib": 16, + "description": "Balanced compute: 4 cores, 16 GiB RAM", + "state": "ACTIVE" +} +``` + +Example of a DEPRECATED instance type with deprecation metadata: +```json +{ + "name": "standard-2-4", + "metadata": { + "name": "standard-2-4" + }, + "cores": 2, + "memory_gib": 4, + "description": "Small balanced compute", + "state": "DEPRECATED", + "deprecation": { + "state": "DEPRECATED", + "replacement": "standard-2-8", + "deprecated": "2026-03-15T10:00:00Z", + "obsolete": "2026-12-31T23:59:59Z" + } +} +``` + +**Note:** The top-level database `name` column (PRIMARY KEY) mirrors `data.name` and `data.metadata.name` for efficient querying and indexing. All three values must be identical for a given InstanceType record. + +#### Instance Type Resolution Strategy + +**Expansion at API boundary:** + +Instance types are resolved and expanded by fulfillment-service when creating the ComputeInstance CR. The CR spec contains the expanded compute resources (cores, memory_gib), not the instance type reference. This design choice provides: + +- **Zero changes to osac-operator:** Controller reads cores/memory_gib as it does today +- **No runtime dependencies:** osac-operator doesn't need to call fulfillment-service API +- **Enforcement at boundary:** Organization Users access only the fulfillment-service API (not k8s APIs), so instance type validation cannot be bypassed +- **Audit trail:** Original instance_type name stored in CR label + +**fulfillment-service flow:** +1. User creates ComputeInstance via API with `instance_type: "standard-4-16"` (name reference) +2. fulfillment-service queries database for InstanceType with `name = "standard-4-16"` +3. fulfillment-service validates instance type exists and state is ACTIVE or DEPRECATED +4. If state is DEPRECATED, add warning to response +5. fulfillment-service resolves cores=4, memory_gib=16 from InstanceType record +6. fulfillment-service creates ComputeInstance CR with: + - `spec.cores: 4` + - `spec.memory_gib: 16` + - `metadata.labels["osac.io/instance-type-name"]: "standard-4-16"` (audit trail and deletion protection lookup) + +**osac-operator reconciliation (unchanged):** +1. Read ComputeInstance.spec.cores and spec.memory_gib +2. Pass values to KubeVirt VM spec +3. Update ComputeInstance.status + +**Future extensibility:** + +Changing instance types is blocked in Phase 1 (spec.cores and spec.memory_gib are immutable). In the future, we could support instance type updates by: +1. Allowing updates to cores/memory_gib fields +2. Updating the `osac.io/instance-type-name` label +3. Triggering VM resize (if supported by underlying platform) + +Advanced OpenShift Virtualization features (NUMA topology, hugepages, CPU pinning) remain internal implementation details and are not exposed to Organization Users through the instance type API. + +#### Fulfillment Service Implementation + +**Server structure:** +- `InstanceTypesServer` (public) - wraps `PrivateInstanceTypesServer` with tenant filtering +- `PrivateInstanceTypesServer` (private) - full CRUD using GenericServer pattern +- Both servers in `internal/servers/instance_types_server.go` and `private_instance_types_server.go` + +**Authorization and scoping:** + +Instance type APIs follow the standard fulfillment-service authentication, authorization, and tenant-scoping policy. Instance-type-specific deltas: + +- **Global scope:** All instance types are globally scoped (no tenant filtering) +- **Admin-only write:** CreateInstanceType, UpdateInstanceType, DeleteInstanceType require Cloud Provider Admin role +- **State-based filtering:** ListInstanceTypes returns ACTIVE and DEPRECATED by default (use filter parameter for OBSOLETE) +- **Deletion protection:** DeleteInstanceType rejects if any ComputeInstances reference the type (API-layer query check) + +#### Proto Definition Sketch + +```proto +// proto/public/osac/public/v1/instance_type_type.proto +enum InstanceTypeState { + INSTANCE_TYPE_STATE_UNSPECIFIED = 0; + ACTIVE = 1; // Fully available for new VM creation + DEPRECATED = 2; // Available with warnings, migration recommended + OBSOLETE = 3; // Not available for new VMs, visible for lookups only +} + +message InstanceTypeDeprecation { + InstanceTypeState state = 1; // DEPRECATED or OBSOLETE (matches parent InstanceType.state) + string replacement = 2; // Optional: suggested replacement instance type name + google.protobuf.Timestamp deprecated = 3; // Optional: when deprecation was announced (auto-set on DEPRECATED transition) + google.protobuf.Timestamp obsolete = 4; // Optional: when it becomes/became obsolete (admin-specified or auto-set) +} + +message InstanceType { + string name = 1; // Primary identifier (globally unique, immutable, e.g., "standard-4-16") + Metadata metadata = 2; // Standard metadata (name field matches top-level name) + int32 cores = 3; // Immutable compute specification + int32 memory_gib = 4; // Immutable compute specification + string description = 5; // Mutable descriptive text + InstanceTypeState state = 6; // Mutable lifecycle state + InstanceTypeDeprecation deprecation = 7; // Optional: only set when state is DEPRECATED or OBSOLETE +} + +// proto/public/osac/public/v1/instance_types_service.proto +service InstanceTypes { + rpc List(ListInstanceTypesRequest) returns (ListInstanceTypesResponse) { + option (google.api.http) = { + get: "/api/fulfillment/v1/instance_types" + }; + } + rpc Get(GetInstanceTypeRequest) returns (InstanceType) { + option (google.api.http) = { + get: "/api/fulfillment/v1/instance_types/{name}" + }; + } +} + +// proto/private/osac/private/v1/instance_types_service.proto +service InstanceTypes { + rpc Create(CreateInstanceTypeRequest) returns (InstanceType); + rpc Update(UpdateInstanceTypeRequest) returns (InstanceType); + rpc Delete(DeleteInstanceTypeRequest) returns (google.protobuf.Empty); + rpc List(ListInstanceTypesRequest) returns (ListInstanceTypesResponse); + rpc Get(GetInstanceTypeRequest) returns (InstanceType); +} + +// GetInstanceTypeRequest references by name +message GetInstanceTypeRequest { + string name = 1; // Instance type name (e.g., "standard-4-16") +} + +// DeleteInstanceTypeRequest references by name +message DeleteInstanceTypeRequest { + string name = 1; // Instance type name (e.g., "standard-4-16") +} +``` + +**API vs CR Schema:** + +The fulfillment-service API and the Kubernetes CR have different schemas to support instance type expansion at the API boundary. + +**Public API Schema (proto/public/osac/public/v1/):** +```proto +message ComputeInstanceSpec { + string template = 1; + map template_parameters = 2; + optional google.protobuf.Timestamp restart_requested_at = 3; + optional ComputeInstanceImage image = 4; + + // NEW: Required instance type reference (API only) + string instance_type = 5; + + // REMOVED from API: cores and memory_gib (replaced by instance_type) + + optional string ssh_key = 7; + optional ComputeInstanceDisk boot_disk = 8; + repeated ComputeInstanceDisk additional_disks = 9; + optional string run_strategy = 10; + optional string user_data = 11; + optional string subnet = 12; + repeated string security_groups = 13; +} + +// CreateComputeInstanceResponse includes warnings for DEPRECATED instance types +message CreateComputeInstanceResponse { + ComputeInstance compute_instance = 1; + repeated string warnings = 2; // Warning messages (e.g., "Instance type 'standard-2-4' is deprecated and will become obsolete on 2026-12-31. Consider migrating to 'standard-2-8'.") +} +``` + +**Kubernetes CR Schema (osac-operator CRD):** +```yaml +# ComputeInstance CR (created by fulfillment-service) +apiVersion: osac.openshift.io/v1alpha1 +kind: ComputeInstance +metadata: + labels: + osac.io/instance-type-name: "standard-4-16" # Audit trail +spec: + template: "..." + cores: 4 # Expanded from instance type + memory_gib: 16 # Expanded from instance type + boot_disk: + size_gib: 50 + # ... other fields unchanged +``` + +The CR retains cores/memory_gib fields in spec (unchanged from today) so osac-operator requires no modifications. The instance_type field exists only in the API layer and is stored as a label in the CR. + +### Risks and Mitigations + +**Risk: Breaking change for existing API consumers** +- Mitigation: Since OSAC is pre-GA, breaking changes are acceptable + +**Risk: Instance type reference becomes stale if type is deleted** +- Mitigation: fulfillment-service API checks prevent deletion of InstanceTypes referenced by active VMs +- Mitigation: Admin must set type to OBSOLETE, wait for VMs to be deleted, then delete type + +**Risk: Users need a specific core/memory combination not offered** +- Mitigation: Document process for requesting new instance types from Cloud Provider Admin +- Mitigation: Phase 2 will introduce flexible mode with custom configurations within series bounds +- Mitigation: Admins can create arbitrary instance types to meet tenant needs + +**Risk: Capacity planning becomes harder without understanding actual resource allocation** +- Mitigation: ListInstanceTypes provides full spec visibility +- Mitigation: Future quota system will track instance type usage +- Mitigation: Admin APIs expose compute resource consumption per tenant + +### Drawbacks + +**Reduced flexibility for Organization Users** +- Users can no longer specify arbitrary core/memory combinations +- Trade-off: Simplicity and standardization vs. maximum flexibility +- Justification: Cloud providers typically constrain VM sizes for operational reasons (capacity planning, SLA guarantees, cost optimization) + +**Additional API resource and complexity** +- Introduces new InstanceType resource with CRUD operations +- Trade-off: More API surface area vs. simplified VM creation UX +- Justification: The pattern is well-established (AWS, Azure, GCP) and familiar to users + +## Alternatives (Not Implemented) + +**Alternative 1: Keep cores/memory_gib fields alongside instance_type** +- Allow users to either specify instance_type OR cores/memory_gib +- Rejected: Creates API ambiguity (what if both are specified?) +- Rejected: Doesn't achieve goal of standardization and governance +- Rejected: Complicates capacity planning and quota enforcement + +**Alternative 2: Make instance types optional (flexible mode from start)** +- Allow instance_type as a convenience but keep cores/memory_gib as primary fields +- Rejected: Doesn't enforce standardization +- Rejected: Deferred to Phase 2 with proper series-based constraints + +**Alternative 3: Use HostType instead of creating new InstanceType resource** +- Reuse existing HostType resource for VMs +- Rejected: HostType is intentionally opaque (only id/title/description) +- Rejected: HostType serves bare metal inventory labeling, not VM compute specs +- Rejected: Need structured cores/memory_gib for capacity planning and quotas + +**Alternative 4: Store instance type inline in ComputeInstanceSpec** +- Embed full instance type spec (cores, memory) directly in each ComputeInstance +- Rejected: Updates to instance type definitions wouldn't propagate to existing VMs +- Rejected: Breaks immutability guarantees (users could modify embedded spec) +- Rejected: Increases storage overhead and denormalization + +**Alternative 5: Dynamic instance type generation based on requested resources** +- Auto-create instance types on-demand when users specify custom cores/memory +- Rejected: Defeats purpose of governance and standardization +- Rejected: Infinite instance type growth makes management infeasible +- Rejected: Deferred to Phase 2 as explicit "custom instance types within series" + +## Test Plan + +**Unit Tests:** +- InstanceType CRUD operations via GenericServer +- ComputeInstance validation rejects missing instance_type +- ComputeInstance validation rejects OBSOLETE instance_type +- ComputeInstance validation rejects non-existent instance_type +- ComputeInstance with DEPRECATED instance_type succeeds with warning (includes obsolete timestamp in warning) +- InstanceType state transitions (ACTIVE ↔ DEPRECATED ↔ OBSOLETE, including reactivation) +- InstanceType deprecation metadata auto-population (deprecated timestamp on DEPRECATED transition, obsolete timestamp on OBSOLETE transition) +- InstanceType validation rejects obsolete timestamp in the past when transitioning to DEPRECATED +- InstanceType deletion rejected when VMs reference it +- InstanceType cores/memory_gib/name immutability enforcement +- InstanceType deprecation field mutability (replacement, timestamps) +- ListInstanceTypes filter behavior (default excludes OBSOLETE, filter includes it) +- ListInstanceTypes returns deprecation metadata for DEPRECATED/OBSOLETE types + +**Integration Tests (fulfillment-service/it/):** +- Create InstanceType, verify it appears in ListInstanceTypes +- Create ComputeInstance with valid instance_type, verify VM provisions with correct resources +- Attempt to create ComputeInstance with OBSOLETE instance_type, verify 409 Conflict +- Set InstanceType to DEPRECATED with future obsolete timestamp, verify warning includes obsolete date +- Create VM with DEPRECATED instance_type, verify warning message includes deprecation timeline +- Set InstanceType to OBSOLETE, verify it disappears from org user ListInstanceTypes +- Verify GetInstanceType returns deprecation metadata for DEPRECATED and OBSOLETE types +- Attempt to delete InstanceType with active VMs, verify rejection +- Delete all VMs, then delete InstanceType, verify success +- Create VM, update instance_type, verify rejection (immutability) + +**End-to-End Tests:** +- Cloud Provider Admin creates instance type via CLI +- Organization User lists instance types and sees new type +- Organization User creates VM using instance type +- osac-operator provisions KubeVirt VM with correct cores/memory +- VM reaches Running state +- Cloud Provider Admin sets instance type to DEPRECATED with replacement and future obsolete timestamp +- Organization User lists instance types and sees DEPRECATED status with deprecation metadata +- Organization User creates new VM with DEPRECATED type, receives warning with obsolete date +- Cloud Provider Admin sets instance type to OBSOLETE +- Organization User no longer sees type in list (unless filtered) +- Attempt to create new VM with OBSOLETE type fails with 409 Conflict +- Existing VM continues running + +**Migration Testing:** +- Run migration on test database with existing ComputeInstances +- Verify all VMs get assigned instance_type +- Verify compute resources match pre-migration values +- Verify new VM creation works post-migration + +## Graduation Criteria + +This enhancement is not targeting a specific OSAC release at this time. When targeting a release, graduation criteria will be defined based on production deployment feedback and user validation. + +Expected maturity progression: +- **Dev Preview:** Initial implementation with basic instance type CRUD and VM creation +- **Tech Preview:** Production deployment at pilot sites with migration tooling +- **GA:** Full production deployment with documented migration path and operational runbooks + +Success signals for graduation: +- Zero incidents related to instance type reference resolution +- Migration tooling successfully used at 2+ pilot sites +- Positive feedback from Cloud Provider Admins on manageability +- Positive feedback from Organization Users on simplified VM creation UX +- Documentation complete and validated by users + +## Upgrade/Downgrade Strategy + +Not applicable - OSAC is pre-GA. This is a breaking API change. + +## Version Skew Strategy + +Not applicable - OSAC is pre-GA. Components should be upgraded together. + +## Support Procedures + +### Operational Impact of API Extensions + +**New API resources:** +- InstanceType (CRD-like resource in fulfillment-service, not a Kubernetes CRD) +- Creates additional database table and API endpoints +- Increases fulfillment-service memory footprint (caching layer for instance type lookups) + +**Modified API resources:** +- ComputeInstance.spec.instance_type (new required field) +- Breaking change: cores/memory_gib fields removed + +**Failure Modes and Troubleshooting:** + +**ComputeInstance stuck in Pending with invalid resource values** +- **Detection:** `kubectl get computeinstance -o yaml` shows invalid cores/memory_gib values +- **Diagnosis:** Instance type definition was corrupted or manually modified in database +- **Resolution:** Delete and recreate VM via API with valid instance type, or fix database corruption +- **Prevention:** Only modify instance types via fulfillment-service API, never directly in database + +**ComputeInstance creation rejected: "instance type is obsolete"** +- **Detection:** API returns 409 Conflict with message indicating instance type is OBSOLETE +- **Diagnosis:** Cloud Provider Admin marked the instance type as OBSOLETE +- **Resolution:** Choose a different instance type (check replacement suggestion if available) or ask admin to change state back to ACTIVE/DEPRECATED +- **Prevention:** Communicate instance type lifecycle changes (OBSOLETE transitions) to users before making the change + +**osac-operator reconciliation failures** +- **Detection:** Controller logs show errors provisioning VM or invalid spec values +- **Diagnosis:** ComputeInstance CR has invalid cores/memory_gib values (data corruption or manual edit) +- **Resolution:** Check CR spec.cores and spec.memory_gib; if invalid, delete CR and recreate VM via API +- **Prevention:** Never manually edit ComputeInstance CRs; always use fulfillment-service API + +**InstanceType cannot be deleted: "referenced by active VMs"** +- **Detection:** DELETE API call returns 409 Conflict +- **Diagnosis:** fulfillment-service database query found ComputeInstances using this instance type name +- **Resolution:** Delete all referencing VMs first, then delete instance type. Or set instance type to OBSOLETE and delete later. +- **Prevention:** Use List API or query database to find referencing VMs before attempting deletion + +### Disabling / Rollback + +Not supported. + +## Infrastructure Needed + +No additional infrastructure is required for this enhancement. From cba4bd481b1ab303684c15e8dea218285ebb7ab9 Mon Sep 17 00:00:00 2001 From: Ygal Blum Date: Wed, 3 Jun 2026 06:27:57 -0400 Subject: [PATCH 003/450] OSAC-979: Add image management enhancement proposal (#44) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What Add a ComputeImage API resource for managing VM images as first-class entities in OSAC. ## Why Today, ComputeInstance creation accepts arbitrary image URLs with no access control, discoverability, or governance. Every major cloud provider treats VM images as managed resources — OSAC needs the same to be a credible VM-as-a-Service platform. ## Changes - Introduce ComputeImage resource with CRUD API for registering, listing, and managing VM image metadata - Support two-tier visibility: provider-global images (all tenants) and tenant-scoped images - Define image lifecycle states: AVAILABLE, DEPRECATED (with warnings), and OBSOLETE (blocks new VMs) - Update ComputeInstance to reference a registered ComputeImage by ID instead of accepting arbitrary URLs - Include deletion protection preventing removal of images referenced by active ComputeInstances - Define authorization model for provider admins, tenant admins, and tenant users - Specify database schema, list filtering, deprecation metadata, and migration strategy ## Testing Comprehensive test plan covering unit tests for CRUD and validation, integration tests for multi-tenant visibility and lifecycle transitions, and end-to-end tests for the full image registration through VM provisioning workflow. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.6 Signed-off-by: Ygal Blum --- enhancements/image-management/README.md | 890 ++++++++++++++++++++++++ 1 file changed, 890 insertions(+) create mode 100644 enhancements/image-management/README.md diff --git a/enhancements/image-management/README.md b/enhancements/image-management/README.md new file mode 100644 index 000000000..a941ddef2 --- /dev/null +++ b/enhancements/image-management/README.md @@ -0,0 +1,890 @@ +--- +title: image-management +authors: + - yblum@redhat.com +creation-date: 2026-05-26 +last-updated: 2026-05-26 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-979 +see-also: + - "/enhancements/vmaas" + - "/enhancements/catalog-items" +replaces: + - N/A +superseded-by: + - N/A +--- + +# OSAC Image Management + +## Summary + +This enhancement introduces a **ComputeImage** API resource for managing virtual machine images as first-class entities in OSAC. Instead of users specifying arbitrary image URLs when creating a ComputeInstance, administrators register images that point to OCI-compatible artifacts in a registry, and users select from this curated list of images. This first milestone focuses on metadata-level image management with out-of-band image upload — OSAC tracks and serves the list of images but does not handle the binary upload itself. The design supports both provider-global images (available to all tenants) and tenant-scoped images (visible only within a single tenant). + +## Terminology + +- **ComputeImage**: An OSAC API resource representing a registered VM image. It contains metadata (name, description, OS info) and a reference to an OCI artifact in a registry. ComputeImages are the unit of selection when creating a ComputeInstance. +- **OCI artifact**: An Open Container Initiative-compatible image stored in a container registry. VM disk images can be packaged as OCI artifacts and imported by KubeVirt into DataVolumes for use as root disks. +- **Global image**: A ComputeImage created by a provider administrator with no tenant set (`metadata.tenant` is empty). Visible to all tenants. +- **Tenant image**: A ComputeImage created by a tenant administrator. The server automatically sets `metadata.tenant` to the caller's tenant, scoping visibility to that tenant only. +- **Out-of-band upload**: The process of pushing an OCI artifact to a registry outside of OSAC (e.g., using `podman push`, `skopeo copy`, or Quay UI). OSAC does not mediate the upload in this milestone. +- **Image reference**: The OCI image reference string (e.g., `registry.example.com/images/rhel9:latest`) that points to the actual artifact in a registry. + +## Motivation + +Today, ComputeInstance creation accepts an arbitrary image URL via the `source_ref` field. This has several problems: + +1. **No access control**: Any user can point to any image, including untrusted or unauthorized sources. +2. **No discoverability**: Users must know the exact OCI reference — there is no way to list available images. +3. **No governance**: Administrators cannot restrict which images are available to tenants. +4. **No metadata**: Users see raw OCI references with no human-readable context (OS type, version, description). +5. **No consistency**: Different users may use different URLs for the same logical image, making auditing difficult. + +Every major cloud provider (AWS AMI, GCP Images, Azure VM Images) treats VM images as managed resources with dedicated APIs. OSAC needs the same to be a credible VM-as-a-Service platform. + +### User Stories + +**Provider Administrator:** + +* As a provider administrator, I want to register base OS images (RHEL, CentOS, Ubuntu) as global ComputeImages so that all tenants can discover and use them without needing to know the underlying OCI reference. + +* As a provider administrator, I want to set a human-readable display name and description on each ComputeImage so that users can make informed selections without understanding OCI artifact conventions. + +* As a provider administrator, I want to list all registered images (global and tenant-scoped) so that I can audit the list across the platform. + +* As a provider administrator, I want to update the metadata of a global image (e.g., change description, fix a typo in the display name) so that I can maintain the platform's list of base images. + +* As a provider administrator, I want to delete a global image that is no longer supported so that tenants are not using outdated or insecure base images. + +* As a provider administrator, I want to deprecate a global image and specify a replacement so that tenants are warned to migrate before the image becomes obsolete. + +* As a provider administrator, I want to mark an obsolete global image so that new VM creation with that image is blocked while existing VMs remain unaffected. + +* As a provider administrator, I want to reactivate a previously deprecated or obsolete global image so that tenants can resume using it if circumstances change (e.g., a replacement image is recalled, or the original image receives a critical security patch). + +**Tenant Administrator:** + +* As a tenant administrator, I want to register custom images for my organization so that my users can create VMs from our internally approved golden images. + +* As a tenant administrator, I want to list all images available to my tenant (both global and tenant-scoped) so that I can verify which images my users will see. + +* As a tenant administrator, I want to update the metadata of a tenant-scoped image (e.g., change description, deprecate it) so that I can manage my organization's image lifecycle. + +* As a tenant administrator, I want to delete a tenant-scoped image that is no longer needed so that my users are not confused by stale entries. + +* As a tenant administrator, I want to deprecate a tenant-scoped image and point users to a replacement so that my team migrates to newer images on a clear timeline. + +* As a tenant administrator, I want to mark a tenant-scoped image as obsolete so that my users cannot create new VMs with an unsupported image. + +* As a tenant administrator, I want to reactivate a previously deprecated or obsolete tenant-scoped image so that my users can resume using it if needed. + +**Tenant User:** + +* As a tenant user, I want to list all images available to me so that I can choose the right base image for my VM. + +* As a tenant user, I want to see the display name, description, OS type, and architecture of each image so that I can make an informed choice without consulting documentation or administrators. + +* As a tenant user, I want to create a ComputeInstance that references a ComputeImage by ID so that I use only approved, registered images. + +### Goals + +1. Provide a dedicated ComputeImage API for registering, listing, and managing VM image metadata. +2. Support two-tier visibility: provider-global images (visible to all tenants) and tenant-scoped images (visible only within a tenant). +3. Allow provider and tenant administrators to register images that reference OCI artifacts in a registry. +4. Expose human-readable metadata (display name, description, OS type, architecture) alongside the technical OCI reference. +5. Update ComputeInstance to reference a registered ComputeImage instead of accepting arbitrary image URLs. +6. Enforce that only registered images can be used for VM creation, preventing use of unvetted sources. + +### Non-Goals + +1. **Image upload/push**: This milestone does not provide an API for uploading image binaries to a registry. Upload is performed out-of-band. +2. **VM snapshot/export**: Creating images from running VMs (snapshot-to-image workflow) is a future capability. +3. **Image scanning or vulnerability analysis**: Integration with security scanning tools is deferred. +4. **Registry management**: OSAC does not deploy, configure, or manage the underlying container registry in this milestone. +5. **Container image management**: This proposal covers VM disk images only, not general container images or AI model artifacts. +6. **Quota or rate limiting**: Resource quotas on the number of images per tenant are deferred to the quota system proposal. +7. **Image versioning or tagging**: Tracking multiple versions of the same logical image (e.g., RHEL 9.1, 9.2, 9.3) is left to the administrator's naming conventions for now. +8. **Image caching on workload clusters**: This milestone does not change the osac-operator's image handling. Pre-pulling or caching images on target clusters for faster VM startup is out of scope. +9. **Private registry authentication**: This milestone assumes the OCI registry referenced by `source_ref` is accessible without additional credentials from the workload cluster. Authentication to private registries (e.g., pull secrets, registry credentials management) is not addressed. A future milestone will either add private registry support with credential management, or make it unnecessary by requiring all images to be uploaded to an OSAC-managed registry via the planned upload API. +10. **Multi-disk image support**: Since ComputeInstance currently supports only a single disk image, ComputeImage follows the same single-disk model. Future implementations may add support for multi-disk images similar to AWS AMIs, where a single image definition bundles multiple block device mappings. + +## Proposal + +### Overview + +A new **ComputeImage** resource is added to the OSAC fulfillment-service API. ComputeImages are registered by administrators (provider or tenant level) and represent VM images stored as OCI artifacts in a container registry. Each ComputeImage record holds: + +- A human-readable display name and description +- The OCI image reference pointing to the actual artifact +- OS metadata (type, version, architecture) +- Tenant scoping via `metadata.tenant` (empty = global, set = tenant-scoped) + +Users interact with ComputeImages through List and Get operations. When creating a ComputeInstance, the user specifies a ComputeImage ID instead of a raw image URL. + +### Workflow Description + +**Actors:** + +- **Provider administrator**: Manages the OSAC platform. Can create global ComputeImages visible to all tenants. +- **Tenant administrator**: Manages resources within a tenant. Can create tenant-scoped ComputeImages. +- **Tenant user**: Consumes resources within a tenant. Can list/get ComputeImages and reference them in ComputeInstances. + +#### Image Management (Provider Administrator) + +**Creating a Global ComputeImage** + +1. Provider administrator uploads a VM disk image to the platform's OCI registry out-of-band (e.g., using `skopeo copy` or the Quay UI). +2. Provider administrator uses the OSAC CLI to register the image: + ```bash + osac create compute-image \ + --display-name "Red Hat Enterprise Linux 9.4" \ + --description "RHEL 9.4 base image with cloud-init support" \ + --source-ref "registry.internal.example.com/osac-images/rhel9:9.4" \ + --os-type linux \ + --os-version "RHEL 9.4" \ + --architecture amd64 \ + --min-cpus 2 \ + --min-memory-mib 2048 \ + --min-disk-gib 20 + ``` +3. The Fulfillment Service validates the request and creates the ComputeImage resource with `metadata.tenant` empty (global). +4. The image is immediately available (state=AVAILABLE by default) to all tenants. + +**Note:** The fulfillment-service does not verify the existence or accessibility of the OCI artifact at `source_ref` during image registration. The image may exist at creation time and be deleted later, or may not yet be pushed at registration time. It is the responsibility of the administrator (provider or tenant) to ensure the address is correct and the image remains available in the registry for as long as it is registered with OSAC. + +**Deprecating a ComputeImage** + +1. Provider administrator marks a ComputeImage as deprecated with optional transition timeline: + ```bash + osac update compute-image \ + --state DEPRECATED \ + --replacement \ + --obsolete-at 2026-12-31T23:59:59Z + ``` +2. The Fulfillment Service updates the ComputeImage state and records deprecation metadata (current timestamp as `deprecated`, future timestamp as `obsolete`). +3. The image remains visible in `ListComputeImages` for all users. +4. New VM creation requests succeed but return a warning: *"ComputeImage 'rhel9-9.3' is deprecated and will become obsolete on 2026-12-31. Consider migrating to 'rhel9-9.4'."* +5. Existing ComputeInstances are unaffected — they retain their resolved OCI reference. +6. **Note:** Phase 1 does not include automatic state transitions — the administrator must manually transition to OBSOLETE on or after the obsolete date. + +**Obsoleting a ComputeImage** + +1. Provider administrator marks a ComputeImage as obsolete to prevent new VM creation: + ```bash + osac update compute-image --state OBSOLETE + ``` +2. The Fulfillment Service updates the ComputeImage state and records the obsolete timestamp. +3. The image is hidden from `ListComputeImages` by default (unless explicitly filtered by state). +4. `GetComputeImage` returns any image regardless of state, so direct lookups still work. +5. Existing ComputeInstances using this image continue to run unchanged. +6. New VM creation requests with this image are rejected with 409 Conflict. + +**Reactivating a ComputeImage** + +1. Provider administrator reactivates a previously deprecated or obsolete ComputeImage: + ```bash + osac update compute-image --state AVAILABLE + ``` +2. The Fulfillment Service updates the ComputeImage state to AVAILABLE. +3. The image becomes fully available for new VM creation without warnings. +4. Deprecation metadata (timestamps, replacement) is retained for historical reference. +5. The image appears in default `ListComputeImages` results again (if it was OBSOLETE and previously hidden). + +**Updating a ComputeImage** + +1. Provider administrator updates mutable fields on a ComputeImage: + ```bash + osac update compute-image \ + --display-name "Red Hat Enterprise Linux 9.4 (Updated)" \ + --description "RHEL 9.4 base image with cloud-init and security patches" + ``` +2. The Fulfillment Service validates and applies the update. Mutable fields: `display_name`, `description`, `os_version`, `architecture`, `min_requirements`. +3. The `source_ref` is immutable — to point to a new artifact version, create a new ComputeImage and delete the old one. + +**Deleting a ComputeImage** + +1. Provider administrator attempts to delete a ComputeImage: + ```bash + osac delete compute-image + ``` +2. The Fulfillment Service checks if any active ComputeInstances reference this image. +3. If in use: deletion is rejected with 409 Conflict: *"ComputeImage is in use by at least one ComputeInstance"*. +4. If not in use: soft-delete succeeds (sets `deleted_at`, record is excluded from list/get queries but retained in the database). + +#### Image Management (Tenant Administrator) + +**Creating a Tenant-Scoped ComputeImage** + +1. Tenant administrator uploads a custom image to the registry out-of-band. +2. Tenant administrator uses the OSAC CLI to register the image: + ```bash + osac create compute-image \ + --display-name "Custom RHEL 9 with GPU Drivers" \ + --description "RHEL 9.4 with NVIDIA GPU drivers pre-installed" \ + --source-ref "registry.internal.example.com/tenant-acme/rhel9-gpu:v2" \ + --os-type linux \ + --os-version "RHEL 9.4" \ + --architecture amd64 + ``` +3. The Fulfillment Service validates the request and automatically sets `metadata.tenant` to the caller's tenant. +4. The image is visible only to users within that tenant. + +**Note:** The fulfillment-service does not verify the existence or accessibility of the OCI artifact at `source_ref` during image registration. The image may exist at creation time and be deleted later, or may not yet be pushed at registration time. It is the responsibility of the tenant administrator to ensure the address is correct and the image remains available in the registry for as long as it is registered with OSAC. + +Tenant administrators can also deprecate, obsolete, update, and delete their own tenant-scoped images using the same commands shown above for provider administrators. Tenant administrators cannot modify or delete global images or other tenants' images. + +#### VM Creation Using a ComputeImage (Tenant User) + +**Listing Available ComputeImages** + +1. Tenant user lists available images: + ```bash + osac list compute-images + ``` +2. The Fulfillment Service returns AVAILABLE and DEPRECATED images (OBSOLETE images can be included via filter parameter). Results include both global images and the user's tenant-scoped images: + ```json + { + "items": [ + { + "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "metadata": { + "tenant": "" + }, + "display_name": "Red Hat Enterprise Linux 9.4", + "description": "RHEL 9.4 base image with cloud-init support", + "source_ref": "registry.internal.example.com/osac-images/rhel9:9.4", + "os_type": "linux", + "os_version": "RHEL 9.4", + "architecture": "amd64", + "min_requirements": { + "min_cpus": 2, + "min_memory_mib": 2048, + "min_disk_gib": 20 + }, + "state": "AVAILABLE" + }, + { + "id": "948c850b-5670-478d-b4be-e16642f88b9b", + "metadata": { + "tenant": "" + }, + "display_name": "Red Hat Enterprise Linux 9.3", + "description": "RHEL 9.3 base image (deprecated)", + "source_ref": "registry.internal.example.com/osac-images/rhel9:9.3", + "os_type": "linux", + "os_version": "RHEL 9.3", + "architecture": "amd64", + "state": "DEPRECATED", + "deprecation": { + "replacement": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "deprecated": "2026-06-01T00:00:00Z", + "obsolete": "2026-12-31T23:59:59Z" + } + }, + { + "id": "6997aa1a-7bcc-4b14-bf1e-18d91fb64e33", + "metadata": { + "tenant": "0c9e7c27-8558-4641-84cb-c72daf811de7" + }, + "display_name": "Custom RHEL 9 with GPU Drivers", + "description": "RHEL 9.4 with NVIDIA GPU drivers pre-installed", + "source_ref": "registry.internal.example.com/tenant-acme/rhel9-gpu:v2", + "os_type": "linux", + "os_version": "RHEL 9.4", + "architecture": "amd64", + "state": "AVAILABLE" + } + ] + } + ``` + +**Creating a VM with a ComputeImage** + +1. Tenant user creates a VM specifying a ComputeImage: + ```bash + osac create compute-instance my-vm \ + --compute-image a1b2c3d4-e5f6-7890-abcd-ef1234567890 \ + --instance-type standard-4-16 \ + --boot-disk-gib 50 + ``` +2. The Fulfillment Service validates and resolves: + - ComputeImage `a1b2c3d4-...` exists and is visible to the user's tenant + - Image state is AVAILABLE or DEPRECATED + - Retrieves the `source_ref` from the ComputeImage + - If state is DEPRECATED, includes warning in `CreateComputeInstanceResponse.warnings` field +3. The API returns `CreateComputeInstanceResponse`: + ```json + { + "compute_instance": { /* ComputeInstance resource */ }, + "warnings": [] + } + ``` +4. The Fulfillment Service creates the ComputeInstance CR with the resolved OCI reference: + ```yaml + apiVersion: osac.openshift.io/v1alpha1 + kind: ComputeInstance + metadata: + name: my-vm + spec: + compute_image: "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + instance_type: "standard-4-16" + boot_disk: + size_gib: 50 + image: + source_type: "registry" + source_ref: "registry.internal.example.com/osac-images/rhel9:9.4" + ``` +5. The osac-operator reconciles the ComputeInstance, reading the resolved `source_ref` from the image spec. +6. The VM is provisioned via KubeVirt using the OCI artifact from the registry. + +**Note:** The fulfillment-service does not verify the existence or accessibility of the OCI artifact at the ComputeImage's `source_ref` during VM creation. If the artifact is unreachable or invalid, the error will surface at VM provisioning time from KubeVirt/osac-operator, not from the fulfillment-service API. + +**Error Cases** + +**Non-existent ComputeImage:** +```bash +$ osac create compute-instance my-vm --compute-image nonexistent-id +Error: compute image "nonexistent-id" not found +``` + +**ComputeImage not visible to tenant:** +```bash +$ osac create compute-instance my-vm --compute-image +Error: compute image "" not found +``` + +**OBSOLETE ComputeImage:** +```bash +$ osac create compute-instance my-vm --compute-image +Error: ComputeImage 'rhel9-9.3' is obsolete and cannot be used for new VMs +``` + +**Missing ComputeImage:** +```bash +$ osac create compute-instance my-vm --instance-type standard-4-16 +Error: compute_image field is required +``` + +#### Image Lifecycle Summary + +**API behavior by state:** + +| State | VM creation | List visibility | Get visibility | +|-------|-------------|-----------------|----------------| +| AVAILABLE | Succeeds | Shown by default | Always returned | +| DEPRECATED | Succeeds with warning | Shown by default | Always returned | +| OBSOLETE | Rejected (409 Conflict) | Hidden by default (requires explicit filter) | Always returned | + +**State transitions are bidirectional.** All transitions between the three states are supported: + +| From \ To | AVAILABLE | DEPRECATED | OBSOLETE | +|-----------|-----------|------------|----------| +| AVAILABLE | — | Yes | Yes | +| DEPRECATED | Yes (reactivation) | — | Yes | +| OBSOLETE | Yes (reactivation) | Yes | — | + +When reactivating a ComputeImage (transitioning back to AVAILABLE from DEPRECATED or OBSOLETE), the image becomes fully available for new VM creation without warnings. Deprecation metadata (timestamps, replacement) is retained for historical reference. + +**Validation rules:** + +- When transitioning to DEPRECATED: if an `obsolete` timestamp is provided, it must be in the future. +- When transitioning to OBSOLETE: the `obsolete` timestamp can be in the past or future (it represents when the image became or becomes obsolete). +- The `deprecated` timestamp is auto-populated with the current time on the DEPRECATED transition if not explicitly set. +- Phase 1 does not include automatic state transitions — the administrator must manually transition an image to OBSOLETE on or after the planned obsolete date. + +### API Extensions + +This enhancement introduces a new ComputeImage resource and modifies existing OSAC APIs: + +#### New Resources + +**ComputeImage** (`proto/public/osac/public/v1/compute_image_type.proto`) +- Represents a registered VM image stored as an OCI artifact in a container registry +- Contains metadata (display name, description, OS info) and a reference to an OCI artifact +- Supports two-tier visibility: global (visible to all tenants) and tenant-scoped + +**ComputeImages Service** (`proto/public/osac/public/v1/compute_images_service.proto`) +- Public API: ListComputeImages, GetComputeImage, CreateComputeImage, UpdateComputeImage, DeleteComputeImage +- Private API: Same as public, plus SignalComputeImage (included for consistency with the established private API pattern; no active consumers in this milestone) + +#### Modified Resources + +**ComputeInstanceSpec** (`proto/public/osac/public/v1/compute_instance_type.proto`) +- Replace: `image` field (ComputeInstanceImage with `source_type`/`source_ref`) with `compute_image` field (string, required) — reference to ComputeImage by ID +- Note: The fulfillment-service resolves the `compute_image` ID to the OCI `source_ref` when creating the CR. The osac-operator continues to receive the resolved OCI reference, unchanged from today. + +**CreateComputeInstanceResponse** (`proto/public/osac/public/v1/compute_instances_service.proto`) +- Add: `warnings` field (`repeated string`) — surfaces deprecation notices when creating a VM with a DEPRECATED image + +**ComputeInstanceTemplate** (`proto/public/osac/public/v1/compute_instance_template_type.proto`) +- Add: `compute_image` field (string) — reference to ComputeImage by ID, replacing inline image specification + +#### Proto Definition Sketch + +```proto +// proto/public/osac/public/v1/compute_image_type.proto +enum ComputeImageState { + COMPUTE_IMAGE_STATE_UNSPECIFIED = 0; + AVAILABLE = 1; // Fully available for new VM creation + DEPRECATED = 2; // Available with warnings, migration recommended + OBSOLETE = 3; // Not available for new VMs, visible for direct lookups only +} + +message ComputeImageDeprecation { + string replacement = 1; // Optional: suggested replacement ComputeImage ID + optional google.protobuf.Timestamp deprecated = 2; // When deprecation was announced (auto-set on DEPRECATED transition) + optional google.protobuf.Timestamp obsolete = 3; // When it becomes/became obsolete (admin-specified or auto-set) +} + +message ComputeImageRequirements { + optional int32 min_cpus = 1; // Minimum CPU cores required + optional int32 min_memory_mib = 2; // Minimum memory in MiB + optional int32 min_disk_gib = 3; // Minimum disk size in GiB +} + +message ComputeImageSpec { + string display_name = 1; // Human-readable name (e.g., "Red Hat Enterprise Linux 9.4") + string description = 2; // Detailed description of the image + string source_ref = 3 [(google.api.field_behavior) = IMMUTABLE]; // OCI image reference (immutable) + string os_type = 4 [(buf.validate.field).string.min_len = 1]; // "linux", "windows" + optional string os_version = 5; // e.g., "RHEL 9.4", "Windows Server 2022" + string architecture = 6 [(buf.validate.field).string.min_len = 1]; // "amd64", "arm64" + optional ComputeImageRequirements min_requirements = 7; +} + +message ComputeImageStatus { + ComputeImageState state = 1; // Lifecycle state + optional ComputeImageDeprecation deprecation = 2; // Present when state is DEPRECATED or OBSOLETE +} + +message ComputeImage { + string id = 1; // System-generated UUID (primary identifier) + Metadata metadata = 2; // Standard metadata (tenant, labels, annotations, timestamps) + ComputeImageSpec spec = 3; // Image specification + ComputeImageStatus status = 4; // Current status +} + +// proto/public/osac/public/v1/compute_images_service.proto +service ComputeImages { + rpc List(ListComputeImagesRequest) returns (ListComputeImagesResponse) { + option (google.api.http) = { + get: "/api/fulfillment/v1/compute_images" + }; + }; + rpc Get(GetComputeImageRequest) returns (ComputeImage) { + option (google.api.http) = { + get: "/api/fulfillment/v1/compute_images/{id}" + }; + }; + rpc Create(CreateComputeImageRequest) returns (ComputeImage) { + option (google.api.http) = { + post: "/api/fulfillment/v1/compute_images" + body: "*" + }; + }; + rpc Update(UpdateComputeImageRequest) returns (ComputeImage) { + option (google.api.http) = { + patch: "/api/fulfillment/v1/compute_images/{id}" + body: "*" + }; + }; + rpc Delete(DeleteComputeImageRequest) returns (google.protobuf.Empty) { + option (google.api.http) = { + delete: "/api/fulfillment/v1/compute_images/{id}" + }; + }; +} + +// proto/private/osac/private/v1/compute_images_service.proto +service ComputeImages { + rpc Create(CreateComputeImageRequest) returns (ComputeImage); + rpc Update(UpdateComputeImageRequest) returns (ComputeImage); + rpc Delete(DeleteComputeImageRequest) returns (google.protobuf.Empty); + rpc List(ListComputeImagesRequest) returns (ListComputeImagesResponse); + rpc Get(GetComputeImageRequest) returns (ComputeImage); + rpc Signal(SignalComputeImageRequest) returns (SignalComputeImageResponse); +} +``` + +**API vs CR Schema:** + +The fulfillment-service API and the Kubernetes CR have different schemas. The API introduces `compute_image` as the image selection field, while the CR retains the existing `image` block with the resolved OCI reference. + +**Public API Schema (proto/public/osac/public/v1/):** +```proto +message ComputeInstanceSpec { + string template = 1; + map template_parameters = 2; + optional google.protobuf.Timestamp restart_requested_at = 3; + string compute_image = 4 [(buf.validate.field).string.min_len = 1]; + string instance_type = 5; + optional string ssh_key = 7; + optional ComputeInstanceDisk boot_disk = 8; + repeated ComputeInstanceDisk additional_disks = 9; + optional string run_strategy = 10; + optional string user_data = 11; + optional string subnet = 12; + repeated string security_groups = 13; +} + +// CreateComputeInstanceResponse includes warnings for DEPRECATED images +message CreateComputeInstanceResponse { + ComputeInstance compute_instance = 1; + repeated string warnings = 2; // e.g., "ComputeImage 'rhel9-9.3' is deprecated and will become obsolete on 2026-12-31. Consider migrating to 'rhel9-9.4'." +} +``` + +**Kubernetes CR Schema (osac-operator CRD):** +```yaml +# ComputeInstance CR (created by fulfillment-service) +apiVersion: osac.openshift.io/v1alpha1 +kind: ComputeInstance +metadata: + name: my-vm +spec: + template: "..." + compute_image: "a1b2c3d4-e5f6-7890-abcd-ef1234567890" # ComputeImage ID for audit trail + image: + source_type: "registry" + source_ref: "registry.internal.example.com/osac-images/rhel9:9.4" # Resolved from ComputeImage + boot_disk: + size_gib: 50 + # ... other fields unchanged +``` + +The CR retains the `image` block with the resolved `source_ref` (unchanged from today) so osac-operator requires no modifications. The `compute_image` field is stored in the CR for audit purposes. The indirection from ComputeImage ID to OCI reference is handled entirely in the fulfillment-service. + +#### Validation + +**Public API (Tenant Users — ComputeInstance creation):** +- CreateComputeInstance: Require `compute_image` field (ComputeImage ID), reject if missing +- CreateComputeInstance: Validate ComputeImage exists and is visible to the caller's tenant (global or same tenant) +- CreateComputeInstance: If state is AVAILABLE, proceed normally +- CreateComputeInstance: If state is DEPRECATED, succeed but include warning in `CreateComputeInstanceResponse.warnings` with replacement suggestion (if set) and obsolete timestamp (if set) +- CreateComputeInstance: If state is OBSOLETE, reject with HTTP 409 Conflict ("ComputeImage is obsolete and cannot be used for new VMs") +- CreateComputeInstance: If ComputeImage ID not found or not visible to tenant, reject with HTTP 404 Not Found +- CreateComputeInstance: Resolve ComputeImage `source_ref` and pass to underlying provisioning system +- UpdateComputeInstance: Reject changes to `compute_image` field (immutable) + +**Public API (Administrators — ComputeImage management):** +- CreateComputeImage: Require `source_ref`, `os_type`, and `architecture` fields +- CreateComputeImage: Validate `source_ref` is a valid OCI image reference format +- CreateComputeImage: Default state to AVAILABLE if not specified +- CreateComputeImage: Provider admin creates global images (`metadata.tenant` empty); tenant admin creates tenant-scoped images (`metadata.tenant` auto-set to caller's tenant) +- CreateComputeImage: Tenant users cannot create images (authorization rejection) +- UpdateComputeImage: Reject changes to `source_ref` (immutable after creation) +- UpdateComputeImage: Allow changes to `display_name`, `description`, `os_version`, `architecture`, `min_requirements`, `state`, and `deprecation` fields +- UpdateComputeImage: State transitions are bidirectional — AVAILABLE↔DEPRECATED, AVAILABLE↔OBSOLETE, DEPRECATED↔OBSOLETE are all valid +- UpdateComputeImage: When transitioning to AVAILABLE (reactivation), deprecation metadata is retained for historical reference +- UpdateComputeImage: When transitioning to DEPRECATED, auto-populate `deprecation.deprecated` timestamp (set to current time) if not provided +- UpdateComputeImage: When transitioning to OBSOLETE, auto-populate `deprecation.obsolete` timestamp (set to current time) if not provided +- UpdateComputeImage: If `deprecation.obsolete` is provided when transitioning to DEPRECATED, validate it is in the future +- UpdateComputeImage: If `deprecation.obsolete` is provided when transitioning to OBSOLETE, allow any timestamp (past or future) +- UpdateComputeImage: API behavior is based on state field, not timestamps +- UpdateComputeImage: Tenant admin can only update own tenant's images; provider admin can update any +- DeleteComputeImage: Reject if any active ComputeInstances reference this image (409 Conflict) +- DeleteComputeImage: Tenant admin can only delete own tenant's images; provider admin can delete any + +#### Deletion Protection + +**ComputeImage deletion checks:** +- fulfillment-service queries database for active ComputeInstances referencing this ComputeImage ID +- If any references exist, deletion is rejected with 409 Conflict +- Returns descriptive error message: *"ComputeImage is in use by at least one ComputeInstance"* +- Deletion succeeds only when no active ComputeInstances reference the image +- Alternative to deletion: transition image to OBSOLETE to prevent new usage while existing VMs wind down + +#### Authorization Model + +| Operation | Provider Admin | Tenant Admin | Tenant User | +|-----------|---------------|--------------|-------------| +| CreateComputeImage (global) | Yes | No | No | +| CreateComputeImage (tenant-scoped) | Yes | Yes (own tenant) | No | +| ListComputeImages | All | Global + own tenant | Global + own tenant | +| GetComputeImage | All | Global + own tenant | Global + own tenant | +| UpdateComputeImage | All | Own tenant only | No | +| DeleteComputeImage | All | Own tenant only | No | + +### Implementation Details/Notes/Constraints + +#### Database Schema + +A new `compute_images` table in the fulfillment-service PostgreSQL database: + +| Column | Type | Notes | +|--------|------|-------| +| id | UUID | Primary key | +| creation_timestamp | TIMESTAMPTZ | Creation time | +| deletion_timestamp | TIMESTAMPTZ | Soft delete | +| finalizers | TEXT[] | Finalizer list | +| creators | TEXT[] | Who created the record | +| tenants | TEXT[] | Tenant scoping; empty array = global (visible to all tenants) | +| labels | JSONB | Indexed labels | +| annotations | JSONB | Annotations | +| data | JSONB | Serialized ComputeImage protobuf (display_name, description, source_ref, os_type, os_version, architecture, min_requirements) | + +The `data` JSONB column contains the serialized ComputeImage protobuf. Resource-specific fields are stored inside `data` rather than as individual columns, following the fulfillment-service convention. + +Example of a global ComputeImage `data` value: +```json +{ + "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "display_name": "Red Hat Enterprise Linux 9.4", + "description": "RHEL 9.4 base image for general-purpose workloads", + "source_ref": "registry.example.com/osac-images/rhel9:9.4", + "os_type": "linux", + "os_version": "RHEL 9.4", + "architecture": "amd64", + "min_requirements": { + "min_cpus": 2, + "min_memory_mib": 2048, + "min_disk_gib": 20 + }, + "state": "COMPUTE_IMAGE_STATE_AVAILABLE", +} +``` + +Example of a tenant ComputeImage `data` value: +```json +{ + "id": "6997aa1a-7bcc-4b14-bf1e-18d91fb64e33", + "metadata": { + "tenant": "0c9e7c27-8558-4641-84cb-c72daf811de7" + }, + "display_name": "Red Hat Enterprise Linux 9.4", + "description": "RHEL 9.4 base image for general-purpose workloads", + "source_ref": "registry.example.com/osac-images/rhel9:9.4", + "os_type": "linux", + "os_version": "RHEL 9.4", + "architecture": "amd64", + "min_requirements": { + "min_cpus": 2, + "min_memory_mib": 2048, + "min_disk_gib": 20 + }, + "state": "COMPUTE_IMAGE_STATE_AVAILABLE", +} +``` + +Example of a global ComputeImage `data` value with deprecation metadata: +```json +{ + "id": "948c850b-5670-478d-b4be-e16642f88b9b", + "display_name": "Red Hat Enterprise Linux 9.4", + "description": "RHEL 9.4 base image for general-purpose workloads", + "source_ref": "registry.example.com/osac-images/rhel9:9.4", + "os_type": "linux", + "os_version": "RHEL 9.4", + "architecture": "amd64", + "min_requirements": { + "min_cpus": 2, + "min_memory_mib": 2048, + "min_disk_gib": 20 + }, + "state": "COMPUTE_IMAGE_STATE_DEPRECATED", + "deprecation": { + "replacement": "b2c3d4e5-f6a7-8901-bcde-f12345678901", + "deprecated": "2026-06-01T00:00:00Z", + "obsolete": "2026-12-31T23:59:59Z" + } +} +``` + +#### List Filtering + +The `ListComputeImages` RPC supports filtering via the standard OSAC SQL-like filter syntax: + +- `os_type = 'linux'` — only Linux images +- `architecture = 'amd64'` — only amd64 images +- `display_name LIKE '%RHEL%'` — search by name +- `state IN (AVAILABLE, DEPRECATED, OBSOLETE)` — include obsolete images in results + +By default, `ListComputeImages` returns images in AVAILABLE and DEPRECATED states. OBSOLETE images are hidden from default results and require an explicit state filter. `GetComputeImage` returns any image regardless of state. + +The List operation automatically scopes results: tenant users see global images plus their own tenant's images. Provider administrators see all images. + +#### ComputeInstance Creation Flow + +When a user creates a ComputeInstance referencing a ComputeImage: + +1. The fulfillment-service resolves the `compute_image` ID. +2. It verifies the ComputeImage exists and is visible to the user's tenant. +3. It checks the image's state: + - **AVAILABLE**: Proceeds normally. + - **DEPRECATED**: Proceeds, but includes a warning in the `CreateComputeInstanceResponse.warnings` field with the deprecation details (obsolete date and replacement suggestion, if set). + - **OBSOLETE**: Rejects the request with 409 Conflict. +4. It retrieves the `source_ref` from the ComputeImage and passes it to the underlying provisioning system (osac-operator/KubeVirt). +5. If the ComputeImage is not found or not visible to the tenant, the request is rejected with a descriptive error. If the OCI artifact is unreachable or invalid, the error surfaces at VM provisioning time (from KubeVirt). + +#### osac-operator Integration + +The osac-operator does not need a new CRD for ComputeImage — image management is purely a fulfillment-service concern for this milestone. The operator continues to receive the resolved OCI reference in the ComputeInstance CR's spec, just as it does today. The indirection from ComputeImage ID to OCI reference is handled entirely in the fulfillment-service. + +#### Impact on ComputeInstanceTemplate + +The `ComputeInstanceTemplate` resource also uses `ComputeInstanceImage`. It will be updated to reference a `compute_image` (ComputeImage ID) instead. Templates that specify an image will point to a registered ComputeImage, ensuring that images are chosen from the list. + +### Risks and Mitigations + +| Risk | Impact | Mitigation | +|------|--------|------------| +| Global images visible across all tenants | Potential information leakage if image names/descriptions contain sensitive info | Provider admin is trusted; document that global image metadata is visible to all tenants. | +| Orphaned ComputeImages | Deleted images still referenced by running VMs | Running VMs retain the resolved OCI reference. Only new VM creation is blocked. Document this behavior. | +| Performance of List with many images | A large image list could slow listing | Standard pagination support via `page` and `size` parameters. Index on tenants array. | + +### Drawbacks + +- **Additional indirection**: Users must now discover and select a ComputeImage before creating a VM, adding a step to the workflow. This is intentional — it trades convenience for governance — but may feel heavy for simple deployments with few images. +- **No upload API**: Requiring out-of-band upload for this milestone means administrators must use separate tooling (skopeo, Quay UI) to push images. This is a deliberate scope limitation but may frustrate administrators who expect a unified experience. + +## Alternatives (Not Implemented) + +### Alternative 1: Extend ComputeInstance with Image Validation Only + +Instead of a new resource, add validation rules to the existing `source_ref` field — for example, an allowlist of permitted registries or image patterns. + +**Rejected because**: This provides no discoverability (users can't list available images), no human-readable metadata, and no centralized governance. It addresses only the "prevent arbitrary URLs" goal without solving the list of offering problem. + +### Alternative 2: Use CatalogItems for Image Selection + +Leverage the proposed CatalogItems system to present images as catalog entries rather than a dedicated ComputeImage resource. + +**Rejected because**: CatalogItems are broader (they wrap entire templates with parameter variations) and serve a different purpose. Images are a fundamental building block that should exist as a standalone resource. CatalogItems may reference ComputeImages in the future, but images need their own API for registration and management. + +### Alternative 3: ImageClass + ComputeImage (Two-Resource Model) + +Introduce an `ImageClass` resource (like NetworkClass) representing different image backends or registries, with ComputeImage referencing an ImageClass. + +**Deferred**: For the first milestone with a single registry and metadata-only management, ImageClass adds complexity without clear benefit. If OSAC later supports multiple registries with different capabilities (e.g., scanning-enabled vs. basic), ImageClass can be introduced as a backward-compatible addition. + +### Alternative 4: Full Upload API from Day One + +Build the ComputeImage API with integrated binary upload (similar to AWS EC2 `ImportImage`). + +**Deferred**: Building a reliable image upload pipeline (chunked upload, resumption, progress tracking, virus scanning) is a significant effort that would delay the core image list functionality. The out-of-band upload approach unblocks VM governance immediately while the upload API is designed separately. + +## Open Questions + +### Should ComputeImage have a structured `name` field (DNS-1035)? + +ComputeImage currently has a system-generated `id` (UUID) and a free-text `display_name`. Should it also have a `name` field with DNS-1035 constraints (lowercase alphanumeric and hyphens, max 63 characters, e.g., `rhel9-9-4-base`)? + +If so, should `name` replace `id` as the primary reference in `ComputeInstanceSpec.compute_image`? This would allow users to write `compute_image: "rhel9-9-4-base"` instead of `compute_image: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"`, making templates and CLI usage significantly more readable. + +**Considerations:** + +- **Usability**: Meaningful names are easier to type, remember, and use in automation scripts and templates than UUIDs. +- **Uniqueness scope**: Names would need to be unique within a scope — globally for global images, per-tenant for tenant-scoped images. This mirrors how Kubernetes resources use names within namespaces. +- **Immutability**: If `name` is used as a reference key, it should be immutable after creation (same as `source_ref`), since renaming would break existing ComputeInstance references. +- **Consistency**: Other OSAC resources (VirtualNetwork, Subnet, ComputeInstance) use UUIDs as their primary identifier. Introducing name-based references for ComputeImage alone may be inconsistent — or it may set a precedent for improving usability across the API. + +## Future Work + +This enhancement establishes the foundation for a complete image management lifecycle. Planned future capabilities include: + +- **Image upload API**: A dedicated API for uploading image binaries through OSAC, eliminating the need for out-of-band registry access. +- **VM snapshot/export**: Creating a new ComputeImage from a running ComputeInstance's disk state. +- **Image scanning integration**: Connecting with vulnerability scanning tools and exposing scan results as image metadata. +- **Cross-region image replication**: Automatically replicating images across multiple registries for multi-region deployments. +- **Image policies**: Tenant-level policies restricting which global images are available or requiring approval for new tenant images. +- **User-created images**: Allowing tenant users (not just admins) to register images, with optional admin approval workflows. +- **CatalogItems integration**: CatalogItems (see `/enhancements/catalog-items`) may reference ComputeImages as building blocks — e.g., a catalog entry could combine a ComputeImage with a predefined instance size and network configuration into a one-click deployment template. + +## Test Plan + +**Unit Tests:** + +- ComputeImage CRUD operations via GenericServer +- ComputeImage proto validation rejects missing required fields (source_ref, os_type, architecture) +- ComputeImage source_ref immutability enforcement on update +- ComputeImage mutable field updates (display_name, description, os_version, architecture, min_requirements) +- ComputeInstance validation rejects missing compute_image +- ComputeInstance validation rejects non-existent compute_image +- ComputeInstance validation rejects compute_image not visible to caller's tenant +- ComputeInstance validation rejects OBSOLETE compute_image +- ComputeInstance with DEPRECATED compute_image succeeds with warning (includes obsolete timestamp and replacement in warning) +- ComputeImage state transitions are bidirectional (AVAILABLE↔DEPRECATED, AVAILABLE↔OBSOLETE, DEPRECATED↔OBSOLETE) +- ComputeImage reactivation (DEPRECATED → AVAILABLE, OBSOLETE → AVAILABLE) retains deprecation metadata for historical reference +- ComputeImage deprecation metadata auto-population (deprecated timestamp on DEPRECATED transition, obsolete timestamp on OBSOLETE transition) +- ComputeImage validation rejects obsolete timestamp in the past when transitioning to DEPRECATED +- ComputeImage deletion rejected when active ComputeInstances reference it +- ListComputeImages default filter excludes OBSOLETE, explicit filter includes it +- ListComputeImages returns deprecation metadata for DEPRECATED/OBSOLETE images +- ListComputeImages tenant scoping: tenant user sees global + own tenant images only +- Provider admin CreateComputeImage with no tenant creates global image +- Tenant admin CreateComputeImage auto-sets metadata.tenant to caller's tenant +- Tenant user CreateComputeImage rejected (authorization) +- Tenant admin cannot update or delete global images +- Tenant admin cannot update or delete another tenant's images + +**Integration Tests (fulfillment-service/it/):** + +- Create global ComputeImage as provider admin, verify it appears in ListComputeImages for all tenants +- Create tenant-scoped ComputeImage as tenant admin, verify it appears only for that tenant +- Verify tenant A cannot see tenant B's tenant-scoped images +- Create ComputeInstance with valid compute_image, verify VM provisions with correct OCI reference +- Attempt to create ComputeInstance with OBSOLETE compute_image, verify 409 Conflict +- Create ComputeInstance with DEPRECATED compute_image, verify warning message includes deprecation timeline and replacement +- Set ComputeImage to DEPRECATED with future obsolete timestamp and replacement, verify deprecation metadata persisted +- Set ComputeImage to OBSOLETE, verify it disappears from default ListComputeImages results +- Reactivate DEPRECATED ComputeImage to AVAILABLE, verify VM creation succeeds without warnings +- Reactivate OBSOLETE ComputeImage to AVAILABLE, verify it reappears in default ListComputeImages and VM creation succeeds +- Transition OBSOLETE ComputeImage to DEPRECATED, verify VM creation succeeds with warning +- Verify reactivated ComputeImage retains deprecation metadata for historical reference +- Verify GetComputeImage returns any image regardless of state (AVAILABLE, DEPRECATED, OBSOLETE) +- Attempt to delete ComputeImage with active ComputeInstances, verify 409 Conflict rejection +- Delete all referencing ComputeInstances, then delete ComputeImage, verify success +- Attempt to update source_ref on existing ComputeImage, verify rejection (immutability) +- ListComputeImages filtering by os_type, architecture, display_name LIKE, and state +- ListComputeImages pagination with page and size parameters + +**End-to-End Tests:** + +- Provider admin registers a global ComputeImage via API +- Tenant user lists images and sees the global image +- Tenant admin registers a tenant-scoped ComputeImage +- Tenant user lists images and sees both global and tenant-scoped images +- Tenant user creates ComputeInstance using a ComputeImage reference +- osac-operator provisions KubeVirt VM with the resolved OCI reference from the ComputeImage +- VM reaches Running state +- Provider admin sets global ComputeImage to DEPRECATED with replacement and future obsolete timestamp +- Tenant user lists images and sees DEPRECATED status with deprecation metadata +- Tenant user creates new VM with DEPRECATED image, receives warning with obsolete date and replacement suggestion +- Provider admin sets global ComputeImage to OBSOLETE +- Tenant user no longer sees image in default list (unless explicitly filtered by state) +- Attempt to create new VM with OBSOLETE image fails with 409 Conflict +- Provider admin reactivates OBSOLETE ComputeImage back to AVAILABLE +- Tenant user sees image in default list again +- Tenant user creates new VM with reactivated image, succeeds without warnings +- Existing VM continues running unaffected +- Multi-tenant isolation: tenant B cannot see tenant A's tenant-scoped images + +## Graduation Criteria + +Graduation criteria will be defined when targeting a release. Expected stages: + +- **Dev Preview**: ComputeImage CRUD API available. ComputeInstance accepts `compute_image` reference. +- **Tech Preview**: Multi-tenant visibility enforced. +- **GA**: Production-hardened. Performance validated at scale. Upgrade/downgrade tested. Documentation complete. + +## Upgrade / Downgrade Strategy + +Not applicable - OSAC is pre-GA. This is a breaking API change. + +## Version Skew Strategy + +Not applicable - OSAC is pre-GA. This is a breaking API change. + +## Support Procedures + +- **User cannot see expected images**: Verify the image's `metadata.tenant` field. Global images have an empty tenant and should be visible to all tenants. Tenant-scoped images should have `metadata.tenant` matching the user's tenant. Use `ListComputeImages` as a provider admin to see all images. +- **ComputeInstance creation fails with "image not found"**: Verify the referenced ComputeImage exists and is visible to the user's tenant. +- **VM fails to start after ComputeInstance creation**: The ComputeImage's `source_ref` may point to an unreachable or invalid OCI artifact. Check KubeVirt/osac-operator logs for image pull errors. Verify the OCI reference is correct and the registry is accessible from the workload cluster. +- **Deleting a ComputeImage**: `DeleteComputeImage` is rejected with 409 Conflict when any active ComputeInstances reference the ComputeImage ID, returning the error *"ComputeImage is in use by at least one ComputeInstance"*. Once all referencing ComputeInstances are removed, deletion proceeds as a soft-delete — the record is marked with `deletion_timestamp` and excluded from queries, but retained in the database. As an alternative to deletion, transition the image to OBSOLETE to prevent new usage while existing VMs wind down. +- **ComputeInstance creation rejected: "ComputeImage is obsolete"**: The API returns 409 Conflict when attempting to create a VM with an OBSOLETE image. Resolution: choose a different image, or ask the administrator to reactivate the image by transitioning it back to AVAILABLE or DEPRECATED (state transitions are bidirectional). Prevention: communicate image lifecycle changes (OBSOLETE transitions) to users before making the change. +- **User sees deprecation warning on VM creation**: This is expected behavior when the referenced ComputeImage is in DEPRECATED state. The warning includes the planned obsolete date and suggested replacement (if set). Users should plan to migrate to the replacement image before the obsolete date. +- **User cannot find an image they previously used**: The image may have been transitioned to OBSOLETE state, which hides it from default `ListComputeImages` results. Use `GetComputeImage` with the image ID for a direct lookup (returns any state), or filter the list explicitly with `state IN (AVAILABLE, DEPRECATED, OBSOLETE)` to include obsolete images. + +## Infrastructure Needed + +No additional infrastructure is required for this enhancement. All work will be done in the existing `fulfillment-service` repository. The container registry is assumed to be pre-deployed as part of the OSAC landing zone installation. From 667ef673b9fa0c50f53f9885e7724e859aa4caba Mon Sep 17 00:00:00 2001 From: Tzu-Mainn Chen Date: Thu, 4 Jun 2026 09:16:51 -0400 Subject: [PATCH 004/450] Update the bare metal fulfillment proposal (#31) This update provides a more detailed architecture that more closely matches k8s patterns. --- enhancements/bare-metal-fulfillment/README.md | 683 +++++++----------- 1 file changed, 264 insertions(+), 419 deletions(-) diff --git a/enhancements/bare-metal-fulfillment/README.md b/enhancements/bare-metal-fulfillment/README.md index ed5d96ce2..a3e3b8f78 100644 --- a/enhancements/bare-metal-fulfillment/README.md +++ b/enhancements/bare-metal-fulfillment/README.md @@ -3,7 +3,7 @@ title: bare-metal-fulfillment authors: - Tzu-Mainn Chen creation-date: 2025-09-08 -last-updated: 2025-09-15 +last-updated: 2026-06-02 tracking-link: - None see-also: @@ -18,480 +18,325 @@ superseded-by: ## Summary -Bare metal fulfillment refers to the process through which a tenant acquires bare metal hosts and configures their networking. -Afterwards, the tenant will be able to perform selected operations on the host: inventory information, power control, and -serial console access. Any further operations - for example, image-based provisioning - are outside the scope of the O-SAC -solution and must be integrated by the cloud provider. - -We believe this feature set is generic enough to support a wide range of initial bare metal cluster use cases, including: - -* support for an environment where developers have the flexibility to install their own operating systems and software -* support for alternative clusters such as SLURM -* support for bare metal configuration during O-SAC cluster fulfillment - -In this context, a “tenant” is an organization that can perform selected actions on the requested bare metal resources in -order to achieve these use cases; at the MOC, a tenant would be led by a ColdFront PI or manager. +Bare metal fulfillment refers to the process through which bare metal hosts are acquired and configured. Requests for bare metal fulfillment could be triggered by an OSAC tenant, or by a provider workflow, or by other OSAC fulfillment processes (such as cluster fulfillment). ## Motivation -Bare metal fulfillment is a fundamental requirement for O-SAC, identified as an explicit need for the MOC. It will also serve -as a component for other fulfillment workflows, such as VDC and OpenShift cluster fulfillment. +OSAC requires the manipulation of bare metal, whether for cluster fulfillment or agent provisioning or to satisfy tenant requests. Bare metal fulfillment provides a unified API for this manipulation while also giving us a single point of integration with backend bare metal inventory and management systems. ### User Stories -* As a provider, I want to define the available bare metal resource classes. -* As a provider, I want to mark hosts as available for bare metal fulfillment. -* As a tenant, I want to see available bare metal resource classes. -* As a tenant, I want to select hosts for a bare metal cluster by resource class and other possible filters (such as rack location). -* As a tenant, I want to increase or decrease the number of hosts for an existing bare metal cluster. -* As a tenant, I want to view inventory details for my hosts. -* As a tenant, I want to connect to the serial console of my hosts. -* As a tenant, I want to perform basic power control (power on/power off/reset) of my hosts. -* As a tenant, I want to specify network attachments for a host when I first acquire it. -* As a tenant, I want to modify the network attachments of a host that I have already acquired. -* As a tenant, I want to manage connectivity at the interface level in order to support hardware that may have both regular and high performance interfaces (for example, some hosts may have high-bandwidth connectivity for GPU workloads, and it may be necessary to attach specific networks to these interfaces). +* As a Cloud Provider Admin, I want a bare metal fulfillment architecture that allows support for multiple bare metal backends +* As a Cloud Provider Admin, I want to integrate OSAC with an external bare metal inventory that can still be used as a source of truth regarding the utilization of bare metal. +* As a Cloud Provider Admin, I want to utilize bare metal fulfillment in workflows such as cluster fulfillment and agent provisioning +* As a tenant user, I want to utilize bare metal fulfillment to acquire and configure hardware ### Goals -We will implement bare metal fulfillment in the O-SAC solution, covering the user stories described above. This implementation will -be first deployed and tested in the MOC; as a result, the implementation will rely upon ESI for bare metal management (although -it will be designed to allow easy replacement of the bare metal management layer). While not addressed in this proposal, a long term goal -of bare metal fulfillment is to have it be used as the basis of bare metal configuration during cluster fulfillment. +We will implement a bare metal fulfillment architecture that: -### Non-Goals +* Enables support for multiple bare metal backends +* Integrates with an external bare metal inventory and maintains it as a source of truth regarding bare metal utilization +* Allows providers to specify bare metal workflows that automate bare metal configuration (such as image provisioning) +* Is usable by tenants, providers, and within other OSAC fulfillment workflows (such as cluster fulfillment) -We will not look into alternatives for ESI; that work will be covered through a future enhancement. +### Non-Goals -We will not substantially modify the existing cluster fulfillment workflow at this time. +* This proposal does not address bare metal networking; that topic depends upon a completed OSAC Networking API, which is still in discussion and development. +* This proposal does not implement serial console access to bare metal hosts. +* This proposal does not accommodate a single OSAC environment with multiple bare metal inventories (although the architecture is designed to support this feature in the future). +* This proposal does not address reporting and billing requirements; those will be addressed in a future enhancement that exposes these resources in the fulfillment service. ## Proposal -The implementation of bare metal fulfillment relies on three concepts: +### CRDs and Operators -* HostPool: A collection of requested bare metal machines (potentially from multiple resource classes), along with their desired network configuration. -* Host: An ephemeral resource created when bare metal resources are acquired from the underlying bare metal management layer. They represent the assignment of a bare metal resource to a HostPool, and provide an endpoint for host-specific operations. -* HostClass: Defines a set of bare metal properties, such as memory, CPU, and network interfaces. Each Host has a single HostClass attribute. HostClasses are defined by the provider. +We propose creating the following CRDs (full specifications can be found in the [API Extensions section](#API Extensions) below): -At a high level, tenants will request a HostPool by specifying: +* **Host Lease:** Represents a lease on a host; an ephemeral resource that can be used to manipulate the host after acquisition, and which can be safely deleted when the host is released. -* the number and HostClass of hosts, as well as any desired filters (for example: "hosts on rack X" or "not host with name Y") -* the network configuration to be applied to those hosts (note that this configuration applies to each individual host in the HostPool) + * **Host Class:** Specifies how a host can be managed (OpenStack, Carbide, etc). Each Host is associated with one Host Class. + * **Host Type:** Describes the type of hardware represented by this host. Analogous to "instance type" in AWS or "resource class" in Ironic. E.g. `fc430`, `h100` + * **Host Template:** An Ansible role that details setup and teardown steps for an individual host. -The O-SAC solution will find available hosts matching the request from the system's inventory, which is managed separately by the cloud provider. O-SAC -will then assign them to the tenant, creating matching Host resources for each host. The tenant can then perform available operations upon the Host. +* **Bare Metal Pool:** Represents a collection of bare metal hosts that share a common desired configuration, and which cannot be used by others for the lifetime of the bare metal pool. It specifies three items: -We expect bare metal fulfillment to follow the same workflow used for cluster fulfillment; as such, we expect to update the -following existing O-SAC components: + * **Host Sets:** A collection of desired hosts + * **Profile:** A configuration that specifies how the bare metal pool should behave. It specifies additional restrictions when selecting hosts, the template that should be applied to the bare metal pool, and the template that should be applied to each host. Profiles will initially be stored in a ConfigMap; a future enhancement may create Profile CRDs instead. + * **Bare Metal Pool Template:** An Ansible role that details setup and teardown steps for a bare metal pool as a whole. -* Fulfillment Service: Define the API for HostPools, Hosts, and HostClasses -* OSAC CLI: Give the tenant access to the API -* O-SAC Operator: Manage and reconcile the Custom Resources for HostPools and Hosts -* O-SAC AAP: Use Ansible playbooks to perform the requested reconciliation operations of HostPools and Hosts by calling Bare Metal Management APIs. -* Bare Metal Management: Manage the hardware; we will use ESI for now +These CRDs will be managed by the following operators: -Much of the development work can simply follow the established patterns set by the cluster fulfillment workflow. However, the -following topics merit further discussion in the "Implementation Details/Notes/Constraints" below: +* **Host Operator:** -* ESI playbooks -* Serial console support + * Queries the backend inventory, assigns a host, and populates the Host Lease CR with information about that host (including the Host Class). + * Executes setup/teardown templates applicable to an individual host. + * Allows bare metal management operations to be performed upon the associated host through modification of the Host Lease CR's `spec`. These operations will be limited to those that a Cloud Provider Admin will not need to customize; initially we will only support power management. -### Workflow Description +* **Bare Metal Pool Operator:** -#### Host Pool Creation + * Creates/deletes Host Lease CRs corresponding to the hosts desired for the bare metal pool. + * Executes setup/teardown templates applicable to the entire bare metal pool (and not to individual hosts). -1. The tenant uses the OSAC CLI to request a HostPool by specifying the desired number of hosts and resource classes, as well as the desired network configuration. -2. The Fulfillment Service receives the request and creates a new HostPool custom resource (CR) in the appropriate Hub. -3. The O-SAC Operator begins the reconciliation process for the new HostPool CR, finding the requested hosts and creating matching Host CRs, and performing the requested network configuration. -4. The O-SAC Operator monitors the status of the reconciliation process and updates the status of the HostPool CR to reflect the current state. -5. The tenant uses the OSAC CLI to check the status of their requested HostPool. +### Bare Metal Inventory -#### Host Pool Network Update +The host operator assumes the existence of an external bare metal inventory. This inventory must meet the following requirements: -1. The tenant uses the OSAC CLI to update the network configuration of a HostPool. -2. The Fulfillment Service receives the request and updates the existing HostPool CR in the appropriate Hub. -3. The O-SAC Operator begins the reconciliation process for the HostPool CR, noting the discrepency between the current network configuration and the desired networking confiuration, and updating the network configuration of the hosts. -4. The O-SAC Operator monitors the status of the reconciliation process and updates the status of the HostPool CR to reflect the current state. -5. The tenant uses the OSAC CLI to check the status of the HostPool. +* hosts that specify a host type (`fc430`, `h100`, etc) +* capability for arbitrary label assignment upon a host -#### Host Pool Expansion +As an example, OpenStack Ironic fulfills the first requirement through its node's `resource_class` attribute, and the second through the node's `extra` dictionary attribute. -1. The tenant uses the OSAC CLI to increase the number of requested hosts specified in a HostPool, updating any desired filters. -2. The Fulfillment Service receives the request and updates the existing HostPool CR in the appropriate Hub. -3. The O-SAC Operator begins the reconciliation process for the HostPool CR, noting the discrepency between the current host count and the desired host count, and adding the requested hosts. -4. The O-SAC Operator monitors the status of the reconciliation process and updates the status of the HostPool CR to reflect the current state. -5. The tenant uses the OSAC CLI to check the status of the HostPool. +The operator supports multiple types of inventories through a Go interface. This interface requires the following functions to be implented: -#### Host Pool Reduction +* *FindFreeHost(ctx context.Context, matchExpressions map[string]string)*: Finds an unassigned host filtered by the expressions in `matchExpressions`. +* *AssignHost(ctx context.Context, inventoryHostID string, bareMetalPoolID string, bareMetalPoolHostID string, labels map[string]string)*: Assigns the host identified in the inventory as `inventoryHostID` to the bare metal pool and host lease specified by `bareMetalPoolID` and `bareMetalPoolHostID`, while also applying the additional labels specified by `labels`. +* *UnassignHost(ctx context.Context, inventoryHostID string, labels []string)*: Unassigns the host identified in the inventory as `inventoryHostID`, while also unsetting any labels listed in `labels`. -1. The tenant uses the OSAC CLI to decrease the number of requested hosts specified in a HostPool; they also optionally mark specific hosts for removal (this is detailed further below). -2. The Fulfillment Service receives the request and updates the existing HostPool CR in the appropriate Hub. -3. The O-SAC Operator begins the reconciliation process for the HostPool CR, noting the discrepency between the current host count and the desired host count, and removing the requested hosts. -4. The O-SAC Operator monitors the status of the reconciliation process and updates the status of the HostPool CR to reflect the current state. -5. The tenant uses the OSAC CLI to check the status of the HostPool. +Note that an inventory implementation may break apart the match expressions in order to perform host selection in a way specific to that implementation; for example, the OpenStack implementation will compare a `hostType` match expression against an Ironic node's `resourceClass`, while comparing any other match expressions to values found in the `extra` attribute. -#### Host Operations +A bare metal inventory will be specified in OSAC as follows: -1. The tenant uses the OSAC CLI to view their Hosts. -2. The tenant uses the OSAC CLI to perform available operations upon the desired Host. Initially, these actions will be limited to power control; additional actions (such as console enablement) may be added later. -3. The Fulfillment Service receives the request and updates the existing Host CR in the appropriate Hub. -3. The O-SAC Operator begins the reconciliation process for the Host CR, noting any discrepency between the current state and the desired state, and calling the bare metal service to perform any needed host operations. -4. The O-SAC Operator monitors the status of the reconciliation process and updates the status of the Host CR to reflect the current state. -5. The tenant uses the OSAC CLI to check the status of the Host. +``` yaml +name: bare-metal-inventory +type: openstack # corresponds to a bare metal inventory implementation +options: +hostClass: openstack # applied to a Host Lease after a host is assigned +networkClass: openstack # applied to a Host Lease after a host is assigned +``` +An inventory will be managed as a Secret, although a later enhancement may create a Bare Metal Inventory CR instead. -#### Host Pool Deletion +For now, we assume that there is exactly one bare metal inventory specified; a later enhancement may support multiple inventories. -1. The tenant uses the OSAC CLI to delete a HostPool. -2. The Fulfillment Service receives the request and deletes the existing HostPool CR in the appropriate Hub. -3. The deletion of the HostPool CR triggers cascading deletes of associated Hosts. -4. The O-SAC Operator detects these deletions and calls the bare metal service to clean these hosts. +### Bare Metal Management -### API Extensions +The host operator will support multiple bare metal management systems through a bare metal management interface. This interface requires the following function to be implented: -#### HostPools - -A tenant requests bare metal resources by creating a HostPool with a desired specification. Once created, O-SAC -will repeatedly attempt to fulfill the request until the state of the HostPool matches its specification. Tenants -can update a HostPool specification, and the same reconciliation process will perform the needed operations to change -the state of the HostPool. - -In the following example, the tenant uses the OSAC CLI to request a HostPool with two fc430 hosts and one h100. -Each host will have `network1` attached as a native VLAN on one physical interface; and a trunk port with `network2` as a native -VLAN and `network3` as a tagged VLAN on a second physical interface. It is assumed that the tenant will have knowledge of their -available networks from a separate O-SAC network service (whose implementation is outside the scope of this proposal). - - $ ./osac create hostpool \ - --host-set workers=host_class:fc430,size:2 \ - --host-set gpus=host_class:h100,size:1 \ - --network-attachments primary:network1 \ - --network-attachments primary:network2,vlans:network3 - -The OSAC CLI sends this JSON request to the Fulfillment Service: - - { - "object": { - "spec": { - "host_sets": { - "workers": { - "resource_class": "fc430", - "replicas": 2 - }, - "gpus": { - "resource_class": "h100", - "replicas": 1 - } - }, - "network_attachments": [ - { - "primary": "network1" - }, - { - "primary": "network2", - "vlans": ["network3"] - }, - ] - } - } - } - -The Fulfillment Service creates the following HostPool CR: - - apiVersion: o-sac.openshift.io/v1alpha1 - kind: HostPool - metadata: - name: examplehostpool - spec: - hostSets: - workers: - resourceClass: fc430 - replicas: 2 - gpus - resourceClass: h100 - replicas: 1 - selector: - matchLabels: - hostPoolUID: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - - # The networkAttachments section controls how networks are connected to - # physical interfaces. - networkAttachments: - # This configuration will match any available interface. - - primary: network1 - - primary: network2 - vlans: - - network3 - -Note that the selector values are supplied by the Fulfillment Service; these values are used to label the -created Hosts without any input needed from the tenant. - -If the tenant wishes to specify host properties, they can filter hosts by specifying hostSelectors that use standard -Kubernetes `matchLabel` and `matchExpression` syntax; for example, this HostPool CR requires that the fc430 hosts be -located on rack R2, but not in cabinet C2: - - apiVersion: o-sac.openshift.io/v1alpha1 - kind: HostPool - metadata: - name: examplehostpool - spec: - hostSets: - workers: - resourceClass: fc430 - replicas: 2 - hostSelectors: - matchLabels: - row: R2 - matchExpressions: - - op: NotIn - key: cabinet - values: ["C2"] - gpus: - resourceClass: h100 - replicas: 1 - selector: - matchLabels: - hostPoolUID: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - - # The networkAttachments section controls how networks are connected to - # physical interfaces. - networkAttachments: - # This configuration will match any available interface. - - primary: network1 - - primary: network2 - vlans: - - network3 - -Note that initially, information regarding valid key/value pairs will have to be provided out-of-band by the -cloud provider. A later enhancement may allow O-SAC to provide this information through an API. - -If the tenant wishes to remove a host from their HostPool, then they will update their HostPool specification to reduce -the number of Hosts of a resource class, resulting in the following CR. The reconciliation process will remove an arbitrary -Host of that resource class from the HostPool: - - apiVersion: o-sac.openshift.io/v1alpha1 - kind: HostPool - metadata: - name: examplehostpool - spec: - hostSets: - workers: - resourceClass: fc430 - replicas: 1 - gpus: - resourceClass: h100 - replicas: 1 - selector: - matchLabels: - hostPoolUID: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - - # The networkAttachments section controls how networks are connected to - # physical interfaces. - networkAttachments: - # This configuration will match any available interface. - - primary: network1 - - primary: network2 - vlans: - - network3 - -If the tenant wishes to remove a specific host from their HostPool, then they will first mark a Host for removal -using an O-SAC API that annotates the Host with a marker for deletion before reducing the number of Hosts within a HostPool. -The reconciliation process will favor removing Hosts with that annotation. - -If a tenant wishes to remove a specific host and prevent it from ever being re-added to the HostPool, they can add -a host selector that excludes a particular host by name. Note that this also allows for a tenant to replace a host by keeping the -number of hosts constant while also excluding the unwanted host. - - apiVersion: o-sac.openshift.io/v1alpha1 - kind: HostPool - metadata: - name: examplehostpool - spec: - hostSets: - workers: - resourceClass: fc430 - replicas: 1 - hostSelectors: - matchExpressions: - - op: NotIn - key: hostName - values: ["examplehost"] - gpus: - resourceClass: h100 - replicas: 1 - selector: - matchLabels: - hostPoolUID: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - - # The networkAttachments section controls how networks are connected to - # physical interfaces. - networkAttachments: - # This configuration will match any available interface. - - primary: network1 - - primary: network2 - vlans: - - network3 - -Tenants can also attach networks to interfaces matching a specific property. In this example, -each fc430 will be configured with `network1` attached to any available physical interface, -while `storage-network` will only be attached to an interface with the `25gb` property. - - apiVersion: o-sac.openshift.io/v1alpha1 - kind: HostPool - metadata: - name: example - spec: - hostSets: - workers: - resourceClass: fc430 - replicas: 2 - gpus: - resourceClass: h100 - replicas: 1 - selector: - matchLabels: - hostPoolUID: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - - # The networkAttachments section controls how networks are connected to - # physical interfaces. - networkAttachments: - # This configuration will match any available interface. - - primary: network1 - # This configuration will only match interfaces with the `25gb` property - - matches: - property: 25gb - primary: storage-network - -#### Hosts - -Tenants do not create Hosts directly; instead, they are created as part of the HostPool fulfillment workflow, resulting in a Host CR. -For example: - - apiVersion: cloudkit.openshift.io/v1alpha1 - kind: Host - metadata: - name: examplehost - ownerReferences: - - apiVersion: o-sac.openshift.io/v1alpha1 - blockOwnerDeletion: true - controller: true - kind: HostPool - name: examplehostpool - uid: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - labels: - hostPoolUID: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - spec: - powerState: PowerOff - serialConsole: Enabled - status: - powerState: PowerOff - serialConsole: Enabled - name: examplehost - properties: - cpus: 512 - memory_mb: 1572864 - accelerators: - - "NVIDIA Corporation GH100" - -The listed properties will depend upon the attributes returned by the underlying bare metal management service. - -Note that each Host has an ownerReferences entry to the parent HostPool; this will enable both cascading resource deletion, while -also preventing the parent HostPool from being deleted until its child Hosts are deleted. - -Once created, tenants can use the OSAC CLI to perform various operations against the host: - -* Power control -* Serial console access -* Inventory information - -A Hosts API will support these operations. For example, the tenant can edit the Host spec to change the power state, resulting -in the following Host CR: - - apiVersion: cloudkit.openshift.io/v1alpha1 - kind: Host - metadata: - name: examplehost - ownerReferences: - - apiVersion: o-sac.openshift.io/v1alpha1 - blockOwnerDeletion: true - controller: true - kind: HostPool - name: examplehostpool - uid: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - labels: - hostPoolUID: 66b8ed6f-1af2-4892-ac12-47bd47dacd40 - spec: - powerState: PowerOn - serialConsole: Enabled - status: - powerState: PowerOff - serialConsole: Enabled - name: examplehost - properties: - cpus: 512 - memory_mb: 1572864 - accelerators: - - "NVIDIA Corporation GH100" - -The O-SAC Operator will detect the difference in powerState between the spec and the status, and call the bare metal service -to power the host on before updating the Host's status to PowerOn. +* *SetHostPowerState(ctx context.Context, inventoryHostID string, targetPowerState string)*: Sets the target power state of the host identified in the inventory as `inventoryHostID` -### Implementation Details/Notes/Constraints +Each `hostClass` will correspond to a specific bare metal management implementation. -#### ESI Playbooks +### Workflow Description -The ESI API supports all of the needed bare metal functionality (the cluster fulfillment workflow already relies upon ESI -playbooks for bare metal network configuration) . However, it is important to recognize that O-SAC will not use the ESI concept -of tenancy; from an ESI perspective, hosts will always remain leased to the O-SAC project. As a result, we will need to introduce -a new method of keeping track of O-SAC tenancy. +```mermaid +flowchart TD + U1([Tenant]) --> A1 + A1[Fulfillment Service] -->B{Bare Metal Pool CR} + A2[OSAC Cluster Operator] -->B + B <--> C[Bare Metal Pool Operator] + C --> D{Host CR} + D <--> E[Host Operator] +``` + +Bare metal fulfillment begins with the creation of a Bare Metal Pool CR. This creation might be triggered by a tenant (through the fulfillment service), a provider (through their own workflow), or another OSAC fulfillment workflow (such as cluster fulfillment). + +#### Bare Metal Pool Creation + +1. The Bare Metal Pool CR is created, specifying the desired Host Set and Profile +2. The Bare Metal Pool Operator: + * Iterates through `spec.hostSets` and creates a Host Lease CR for each requested host + * Uses the following values when creating the Host Lease CRs + * Adds the host set’s host type to the Host Lease CRs `spec.selector` dictionary + * Adds the values in the profile’s `matchLabels` attribute to the Host Lease CRs `spec.selector` dictionary + * Sets `spec.template` + * The template ID is taken from the profile’s `hostTemplate` value + * The template input is taken from `spec.profile.input` + * Executes the provisioning step of the template specified by the profile’s `bareMetalPoolTemplate` +3. The Host Operator: + * Assigns a host to individual Host Lease CRs by: + * Querying configured Bare Metal Inventory for a host that matches the Host Lease CR’s `spec.selector` dictionary and assigns it to the Bare Metal Pool + * Updating the Host Lease CR with information returned from the above query, as well as the `hostClass` and `networkClass` specified by the operator’s inventory configuration + * Executes the provisioning step of the template specified by the Host Lease CR’s `spec.template` + +#### Bare Metal Pool Scale Up + +1. The Bare Metal Pool CR is modified to add an additional host in the specified Host Set +2. The Bare Metal Pool Operator: + * Counts the pool’s existing Host Lease CRs against requested hosts, and creates Host Lease CRs as required + * Uses the following values when creating the Host Lease CRs + * Adds the host set’s host type to the Host Lease CRs `spec.selector` dictionary + * Adds the values in the profile’s `matchLabels` attribute to the Host Lease CRs `spec.selector` dictionary + * Sets `spec.template` + * The template ID is taken from the profile’s `hostTemplate` value + * The template input is taken from `spec.profile.input` +3. The Host Operator: + * Assigns a host to new Host Lease CRs by: + * Querying configured Bare Metal Inventory for a host that matches the Host Lease CR’s `spec.selector` dictionary and assigns it to the Bare Metal Pool + * Updating the Host Lease CR with information returned from the above query, as well as the `hostClass` and `networkClass` specified by the operator’s inventory configuration + * Executes the provisioning step of the template specified by the Host Lease CR’s `spec.template` + +#### Bare Metal Pool Scale Down + +1. The Bare Metal Pool CR is modified to remove a host in the specified Host Set +2. The Bare Metal Pool Operator: + * Counts pool’s Host Lease CRs against requested hosts, and identifies Host Lease CRs to be removed + * The operator may prioritize Host Lease CRs with a “to-be-removed” annotation + * Deletes selected Host Lease CRs +3. Host Operator + * Executes the deprovisioning step of the template specified by the Host Lease CR’s `spec.template` + * Unsets the Host Lease CR’s `hostClass` + * Updates the bare metal inventory to unassign the bare metal pool from the host + * Allows Host Lease CR to be deleted + +#### Bare Metal Pool Deletion + +1. The Bare Metal Pool CR is deleted +2. The Bare Metal Pool Operator: + * Deletes the pool’s Host Lease CRs + * Executes the deprovisioning step of the template specified by the profile’s `bareMetalPoolTemplate` + * Allows Bare Metal Pool CR to be deleted after the above operations are complete +3. The Host Operator: + * Executes the deprovisioning step of the template specified by the Host Lease CR’s `spec.template` + * Unsets the Host Lease CR’s `hostClass` + * Updates the bare metal inventory to unassign the bare metal pool from the host + * Allows Host Lease CR to be deleted -One solution would be to rely upon Ironic’s extra field, which allows an ESI project to set arbitrary key/value pairs upon a -specified host. Doing so would not be difficult; it simply creates additional considerations when implementing playbooks related -to host assignment and unassignment. +### API Extensions -#### Serial Console Support +#### Bare Metal Pool + +The Bare Metal Pool CR acts as the entrypoint into bare metal fulfillment. For example: + +``` yaml +apiVersion: osac.openshift.io/v1alpha1 +kind: BareMetalPool +metadata: + name: bm-pool-12345 + namespace: osac-namespace +spec: + hostSets: + - hostType: fc430 + replicas: 2 + - hostType: h100 + replicas: 1 + profile: + name: imageProvisioning + templateParameters: some-json-string +``` + +The profile allows for the automated configuration of a Bare Metal Pool. For example, an `imageProvisioning` profile might look like the following: + +``` yaml +imageProvisioning: + matchLabels: + managedBy: None + provisionState: available + expectedParameters: ["imageURL"] + bareMetalPoolTemplate: osac.templates.bm_private_network_create + hostTemplate: osac.templates.bm_host_image_provision +``` + +* `matchLabels`: additional constraints to be used when selecting hosts for the Bare Metal Pool +* `bareMetalPoolTemplate`: similar to an OSAC cluster template; specifies an Ansible role with setup and tear down actions to be applied for the bare metal pool as a whole +* `hostTemplate`: similar to an OSAC cluster template; specifies an Ansible role with setup and tear down actions to be applied for an individual host +* `expectedParameters`: parameters expected by the above templates + +Profiles are specified by the provider in a configuration file that can be read by the Bare Metal Pool Operator. + +#### Host Lease + +Host Lease CRs are created by the Bare Metal Pool Operator when it reconciles a Bare Metal Pool CR: + +``` yaml +apiVersion: osac.openshift.io/v1alpha1 +kind: HostLease +metadata: + name: host-54321 + namespace: osac-namespace + ownerReferences: + - apiVersion: osac.openshift.io/v1alpha1 + kind: BareMetalPool + name: bm-pool-12345 + namespace: osac-namespace +spec: + selector: + hostSelector: + hostType: fc430 + managedBy: None + provisionState: available + templateID: osac.templates.bm_host-image-provision + templateParameters: some-json-string +``` + +When a Host Lease CR is created, it is not initially associated with a host from the backend inventory. That association only occurs once the Host Operator performs reconciliation: + +``` yaml +apiVersion: osac.openshift.io/v1alpha1 +kind: HostLease +metadata: + name: host-54321 + namespace: osac-namespace + ownerReferences: + - apiVersion: osac.openshift.io/v1alpha1 + kind: BareMetalPool + name: bm-pool-12345 + namespace: osac-namespace +spec: + selector: + hostType: fc430 + managedBy: None + provisionState: available + templateID: osac.templates.bm_host-image-provision + templateParameters: some-json-string + externalID: host-id + externalName: host-name + hostClass: openstack + networkClass: openstack +status: + poweredOn: false +``` + +That assignment also sets the Host Lease CR's `hostClass` and `networkClass`. Once the `hostClass` is set, the Host Lease CR allows management operations against the assigned host (limited to power control for this proposal): + +``` yaml +apiVersion: osac.openshift.io/v1alpha1 +kind: HostLease +metadata: + name: host-54321 + namespace: osac-namespace + ownerReferences: + - apiVersion: osac.openshift.io/v1alpha1 + kind: BareMetalPool + name: bm-pool-12345 + namespace: osac-namespace +spec: + selector: + hostType: fc430 + managedBy: None + provisionState: available + templateID: osac.templates.bm_host-image-provision + templateParameters: some-json-string + externalID: host-id + externalName: host-name + hostClass: openstack + networkClass: openstack + poweredOn: true +status: + poweredOn: false +``` + +`networkClass` is not directly utilized by the bare metal operators; however, it may be used within the templates that initialize a Host Lease. -We would like to implement a solution for serial console access that can be utilized by both the bare metal service and the virtual machine service. -We will propose a design for that solution in a future enhancement proposal. +### Implementation Details/Notes/Constraints -#### Task Groupings +Bare metal fulfillment is designed to support varying bare metal inventory and management backends. These backends may often be one and the same - OpenStack Ironic, Carbide - but they may also be different - Netbox for inventory, and OpenShift BareMetalHosts for management. -We can perform this implementation in smaller steps: +We integrate with these backends through two different interfaces defined by the host operator: a bare metal inventory interface, and a bare metal management backend. -* HostPool and Host - * Tenants can create/update/delete HostPools - * No network attachments - * Hosts are created when a HostPool is fulfilled - * Tenants can view their Hosts - * Tenants have power control on their Hosts -* Implement HostPool hostSelectors - * filter by name (to support removal of specific hosts from a HostPool) - * filter by rack - * affinity/anti-affinity -* Network Attachments - * Tenants can specify network attachments when creating/updating HostPools - * Requires network service -* Serial Console - * Tenants can enable/disable serial console on a Host +We assume that the external inventory should always contain an accurate representation of the current state of any bare metal host. As a result, we do not have an intermediate host representation within OSAC, as that merely adds an additional sync requirement. ### Risks and Mitigations -How do we prevent a tenant from performing unauthorized bare metal operations? The first layer of protection -is the fact that the tenant never has direct access to the underlying bare metal management service; they -only have access to the fulfillment service and the limited operations provided by the fulfillment API. -Naturally, that API will need strong multi-tenancy support in order to prevent a tenant from accessing -resources allocated to a different tenant; that feature is being implemented outside of this proposal. +We need to integrate with a variety of bare metal backends. If our design does not allow us to do so easily, then we may find ourselves in a position where we need to restructure and recode a significant +chunk of our architecture. In order to mitigate this risk, we will clearly identify the process for integrating with a backend; and we will isolate the integration points to both minimize the needed integration work, and to reduce the impact of portions of this architecture need to be reworked. ### Drawbacks -?? +TBD ## Alternatives (Not Implemented) -* One alternative would be to focus on an ESI replacement before developing the bare metal fulfillment functionality. However, we anticipate that the MOC will continue to use ESI for now; for that reason, it makes sense to perform this implementation with ESI in mind, and simply make it easy to replace ESI if/when needed. Note that it is also possible that we will continue to use ESI if the issues pushing us towards a replacement are resolved. -* We could limit a HostPool to hosts from a single resource class; a tenant in need of hosts from multiple resource classes would simply create multiple HostPools. However allowing multiple resource classes in HostPool does provide additional conveniences. For example, cluster fulfillment allows for OpenShift clusters with mixed resource classes; if we move that workflow towards using HostPools for bare metal configuration, then associating that cluster with a single HostPool allows for far easier bare metal tracking and management. - -## Open Questions [optional] - -* Can we manage network attachment by MAC address? -* Do we allow a network to be attached multiple times to a host (on different interfaces)? +One question that has been raised is whether it's possible to standardize on a single bare metal management backend, with OpenShift BareMetalHosts identified as an ideal backend. However BareMetalHosts do not represent the temporary access to a bare metal resource that Host Leases do. In addition there are some gaps with OSAC requirements, such as the capability to support alternative bare metal backends. ## Test Plan From 26e17d56e920c6857fcb858f608e70c17ca8825e Mon Sep 17 00:00:00 2001 From: Adrien Gentil Date: Mon, 8 Jun 2026 09:50:43 +0200 Subject: [PATCH 005/450] OSAC-1118: Add BareMetal Instance API enhancement proposal (#45) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * OSAC-1118: Add BareMetal Instance API enhancement proposal Signed-off-by: Adrien Gentil * Address review: refine user stories and workflow descriptions Signed-off-by: Adrien Gentil * Address CodeRabbit review comments Signed-off-by: Adrien Gentil * Address review: immutable ssh_key/user_data, quota as non-goal Signed-off-by: Adrien Gentil * OSAC-1118: update initial backend from OpenStack to BCM Scope change: BCM (NVIDIA Base Command Manager) is the first supported backend, replacing OpenStack/MOC references throughout the EP. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: keep API backend-agnostic, BCM only in admin/impl context Public API (proto field comments, tenant user stories, non-goals) must not leak BCM as the backend. BCM is referenced only where relevant: Cloud Infrastructure Admin user story, workflow actors, risks, support procedures, and infrastructure section. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: simplify proto field comments in BaremetalInstanceTemplateSpecDefaults Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: address CodeRabbit review comments - Non-goals: document interim workarounds for deferred networking (user_data/cloud-init) and custom profile deferral - Pluggable provider: add HostLease → BaremetalInstanceStatus mapping table - Test plan: remove quota enforcement from unit tests (quota is non-goal) - Credential risk: defer to fulfillment-service cross-cutting policy instead of prescribing storage mechanisms Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: defer credential management to a dedicated security enhancement Credential storage is undefined and not owned by fulfillment-service. Only constraint stated: no credentials in CRDs or API responses. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: remove HostLease→BaremetalInstance mapping table (HostLease WIP) Mapping will be defined once the HostLease CRD schema is finalized. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: fix networking non-goal — network config owned by admin, not tenant Tenants have no mechanism to configure networking; it is defined by the Cloud Provider Admin in the BaremetalInstanceTemplate. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: remove Pluggable Provider Architecture section Pluggable backend interface definition is owned by OSAC-1032. This EP covers BCM integration against that interface, not the interface itself. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: clarify networking non-goal is phase-scoped, not permanent Network config via VirtualNetwork/Subnet/SecurityGroup is deferred to a dedicated networking enhancement, not removed from scope entirely. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: close open question on BaremetalInstance vs HostLease coupling HostLease remains distinct from BaremetalInstance because it serves cases outside fulfillment-service scope (e.g. cluster BM nodes). Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: empty BaremetalInstanceTemplateSpecDefaults, no fields yet spec_defaults follows template convention but no tenant-overridable fields are defined in this initial version. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: clarify BaremetalPool+HostLease may both be used initially Making pool optional is not a prerequisite; both CRs may exist in the initial implementation as internal details of the BMF component. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: rearchitect provisioning flow — FS creates HostLease directly - fulfillment-service creates HostLease CR directly (no BareMetalPool, no osac-operator CR creation step) - baremetal-fulfillment-operator owns inventory assignment and AAP provisioning trigger - osac-operator watches HostLease for status and signals back via Signal RPC - Add provisioning sequence diagram - Update actors, workflow steps, drawbacks, alternatives, version skew, and support procedures to match Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: address carbonin review comments - Remove pluggable provider abstraction from Goals (implementation concern of baremetal fulfillment component, not API scope) - Remove "no intermediate CRD" callout per carbonin's suggestion - Fix step numbering gap in provisioning sequence (9→8) - Note that delete+recreate retry may assign a different host - Remove ip_address from BaremetalInstanceStatus and mermaid diagram; deferred until networking integration is scoped - Simplify Drawbacks: drop consolidation language (BaremetalInstance and HostLease would never be the same object) - Simplify Alternatives: same reasoning, remove "future consolidation breaking change" framing Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: clarify run_strategy field comment Explain that run_strategy controls power state and note why an enum is used over a boolean, per tzumainn's review feedback. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: describe BMF/osac-operator division of responsibility Per carbonin's feedback, add a note near the non-goals clarifying the intended pattern: baremetal-fulfillment-operator handles host assignment and provisioning; osac-operator handles status feedback via Signal RPC. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: fix non-goals formatting, inline BMF/operator note Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: remove osac-operator non-goal bullet and inline note Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: clear drawbacks section Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1118: remove HostLease-as-API alternative Assisted-by: Claude Code Signed-off-by: Adrien Gentil --------- Signed-off-by: Adrien Gentil --- enhancements/baremetal-instance-api/README.md | 370 ++++++++++++++++++ 1 file changed, 370 insertions(+) create mode 100644 enhancements/baremetal-instance-api/README.md diff --git a/enhancements/baremetal-instance-api/README.md b/enhancements/baremetal-instance-api/README.md new file mode 100644 index 000000000..3f48da5f8 --- /dev/null +++ b/enhancements/baremetal-instance-api/README.md @@ -0,0 +1,370 @@ +--- +title: baremetal-instance-api +authors: + - agentil@redhat.com +creation-date: 2026-05-29 +last-updated: 2026-06-04 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-1118 +see-also: + - /enhancements/bare-metal-fulfillment +replaces: + - N/A +superseded-by: + - N/A +--- + +# BareMetal Instance API + +## Summary + +This enhancement introduces `BaremetalInstance` and `BaremetalInstanceTemplate` resources to the OSAC fulfillment-service public API, enabling tenants to provision and manage physical bare metal servers through a self-service interface. Templates encode the default instance type, OS base image, and network configuration and are published by Cloud Provider Admins via `osac-aap`. The design adopts a pluggable provider architecture — implemented in a dedicated baremetal fulfillment component — so that future bare metal backends can be integrated without breaking the API. This EP is scoped to the fulfillment-service API layer; operator, provisioning, UX, and E2E concerns are tracked as companion work items under OSAC-1118. + +## Motivation + +OSAC currently provides no fulfillment path for workloads requiring direct hardware access. Tenants running GPU-intensive, high-performance networking, or latency-sensitive workloads have no self-service mechanism to request and manage bare metal nodes through OSAC. + +### User Stories + +* As a **Tenant User**, I want to browse available `BaremetalInstanceTemplate` objects so that I can select the one that matches my workload requirements. +* As a **Tenant User**, I want to provision a bare metal server by specifying a `BaremetalInstanceTemplate` so that I can run workloads that require direct hardware access without manual coordination with the cloud provider. +* As a **Tenant User**, I want to monitor the lifecycle state of my bare metal server (provisioning, running, failed) so that I can take corrective action if provisioning fails. +* As a **Tenant User**, I want to control the run strategy (power on/off) of my bare metal server so that I can manage its operational state without deprovisioning it. +* As a **Tenant User**, I want to restart my bare metal server so that I can perform maintenance without deprovisioning it. +* As a **Tenant User**, I want to deprovision my bare metal server when my workload is complete so that resources are released. +* As a **Cloud Provider Admin**, I want to define and publish `BaremetalInstanceTemplate` objects so that I can control which hardware profiles and OS images are available to tenants. +* As a **Cloud Infrastructure Admin**, I want the OSAC stack to integrate with BCM (NVIDIA Base Command Manager) so that bare metal provisioning is automated through the existing control plane. + +### Goals + +* Provide a self-service API for tenants to create `BaremetalInstance` resources referencing a `BaremetalInstanceTemplate`. +* Define `BaremetalInstanceTemplate` as a provider-managed global catalog resource exposing the available hardware profile, OS base image, and default network configuration. +* Align the `BaremetalInstance` API shape with the existing `ComputeInstance` API (same `id`, `metadata`, `spec`, `status` envelope, same condition and state patterns, same run strategy and restart signal mechanism). +* Expose the API through both gRPC and the existing REST gateway. + +### Non-Goals + +* Integration with OSAC networking resources (`VirtualNetwork`, `Subnet`, `SecurityGroup`) — deferred to a future enhancement; in this initial phase, network configuration is fixed by the Cloud Provider Admin as part of the `BaremetalInstanceTemplate` and tenants have no mechanism to configure networking at provision time. A dedicated networking enhancement will enable tenants to create their own `Subnet` and attach it to a `BaremetalInstance`. +* Custom hardware profile or OS image selection by tenants at provision time — fixed by the template. Tenants requiring a different profile must request the Cloud Provider Admin to publish a new template. +* Per-organization template scoping — templates are global; a BMaaS catalog feature is deferred to a future enhancement. +* AAP playbook implementation — covered in companion work. +* Baremetal fulfillment component implementation — covered in companion work. +* UI and UX — covered in companion work. +* E2E test implementation — covered in companion work. +* Support for multiple bare metal backends in this initial release — the architecture is designed for future extensibility. +* Quota enforcement for `BaremetalInstance` — deferred to a future enhancement. + +## Proposal + +The proposal introduces two new resource types to the fulfillment-service public API: + +**`BaremetalInstanceTemplate`** is a provider-defined, read-only resource that encodes a validated combination of hardware profile (instance flavor), OS base image, and default network configuration. Templates are defined by Cloud Provider Admins as Ansible roles in `osac-aap` and published to the fulfillment service via the provisioning pipeline. Tenants discover available templates via List/Get and reference one when creating a `BaremetalInstance`. This mirrors the role of `ComputeInstanceTemplate` for VMs. + +**`BaremetalInstance`** is a tenant-created resource representing a provisioned bare metal server. Its `spec` references a template and carries provisioning parameters (SSH public key, user data, run strategy, restart signal). Its `status` exposes the lifecycle state and conditions aligned with the `ComputeInstance` pattern. + +Provisioning is driven by a chain of components: the fulfillment service creates a `HostLease` CR in the management cluster when a `BaremetalInstance` is created. The baremetal fulfillment operator picks up the `HostLease`, finds and assigns a free host from the inventory backend, and triggers `osac-aap` for host-level setup (OS image, SSH key, user data). The osac-operator watches `HostLease` CRs for status changes and pushes updates back to the fulfillment service via the `Signal` RPC. `HostLease` is an internal implementation detail; tenants never interact with it directly. + +### Workflow Description + +**Actors:** +- **Cloud Provider Admin** — defines `BaremetalInstanceTemplate` objects as Ansible roles in `osac-aap`; templates are published to the fulfillment service via the provisioning pipeline. +- **Tenant User** — creates and manages `BaremetalInstance` resources via the public API. +- **Fulfillment Service** — handles `BaremetalInstance` CRUD and creates `HostLease` CRs directly in the management cluster. +- **osac-operator** — watches `HostLease` CRs for status changes and pushes them to the fulfillment service via the `Signal` RPC; does not create any CRs in this flow. +- **baremetal-fulfillment-operator** — reconciles `HostLease` CRs: finds and assigns a free host from the inventory backend, then triggers `osac-aap` for host-level provisioning. +- **osac-aap** — executes the provisioning and deprovisioning Ansible roles (OS image, SSH key, user data); triggered by the baremetal-fulfillment-operator. +- **Inventory Backend** — the bare metal inventory used to find and assign free hosts (e.g. OpenStack Ironic or BCM). + +#### Provisioning + +1. The Cloud Provider Admin defines one or more `BaremetalInstanceTemplate` objects as Ansible roles in `osac-aap`. The provisioning pipeline applies these roles and publishes the resulting templates to the fulfillment service. +2. The Tenant User lists available templates: + ``` + GET /api/fulfillment/v1/baremetal_instance_templates + ``` +3. The Tenant User creates a `BaremetalInstance` referencing the desired template: + ``` + POST /api/fulfillment/v1/baremetal_instances + ``` +4. The fulfillment service creates a `HostLease` CR in the management cluster; `BaremetalInstance.status.state` is set to `BAREMETAL_INSTANCE_STATE_PROVISIONING`. +5. The baremetal-fulfillment-operator picks up the `HostLease` and queries the inventory backend to find and assign a free host matching the requested host type and selector. +6. The baremetal-fulfillment-operator triggers `osac-aap` to run the host-level provisioning template (OS image, SSH key, user data) and updates the `HostLease` status on completion. +7. The osac-operator watches the `HostLease` CR and pushes status updates to the fulfillment service via the `Signal` RPC; the fulfillment service reflects this in `BaremetalInstance.status`. +8. The Tenant User polls until `status.state` is `BAREMETAL_INSTANCE_STATE_RUNNING`: + ``` + GET /api/fulfillment/v1/baremetal_instances/{id} + ``` + +#### Failure Handling + +If any step in the provisioning chain fails (playbook error, BCM API failure, `HostLease` stuck), the osac-operator sets `BaremetalInstance.status.state` to `BAREMETAL_INSTANCE_STATE_FAILED` via the `Signal` RPC. The tenant can inspect the `conditions` field for details. To retry, the tenant deletes and recreates the `BaremetalInstance`; note that recreating may result in a different physical host being assigned from the inventory. + +#### Deprovisioning + +1. The Tenant User deletes the instance: + ``` + DELETE /api/fulfillment/v1/baremetal_instances/{id} + ``` +2. The fulfillment service deletes the `HostLease` CR; `BaremetalInstance.status.state` transitions to `BAREMETAL_INSTANCE_STATE_DELETING`. +3. The baremetal-fulfillment-operator reconciles the `HostLease` deletion: triggers `osac-aap` for the host-level deprovisioning template, then unassigns the host from the inventory backend. +4. The osac-operator watches the `HostLease` CR deletion and pushes final status via the `Signal` RPC. +5. The `BaremetalInstance` resource is removed once deprovisioning completes. + +#### Provisioning Sequence + +```mermaid +sequenceDiagram + actor TU as Tenant User + participant FS as Fulfillment Service + participant MC as Management Cluster + participant OP as osac-operator + participant BMF as baremetal-fulfillment-operator + participant AAP as osac-aap + participant INV as Inventory Backend + + TU->>FS: GET /baremetal_instance_templates + FS-->>TU: list of available templates + + TU->>FS: POST /baremetal_instances {template_id, ssh_key, ...} + FS->>MC: create HostLease CR (hostType, profile, templateID, templateParameters) + FS-->>TU: 201 Created {id, state: PROVISIONING} + + MC-->>BMF: watch: HostLease CR created (no ExternalHostID yet) + BMF->>INV: FindFreeHost(hostType, matchLabels) + INV-->>BMF: free host found + BMF->>INV: AssignHost(externalHostID, hostLeaseID, labels) + INV-->>BMF: host assigned (HostClass, NetworkClass) + BMF->>MC: update HostLease CR (ExternalHostID, HostClass, NetworkClass) + BMF->>AAP: run hostTemplate role (OS image, SSH key, user data) + AAP->>INV: provision node + INV-->>AAP: provisioning complete + AAP-->>BMF: host setup complete + BMF->>MC: update HostLease status (phase: Ready) + + MC-->>OP: watch: HostLease CR Ready + OP->>FS: Signal RPC (state: RUNNING) + + TU->>FS: GET /baremetal_instances/{id} + FS-->>TU: {state: RUNNING} +``` + +### API Extensions + +**New gRPC services (public API):** +- `BaremetalInstances` — CRUD for tenant-managed bare metal instances. +- `BaremetalInstanceTemplates` — List/Get for provider-defined templates (read-only for tenants). + +**New gRPC service (private API):** +- `BaremetalInstances` (private) — adds the `Signal` RPC for `osac-operator` feedback, following the existing `ComputeInstances` private API pattern. + +**REST gateway routes:** +- `GET /api/fulfillment/v1/baremetal_instance_templates` +- `GET /api/fulfillment/v1/baremetal_instance_templates/{id}` +- `GET /api/fulfillment/v1/baremetal_instances` +- `GET /api/fulfillment/v1/baremetal_instances/{id}` +- `POST /api/fulfillment/v1/baremetal_instances` +- `PATCH /api/fulfillment/v1/baremetal_instances/{object.id}` — `{object.id}` is the gRPC-gateway convention for PATCH: the request body carries the full object and the ID is bound via `object.id`, matching the `ComputeInstance` pattern. +- `DELETE /api/fulfillment/v1/baremetal_instances/{id}` + +**PATCH semantics:** The `PATCH` endpoint supports partial updates to mutable fields only: `run_strategy` and `restart_requested_at`. The `template`, `ssh_key`, and `user_data` fields are immutable after creation; requests that attempt to modify them are rejected with `400 Bad Request`. A `FieldMask` is applied automatically from the fields present in the request body. + +### Implementation Details/Notes/Constraints + +#### Proto: BaremetalInstanceTemplate + +```protobuf +message BaremetalInstanceTemplate { + string id = 1; + Metadata metadata = 2; + + // Human-friendly short description (CLI/UI single-line display). + string title = 3; + + // Human-friendly long description in Markdown format. + string description = 4; + + // Default spec values applied when creating a BaremetalInstance from this template. + BaremetalInstanceTemplateSpecDefaults spec_defaults = 5; +} + +// BaremetalInstanceTemplateSpecDefaults follows the same convention as other OSAC template types. +// No overridable spec fields are defined in this initial version; fields will be added in future +// enhancements as tenant-configurable options are introduced (e.g. networking integration). +message BaremetalInstanceTemplateSpecDefaults {} +``` + +#### Proto: BaremetalInstance + +```protobuf +message BaremetalInstance { + string id = 1; + Metadata metadata = 2; + BaremetalInstanceSpec spec = 3; + BaremetalInstanceStatus status = 4; +} + +message BaremetalInstanceSpec { + // Reference to the BaremetalInstanceTemplate. Required on create; immutable after creation. + string template = 1; + + // SSH public key injected into the OS at provisioning time. Immutable after creation. + // Must be a valid SSH public key in OpenSSH format (ssh-rsa, ssh-ed25519, etc.). + // Invalid keys are rejected at create time with a 400 error. + optional string ssh_key = 2; + + // User data (e.g. cloud-init). Passed to the OS at first boot. Immutable after creation. + // Maximum size: 64 KB. + optional string user_data = 3; + + // Run strategy controls the power state of the bare metal instance (on/off). + // Named run_strategy for alignment with ComputeInstance; an enum is used rather + // than a boolean to leave room for future states (e.g. suspended backends). + optional BaremetalInstanceRunStrategy run_strategy = 4; + + // RestartRequestedAt is a timestamp signal to request a power cycle. + // Set to the current time to trigger an immediate restart. + // The controller executes the restart if this timestamp is greater than status.last_restarted_at. + optional google.protobuf.Timestamp restart_requested_at = 5; +} + +message BaremetalInstanceStatus { + BaremetalInstanceState state = 1; + repeated BaremetalInstanceCondition conditions = 2; + + // LastRestartedAt records when the last restart was initiated by the controller. + optional google.protobuf.Timestamp last_restarted_at = 3; +} + +enum BaremetalInstanceRunStrategy { + BAREMETAL_INSTANCE_RUN_STRATEGY_UNSPECIFIED = 0; + + // The instance is kept powered on. + BAREMETAL_INSTANCE_RUN_STRATEGY_ALWAYS = 1; + + // The instance is powered off. + BAREMETAL_INSTANCE_RUN_STRATEGY_HALTED = 2; +} + +enum BaremetalInstanceState { + BAREMETAL_INSTANCE_STATE_UNSPECIFIED = 0; + BAREMETAL_INSTANCE_STATE_PROVISIONING = 1; + BAREMETAL_INSTANCE_STATE_RUNNING = 2; + BAREMETAL_INSTANCE_STATE_FAILED = 3; + BAREMETAL_INSTANCE_STATE_DELETING = 4; +} + +enum BaremetalInstanceConditionType { + BAREMETAL_INSTANCE_CONDITION_TYPE_UNSPECIFIED = 0; + + // Infrastructure has been allocated to this instance by the baremetal fulfillment component. + BAREMETAL_INSTANCE_CONDITION_TYPE_PROVISIONED = 1; + + // OS image and user configuration have been applied by the provisioning provider. + BAREMETAL_INSTANCE_CONDITION_TYPE_CONFIGURATION_APPLIED = 2; + + // The machine is available and assigned to the tenant (HostLease active). + // Does not indicate OS readiness — the tenant is responsible for OS-level checks. + BAREMETAL_INSTANCE_CONDITION_TYPE_READY = 3; + + // A power cycle (restart) is currently in progress. + BAREMETAL_INSTANCE_CONDITION_TYPE_RESTART_IN_PROGRESS = 4; + + // A restart request has failed. + BAREMETAL_INSTANCE_CONDITION_TYPE_RESTART_FAILED = 5; + + // Configuration changes require a restart to take effect. + BAREMETAL_INSTANCE_CONDITION_TYPE_RESTART_REQUIRED = 6; +} +``` + +#### Alignment with ComputeInstance + +`BaremetalInstance` intentionally mirrors `ComputeInstance`: +- Same `id` + `Metadata` + `spec` + `status` envelope. +- Same condition shape (`ConditionStatus`, `last_transition_time`, `reason`, `message`). +- Same CRUD service shape and REST route pattern (`/api/fulfillment/v1/`). +- Same private API `Signal` RPC for `osac-operator` feedback loop. +- Same run strategy (`run_strategy`) and restart signal mechanism (`restart_requested_at`, `last_restarted_at`). + +Fields specific to VMs (network attachments) are absent from `BaremetalInstance` in this initial version and may be added in future enhancements. + +### Risks and Mitigations + +**Risk:** Backend API changes break the provisioning path. +**Mitigation:** The provider abstraction in the baremetal fulfillment component isolates backend-specific code. Integration tests run against a BCM-compatible environment in CI. + +**Risk:** Long provisioning times (bare metal typically takes 10–30 minutes) confuse tenants or expose timeout issues. +**Mitigation:** The `PROVISIONING` state and condition set give clear asynchronous progress signals. No API call blocks on physical provisioning; the osac-operator and baremetal fulfillment component reconcile asynchronously. + +**Risk:** Tenant isolation errors — one tenant accesses or deprovisions another tenant's bare metal instance. +**Mitigation:** Follows the same OPA-based authorization model as `ComputeInstance`. Tenant scoping is enforced at the fulfillment-service layer via existing middleware. + +**Risk:** Tenants provision unlimited bare metal instances, exhausting backend capacity. +**Mitigation:** Quota enforcement for `BaremetalInstance` is deferred to a future enhancement. Until quotas are implemented, capacity limits must be managed out-of-band by the Cloud Provider Admin. + +**Risk:** Compromised backend credentials expose the bare metal backend to unauthorized access. +**Mitigation:** Credential storage and management for the baremetal backend are out of scope for this EP and will be defined in a dedicated security enhancement. The baremetal fulfillment component must not embed credentials in CRDs or API responses. + +### Drawbacks + +None identified. + +## Alternatives (Not Implemented) + +**Map `BaremetalInstance` to the existing `ComputeInstance` resource with a baremetal flag.** This avoids a new resource type but conflates VM and bare metal semantics, complicates template definitions, and requires dispatching on a field value rather than resource type. A dedicated resource type provides cleaner separation of concerns and allows independent API evolution. + +**Expose the bare metal backend API directly to tenants.** This bypasses OSAC entirely, eliminating tenant isolation, quota enforcement, and the pluggable backend architecture. Not viable for a multi-tenant cloud platform. + +## Open Questions + +1. ~~Should `BaremetalInstance` and `HostLease` be the same object, or remain distinct?~~ **Closed:** Distinct. The fulfillment service creates a `HostLease` as the internal backend CRD, but `HostLease` is also used by other workflows (cluster-as-a-service, agent provisioning) that bypass the fulfillment service entirely. Coupling the public API schema to `HostLease` would make future consolidation a breaking change. + +## Test Plan + +Test plan will be finalized during the implementation phase. Expected coverage: + +- **Unit tests:** Proto field validation, state machine transitions, provider interface mocking. +- **Integration tests:** `BaremetalInstance` CRUD via gRPC, template List/Get, Signal RPC feedback loop, OPA authorization enforcement, PATCH immutability enforcement. +- **E2E tests:** Full provisioning and deprovisioning workflow against BCM; CI pipeline configured to run E2E tests on merge. + +Tricky areas: asynchronous provisioning lifecycle (tests must handle delays or mock the provider), template immutability enforcement after create, tenant isolation boundary checks, and failure-path recovery (FAILED state → delete → recreate). + +## Graduation Criteria + +Graduation criteria will be defined when targeting a release. Expected progression: + +- **Dev Preview:** API available, BCM backend functional, basic CRUD working end-to-end. +- **Tech Preview:** E2E tests passing in CI, UX available, pluggable backend architecture validated with at least one additional backend stub. +- **GA:** Production deployment, full test coverage, support procedures operationally validated. + +## Upgrade / Downgrade Strategy + +This is a new API with no impact on existing resources. Downgrading requires deleting all `BaremetalInstance` and `BaremetalInstanceTemplate` resources and uninstalling the new CRDs from the management cluster before reverting the fulfillment-service image. + +## Version Skew Strategy + +The fulfillment-service, baremetal-fulfillment-operator, and osac-operator must be upgraded together. If the baremetal-fulfillment-operator is not running, `HostLease` CRs will not be reconciled and new `BaremetalInstance` create requests will remain in `PROVISIONING` indefinitely. If the osac-operator baremetal controller is not running, status updates will not propagate back to the fulfillment service. + +## Support Procedures + +**Symptom:** `BaremetalInstance` stuck in `BAREMETAL_INSTANCE_STATE_PROVISIONING`. +**Diagnosis:** Check baremetal-fulfillment-operator logs for inventory or AAP errors. Check osac-aap job logs for playbook failures. Verify the `HostLease` CR exists and that `ExternalHostID` has been set (indicates host was found in inventory). +**Resolution:** If the host was allocated but provisioning failed, delete the `HostLease` CR manually and delete the `BaremetalInstance` to trigger a clean retry. + +**Symptom:** All `BaremetalInstance` creations remain in `BAREMETAL_INSTANCE_STATE_PROVISIONING` with no `HostLease` created. +**Diagnosis:** The fulfillment service may not be able to reach the management cluster API. Check fulfillment-service logs for Kubernetes API errors. +**Resolution:** Verify management cluster connectivity and credentials used by the fulfillment service. + +**Symptom:** All `BaremetalInstance` creations transition to `BAREMETAL_INSTANCE_STATE_FAILED`. +**Diagnosis:** Check baremetal-fulfillment-operator logs for inventory API or AAP errors. Validate inventory backend credentials, verify AAP availability, and confirm network connectivity. +**Resolution:** Rotate credentials if expired, resolve network connectivity issues, then retry by deleting the failed `BaremetalInstance` and recreating it. + +**Symptom:** Create returns `404 Not Found` for the template ID. +**Diagnosis:** The `BaremetalInstanceTemplate` does not exist or is not visible to the tenant's organization. +**Resolution:** Cloud Provider Admin must create and publish the appropriate template via `osac-aap`. + +**Disabling the API:** Scale the baremetal-fulfillment-operator and the osac-operator baremetal controller to 0 replicas. Existing `HostLease` CRs persist but are not reconciled. Re-enabling both resumes reconciliation without data loss. + +## Infrastructure Needed + +BCM (NVIDIA Base Command Manager) access (credentials, API endpoint) is required for integration and E2E testing. This infrastructure is managed by the cloud provider and must be provisioned as part of the OSAC CI environment setup. From f4fedae27e65893bfb77c95293d3d53a42a1736d Mon Sep 17 00:00:00 2001 From: Ygal Blum Date: Mon, 8 Jun 2026 12:36:17 -0400 Subject: [PATCH 006/450] OSAC-55: Add VM Snapshots enhancement proposal (#48) Introduce a Snapshot API for OSAC ComputeInstance resources, enabling tenants to create, list, get, delete, and restore from point-in-time snapshots of their VMs through the fulfillment-service API. Key design decisions: - New Snapshot resource (proto + CRD + dual controller) - Both online and offline snapshot modes via KubeVirt - In-place restore via declarative signal on ComputeInstance - Cascade deletion when parent ComputeInstance is deleted Assisted-by: Claude Code Signed-off-by: Ygal Blum --- enhancements/vm-snapshots/README.md | 999 ++++++++++++++++++++++++++++ 1 file changed, 999 insertions(+) create mode 100644 enhancements/vm-snapshots/README.md diff --git a/enhancements/vm-snapshots/README.md b/enhancements/vm-snapshots/README.md new file mode 100644 index 000000000..ca3258d53 --- /dev/null +++ b/enhancements/vm-snapshots/README.md @@ -0,0 +1,999 @@ +--- +title: vm-snapshots +authors: + - yblum@redhat.com +creation-date: 2026-06-01 +last-updated: 2026-06-01 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-55 +see-also: + - "/enhancements/vmaas" + - "/enhancements/computeinstance-phase-condition-expansion" + - "/enhancements/networking" +replaces: + - N/A +superseded-by: + - N/A +--- + +# VM Snapshots + +## Summary + +This enhancement introduces a ComputeInstanceSnapshot API for OSAC ComputeInstance resources, enabling tenants to create, list, get, delete, and restore from point-in-time snapshots of their virtual machines. Since OSAC tenants have no direct access to the clusters where their VMs run, snapshot management must be exposed through the fulfillment-service API and orchestrated by the osac-operator, which interacts with KubeVirt's `VirtualMachineSnapshot` and `VirtualMachineRestore` primitives on the workload cluster. The feature supports both online snapshots (taken while the VM is running, with optional guest agent quiescing) and offline snapshots (taken while the VM is stopped). Restore is in-place only, reverting the VM's disks and configuration to the state captured in a selected snapshot. + +## Terminology + +- **Snapshot**: A point-in-time capture of a ComputeInstance's disk state and VM configuration. Snapshots are tied to the specific ComputeInstance they were taken from and cannot exist independently. +- **Online Snapshot**: A snapshot taken while the VM is running. KubeVirt uses the QEMU guest agent (when available) to quiesce the guest filesystem before capturing disk state, ensuring crash-consistent or application-consistent snapshots. +- **Offline Snapshot**: A snapshot taken while the VM is stopped. The disk state is captured at rest, guaranteeing full consistency. +- **In-place Restore**: Reverting a ComputeInstance's disks and VM configuration to the state captured in a snapshot. The VM must be stopped before restoring. The restore overwrites the current disk contents; it does not create a new ComputeInstance. +- **VirtualMachineSnapshot (KubeVirt)**: The underlying KubeVirt custom resource that captures a point-in-time snapshot of a `VirtualMachine`, including its `DataVolume`/`PVC` disk state and VM spec. +- **VirtualMachineRestore (KubeVirt)**: The underlying KubeVirt custom resource that restores a `VirtualMachine` from a `VirtualMachineSnapshot`. + +## Motivation + +OSAC tenants run virtual machines through the ComputeInstance API but have no direct access to the Kubernetes clusters where those VMs execute. This means tenants cannot use KubeVirt's native snapshot capabilities (`VirtualMachineSnapshot`, `VirtualMachineRestore`) because they have no access to the workload cluster's API server. Without an OSAC-level snapshot API, tenants have no way to: + +- Capture the state of a VM before making risky changes (software upgrades, configuration changes) +- Roll back a VM to a known-good state after a failed update +- Create recovery points as part of a maintenance workflow + +This gap forces tenants to rely on application-level backup strategies or external tooling, neither of which can capture the full VM state (disks, configuration) in a single atomic operation. + +### User Stories + +- As a Cloud Provider Admin, I want snapshots to be automatically deleted when a ComputeInstance is deleted so that orphaned snapshots do not consume storage indefinitely. +- As a Cloud Provider Admin, I want to monitor snapshot operations (creation, deletion, restore) so that I can detect and troubleshoot failures. +- As a Cloud Provider Admin, I want a configurable default limit on snapshots per ComputeInstance so that I can prevent storage exhaustion before a formal quota system is implemented. +- As a tenant, I want to create a snapshot of my running ComputeInstance so that I can capture its current state before applying a software upgrade. +- As a tenant, I want to create a snapshot of my stopped ComputeInstance so that I can preserve its disk state before making configuration changes. +- As a tenant, I want to list all snapshots of a specific ComputeInstance so that I can see available recovery points and their creation times. +- As a tenant, I want to delete a snapshot I no longer need so that I can free up storage resources. +- As a tenant, I want to restore my ComputeInstance from a specific snapshot so that I can roll back to a known-good state after a failed update. +- As a tenant, I want to see the progress and status of my snapshot creation and restore operations so that I know when they complete or if they fail. + +### Goals + +- Provide a self-service API for tenants to create, list, get, and delete snapshots of their ComputeInstances through the fulfillment-service. +- Support both online (VM running) and offline (VM stopped) snapshots, with filesystem quiescing via the QEMU guest agent when available. +- Enable tenants to restore a ComputeInstance in-place from a selected snapshot via the fulfillment-service API. +- Automatically cascade-delete all snapshots when the parent ComputeInstance is deleted. +- Expose snapshot and restore lifecycle status through conditions on the respective resources, following existing OSAC condition patterns. +- Maintain tenant isolation: tenants can only see and manage snapshots of their own ComputeInstances. + +### Non-Goals + +The following are explicitly out of scope for this proposal and may be addressed in future enhancements: + +- **Exporting snapshots as images**: Converting a snapshot into a reusable VM image (e.g., for use as a `ComputeInstanceTemplate` source or for distribution via an image registry) is out of scope. This would require additional infrastructure for image conversion, storage, and registry integration. +- **Creating a new ComputeInstance from a snapshot**: Restoring a snapshot into a brand-new ComputeInstance (clone) is not supported in this proposal. Only in-place restore to the original ComputeInstance is supported. +- **Scheduled or automated snapshots**: Automatic snapshot creation on a schedule or triggered by events (e.g., before an upgrade) is not included. Tenants must explicitly create snapshots. +- **Snapshot quotas or storage limits**: Limiting the number of snapshots per ComputeInstance or the total snapshot storage per tenant will be addressed in a separate quota proposal. +- **Cross-cluster snapshot migration**: Moving snapshots between workload clusters is not supported. +- **Application-consistent snapshots with custom hooks**: While online snapshots leverage the QEMU guest agent for filesystem quiescing, custom pre/post-snapshot hooks inside the guest are not supported. + +## Proposal + +The VM Snapshot feature introduces a new `ComputeInstanceSnapshot` resource as a first-class OSAC entity, following the established pattern of all other OSAC resources (proto types + gRPC service in fulfillment-service, CRD + dual controller in osac-operator). Snapshots are child resources of a ComputeInstance — they are created against a specific VM and cannot exist independently. + +The restore operation is modeled as a declarative signal on the ComputeInstance resource, following the same timestamp-based pattern used for VM restarts (`spec.restart_requested_at` / `status.last_restarted_at`). This keeps the restore mechanism consistent with existing OSAC conventions and avoids introducing a separate Restore resource for what is fundamentally an operation on the ComputeInstance. + +### Workflow Description + +**Actors:** + +- **Tenant**: A user who manages ComputeInstances and their snapshots through the fulfillment-service API. +- **Provider/Admin**: An operator administrator who monitors and troubleshoots the system. +- **osac-operator**: The Kubernetes operator that reconciles OSAC CRDs into KubeVirt resources on the workload cluster. +- **fulfillment-service**: The gRPC/REST API server that stores OSAC resource state and exposes tenant-facing APIs. + + +#### Creating a Snapshot + +1. The tenant creates a snapshot of their ComputeInstance: + + ```bash + osac create computeinstancesnapshot --compute-instance --description "Before upgrade" + ``` + +2. The fulfillment-service validates that the referenced ComputeInstance exists and belongs to the tenant, creates a `ComputeInstanceSnapshot` record in PostgreSQL with state `CREATING`, and creates a corresponding `ComputeInstanceSnapshot` CR on the hub cluster. +3. The osac-operator's ComputeInstanceSnapshot controller detects the new `ComputeInstanceSnapshot` CR. +4. The controller looks up the parent ComputeInstance CR to find the `VirtualMachineReference` (the KubeVirt VM name and namespace on the workload cluster). +5. The controller creates a KubeVirt `VirtualMachineSnapshot` CR on the workload cluster, targeting the KubeVirt `VirtualMachine`. +6. The controller watches the `VirtualMachineSnapshot` status and updates the OSAC `ComputeInstanceSnapshot` CR's phase and conditions accordingly: + - While KubeVirt reports the snapshot is in progress: phase = `Creating`, condition `Ready` = `False`. + - When KubeVirt reports the snapshot is ready: phase = `Ready`, condition `Ready` = `True`, `creationTime` and `online` fields populated. + - If KubeVirt reports failure: phase = `Failed`, condition `Ready` = `False` with error message. +7. The feedback controller syncs the OSAC `ComputeInstanceSnapshot` CR status back to the fulfillment-service via the private gRPC API. +8. The tenant polls the snapshot status until it reaches `READY`: + + ```bash + osac get computeinstancesnapshot + ``` + + +```mermaid +sequenceDiagram + participant T as Tenant + participant FS as Fulfillment Service + participant Hub as Hub Cluster (CRs) + participant Op as osac-operator + participant WC as Workload Cluster (KubeVirt) + + Note over T,WC: Create Snapshot + T->>FS: ComputeInstanceSnapshots.Create(compute_instance_id) + FS->>Hub: Create ComputeInstanceSnapshot CR + Op->>Hub: Watch ComputeInstanceSnapshot CR + Op->>WC: Create VirtualMachineSnapshot + WC-->>Op: VMS status: Ready + Op->>Hub: Update ComputeInstanceSnapshot CR phase=Ready + Op->>FS: Feedback: ComputeInstanceSnapshot READY +``` + +##### Create Error Cases + +###### Non-existent ComputeInstance + +```bash +$ osac create computeinstancesnapshot --compute-instance nonexistent-id --description "Before upgrade" +Error: compute instance "nonexistent-id" not found +``` + +###### ComputeInstance not owned by tenant + +```bash +$ osac create computeinstancesnapshot --compute-instance --description "Before upgrade" +Error: compute instance "" not found +``` + +(Same "not found" error to avoid leaking existence of other tenants' resources.) + +###### ComputeInstance not yet provisioned (no VM on workload cluster) + +```bash +$ osac create computeinstancesnapshot --compute-instance --description "Before upgrade" +Error: compute instance "" is not yet provisioned +``` + + +#### Listing Snapshots + +1. The tenant lists snapshots, optionally filtering by ComputeInstance: + + ```bash + osac list computeinstancesnapshots --compute-instance + ``` + +2. The fulfillment-service returns all snapshots belonging to the tenant's ComputeInstances, with support for pagination, filtering (CEL expressions), and ordering. + + + + +#### Restoring from a Snapshot + +1. The tenant stops the ComputeInstance and confirms it reaches `Stopped` state: + + ```bash + osac update computeinstance --run-strategy Halted + osac get computeinstance + ``` + +2. The tenant triggers a restore from a specific snapshot: + + ```bash + osac update computeinstance --restore-from-snapshot + ``` + +3. The fulfillment-service validates that the snapshot exists, belongs to the same ComputeInstance, and is in `READY` state. It updates the ComputeInstance record and CR. The fulfillment-service intentionally does not validate VM running state because its view of VM state is eventually consistent via the feedback loop; the ComputeInstance controller is the authoritative enforcer (see step 5). +4. The ComputeInstance controller detects that `spec.restore_requested_at` is greater than `status.last_restored_at` and that `spec.restore_from_snapshot` is set. +5. The controller verifies the VM is stopped. If the VM is running, it sets the `RestoreFailed` condition with a message indicating the VM must be stopped first. +6. The controller creates a KubeVirt `VirtualMachineRestore` CR on the workload cluster, referencing the KubeVirt `VirtualMachineSnapshot` associated with the OSAC ComputeInstanceSnapshot. +7. The controller sets the `RestoreInProgress` condition to `True` on the ComputeInstance CR. +8. The controller watches the `VirtualMachineRestore` status: + - When KubeVirt reports the restore is complete: set `status.last_restored_at` to `spec.restore_requested_at`, clear `RestoreInProgress`, set `RestoreInProgress` condition to `False`. + - If KubeVirt reports failure: set `RestoreFailed` condition to `True` with error details. +9. The feedback controller syncs the updated conditions and status back to the fulfillment-service. +10. The tenant starts the ComputeInstance: + + ```bash + osac update computeinstance --run-strategy Always + ``` + + +```mermaid +sequenceDiagram + participant T as Tenant + participant FS as Fulfillment Service + participant Hub as Hub Cluster (CRs) + participant Op as osac-operator + participant WC as Workload Cluster (KubeVirt) + + Note over T,WC: Restore from Snapshot + T->>FS: ComputeInstances.Update(restore_from_snapshot, restore_requested_at) + FS->>Hub: Update ComputeInstance CR + Op->>Hub: Watch ComputeInstance CR + Op->>WC: Create VirtualMachineRestore + WC-->>Op: VMR status: Complete + Op->>Hub: Update CI conditions, last_restored_at + Op->>FS: Feedback: Restore complete +``` + +##### Restore Error Cases + +###### Snapshot not in READY state + +```bash +$ osac update computeinstance --restore-from-snapshot +Error: compute instance snapshot "" is not ready +``` + +###### Snapshot not found or does not belong to this ComputeInstance + +```bash +$ osac update computeinstance --restore-from-snapshot +Error: compute instance snapshot "" not found +``` + +(Same "not found" error for non-existent snapshots and snapshots belonging to a different ComputeInstance, to avoid leaking existence of other resources.) + + +#### Deleting a Snapshot + + +1. The tenant deletes a snapshot: + + ```bash + osac delete computeinstancesnapshot + ``` + +2. The fulfillment-service marks the snapshot as `DELETING` and updates the `ComputeInstanceSnapshot` CR on the hub cluster. +3. The ComputeInstanceSnapshot controller deletes the corresponding KubeVirt `VirtualMachineSnapshot` on the workload cluster. +4. Once the KubeVirt snapshot is removed, the controller removes its finalizer from the OSAC `ComputeInstanceSnapshot` CR, allowing Kubernetes garbage collection to proceed. +5. The feedback controller syncs the deletion state to fulfillment-service. + + + +##### Delete Error Cases + +###### Deleting non-existent snapshot + +```bash +$ osac delete computeinstancesnapshot nonexistent-id +Error: compute instance snapshot "nonexistent-id" not found +``` + +###### Snapshot not owned by tenant + +```bash +$ osac delete computeinstancesnapshot +Error: compute instance snapshot "" not found +``` + +(Same "not found" error to avoid leaking existence of other tenants' resources.) + +#### Cascade Deletion + +When a ComputeInstance is deleted: + +1. The ComputeInstance controller's deletion handler queries for all `ComputeInstanceSnapshot` CRs that reference the ComputeInstance. + **Security requirement:** The query must filter by both `spec.computeInstanceRef` AND the `osac.openshift.io/tenant` annotation to ensure only snapshots owned by the same tenant are deleted. +2. Each `ComputeInstanceSnapshot` CR is deleted, triggering the ComputeInstanceSnapshot controller's deletion flow (which deletes the underlying KubeVirt `VirtualMachineSnapshot`). +3. Only after all snapshots are fully deleted does the ComputeInstance controller proceed with its own deletion finalizer logic. + +```mermaid +sequenceDiagram + participant T as Tenant + participant FS as Fulfillment Service + participant Hub as Hub Cluster (CRs) + participant Op as osac-operator + participant WC as Workload Cluster (KubeVirt) + + Note over T,WC: Cascade Deletion (ComputeInstance deleted) + T->>FS: ComputeInstances.Delete(id) + FS->>Hub: Delete ComputeInstance CR + Op->>Hub: Watch ComputeInstance CR (DeletionTimestamp set) + Op->>Hub: List ComputeInstanceSnapshot CRs for ComputeInstance + Op->>Hub: Delete each ComputeInstanceSnapshot CR + Op->>WC: Delete VirtualMachineSnapshot (per snapshot) + WC-->>Op: VMS deleted + Op->>Hub: Remove ComputeInstanceSnapshot finalizers + Op->>Hub: All ComputeInstanceSnapshots deleted + Op->>WC: Delete VirtualMachine + WC-->>Op: VM deleted + Op->>Hub: Remove ComputeInstance finalizer + Op->>FS: Feedback: ComputeInstance deleted +``` + +### API Extensions + +#### New gRPC Services (fulfillment-service) + +**Public API** (`proto/public/osac/public/v1/`): + +- `compute_instance_snapshot_type.proto` — `ComputeInstanceSnapshot` message with `ComputeInstanceSnapshotSpec`, `ComputeInstanceSnapshotStatus`, `ComputeInstanceSnapshotState` enum, `ComputeInstanceSnapshotCondition`, `ComputeInstanceSnapshotConditionType` enum +- `compute_instance_snapshots_service.proto` — `ComputeInstanceSnapshots` service with `Create`, `Get`, `List`, `Delete` RPCs and HTTP annotations + +**Private API** (`proto/private/osac/private/v1/`): + +- `compute_instance_snapshot_type.proto` — mirrors public type +- `compute_instance_snapshots_service.proto` — `ComputeInstanceSnapshots` service with `Create`, `Get`, `List`, `Delete`, `Update`, and `Signal` RPCs + +#### Modified gRPC Types (fulfillment-service) + +**ComputeInstance type** (both public and private): + +- `ComputeInstanceSpec`: Add `restore_from_snapshot` (optional string) and `restore_requested_at` (optional timestamp) fields +- `ComputeInstanceStatus`: Add `last_restored_at` (optional timestamp) field +- `ComputeInstanceConditionType` enum: Add `RESTORE_IN_PROGRESS` and `RESTORE_FAILED` values + +#### New CRD (osac-operator) + +- `ComputeInstanceSnapshot` CRD (`api/v1alpha1/computeinstancesnapshot_types.go`) — Kubernetes representation of a snapshot, with spec referencing the parent ComputeInstance and status tracking the KubeVirt VirtualMachineSnapshot lifecycle + +#### Modified CRD (osac-operator) + +- `ComputeInstance` CRD: Add `restoreFromSnapshot` and `restoreRequestedAt` fields to spec, `lastRestoredAt` to status, new condition types `RestoreInProgress` and `RestoreFailed` + +#### New Controllers (osac-operator) + +- `computeinstancesnapshot_controller.go` — Provisions/deprovisions KubeVirt `VirtualMachineSnapshot` on the workload cluster +- `computeinstancesnapshot_feedback_controller.go` — Syncs ComputeInstanceSnapshot CR status to fulfillment-service via private gRPC API + +#### Modified Controllers (osac-operator) + +- `computeinstance_controller.go` — Add restore handling: detect `restoreRequestedAt` signal, create KubeVirt `VirtualMachineRestore`, update conditions and `lastRestoredAt`. Add cascade deletion of ComputeInstanceSnapshot CRs. + +#### Finalizers + +- `osac.openshift.io/computeinstancesnapshot-finalizer` — on `ComputeInstanceSnapshot` CR, ensures KubeVirt `VirtualMachineSnapshot` is deleted before the OSAC ComputeInstanceSnapshot CR is removed +- `osac.openshift.io/computeinstancesnapshot-feedback-finalizer` — on `ComputeInstanceSnapshot` CR, ensures final state is synced to fulfillment-service before removal + +#### RBAC + +The osac-operator will need additional RBAC permissions: + +```go ++kubebuilder:rbac:groups=snapshot.kubevirt.io,resources=virtualmachinesnapshots;virtualmachinerestores,verbs=get;list;watch;create;delete ++kubebuilder:rbac:groups=osac.openshift.io,resources=computeinstancesnapshots,verbs=get;list;watch;create;update;patch;delete ++kubebuilder:rbac:groups=osac.openshift.io,resources=computeinstancesnapshots/status,verbs=get;update;patch ++kubebuilder:rbac:groups=osac.openshift.io,resources=computeinstancesnapshots/finalizers,verbs=update +``` + +### Implementation Details/Notes/Constraints + +#### ComputeInstanceSnapshot Proto Type (fulfillment-service) + +```proto +// compute_instance_snapshot_type.proto (public) + +message ComputeInstanceSnapshot { + string id = 1; + Metadata metadata = 2; + ComputeInstanceSnapshotSpec spec = 3; + ComputeInstanceSnapshotStatus status = 4; +} + +message ComputeInstanceSnapshotSpec { + // Reference to the ComputeInstance this snapshot belongs to. + // Must be the ID of an existing ComputeInstance owned by the tenant. + // This cannot be modified after the snapshot is created. + string compute_instance = 1; + + // User-provided description of the snapshot. + optional string description = 2; +} + +message ComputeInstanceSnapshotStatus { + // Indicates the overall state of the snapshot. + ComputeInstanceSnapshotState state = 1; + + // Conditions describing the detailed status of the snapshot. + repeated ComputeInstanceSnapshotCondition conditions = 2; + + // Timestamp when the snapshot data was fully captured. + // Empty while the snapshot is still being created. + optional google.protobuf.Timestamp creation_time = 3; + + // Size of the snapshot data in bytes. + // Populated when the snapshot reaches READY state. + optional int64 size_bytes = 4; + + // Whether the snapshot was taken while the VM was running (online) + // or while the VM was stopped (offline). + bool online = 5; +} + +enum ComputeInstanceSnapshotState { + COMPUTE_INSTANCE_SNAPSHOT_STATE_UNSPECIFIED = 0; + COMPUTE_INSTANCE_SNAPSHOT_STATE_CREATING = 1; + COMPUTE_INSTANCE_SNAPSHOT_STATE_READY = 2; + COMPUTE_INSTANCE_SNAPSHOT_STATE_FAILED = 3; + COMPUTE_INSTANCE_SNAPSHOT_STATE_DELETING = 4; +} + +message ComputeInstanceSnapshotCondition { + ComputeInstanceSnapshotConditionType type = 1; + ConditionStatus status = 2; + google.protobuf.Timestamp last_transition_time = 3; + optional string reason = 4; + optional string message = 5; +} + +enum ComputeInstanceSnapshotConditionType { + COMPUTE_INSTANCE_SNAPSHOT_CONDITION_TYPE_UNSPECIFIED = 0; + + // Indicates that the snapshot data has been captured successfully. + COMPUTE_INSTANCE_SNAPSHOT_CONDITION_TYPE_READY = 1; +} +``` + +ComputeInstanceSnapshot naming follows the standard OSAC `Metadata` pattern: tenants provide a name via `metadata.name` at creation time, subject to the same uniqueness validation applied to other OSAC resources. Auto-generated UUIDs are used for the `id` field. + +#### ComputeInstanceSnapshots Service Proto (fulfillment-service) + +```proto +// compute_instance_snapshots_service.proto (public) + +service ComputeInstanceSnapshots { + rpc List(ComputeInstanceSnapshotsListRequest) returns (ComputeInstanceSnapshotsListResponse) { + option (google.api.http) = {get: "/api/fulfillment/v1/compute_instance_snapshots"}; + } + + rpc Get(ComputeInstanceSnapshotsGetRequest) returns (ComputeInstanceSnapshotsGetResponse) { + option (google.api.http) = { + get: "/api/fulfillment/v1/compute_instance_snapshots/{id}" + response_body: "object" + }; + } + + rpc Create(ComputeInstanceSnapshotsCreateRequest) returns (ComputeInstanceSnapshotsCreateResponse) { + option (google.api.http) = { + post: "/api/fulfillment/v1/compute_instance_snapshots" + body: "object" + response_body: "object" + }; + } + + rpc Delete(ComputeInstanceSnapshotsDeleteRequest) returns (ComputeInstanceSnapshotsDeleteResponse) { + option (google.api.http) = {delete: "/api/fulfillment/v1/compute_instance_snapshots/{id}"}; + } +} + +message ComputeInstanceSnapshotsListRequest { + optional int32 offset = 1; + optional int32 limit = 2; + optional string filter = 3; + optional string order = 4; +} + +message ComputeInstanceSnapshotsListResponse { + int32 size = 1; + int32 total = 2; + repeated ComputeInstanceSnapshot items = 3; +} + +message ComputeInstanceSnapshotsGetRequest { + string id = 1; +} + +message ComputeInstanceSnapshotsGetResponse { + ComputeInstanceSnapshot object = 1; +} + +message ComputeInstanceSnapshotsCreateRequest { + ComputeInstanceSnapshot object = 1; +} + +message ComputeInstanceSnapshotsCreateResponse { + ComputeInstanceSnapshot object = 1; +} + +message ComputeInstanceSnapshotsDeleteRequest { + string id = 1; +} + +message ComputeInstanceSnapshotsDeleteResponse {} +``` + +> **Note:** The public `ComputeInstanceSnapshots` service intentionally omits an `Update` RPC. Snapshots are immutable after creation — their spec (including `description`) cannot be modified once the snapshot is taken. This reflects the nature of snapshots as point-in-time captures. + +The private API mirrors the public API and adds `Update` and `Signal` RPCs, following the existing pattern. The private `Update` RPC is used exclusively by the feedback controller to sync status fields (state, conditions, creation_time, size_bytes, online) from the osac-operator back to the fulfillment-service — it does not allow spec modifications. + +#### ComputeInstance Proto Changes (fulfillment-service) + +> **Implementation prerequisite**: The field numbers and enum values below are illustrative based on the proto definitions at the time of writing. Before implementation, verify the current highest field numbers in `ComputeInstanceSpec`, `ComputeInstanceStatus`, and `ComputeInstanceConditionType`, check for any `reserved` declarations or retired numbers (e.g., spec fields 12 and 13 are currently unassigned), and assign the next available numbers to avoid wire-format collisions. + +New fields added to existing `ComputeInstanceSpec`: + +```proto +// The ID of the snapshot to restore from. Set this field together with +// restore_requested_at to initiate an in-place restore. +// The snapshot must belong to this ComputeInstance and be in READY state. +// The VM must be stopped (run_strategy = "Halted") before initiating a restore. +optional string restore_from_snapshot = 16; + +// Timestamp signal to request a restore from snapshot. +// Set this field to the current time to request a restore. +// The controller will execute the restore if this timestamp is greater than +// status.last_restored_at. +// This is a declarative signal mechanism, following the same pattern as +// restart_requested_at. +optional google.protobuf.Timestamp restore_requested_at = 17; +``` + +New field added to existing `ComputeInstanceStatus`: + +```proto +// Records when the last restore was initiated by the controller. +// Set to spec.restore_requested_at when the controller processes a restore request. +// Empty if no restore has been performed. +optional google.protobuf.Timestamp last_restored_at = 6; +``` + +New values added to `ComputeInstanceConditionType` enum: + +```proto +// Indicates that a restore from snapshot is in progress. +// When status is TRUE, a restore operation is currently in progress. +// When status is FALSE, no restore is currently in progress. +COMPUTE_INSTANCE_CONDITION_TYPE_RESTORE_IN_PROGRESS = 7; + +// Indicates that a restore request has failed. +// When status is TRUE, the restore could not be executed. +// The message field will contain details about the failure. +// When status is FALSE, there is no restore failure. +COMPUTE_INSTANCE_CONDITION_TYPE_RESTORE_FAILED = 8; +``` + +#### ComputeInstanceSnapshot CRD (osac-operator) + +```go +// api/v1alpha1/computeinstancesnapshot_types.go + +// ComputeInstanceSnapshotSpec defines the desired state of a ComputeInstanceSnapshot +type ComputeInstanceSnapshotSpec struct { + // ComputeInstanceRef is the name of the ComputeInstance CR this snapshot belongs to. + // Must reference an existing ComputeInstance in the same namespace. + // +kubebuilder:validation:Required + // +kubebuilder:validation:MinLength=1 + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="computeInstanceRef is immutable" + ComputeInstanceRef string `json:"computeInstanceRef"` + + // Description is a user-provided description of the snapshot. + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxLength=256 + Description string `json:"description,omitempty"` +} + +// ComputeInstanceSnapshotPhase represents the phase of a ComputeInstanceSnapshot +// +kubebuilder:validation:Enum=Creating;Ready;Failed;Deleting +type ComputeInstanceSnapshotPhase string + +const ( + ComputeInstanceSnapshotPhaseCreating ComputeInstanceSnapshotPhase = "Creating" + ComputeInstanceSnapshotPhaseReady ComputeInstanceSnapshotPhase = "Ready" + ComputeInstanceSnapshotPhaseFailed ComputeInstanceSnapshotPhase = "Failed" + ComputeInstanceSnapshotPhaseDeleting ComputeInstanceSnapshotPhase = "Deleting" +) + +// ComputeInstanceSnapshotConditionType is a valid value for .status.conditions.type +type ComputeInstanceSnapshotConditionType string + +const ( + // ComputeInstanceSnapshotConditionReady indicates the snapshot data has been captured. + ComputeInstanceSnapshotConditionReady ComputeInstanceSnapshotConditionType = "Ready" +) + +// VirtualMachineSnapshotReference contains a reference to the KubeVirt VirtualMachineSnapshot +type VirtualMachineSnapshotReference struct { + // Name of the KubeVirt VirtualMachineSnapshot CR + Name string `json:"name"` + // Namespace of the KubeVirt VirtualMachineSnapshot CR + Namespace string `json:"namespace"` +} + +// ComputeInstanceSnapshotStatus defines the observed state of a ComputeInstanceSnapshot +type ComputeInstanceSnapshotStatus struct { + // Phase provides a single-value overview of the ComputeInstanceSnapshot state + // +kubebuilder:validation:Optional + Phase ComputeInstanceSnapshotPhase `json:"phase,omitempty"` + + // Conditions holds an array of conditions describing the ComputeInstanceSnapshot state + // +kubebuilder:validation:Optional + Conditions []metav1.Condition `json:"conditions,omitempty"` + + // Reference to the KubeVirt VirtualMachineSnapshot on the workload cluster + // +kubebuilder:validation:Optional + VirtualMachineSnapshotRef *VirtualMachineSnapshotReference `json:"virtualMachineSnapshotRef,omitempty"` + + // CreationTime is when the snapshot data was fully captured + // +kubebuilder:validation:Optional + CreationTime *metav1.Time `json:"creationTime,omitempty"` + + // SizeBytes is the total size of the snapshot data + // +kubebuilder:validation:Optional + SizeBytes *int64 `json:"sizeBytes,omitempty"` + + // Online indicates whether the snapshot was taken while the VM was running + // +kubebuilder:validation:Optional + Online bool `json:"online,omitempty"` + + // Jobs tracks the history of snapshot operations. + // JobStatus is an existing shared type from api/v1alpha1/job_types.go used by other OSAC resources to track operation history. + // +kubebuilder:validation:Optional + Jobs []JobStatus `json:"jobs,omitempty"` +} + +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:shortName=cisnap +// +kubebuilder:printcolumn:name="ComputeInstance",type=string,JSONPath=`.spec.computeInstanceRef` +// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase` +// +kubebuilder:printcolumn:name="Online",type=boolean,JSONPath=`.status.online` +// +kubebuilder:printcolumn:name="CreationTime",type=date,JSONPath=`.status.creationTime` + +// ComputeInstanceSnapshot is the Schema for the computeinstancesnapshots API +type ComputeInstanceSnapshot struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty,omitzero"` + Spec ComputeInstanceSnapshotSpec `json:"spec"` + Status ComputeInstanceSnapshotStatus `json:"status,omitempty,omitzero"` +} + +// +kubebuilder:object:root=true + +// ComputeInstanceSnapshotList contains a list of ComputeInstanceSnapshot +type ComputeInstanceSnapshotList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + Items []ComputeInstanceSnapshot `json:"items"` +} +``` + +#### ComputeInstance CRD Changes (osac-operator) + +New fields added to `ComputeInstanceSpec`: + +```go +// RestoreFromSnapshot is the name of the ComputeInstanceSnapshot CR to restore from. +// Set this field together with RestoreRequestedAt to initiate an in-place restore. +// The ComputeInstanceSnapshot must reference this ComputeInstance and be in Ready phase. +// The ComputeInstance must be stopped (RunStrategy = Halted) before restoring. +// +kubebuilder:validation:Optional +RestoreFromSnapshot string `json:"restoreFromSnapshot,omitempty"` + +// RestoreRequestedAt is a timestamp signal to request a restore from snapshot. +// Set this field to the current time to request a restore. +// The controller will execute the restore if this timestamp is greater than +// status.lastRestoredAt. +// +kubebuilder:validation:Optional +// +kubebuilder:validation:Format=date-time +RestoreRequestedAt *metav1.Time `json:"restoreRequestedAt,omitempty"` +``` + +New field added to `ComputeInstanceStatus`: + +```go +// LastRestoredAt records when the last restore was initiated by the controller. +// Set to spec.restoreRequestedAt when the controller processes a restore. +// Empty if no restore has been performed. +// +kubebuilder:validation:Optional +// +kubebuilder:validation:Format=date-time +LastRestoredAt *metav1.Time `json:"lastRestoredAt,omitempty"` +``` + +New condition type constants: + +```go +// ComputeInstanceConditionRestoreInProgress indicates a restore from snapshot is in progress +ComputeInstanceConditionRestoreInProgress ComputeInstanceConditionType = "RestoreInProgress" + +// ComputeInstanceConditionRestoreFailed indicates a restore request has failed +ComputeInstanceConditionRestoreFailed ComputeInstanceConditionType = "RestoreFailed" +``` + +#### ComputeInstanceSnapshot Controller Logic (osac-operator) + +The ComputeInstanceSnapshot controller follows the dual-controller pattern: + +**Resource Controller** (`computeinstancesnapshot_controller.go`): + +Before any reconciliation logic, the controller checks the `osac.openshift.io/management-state` annotation. If the annotation is present and set to `Unmanaged`, the controller skips reconciliation and returns immediately. This follows the standard OSAC operator convention shared by all controllers. + +1. **On create** (DeletionTimestamp is zero, no finalizer): + - Add finalizer `osac.openshift.io/computeinstancesnapshot-finalizer`. + - Look up the parent ComputeInstance CR by `spec.computeInstanceRef`. + - Validate the ComputeInstance exists and has a `VirtualMachineReference` (VM is provisioned). + - Determine the KubeVirt VM name and namespace from the ComputeInstance's status. + - Create a KubeVirt `VirtualMachineSnapshot` on the workload cluster targeting the KubeVirt `VirtualMachine`. + - Set phase to `Creating`. + +2. **On reconcile** (snapshot exists, KubeVirt VMS created): + - Fetch the KubeVirt `VirtualMachineSnapshot` from the workload cluster. + - Map KubeVirt snapshot status to OSAC ComputeInstanceSnapshot status: + - KubeVirt `ReadyToUse = true` → phase `Ready`, condition `Ready` = `True`, populate `creationTime`, `sizeBytes`, `online`. + - KubeVirt `ReadyToUse = false` and no error → phase `Creating`, condition `Ready` = `False`. + - KubeVirt error or failure deadline exceeded → phase `Failed`, condition `Ready` = `False` with error message. + - Update the OSAC ComputeInstanceSnapshot CR status if changed. + +3. **On delete** (DeletionTimestamp is set): + - Delete the KubeVirt `VirtualMachineSnapshot` from the workload cluster. + - Wait until the KubeVirt resource is gone (requeue if still present). + - Remove the `computeinstancesnapshot-finalizer`. + +**Feedback Controller** (`computeinstancesnapshot_feedback_controller.go`): + +Following the established feedback controller pattern: + +- On phase changes: sync OSAC ComputeInstanceSnapshot phase → fulfillment-service ComputeInstanceSnapshot state via private `ComputeInstanceSnapshots.Update` gRPC. +- Phase-to-state mapping: `Creating` → `CREATING`, `Ready` → `READY`, `Failed` → `FAILED`, `Deleting` → `DELETING`. +- On deletion: send final deletion state via `ComputeInstanceSnapshots.Signal`, then remove the feedback finalizer. +- Handle `NotFound` gracefully during deletion (fulfillment-service may archive before finalizer removal). + +#### Restore Logic in ComputeInstance Controller + +Added to the existing ComputeInstance controller's `handleUpdate` method: + +1. Check if spec.restoreRequestedAt > status.lastRestoredAt (new restore request). +2. If yes: + a. Validate spec.restoreFromSnapshot is set and references an existing ComputeInstanceSnapshot CR in Ready phase. + b. Validate the ComputeInstance is stopped (phase == Stopped). + - If not stopped: set RestoreFailed condition with message "ComputeInstance must be stopped before restoring". + - Return without requeue (tenant must stop the VM and re-trigger). + c. Look up the ComputeInstanceSnapshot CR's VirtualMachineSnapshotRef to find the KubeVirt VMS name. + d. Create a KubeVirt VirtualMachineRestore on the workload cluster, targeting the VM + and referencing the KubeVirt `VirtualMachineSnapshot`. + e. Set RestoreInProgress condition to True with reason "RestoreRequested". + f. Clear RestoreFailed condition if previously set. +3. If RestoreInProgress is True: + a. Check the KubeVirt VirtualMachineRestore status. + b. If complete: set status.lastRestoredAt = spec.restoreRequestedAt, + set RestoreInProgress to False, clear RestoreFailed. + c. If failed: set RestoreFailed to True with error details, + set RestoreInProgress to False. + d. If in progress: requeue after StatusPollInterval. + +> **Note:** After a successful restore, `spec.restoreFromSnapshot` and `spec.restoreRequestedAt` remain set in the ComputeInstance spec. This is consistent with the `restartRequestedAt` pattern — the timestamp guard (`restoreRequestedAt > lastRestoredAt`) prevents re-triggering, and the stale snapshot reference is harmless. To initiate another restore (from the same or a different snapshot), the tenant sets `restoreRequestedAt` to a new timestamp. + +#### Tenant Isolation + +- Snapshots inherit tenant isolation from their parent ComputeInstance. +- The `ComputeInstanceSnapshot` CR must include the `osac.openshift.io/tenant` annotation, copied from the parent ComputeInstance. +- The `osac.io/owner-reference` annotation on the ComputeInstanceSnapshot CR must reference the parent ComputeInstance. +- The fulfillment-service ensures tenants can only access snapshots belonging to their own ComputeInstances. +- The `ComputeInstanceSnapshots.List` and `ComputeInstanceSnapshots.Get` endpoints enforce tenant scoping through the same CEL filter and authorization mechanisms used by other resources. + +#### Database Schema (fulfillment-service) + +A new `compute_instance_snapshots` table following the standard OSAC database pattern: + +```sql +CREATE TABLE compute_instance_snapshots ( + id UUID PRIMARY KEY, + creation_timestamp TIMESTAMPTZ NOT NULL, + update_timestamp TIMESTAMPTZ NOT NULL, + deletion_timestamp TIMESTAMPTZ, + finalizers TEXT[], + creators TEXT[], + tenants TEXT[], + labels JSONB, + annotations JSONB, + compute_instance_id TEXT NOT NULL, + data JSONB NOT NULL -- Serialized ComputeInstanceSnapshot protobuf +); + +CREATE INDEX idx_compute_instance_snapshots_compute_instance_id + ON compute_instance_snapshots (compute_instance_id); +``` + +Notes: + +- The `data` JSONB column contains the serialized ComputeInstanceSnapshot protobuf, following OSAC's standard database pattern. +- `deletion_timestamp` supports soft delete (NULL = active, non-NULL = soft-deleted). +- `tenants` array stores tenant IDs for isolation enforcement. +- `labels` and `annotations` are JSONB columns for metadata, following the standard OSAC resource pattern. +- `finalizers` and `creators` are standard OSAC resource lifecycle columns. +- Index on `compute_instance_id` enables efficient listing of snapshots by ComputeInstance. + +#### Cross-Component Integration + +| Component | Changes | +| --- | --- | +| **fulfillment-service** | New `ComputeInstanceSnapshot` proto types and `ComputeInstanceSnapshots` services (public + private), ComputeInstance proto updates (restore fields + conditions), new database table, server implementation, OPA policies | +| **osac-operator** | New `ComputeInstanceSnapshot` CRD + dual controller, ComputeInstance CRD updates (restore fields + conditions), ComputeInstance controller updates (restore logic, cascade delete), new RBAC for `snapshot.kubevirt.io` | +| **osac-installer** | RBAC updates for the operator ServiceAccount to access `snapshot.kubevirt.io` resources on the workload cluster | + +### Risks and Mitigations + +**Risk: Online snapshot inconsistency without guest agent** +If the QEMU guest agent is not installed in the guest VM, online snapshots may not be filesystem-consistent (crash-consistent only). Tenants may not realize their snapshot is crash-consistent rather than application-consistent. +**Mitigation**: Expose the `online` flag in the ComputeInstanceSnapshot status so tenants can see whether the snapshot was taken online. Document that installing the QEMU guest agent is recommended for consistent online snapshots. Consider adding a condition or warning when the guest agent is not detected. + +**Risk: Restore overwrites data without confirmation** +In-place restore is destructive — it replaces the current disk contents with the snapshot state. If a tenant accidentally restores from the wrong snapshot, data is lost. +**Mitigation**: The two-field declarative signal pattern (`restore_from_snapshot` + `restore_requested_at`) makes accidental restores unlikely — the tenant must explicitly set both fields. The VM must be stopped first, adding another deliberate step. Consider recommending that tenants create a snapshot before restoring as a best practice. + +**Risk: Snapshot storage consumption** +Snapshots consume storage on the workload cluster's storage backend (CSI VolumeSnapshots). Without quotas, a tenant could create an unbounded number of snapshots and exhaust storage. +**Mitigation**: Snapshot quotas are out of scope for this proposal but are listed as a future enhancement. Providers can monitor snapshot storage through standard Kubernetes storage metrics. As an interim measure, the operator could log warnings when a ComputeInstance has more than a configurable threshold of snapshots. + +**Risk: KubeVirt snapshot feature not available on the workload cluster** +The workload cluster may not have the KubeVirt snapshot feature gate enabled or the storage backend may not support CSI VolumeSnapshots. +**Mitigation**: The ComputeInstanceSnapshot controller should check for the presence of the `VirtualMachineSnapshot` CRD on the workload cluster at startup. If not available, the controller should log an error and set snapshots to `Failed` state with a clear message. Document the KubeVirt and storage prerequisites. + +### Drawbacks + +- **Increased API surface**: Adding a new first-class resource (ComputeInstanceSnapshot) with its own CRUD service increases the API surface area. This adds maintenance burden for proto definitions, server implementation, database table, OPA policies, and controllers. However, snapshots are a fundamental VM management capability that tenants expect, and the alternative of not providing them forces tenants into application-level backup strategies. + +- **Storage cost opacity**: Tenants can see their snapshot count and individual sizes, but may not understand the cumulative storage cost or how storage backend deduplication affects actual consumption. This is mitigated by the future quota system. + +- **Restore requires VM downtime**: In-place restore requires stopping the VM first, which means workloads on the VM are interrupted. This is a KubeVirt constraint for in-place restore. The alternative (creating a new VM from a snapshot) would avoid downtime but is deferred to a future enhancement. + +## Alternatives (Not Implemented) + +### Separate Restore Resource + +Instead of using the declarative signal pattern on ComputeInstance (`restore_from_snapshot` + `restore_requested_at`), an alternative is to introduce a separate `SnapshotRestore` resource (similar to KubeVirt's `VirtualMachineRestore`). + +**Pros**: Cleaner separation of concerns; restore operations are independently trackable with their own lifecycle. +**Cons**: Adds another CRD, proto type, service, and controller pair. Over-engineered for in-place-only restore. Not consistent with the existing restart signal pattern. + +**Decision**: The declarative signal pattern is simpler, consistent with `restart_requested_at`, and sufficient for in-place restore. If future enhancements add "restore to new VM" support, a separate Restore resource may become warranted. + +### Restore RPC on ComputeInstanceSnapshots Service + +Instead of updating the ComputeInstance spec to trigger a restore, an alternative is to add a `Restore` RPC to the `ComputeInstanceSnapshots` service. + +**Pros**: More intuitive API ("restore this snapshot") rather than "update this ComputeInstance with a restore signal." +**Cons**: Breaks the declarative resource model — RPCs that trigger side effects are imperative, not declarative. The OSAC pattern uses declarative spec changes to drive controller behavior. A custom RPC would also need special handling in the fulfillment-service (it's not a standard CRUD operation). + +**Decision**: The declarative signal on ComputeInstance is more consistent with OSAC's resource-oriented architecture. + +### ComputeInstanceSnapshots as Sub-resource of ComputeInstance + +Instead of a top-level `ComputeInstanceSnapshot` resource, snapshots could be modeled as a sub-resource of ComputeInstance (e.g., `ComputeInstances/{id}/ComputeInstanceSnapshots`). + +**Pros**: Clearer parent-child relationship in the API path. Simpler authorization (inherits from ComputeInstance). +**Cons**: Not consistent with how other OSAC child resources are modeled (e.g., Subnet is a top-level resource that references VirtualNetwork, not a sub-resource). Sub-resources in gRPC/REST are more complex to implement with grpc-gateway. Top-level resources with filter-by-parent are the established OSAC pattern. + +**Decision**: Follow the existing OSAC pattern of top-level resources with parent references and filtering. + +## Open Questions + +1. **Guest agent quiescing indicator**: The `online` field indicates whether the snapshot was taken while the VM was running, but it does not distinguish between crash-consistent snapshots (guest agent unavailable) and filesystem-consistent snapshots (guest agent quiesced the filesystem). Should the ComputeInstanceSnapshot status include a separate `quiesced` boolean or condition to surface this distinction? KubeVirt's `VirtualMachineSnapshot` status includes indications about whether the guest agent was used. + +2. **Maximum snapshots per VM**: Should there be a hard limit on the number of snapshots per ComputeInstance, even before a full quota system is implemented? A reasonable default (e.g., 20-50) could prevent storage exhaustion while a proper quota system is developed. + +## Test Plan + +### Unit Tests (fulfillment-service) + +- **ComputeInstanceSnapshot server CRUD**: Verify Create, Get, List, Delete operations with valid and invalid inputs. +- **Tenant isolation**: Verify that tenants cannot access snapshots belonging to other tenants' ComputeInstances. +- **Create validation**: Verify that creating a snapshot requires a valid, existing ComputeInstance ID owned by the tenant. +- **Delete behavior**: Verify that deleting a snapshot transitions it through the `DELETING` state. +- **List filtering**: Verify CEL filter expressions work for filtering by `compute_instance`, `state`, and `metadata.name`. +- **ComputeInstance restore fields**: Verify that the `restore_from_snapshot` and `restore_requested_at` fields are correctly persisted and retrieved. +- **Restore validation**: Verify that the fulfillment-service rejects restore requests when the snapshot is not in `READY` state or does not belong to the target ComputeInstance. + +### Unit Tests (osac-operator) + +- **ComputeInstanceSnapshot controller reconciliation**: + - Creating a ComputeInstanceSnapshot CR should create a KubeVirt `VirtualMachineSnapshot` on the workload cluster. + - Deleting a ComputeInstanceSnapshot CR should delete the corresponding KubeVirt `VirtualMachineSnapshot`. + - KubeVirt VMS `ReadyToUse = true` should transition the ComputeInstanceSnapshot phase to `Ready`. + - KubeVirt VMS failure should transition the ComputeInstanceSnapshot phase to `Failed`. + - Missing parent ComputeInstance should set the ComputeInstanceSnapshot to `Failed` with appropriate message. + - Missing VirtualMachineReference on ComputeInstance (VM not yet provisioned) should requeue. +- **ComputeInstanceSnapshot feedback controller**: + - Phase changes should be synced to fulfillment-service via gRPC. + - Deletion should send final state and remove the feedback finalizer. + - `NotFound` from fulfillment-service during deletion should be handled gracefully. +- **ComputeInstance restore logic**: + - Setting `restoreRequestedAt` > `lastRestoredAt` with valid `restoreFromSnapshot` and VM stopped should create a KubeVirt `VirtualMachineRestore`. + - Restore request on a running VM should set `RestoreFailed` condition. + - Restore request with a non-existent or non-Ready snapshot should set `RestoreFailed` condition. + - Successful KubeVirt restore should set `lastRestoredAt` and clear `RestoreInProgress`. + - Failed KubeVirt restore should set `RestoreFailed` condition. +- **Cascade deletion**: + - Deleting a ComputeInstance should trigger deletion of all associated ComputeInstanceSnapshot CRs. + - ComputeInstance finalizer should not be removed until all ComputeInstanceSnapshot CRs are fully deleted. +- **Tenant isolation**: + - ComputeInstanceSnapshot CR must carry the `osac.openshift.io/tenant` annotation from the parent ComputeInstance. + - ComputeInstanceSnapshot CR must carry the `osac.io/owner-reference` annotation referencing the parent ComputeInstance. +- **CRD validation**: + - `computeInstanceRef` should be immutable (reject updates that change it). + - `description` should be limited to 256 characters. + +### Integration Tests (fulfillment-service) + +- **End-to-end snapshot lifecycle**: Create a ComputeInstance, create a snapshot, verify it reaches `READY`, list snapshots filtered by ComputeInstance, delete the snapshot. +- **End-to-end restore lifecycle**: Create a ComputeInstance, create a snapshot, stop the VM, restore from snapshot, verify restore completes, start the VM. +- **Cascade deletion**: Create a ComputeInstance with multiple snapshots, delete the ComputeInstance, verify all snapshots are deleted. +- **Error cases**: Attempt to create a snapshot for a non-existent ComputeInstance, attempt to restore from a snapshot in `CREATING` state, attempt to restore a running VM. + +### Integration Tests (osac-operator) + +- **Kind cluster tests**: Deploy the operator with mock or real KubeVirt, create ComputeInstanceSnapshot CRs, verify KubeVirt VirtualMachineSnapshot resources are created and status is synced. +- **Feedback loop**: Verify that ComputeInstanceSnapshot status changes are correctly propagated to the fulfillment-service mock. + +### E2E Tests + +- **Full snapshot workflow**: Using a deployed OSAC environment with KubeVirt, test the complete tenant workflow: + 1. Create a ComputeInstance and wait for it to be Running. + 2. Write data to the VM's disk (e.g., create a file). + 3. Create a snapshot and wait for it to be Ready. + 4. Modify data on the VM's disk (e.g., delete the file). + 5. Stop the VM. + 6. Restore from the snapshot. + 7. Start the VM. + 8. Verify the data is restored (the file exists again). +- **Online snapshot**: Create a snapshot while the VM is running, verify it completes and the `online` flag is `true`. +- **Offline snapshot**: Stop the VM, create a snapshot, verify it completes and the `online` flag is `false`. +- **Multiple snapshots**: Create multiple snapshots of the same VM, list them, restore from a specific one, verify correctness. + +### Focus Areas + +- **Snapshot consistency**: Verify that online snapshots with the QEMU guest agent produce filesystem-consistent data. +- **Concurrent operations**: Verify behavior when a snapshot is being created while a restore is requested (should fail gracefully). +- **Storage backend compatibility**: Test with the CSI drivers available in the target environment to ensure VolumeSnapshot support works correctly. +- **Large disk snapshots**: Test with VMs that have large disks to verify snapshot creation doesn't time out. + +## Graduation Criteria + +Graduation criteria will be defined when targeting a release. Expected stages: + +- **Dev Preview**: ComputeInstanceSnapshot CRUD and in-place restore functional in development environments. Basic tenant isolation enforced. Limited to a single workload cluster. +- **Tech Preview**: Online and offline snapshot modes validated. Cascade deletion working reliably. Performance tested with multiple concurrent snapshot operations. E2E tests passing in CI. +- **GA**: Snapshot quotas implemented (separate proposal). Production deployment feedback incorporated. Documentation complete. Support procedures validated with real incidents. + +## Upgrade / Downgrade Strategy + +Not applicable - OSAC is pre-GA. + +## Version Skew Strategy + +Not applicable - OSAC is pre-GA. + +## Support Procedures + +### Failure Modes + +- **Snapshot stuck in Creating**: + - Check the ComputeInstanceSnapshot CR status and conditions: `kubectl get computeinstancesnapshot -n -o yaml` + - Check the KubeVirt VirtualMachineSnapshot on the workload cluster: `kubectl get virtualmachinesnapshot -n ` (on workload cluster) + - Common causes: storage backend does not support CSI VolumeSnapshots, insufficient storage quota on the workload cluster, KubeVirt snapshot feature gate not enabled. + - Check osac-operator logs: `kubectl logs -n osac-system deployment/osac-operator-controller-manager -f | grep snapshot` + +- **Snapshot stuck in Deleting**: + - Check if the KubeVirt VirtualMachineSnapshot still exists on the workload cluster. + - Check if the finalizer is present on the ComputeInstanceSnapshot CR. + - Common causes: workload cluster unreachable, RBAC permissions missing for `snapshot.kubevirt.io` resources. + +- **Restore stuck (RestoreInProgress is True indefinitely)**: + - Check the KubeVirt VirtualMachineRestore on the workload cluster: `kubectl get virtualmachinerestore -n ` (on workload cluster) + - Common causes: storage backend restore failure, insufficient storage quota. + - Check ComputeInstance conditions for error messages. + +- **RestoreFailed condition**: + - Check the condition message for details. + - Common causes: VM not stopped before restore, snapshot not in Ready state, snapshot belongs to a different ComputeInstance. + +### Disabling the Feature + +- To disable snapshot operations without removing the CRD: scale the ComputeInstanceSnapshot controller to 0 by disabling it via environment variable (to be added: `OSAC_ENABLE_COMPUTEINSTANCESNAPSHOT_CONTROLLER`). +- Existing ComputeInstanceSnapshot CRs will persist but will not be reconciled. New ComputeInstanceSnapshot CRs will not create KubeVirt VirtualMachineSnapshots. +- The restore functionality on ComputeInstance can be disabled by not setting the restore fields. If a restore is already in progress, it will complete on the KubeVirt side but the operator will not update the ComputeInstance status until re-enabled. + +### Recovery + +- Re-enable snapshot operations by re-enabling the controller. The controller will resume reconciliation of all ComputeInstanceSnapshot CRs, including any that were created while disabled. +- If a restore left the ComputeInstance in an inconsistent state (KubeVirt restore completed but operator didn't update status), re-reconciliation will detect the completed VirtualMachineRestore and update the status accordingly. + +## Infrastructure Needed + +The feature requires KubeVirt snapshot support and a CSI-compatible storage backend on the workload cluster: + +- **KubeVirt snapshot support**: The workload cluster must have KubeVirt with the snapshot feature gate enabled and a CSI driver that supports VolumeSnapshots. This is expected to be available in all OSAC deployment targets. +- **Storage backend with CSI VolumeSnapshot support**: Required for KubeVirt to create disk snapshots. Most production-grade CSI drivers (Ceph RBD, OpenStack Cinder, etc.) support this. From 39a76ae4cfeacca867071c699f289ceb65f0bb15 Mon Sep 17 00:00:00 2001 From: Adrien Gentil Date: Wed, 10 Jun 2026 16:40:19 +0200 Subject: [PATCH 007/450] OSAC-1317: Add BareMetalInstanceCatalogItem in BM enhancement (#53) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * OSAC-1317: rename BaremetalInstance → BareMetalInstance (case fix) BaremetalInstance → BareMetalInstance BaremetalInstanceTemplate → BareMetalInstanceTemplate BAREMETAL_INSTANCE_* → BARE_METAL_INSTANCE_* (enum values) Mechanical rename only; no API or semantic changes. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: adopt BareMetalInstanceCatalogItem API pattern Replace BareMetalInstanceTemplate with BareMetalInstanceCatalogItem as the tenant-facing resource, following the ComputeInstanceCatalogItem pattern. - BareMetalInstanceCatalogItem gains: template (ref to private resource), published, tenant, field_definitions fields - BareMetalInstanceSpec.catalog_item replaces spec.template - Private admin CRUD service added (BareMetalInstanceCatalogItems) - Public API restricted to List/Get on published catalog items - Private REST routes added under /api/private/v1/ - Workflow, proto definitions, mermaid diagram, test plan, non-goals, support procedures, and alignment section updated throughout - see-also: /enhancements/catalog-items added to frontmatter Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: add log hygiene note to Support Procedures Address CodeRabbit No-Sensitive-Data-In-Logs check: operators must redact ssh_key, user_data, and backend credentials before inspecting or sharing logs. Treat any such exposure as a security incident. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: remove stale non-goal about tenant scoping The tenant field is fully defined in the BareMetalInstanceCatalogItem proto (empty = global, set = tenant-scoped) and relied upon in the proposal, test plan, and support procedures. The non-goal claiming it was deferred contradicted the rest of the document. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: add BareMetalInstanceTemplate proto and private service Both BareMetalInstanceTemplate and BareMetalInstanceCatalogItem are now fully specified: - Add BareMetalInstanceTemplate proto (id, metadata, title, description, spec with host_type and os_image) - Add BareMetalInstanceTemplates private gRPC service (full CRUD) - Add private REST routes for baremetal_instance_templates Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: fix summary and consolidate non-goals - Correct template/catalog-item relationship: each catalog item is backed by a BareMetalInstanceTemplate (not the reverse) - Clarify that templates are created via osac-aap; catalog items are managed through the OSAC private API - Consolidate four "covered in companion work" non-goals into one line Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: align BareMetalInstance API with ComputeInstance pattern - Add Tenant Admin user story for tenant-scoped catalog items - Make BareMetalInstanceTemplate and BareMetalInstanceCatalogItem fully public (full CRUD), consistent with ComputeInstanceTemplate and ComputeInstanceCatalogItem - Public BareMetalInstanceCatalogItem omits tenant field (server-managed) - Show public/private proto variants for BareMetalInstanceCatalogItem - Add explicit consistency goal instead of scattered pattern references - Update API Extensions, REST routes, actors, and test plan accordingly Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: show catalog item resolution in provisioning sequence The fulfillment service resolves catalog_item to templateID and derives templateParameters from field_definitions internally before creating the HostLease CR. Update sequence diagram and provisioning step 4 to reflect this — HostLease carries templateID + templateParameters, not catalogItemID. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: fix mermaid syntax in provisioning sequence diagram Replace
with a single-line note — GitHub's mermaid renderer does not support HTML self-closing tags in Note blocks. Assisted-by: Claude Code Signed-off-by: Adrien Gentil * OSAC-1317: restrict public BareMetalInstanceTemplates to List/Get only Templates are managed through the private API by Cloud Provider Admins; tenants can discover available templates via public List/Get only. This matches the ComputeInstanceTemplates authconfig pattern. Update Goals, Proposal, Actors, Provisioning step 1, API Extensions, REST routes, and Alignment section accordingly. Assisted-by: Claude Code Signed-off-by: Adrien Gentil --------- Signed-off-by: Adrien Gentil --- enhancements/baremetal-instance-api/README.md | 292 ++++++++++++------ 1 file changed, 189 insertions(+), 103 deletions(-) diff --git a/enhancements/baremetal-instance-api/README.md b/enhancements/baremetal-instance-api/README.md index 3f48da5f8..ec42a9820 100644 --- a/enhancements/baremetal-instance-api/README.md +++ b/enhancements/baremetal-instance-api/README.md @@ -8,6 +8,7 @@ tracking-link: - https://redhat.atlassian.net/browse/OSAC-1118 see-also: - /enhancements/bare-metal-fulfillment + - /enhancements/catalog-items replaces: - N/A superseded-by: @@ -18,7 +19,7 @@ superseded-by: ## Summary -This enhancement introduces `BaremetalInstance` and `BaremetalInstanceTemplate` resources to the OSAC fulfillment-service public API, enabling tenants to provision and manage physical bare metal servers through a self-service interface. Templates encode the default instance type, OS base image, and network configuration and are published by Cloud Provider Admins via `osac-aap`. The design adopts a pluggable provider architecture — implemented in a dedicated baremetal fulfillment component — so that future bare metal backends can be integrated without breaking the API. This EP is scoped to the fulfillment-service API layer; operator, provisioning, UX, and E2E concerns are tracked as companion work items under OSAC-1118. +This enhancement introduces `BareMetalInstance` and `BareMetalInstanceCatalogItem` resources to the OSAC fulfillment-service public API, enabling tenants to provision and manage physical bare metal servers through a self-service interface. Catalog items are provider-managed entries that expose available hardware profiles, OS base images, and network configurations to tenants; each is backed by a `BareMetalInstanceTemplate`. The design adopts a pluggable provider architecture — implemented in a dedicated baremetal fulfillment component — so that future bare metal backends can be integrated without breaking the API. This EP is scoped to the fulfillment-service API layer; operator, provisioning, UX, and E2E concerns are tracked as companion work items under OSAC-1118. ## Motivation @@ -26,50 +27,50 @@ OSAC currently provides no fulfillment path for workloads requiring direct hardw ### User Stories -* As a **Tenant User**, I want to browse available `BaremetalInstanceTemplate` objects so that I can select the one that matches my workload requirements. -* As a **Tenant User**, I want to provision a bare metal server by specifying a `BaremetalInstanceTemplate` so that I can run workloads that require direct hardware access without manual coordination with the cloud provider. +* As a **Tenant User**, I want to browse available `BareMetalInstanceCatalogItem` objects so that I can select the one that matches my workload requirements. +* As a **Tenant User**, I want to provision a bare metal server by specifying a `BareMetalInstanceCatalogItem` so that I can run workloads that require direct hardware access without manual coordination with the cloud provider. * As a **Tenant User**, I want to monitor the lifecycle state of my bare metal server (provisioning, running, failed) so that I can take corrective action if provisioning fails. * As a **Tenant User**, I want to control the run strategy (power on/off) of my bare metal server so that I can manage its operational state without deprovisioning it. * As a **Tenant User**, I want to restart my bare metal server so that I can perform maintenance without deprovisioning it. * As a **Tenant User**, I want to deprovision my bare metal server when my workload is complete so that resources are released. -* As a **Cloud Provider Admin**, I want to define and publish `BaremetalInstanceTemplate` objects so that I can control which hardware profiles and OS images are available to tenants. +* As a **Cloud Provider Admin**, I want to define and publish `BareMetalInstanceCatalogItem` objects so that I can control which hardware profiles and OS images are available to tenants. +* As a **Tenant Admin**, I want to create organization-specific `BareMetalInstanceCatalogItem` objects from available templates so that I can offer tenant-specific bare metal configurations to users in my organization. * As a **Cloud Infrastructure Admin**, I want the OSAC stack to integrate with BCM (NVIDIA Base Command Manager) so that bare metal provisioning is automated through the existing control plane. ### Goals -* Provide a self-service API for tenants to create `BaremetalInstance` resources referencing a `BaremetalInstanceTemplate`. -* Define `BaremetalInstanceTemplate` as a provider-managed global catalog resource exposing the available hardware profile, OS base image, and default network configuration. -* Align the `BaremetalInstance` API shape with the existing `ComputeInstance` API (same `id`, `metadata`, `spec`, `status` envelope, same condition and state patterns, same run strategy and restart signal mechanism). +* Provide a self-service API for tenants to create `BareMetalInstance` resources referencing a `BareMetalInstanceCatalogItem`. +* Define `BareMetalInstanceTemplate` as a resource managed by Cloud Provider Admins through the private API and readable by tenants through the public API (List/Get); `BareMetalInstanceCatalogItem` as a catalog resource where Cloud Provider Admins publish global entries via the private API and Tenant Admins create tenant-scoped ones via the public API; publishing control and tenant scoping are enforced by the server. +* Maintain API consistency with `ComputeInstance` where possible — same resource shape, service structure, template and catalog item patterns, run strategy and restart signal mechanism, and authorization model. * Expose the API through both gRPC and the existing REST gateway. ### Non-Goals -* Integration with OSAC networking resources (`VirtualNetwork`, `Subnet`, `SecurityGroup`) — deferred to a future enhancement; in this initial phase, network configuration is fixed by the Cloud Provider Admin as part of the `BaremetalInstanceTemplate` and tenants have no mechanism to configure networking at provision time. A dedicated networking enhancement will enable tenants to create their own `Subnet` and attach it to a `BaremetalInstance`. -* Custom hardware profile or OS image selection by tenants at provision time — fixed by the template. Tenants requiring a different profile must request the Cloud Provider Admin to publish a new template. -* Per-organization template scoping — templates are global; a BMaaS catalog feature is deferred to a future enhancement. -* AAP playbook implementation — covered in companion work. -* Baremetal fulfillment component implementation — covered in companion work. -* UI and UX — covered in companion work. -* E2E test implementation — covered in companion work. +* Integration with OSAC networking resources (`VirtualNetwork`, `Subnet`, `SecurityGroup`) — deferred to a future enhancement; in this initial phase, network configuration is fixed by the Cloud Provider Admin as part of the `BareMetalInstanceCatalogItem` and tenants have no mechanism to configure networking at provision time. A dedicated networking enhancement will enable tenants to create their own `Subnet` and attach it to a `BareMetalInstance`. +* Custom hardware profile or OS image selection by tenants at provision time — fixed by the catalog item. Tenants requiring a different profile must request the Cloud Provider Admin to publish a new catalog item. +* AAP playbook, baremetal fulfillment component, UI/UX, and E2E test implementation — covered in companion work. * Support for multiple bare metal backends in this initial release — the architecture is designed for future extensibility. -* Quota enforcement for `BaremetalInstance` — deferred to a future enhancement. +* Quota enforcement for `BareMetalInstance` — deferred to a future enhancement. ## Proposal -The proposal introduces two new resource types to the fulfillment-service public API: +The proposal introduces three new resource types to the fulfillment-service public API: -**`BaremetalInstanceTemplate`** is a provider-defined, read-only resource that encodes a validated combination of hardware profile (instance flavor), OS base image, and default network configuration. Templates are defined by Cloud Provider Admins as Ansible roles in `osac-aap` and published to the fulfillment service via the provisioning pipeline. Tenants discover available templates via List/Get and reference one when creating a `BaremetalInstance`. This mirrors the role of `ComputeInstanceTemplate` for VMs. +**`BareMetalInstanceTemplate`** defines a bare metal hardware profile (host type, OS image, network configuration). Cloud Provider Admins create and manage templates via the private API; tenants can discover available templates via the public API (List/Get only). osac-aap is used for the actual host-level provisioning at runtime, not for template management. -**`BaremetalInstance`** is a tenant-created resource representing a provisioned bare metal server. Its `spec` references a template and carries provisioning parameters (SSH public key, user data, run strategy, restart signal). Its `status` exposes the lifecycle state and conditions aligned with the `ComputeInstance` pattern. +**`BareMetalInstanceCatalogItem`** is a catalog entry that presents an available bare metal configuration to tenants. Cloud Provider Admins publish global catalog items; Tenant Admins can additionally create tenant-scoped catalog items through the public API, referencing available templates. The `published` flag controls visibility and an optional `tenant` field enables scoping to a specific tenant; unpublished or out-of-scope catalog items are invisible to tenant List/Get calls. `FieldDefinition` entries on the catalog item govern which spec fields tenants may override and apply defaults for the rest. -Provisioning is driven by a chain of components: the fulfillment service creates a `HostLease` CR in the management cluster when a `BaremetalInstance` is created. The baremetal fulfillment operator picks up the `HostLease`, finds and assigns a free host from the inventory backend, and triggers `osac-aap` for host-level setup (OS image, SSH key, user data). The osac-operator watches `HostLease` CRs for status changes and pushes updates back to the fulfillment service via the `Signal` RPC. `HostLease` is an internal implementation detail; tenants never interact with it directly. +**`BareMetalInstance`** is a tenant-created resource representing a provisioned bare metal server. Its `spec` references a catalog item via `catalog_item` and carries provisioning parameters (SSH public key, user data, run strategy, restart signal). Its `status` exposes the lifecycle state and conditions. + +Provisioning is driven by a chain of components: the fulfillment service creates a `HostLease` CR in the management cluster when a `BareMetalInstance` is created. The baremetal fulfillment operator picks up the `HostLease`, finds and assigns a free host from the inventory backend, and triggers `osac-aap` for host-level setup (OS image, SSH key, user data). The osac-operator watches `HostLease` CRs for status changes and pushes updates back to the fulfillment service via the `Signal` RPC. `HostLease` is an internal implementation detail; tenants never interact with it directly. ### Workflow Description **Actors:** -- **Cloud Provider Admin** — defines `BaremetalInstanceTemplate` objects as Ansible roles in `osac-aap`; templates are published to the fulfillment service via the provisioning pipeline. -- **Tenant User** — creates and manages `BaremetalInstance` resources via the public API. -- **Fulfillment Service** — handles `BaremetalInstance` CRUD and creates `HostLease` CRs directly in the management cluster. +- **Cloud Provider Admin** — creates and manages `BareMetalInstanceTemplate` objects via the private API and publishes them as global `BareMetalInstanceCatalogItem` entries via the private API. +- **Tenant Admin** — creates tenant-scoped `BareMetalInstanceCatalogItem` entries via the public API, referencing available templates. +- **Tenant User** — creates and manages `BareMetalInstance` resources via the public API. +- **Fulfillment Service** — handles `BareMetalInstance` CRUD and creates `HostLease` CRs directly in the management cluster. - **osac-operator** — watches `HostLease` CRs for status changes and pushes them to the fulfillment service via the `Signal` RPC; does not create any CRs in this flow. - **baremetal-fulfillment-operator** — reconciles `HostLease` CRs: finds and assigns a free host from the inventory backend, then triggers `osac-aap` for host-level provisioning. - **osac-aap** — executes the provisioning and deprovisioning Ansible roles (OS image, SSH key, user data); triggered by the baremetal-fulfillment-operator. @@ -77,27 +78,27 @@ Provisioning is driven by a chain of components: the fulfillment service creates #### Provisioning -1. The Cloud Provider Admin defines one or more `BaremetalInstanceTemplate` objects as Ansible roles in `osac-aap`. The provisioning pipeline applies these roles and publishes the resulting templates to the fulfillment service. -2. The Tenant User lists available templates: +1. The Cloud Provider Admin creates `BareMetalInstanceTemplate` objects via the private API and publishes them as `BareMetalInstanceCatalogItem` entries via the private API, setting `published: true`. +2. The Tenant User lists available catalog items: ``` - GET /api/fulfillment/v1/baremetal_instance_templates + GET /api/fulfillment/v1/baremetal_instance_catalog_items ``` -3. The Tenant User creates a `BaremetalInstance` referencing the desired template: +3. The Tenant User creates a `BareMetalInstance` referencing the desired catalog item: ``` POST /api/fulfillment/v1/baremetal_instances ``` -4. The fulfillment service creates a `HostLease` CR in the management cluster; `BaremetalInstance.status.state` is set to `BAREMETAL_INSTANCE_STATE_PROVISIONING`. +4. The fulfillment service resolves the catalog item to a `templateID` and derives `templateParameters` from the `field_definitions`, then creates a `HostLease` CR in the management cluster with those values plus `ssh_key` and `user_data`; `BareMetalInstance.status.state` is set to `BARE_METAL_INSTANCE_STATE_PROVISIONING`. 5. The baremetal-fulfillment-operator picks up the `HostLease` and queries the inventory backend to find and assign a free host matching the requested host type and selector. 6. The baremetal-fulfillment-operator triggers `osac-aap` to run the host-level provisioning template (OS image, SSH key, user data) and updates the `HostLease` status on completion. -7. The osac-operator watches the `HostLease` CR and pushes status updates to the fulfillment service via the `Signal` RPC; the fulfillment service reflects this in `BaremetalInstance.status`. -8. The Tenant User polls until `status.state` is `BAREMETAL_INSTANCE_STATE_RUNNING`: +7. The osac-operator watches the `HostLease` CR and pushes status updates to the fulfillment service via the `Signal` RPC; the fulfillment service reflects this in `BareMetalInstance.status`. +8. The Tenant User polls until `status.state` is `BARE_METAL_INSTANCE_STATE_RUNNING`: ``` GET /api/fulfillment/v1/baremetal_instances/{id} ``` #### Failure Handling -If any step in the provisioning chain fails (playbook error, BCM API failure, `HostLease` stuck), the osac-operator sets `BaremetalInstance.status.state` to `BAREMETAL_INSTANCE_STATE_FAILED` via the `Signal` RPC. The tenant can inspect the `conditions` field for details. To retry, the tenant deletes and recreates the `BaremetalInstance`; note that recreating may result in a different physical host being assigned from the inventory. +If any step in the provisioning chain fails (playbook error, BCM API failure, `HostLease` stuck), the osac-operator sets `BareMetalInstance.status.state` to `BARE_METAL_INSTANCE_STATE_FAILED` via the `Signal` RPC. The tenant can inspect the `conditions` field for details. To retry, the tenant deletes and recreates the `BareMetalInstance`; note that recreating may result in a different physical host being assigned from the inventory. #### Deprovisioning @@ -105,10 +106,10 @@ If any step in the provisioning chain fails (playbook error, BCM API failure, `H ``` DELETE /api/fulfillment/v1/baremetal_instances/{id} ``` -2. The fulfillment service deletes the `HostLease` CR; `BaremetalInstance.status.state` transitions to `BAREMETAL_INSTANCE_STATE_DELETING`. +2. The fulfillment service deletes the `HostLease` CR; `BareMetalInstance.status.state` transitions to `BARE_METAL_INSTANCE_STATE_DELETING`. 3. The baremetal-fulfillment-operator reconciles the `HostLease` deletion: triggers `osac-aap` for the host-level deprovisioning template, then unassigns the host from the inventory backend. 4. The osac-operator watches the `HostLease` CR deletion and pushes final status via the `Signal` RPC. -5. The `BaremetalInstance` resource is removed once deprovisioning completes. +5. The `BareMetalInstance` resource is removed once deprovisioning completes. #### Provisioning Sequence @@ -122,11 +123,12 @@ sequenceDiagram participant AAP as osac-aap participant INV as Inventory Backend - TU->>FS: GET /baremetal_instance_templates - FS-->>TU: list of available templates + TU->>FS: GET /baremetal_instance_catalog_items + FS-->>TU: list of available catalog items - TU->>FS: POST /baremetal_instances {template_id, ssh_key, ...} - FS->>MC: create HostLease CR (hostType, profile, templateID, templateParameters) + TU->>FS: POST /baremetal_instances {catalog_item, ssh_key, ...} + Note over FS: resolve catalog_item → templateID, apply field_definitions → templateParameters + FS->>MC: create HostLease CR (templateID, templateParameters, ssh_key, user_data) FS-->>TU: 201 Created {id, state: PROVISIONING} MC-->>BMF: watch: HostLease CR created (no ExternalHostID yet) @@ -151,29 +153,53 @@ sequenceDiagram ### API Extensions **New gRPC services (public API):** -- `BaremetalInstances` — CRUD for tenant-managed bare metal instances. -- `BaremetalInstanceTemplates` — List/Get for provider-defined templates (read-only for tenants). +- `BareMetalInstanceTemplates` — List/Get only (read-only for tenants; managed through the private API). +- `BareMetalInstanceCatalogItems` — full CRUD; Tenant Admins manage tenant-scoped entries, Tenant Users get read-only access to published items. +- `BareMetalInstances` — full CRUD for tenant-managed bare metal instances. -**New gRPC service (private API):** -- `BaremetalInstances` (private) — adds the `Signal` RPC for `osac-operator` feedback, following the existing `ComputeInstances` private API pattern. +**New gRPC services (private API):** +- `BareMetalInstanceTemplates` (private) — full CRUD + `Signal` RPC. +- `BareMetalInstanceCatalogItems` (private) — full CRUD + `Signal` RPC. +- `BareMetalInstances` (private) — full CRUD + `Signal` RPC for `osac-operator` feedback. -**REST gateway routes:** +**REST gateway routes (public — tenant-facing):** - `GET /api/fulfillment/v1/baremetal_instance_templates` - `GET /api/fulfillment/v1/baremetal_instance_templates/{id}` +- `GET /api/fulfillment/v1/baremetal_instance_catalog_items` +- `GET /api/fulfillment/v1/baremetal_instance_catalog_items/{id}` +- `POST /api/fulfillment/v1/baremetal_instance_catalog_items` +- `PATCH /api/fulfillment/v1/baremetal_instance_catalog_items/{object.id}` +- `DELETE /api/fulfillment/v1/baremetal_instance_catalog_items/{id}` - `GET /api/fulfillment/v1/baremetal_instances` - `GET /api/fulfillment/v1/baremetal_instances/{id}` - `POST /api/fulfillment/v1/baremetal_instances` -- `PATCH /api/fulfillment/v1/baremetal_instances/{object.id}` — `{object.id}` is the gRPC-gateway convention for PATCH: the request body carries the full object and the ID is bound via `object.id`, matching the `ComputeInstance` pattern. +- `PATCH /api/fulfillment/v1/baremetal_instances/{object.id}` — `{object.id}` is the gRPC-gateway convention for PATCH: the request body carries the full object and the ID is bound via `object.id`. - `DELETE /api/fulfillment/v1/baremetal_instances/{id}` -**PATCH semantics:** The `PATCH` endpoint supports partial updates to mutable fields only: `run_strategy` and `restart_requested_at`. The `template`, `ssh_key`, and `user_data` fields are immutable after creation; requests that attempt to modify them are rejected with `400 Bad Request`. A `FieldMask` is applied automatically from the fields present in the request body. +**REST gateway routes (private — Cloud Provider Admin):** +- `GET /api/private/v1/baremetal_instance_templates` +- `GET /api/private/v1/baremetal_instance_templates/{id}` +- `POST /api/private/v1/baremetal_instance_templates` +- `PATCH /api/private/v1/baremetal_instance_templates/{object.id}` +- `DELETE /api/private/v1/baremetal_instance_templates/{id}` +- `GET /api/private/v1/baremetal_instance_catalog_items` +- `GET /api/private/v1/baremetal_instance_catalog_items/{id}` +- `POST /api/private/v1/baremetal_instance_catalog_items` +- `PATCH /api/private/v1/baremetal_instance_catalog_items/{object.id}` +- `DELETE /api/private/v1/baremetal_instance_catalog_items/{id}` + +**PATCH semantics:** The `PATCH` endpoint supports partial updates to mutable fields only: `run_strategy` and `restart_requested_at`. The `catalog_item`, `ssh_key`, and `user_data` fields are immutable after creation; requests that attempt to modify them are rejected with `400 Bad Request`. A `FieldMask` is applied automatically from the fields present in the request body. ### Implementation Details/Notes/Constraints -#### Proto: BaremetalInstanceTemplate +#### Proto: BareMetalInstanceTemplate + +Public and private types share the same structure; the private type is identical but lives in `osac/private/v1/`. + +**Public** (`osac/public/v1/baremetal_instance_template_type.proto`): ```protobuf -message BaremetalInstanceTemplate { +message BareMetalInstanceTemplate { string id = 1; Metadata metadata = 2; @@ -183,29 +209,86 @@ message BaremetalInstanceTemplate { // Human-friendly long description in Markdown format. string description = 4; - // Default spec values applied when creating a BaremetalInstance from this template. - BaremetalInstanceTemplateSpecDefaults spec_defaults = 5; + // Default spec values applied when creating a BareMetalInstance from this template. + BareMetalInstanceTemplateSpecDefaults spec_defaults = 5; } -// BaremetalInstanceTemplateSpecDefaults follows the same convention as other OSAC template types. -// No overridable spec fields are defined in this initial version; fields will be added in future +// No overridable spec fields in this initial version; fields will be added in future // enhancements as tenant-configurable options are introduced (e.g. networking integration). -message BaremetalInstanceTemplateSpecDefaults {} +message BareMetalInstanceTemplateSpecDefaults {} +``` + +#### Proto: BareMetalInstanceCatalogItem + +Two variants — public type omits the `tenant` field (server-managed); private type includes it: + +**Public** (`osac/public/v1/baremetal_instance_catalog_item_type.proto`): + +```protobuf +message BareMetalInstanceCatalogItem { + string id = 1; + Metadata metadata = 2; + + // Human-friendly short description (CLI/UI single-line display). + string title = 3; + + // Human-friendly long description in Markdown format. + string description = 4; + + // Reference to the underlying BareMetalInstanceTemplate (private resource). + string template = 5; + + // Whether this catalog item is visible to tenants. Only published items are + // returned by tenant-facing List/Get calls. + bool published = 6; + + // Field 7 is omitted: `tenant` is an internal field managed by the server, + // not exposed through the public API. When a Tenant Admin creates a catalog + // item via the public API, the server automatically scopes it to the caller's + // tenant. + + // FieldDefinition controls which BareMetalInstanceSpec fields tenants may + // set at provision time. Non-editable fields are always overridden by the + // catalog default; editable fields are validated against the provided JSON + // Schema and fall back to the default when not supplied by the tenant. + // No overridable fields are defined in this initial version; entries will + // be added in future enhancements (e.g. networking integration). + repeated FieldDefinition field_definitions = 8; +} +``` + +**Private** (`osac/private/v1/baremetal_instance_catalog_item_type.proto`): + +```protobuf +message BareMetalInstanceCatalogItem { + string id = 1; + Metadata metadata = 2; + string title = 3; + string description = 4; + string template = 5; + bool published = 6; + + // Tenant scope for this catalog item. Empty string means the item is global + // and visible to all tenants. + string tenant = 7; + + repeated FieldDefinition field_definitions = 8; +} ``` -#### Proto: BaremetalInstance +#### Proto: BareMetalInstance ```protobuf -message BaremetalInstance { +message BareMetalInstance { string id = 1; Metadata metadata = 2; - BaremetalInstanceSpec spec = 3; - BaremetalInstanceStatus status = 4; + BareMetalInstanceSpec spec = 3; + BareMetalInstanceStatus status = 4; } -message BaremetalInstanceSpec { - // Reference to the BaremetalInstanceTemplate. Required on create; immutable after creation. - string template = 1; +message BareMetalInstanceSpec { + // Reference to the BareMetalInstanceCatalogItem. Required on create; immutable after creation. + string catalog_item = 1; // SSH public key injected into the OS at provisioning time. Immutable after creation. // Must be a valid SSH public key in OpenSSH format (ssh-rsa, ssh-ed25519, etc.). @@ -217,9 +300,8 @@ message BaremetalInstanceSpec { optional string user_data = 3; // Run strategy controls the power state of the bare metal instance (on/off). - // Named run_strategy for alignment with ComputeInstance; an enum is used rather - // than a boolean to leave room for future states (e.g. suspended backends). - optional BaremetalInstanceRunStrategy run_strategy = 4; + // An enum is used rather than a boolean to leave room for future states (e.g. suspended backends). + optional BareMetalInstanceRunStrategy run_strategy = 4; // RestartRequestedAt is a timestamp signal to request a power cycle. // Set to the current time to trigger an immediate restart. @@ -227,66 +309,68 @@ message BaremetalInstanceSpec { optional google.protobuf.Timestamp restart_requested_at = 5; } -message BaremetalInstanceStatus { - BaremetalInstanceState state = 1; - repeated BaremetalInstanceCondition conditions = 2; +message BareMetalInstanceStatus { + BareMetalInstanceState state = 1; + repeated BareMetalInstanceCondition conditions = 2; // LastRestartedAt records when the last restart was initiated by the controller. optional google.protobuf.Timestamp last_restarted_at = 3; } -enum BaremetalInstanceRunStrategy { - BAREMETAL_INSTANCE_RUN_STRATEGY_UNSPECIFIED = 0; +enum BareMetalInstanceRunStrategy { + BARE_METAL_INSTANCE_RUN_STRATEGY_UNSPECIFIED = 0; // The instance is kept powered on. - BAREMETAL_INSTANCE_RUN_STRATEGY_ALWAYS = 1; + BARE_METAL_INSTANCE_RUN_STRATEGY_ALWAYS = 1; // The instance is powered off. - BAREMETAL_INSTANCE_RUN_STRATEGY_HALTED = 2; + BARE_METAL_INSTANCE_RUN_STRATEGY_HALTED = 2; } -enum BaremetalInstanceState { - BAREMETAL_INSTANCE_STATE_UNSPECIFIED = 0; - BAREMETAL_INSTANCE_STATE_PROVISIONING = 1; - BAREMETAL_INSTANCE_STATE_RUNNING = 2; - BAREMETAL_INSTANCE_STATE_FAILED = 3; - BAREMETAL_INSTANCE_STATE_DELETING = 4; +enum BareMetalInstanceState { + BARE_METAL_INSTANCE_STATE_UNSPECIFIED = 0; + BARE_METAL_INSTANCE_STATE_PROVISIONING = 1; + BARE_METAL_INSTANCE_STATE_RUNNING = 2; + BARE_METAL_INSTANCE_STATE_FAILED = 3; + BARE_METAL_INSTANCE_STATE_DELETING = 4; } -enum BaremetalInstanceConditionType { - BAREMETAL_INSTANCE_CONDITION_TYPE_UNSPECIFIED = 0; +enum BareMetalInstanceConditionType { + BARE_METAL_INSTANCE_CONDITION_TYPE_UNSPECIFIED = 0; // Infrastructure has been allocated to this instance by the baremetal fulfillment component. - BAREMETAL_INSTANCE_CONDITION_TYPE_PROVISIONED = 1; + BARE_METAL_INSTANCE_CONDITION_TYPE_PROVISIONED = 1; // OS image and user configuration have been applied by the provisioning provider. - BAREMETAL_INSTANCE_CONDITION_TYPE_CONFIGURATION_APPLIED = 2; + BARE_METAL_INSTANCE_CONDITION_TYPE_CONFIGURATION_APPLIED = 2; // The machine is available and assigned to the tenant (HostLease active). // Does not indicate OS readiness — the tenant is responsible for OS-level checks. - BAREMETAL_INSTANCE_CONDITION_TYPE_READY = 3; + BARE_METAL_INSTANCE_CONDITION_TYPE_READY = 3; // A power cycle (restart) is currently in progress. - BAREMETAL_INSTANCE_CONDITION_TYPE_RESTART_IN_PROGRESS = 4; + BARE_METAL_INSTANCE_CONDITION_TYPE_RESTART_IN_PROGRESS = 4; // A restart request has failed. - BAREMETAL_INSTANCE_CONDITION_TYPE_RESTART_FAILED = 5; + BARE_METAL_INSTANCE_CONDITION_TYPE_RESTART_FAILED = 5; // Configuration changes require a restart to take effect. - BAREMETAL_INSTANCE_CONDITION_TYPE_RESTART_REQUIRED = 6; + BARE_METAL_INSTANCE_CONDITION_TYPE_RESTART_REQUIRED = 6; } ``` #### Alignment with ComputeInstance -`BaremetalInstance` intentionally mirrors `ComputeInstance`: -- Same `id` + `Metadata` + `spec` + `status` envelope. -- Same condition shape (`ConditionStatus`, `last_transition_time`, `reason`, `message`). -- Same CRUD service shape and REST route pattern (`/api/fulfillment/v1/`). -- Same private API `Signal` RPC for `osac-operator` feedback loop. -- Same run strategy (`run_strategy`) and restart signal mechanism (`restart_requested_at`, `last_restarted_at`). +Where possible, the BareMetalInstance API is consistent with ComputeInstance: +- `id` + `Metadata` + `spec` + `status` envelope. +- Condition shape: `ConditionStatus`, `last_transition_time`, `reason`, `message`. +- CRUD service shape and REST route pattern (`/api/fulfillment/v1/`). +- Private `Signal` RPC for `osac-operator` feedback loop. +- `run_strategy` and restart signal mechanism (`restart_requested_at`, `last_restarted_at`). +- `spec.catalog_item` referencing a `BareMetalInstanceCatalogItem` with `FieldDefinition`-based field control. +- `BareMetalInstanceTemplate` with `spec_defaults`; managed via private API, readable via public List/Get. -Fields specific to VMs (network attachments) are absent from `BaremetalInstance` in this initial version and may be added in future enhancements. +Fields specific to VMs (network attachments, image, cores, memory) are absent from `BareMetalInstance` in this initial version. ### Risks and Mitigations @@ -300,7 +384,7 @@ Fields specific to VMs (network attachments) are absent from `BaremetalInstance` **Mitigation:** Follows the same OPA-based authorization model as `ComputeInstance`. Tenant scoping is enforced at the fulfillment-service layer via existing middleware. **Risk:** Tenants provision unlimited bare metal instances, exhausting backend capacity. -**Mitigation:** Quota enforcement for `BaremetalInstance` is deferred to a future enhancement. Until quotas are implemented, capacity limits must be managed out-of-band by the Cloud Provider Admin. +**Mitigation:** Quota enforcement for `BareMetalInstance` is deferred to a future enhancement. Until quotas are implemented, capacity limits must be managed out-of-band by the Cloud Provider Admin. **Risk:** Compromised backend credentials expose the bare metal backend to unauthorized access. **Mitigation:** Credential storage and management for the baremetal backend are out of scope for this EP and will be defined in a dedicated security enhancement. The baremetal fulfillment component must not embed credentials in CRDs or API responses. @@ -311,23 +395,23 @@ None identified. ## Alternatives (Not Implemented) -**Map `BaremetalInstance` to the existing `ComputeInstance` resource with a baremetal flag.** This avoids a new resource type but conflates VM and bare metal semantics, complicates template definitions, and requires dispatching on a field value rather than resource type. A dedicated resource type provides cleaner separation of concerns and allows independent API evolution. +**Map `BareMetalInstance` to the existing `ComputeInstance` resource with a baremetal flag.** This avoids a new resource type but conflates VM and bare metal semantics, complicates catalog item definitions, and requires dispatching on a field value rather than resource type. A dedicated resource type provides cleaner separation of concerns and allows independent API evolution. **Expose the bare metal backend API directly to tenants.** This bypasses OSAC entirely, eliminating tenant isolation, quota enforcement, and the pluggable backend architecture. Not viable for a multi-tenant cloud platform. ## Open Questions -1. ~~Should `BaremetalInstance` and `HostLease` be the same object, or remain distinct?~~ **Closed:** Distinct. The fulfillment service creates a `HostLease` as the internal backend CRD, but `HostLease` is also used by other workflows (cluster-as-a-service, agent provisioning) that bypass the fulfillment service entirely. Coupling the public API schema to `HostLease` would make future consolidation a breaking change. +1. ~~Should `BareMetalInstance` and `HostLease` be the same object, or remain distinct?~~ **Closed:** Distinct. The fulfillment service creates a `HostLease` as the internal backend CRD, but `HostLease` is also used by other workflows (cluster-as-a-service, agent provisioning) that bypass the fulfillment service entirely. Coupling the public API schema to `HostLease` would make future consolidation a breaking change. ## Test Plan Test plan will be finalized during the implementation phase. Expected coverage: - **Unit tests:** Proto field validation, state machine transitions, provider interface mocking. -- **Integration tests:** `BaremetalInstance` CRUD via gRPC, template List/Get, Signal RPC feedback loop, OPA authorization enforcement, PATCH immutability enforcement. +- **Integration tests:** `BareMetalInstance` CRUD via gRPC, catalog item CRUD (public and private), `published` visibility enforcement, tenant-scoping enforcement (Tenant Admin creates scoped items; Cloud Provider Admin creates global items via private API), `FieldDefinition` application, Signal RPC feedback loop, OPA authorization enforcement, PATCH immutability enforcement. - **E2E tests:** Full provisioning and deprovisioning workflow against BCM; CI pipeline configured to run E2E tests on merge. -Tricky areas: asynchronous provisioning lifecycle (tests must handle delays or mock the provider), template immutability enforcement after create, tenant isolation boundary checks, and failure-path recovery (FAILED state → delete → recreate). +Tricky areas: asynchronous provisioning lifecycle (tests must handle delays or mock the provider), `catalog_item` immutability enforcement after create, `published`/`tenant` visibility boundary checks, tenant isolation boundary checks, and failure-path recovery (FAILED state → delete → recreate). ## Graduation Criteria @@ -339,29 +423,31 @@ Graduation criteria will be defined when targeting a release. Expected progressi ## Upgrade / Downgrade Strategy -This is a new API with no impact on existing resources. Downgrading requires deleting all `BaremetalInstance` and `BaremetalInstanceTemplate` resources and uninstalling the new CRDs from the management cluster before reverting the fulfillment-service image. +This is a new API with no impact on existing resources. Downgrading requires deleting all `BareMetalInstance` and `BareMetalInstanceCatalogItem` resources and uninstalling the new CRDs from the management cluster before reverting the fulfillment-service image. ## Version Skew Strategy -The fulfillment-service, baremetal-fulfillment-operator, and osac-operator must be upgraded together. If the baremetal-fulfillment-operator is not running, `HostLease` CRs will not be reconciled and new `BaremetalInstance` create requests will remain in `PROVISIONING` indefinitely. If the osac-operator baremetal controller is not running, status updates will not propagate back to the fulfillment service. +The fulfillment-service, baremetal-fulfillment-operator, and osac-operator must be upgraded together. If the baremetal-fulfillment-operator is not running, `HostLease` CRs will not be reconciled and new `BareMetalInstance` create requests will remain in `PROVISIONING` indefinitely. If the osac-operator baremetal controller is not running, status updates will not propagate back to the fulfillment service. ## Support Procedures -**Symptom:** `BaremetalInstance` stuck in `BAREMETAL_INSTANCE_STATE_PROVISIONING`. +> **Log hygiene:** Before sharing or inspecting logs, operators must verify that `ssh_key`, `user_data`, and backend credentials (inventory API tokens, AAP credentials) are redacted. These values must never appear in fulfillment-service or operator logs; if they do, treat it as a security incident and rotate the affected credentials immediately. + +**Symptom:** `BareMetalInstance` stuck in `BARE_METAL_INSTANCE_STATE_PROVISIONING`. **Diagnosis:** Check baremetal-fulfillment-operator logs for inventory or AAP errors. Check osac-aap job logs for playbook failures. Verify the `HostLease` CR exists and that `ExternalHostID` has been set (indicates host was found in inventory). -**Resolution:** If the host was allocated but provisioning failed, delete the `HostLease` CR manually and delete the `BaremetalInstance` to trigger a clean retry. +**Resolution:** If the host was allocated but provisioning failed, delete the `HostLease` CR manually and delete the `BareMetalInstance` to trigger a clean retry. -**Symptom:** All `BaremetalInstance` creations remain in `BAREMETAL_INSTANCE_STATE_PROVISIONING` with no `HostLease` created. +**Symptom:** All `BareMetalInstance` creations remain in `BARE_METAL_INSTANCE_STATE_PROVISIONING` with no `HostLease` created. **Diagnosis:** The fulfillment service may not be able to reach the management cluster API. Check fulfillment-service logs for Kubernetes API errors. **Resolution:** Verify management cluster connectivity and credentials used by the fulfillment service. -**Symptom:** All `BaremetalInstance` creations transition to `BAREMETAL_INSTANCE_STATE_FAILED`. +**Symptom:** All `BareMetalInstance` creations transition to `BARE_METAL_INSTANCE_STATE_FAILED`. **Diagnosis:** Check baremetal-fulfillment-operator logs for inventory API or AAP errors. Validate inventory backend credentials, verify AAP availability, and confirm network connectivity. -**Resolution:** Rotate credentials if expired, resolve network connectivity issues, then retry by deleting the failed `BaremetalInstance` and recreating it. +**Resolution:** Rotate credentials if expired, resolve network connectivity issues, then retry by deleting the failed `BareMetalInstance` and recreating it. -**Symptom:** Create returns `404 Not Found` for the template ID. -**Diagnosis:** The `BaremetalInstanceTemplate` does not exist or is not visible to the tenant's organization. -**Resolution:** Cloud Provider Admin must create and publish the appropriate template via `osac-aap`. +**Symptom:** Create returns `404 Not Found` for the catalog item ID. +**Diagnosis:** The `BareMetalInstanceCatalogItem` does not exist, is not published, or is scoped to a different tenant. +**Resolution:** Cloud Provider Admin must create the catalog item via the private API, set `published: true`, and verify the `tenant` field matches the requesting tenant (or is empty for global visibility). **Disabling the API:** Scale the baremetal-fulfillment-operator and the osac-operator baremetal controller to 0 replicas. Existing `HostLease` CRs persist but are not reconciled. Re-enabling both resumes reconciliation without data loss. From 5d7c75291112dd27ac16cdae306725a917e770e4 Mon Sep 17 00:00:00 2001 From: Zoltan Szabo Date: Fri, 5 Jun 2026 11:36:55 +0200 Subject: [PATCH 008/450] OSAC-23: Add PRD and design doc for Tenant Storage Onboarding rework Introduces the split PRD + design doc format for the TenantStorage CRD and OSAC Storage Controller extraction from the Tenant controller. Includes the AAP playbook split (OSAC-1145) as part of the delivery scope. Assisted-by: Claude Code Signed-off-by: Zoltan Szabo --- .../design.md | 258 ++++++++++++++++++ .../tenant-storage-onboarding-OSAC-23/prd.md | 99 +++++++ 2 files changed, 357 insertions(+) create mode 100644 enhancements/tenant-storage-onboarding-OSAC-23/design.md create mode 100644 enhancements/tenant-storage-onboarding-OSAC-23/prd.md diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/design.md b/enhancements/tenant-storage-onboarding-OSAC-23/design.md new file mode 100644 index 000000000..e8524601e --- /dev/null +++ b/enhancements/tenant-storage-onboarding-OSAC-23/design.md @@ -0,0 +1,258 @@ +# Rework Tenant Storage Onboarding — Design + +| Field | Value | +|-------------|------------------------------------------------------------| +| Author(s) | Zoltan Szabo, Akshay Nadkarni | +| Jira | [OSAC-23](https://redhat.atlassian.net/browse/OSAC-23) | +| PRD | [prd.md](prd.md) | +| Date | 2026-06-05 | + +# 1. Overview + +This design extracts storage provisioning logic from the Tenant controller into a dedicated OSAC Storage Controller with a new TenantStorage CRD. The Tenant controller creates a TenantStorage CR during onboarding; the OSAC Storage Controller handles Phase 1 backend setup, StorageClass resolution, and teardown independently. The AAP playbooks are split into four lifecycle actions to match. See the [PRD](prd.md) for requirements and motivation. + +# 2. Goals and Non-Goals + +## 2.1 Goals + +- Reuse the existing `provisioning.RunProvisioningLifecycle()` framework and `ProvisioningProvider` interface — no changes to the shared provisioning package. [Codebase: pkg/provisioning/provision_lifecycle.go] +- Follow the established controller pattern (Reconcile → handleUpdate/handleDelete, oldStatus compare, finalizer management). [Codebase: internal/controller/tenant_controller.go] +- Keep the TenantStorage CRD spec minimal (`tenantRef` only) to avoid coupling to tier schema that is still evolving. [Locked: D1] +- Name the controller broadly ("OSAC Storage Controller") so it can absorb future storage responsibilities (backend registration, resource creation storage) without renaming. [Locked: D2] + +## 2.2 Non-Goals + +- Modifying the shared provisioning framework (`pkg/provisioning/`) +- Adding a feedback controller for TenantStorage (no fulfillment-service integration needed for this CRD) +- Changes to the storage provider role logic (`setup`, `ensure_storage_class`, `teardown` tasks) — the same tasks run, only the playbook entry points, dispatcher, and job template names change + +# 3. Motivation / Background + +Storage provisioning currently runs inside the Tenant controller: it checks for hub Secrets, triggers AAP jobs, polls for completion, resolves StorageClasses by label, and tracks storage jobs — all interleaved with namespace creation, UDN reconciliation, and tenant status management. This coupling makes it difficult for the storage working group to iterate on storage onboarding without risking regressions in tenant lifecycle. It also blocks CaaS storage (OSAC-1123), which needs a two-phase model where Phase 1 runs at tenant onboarding and Phase 2 runs after the HyperShift cluster is ready — logic that doesn't belong in the Tenant controller. + +The extraction is mechanical: the same provisioning logic moves from one controller to another, the same AAP provider interface is used, and the same status patterns apply. The primary design decisions are about where to draw the boundary and how the two controllers communicate. + +# 4. Design + +## 4.1 Architecture + +The diagram below shows the component relationships after the extraction. Arrows indicate the direction of control flow or data dependency. + +```mermaid +flowchart TD + TC[Tenant Controller] -->|creates with owner ref| TS[TenantStorage CR] + SC[OSAC Storage Controller] -->|watches| TS + SC -->|provision: triggers| P1[osac-create-tenant-storage] + SC -->|deprovision: triggers| P4[osac-delete-tenant-storage] + SC -->|watches| SEC[Hub Secret] + SC -->|reads| STOCLASS[StorageClasses] + CI[ComputeInstance Controller] -->|reads storageClasses from| TS + P1 -->|creates| SEC + CO[ClusterOrder Controller] -.->|future: triggers| P2[osac-ensure-tenant-storage] + CO -.->|future: triggers| P3[osac-cleanup-tenant-storage] + P2 -->|creates| STOCLASS +``` + +The Tenant controller creates a TenantStorage CR with an owner reference during `handleUpdate`, immediately after ensuring the tenant namespace exists. The OSAC Storage Controller watches TenantStorage CRs and drives the provisioning lifecycle — triggering `osac-create-tenant-storage` (Phase 1) on provision and `osac-delete-tenant-storage` on deletion. The ComputeInstance controller looks up the TenantStorage CR by tenant name to get the resolved StorageClasses for AAP context injection. The owner reference ensures TenantStorage is garbage-collected when the Tenant is deleted. Dashed lines show future CaaS integration (OSAC-1123) where the ClusterOrder controller triggers Phase 2 (`osac-ensure-tenant-storage`) and cleanup. + +### Controller responsibilities after extraction + +| Controller | Responsibilities | +|-----------|-----------------| +| Tenant | Namespace creation, UDN reconciliation, TenantStorage CR creation | +| OSAC Storage | Hub Secret check, Phase 1 AAP provisioning, StorageClass resolution, phase/condition management, teardown | +| ComputeInstance | Read StorageClasses from TenantStorage status (instead of Tenant status) | + +### Reconciliation flow — OSAC Storage Controller + +`handleUpdate`: +1. Add finalizer (`osac.openshift.io/tenantstorage-finalizer`) +2. Look up hub Secret for tenant in `osac-system` namespace +3. If absent → call `provisioning.RunProvisioningLifecycle()` with `osac-create-tenant-storage` template +4. If present → resolve StorageClasses via `getTenantStorageClasses()`, set phase to Ready, set `StorageBackendReady` condition to True + +`handleDelete`: +1. Trigger `osac-delete-tenant-storage` AAP job via `TriggerDeprovisionJob()` +2. Poll until terminal +3. Remove finalizer + +This follows the same structure as the existing Tenant controller's storage handling — the logic moves, the pattern stays identical. [Codebase: internal/controller/tenant_controller.go] + +### Watches + +| Watch | Purpose | +|-------|---------| +| TenantStorage CRs (primary) | Standard controller watch | +| Secrets in tenant namespace | Re-reconcile when hub Secret appears/disappears (filtered by `secretTenantPredicate`) | +| StorageClasses | Re-reconcile when SCs with tenant label are created/deleted (filtered by `storageClassTenantPredicate`) | + +## 4.2 AAP Playbook Split + +The previous single playbook (`playbook_osac_configure_tenant_storage.yml`) executed both Phase 1 (backend setup) and Phase 2 (cluster-side CSI + StorageClasses) in a single run. This design splits it into four playbooks aligned with the storage lifecycle actions. [Locked: D4] + +### Playbook mapping + +| Before | After | Lifecycle Action | +|--------|-------|-----------------| +| `playbook_osac_configure_tenant_storage.yml` (Phase 1 + 2) | `playbook_osac_create_tenant_storage.yml` (Phase 1 only) | Provision backend: create VAST org, VIP pool, credentials, hub Secret | +| (embedded in configure) | `playbook_osac_ensure_tenant_storage.yml` (Phase 2 only) | Cluster-side setup: install CSI driver, create StorageClasses with retry | +| (none) | `playbook_osac_cleanup_tenant_storage.yml` | Cluster-side teardown: remove StorageClasses, VolumeSnapshotClasses, CSI Secret (handles unreachable clusters) | +| `playbook_osac_delete_tenant_storage.yml` | `playbook_osac_delete_tenant_storage.yml` | Backend teardown: delete VAST org, views, quotas, remove hub Secret | + +### Role changes + +The `storage_provider` role dispatcher (`storage_provider/tasks/main.yaml`) gains a `cleanup` action alongside the existing `setup`, `ensure_storage_class`, and `teardown` actions. The `vast_storage` role implements `cleanup.yaml` which removes Kubernetes resources by label (`osac.openshift.io/tenant`), handling unreachable target clusters gracefully (logs warning, continues). + +No changes to the core role logic for `setup`, `ensure_storage_class`, or `teardown` — the same tasks run, just triggered from separate playbook entry points. + +### AAP config-as-code + +Four job templates replace the previous two in `config_as_code/roles/aap/vars/controller.yml`: + +| Job Template | Playbook | Triggered By | +|-------------|----------|-------------| +| `osac-create-tenant-storage` | `playbook_osac_create_tenant_storage.yml` | OSAC Storage Controller (provision) | +| `osac-ensure-tenant-storage` | `playbook_osac_ensure_tenant_storage.yml` | ClusterOrder controller (future, OSAC-1123) | +| `osac-cleanup-tenant-storage` | `playbook_osac_cleanup_tenant_storage.yml` | OSAC Storage Controller (cluster cleanup before teardown) | +| `osac-delete-tenant-storage` | `playbook_osac_delete_tenant_storage.yml` | OSAC Storage Controller (deprovision) | + +The operator references these template names via environment variables: `OSAC_TENANT_STORAGE_AAP_PROVISION_TEMPLATE` (default: `osac-create-tenant-storage`) and `OSAC_TENANT_STORAGE_AAP_DEPROVISION_TEMPLATE` (default: `osac-delete-tenant-storage`). + +### Why four actions instead of two + +The previous two-playbook model (configure + delete) conflated backend provisioning with cluster-side setup. This creates three problems: + +1. **CaaS timing:** Phase 2 cannot run until the HyperShift cluster is ready, but Phase 1 should run at tenant onboarding — splitting them is required for CaaS support. +2. **Cluster teardown vs backend teardown:** When a tenant's cluster is destroyed but the tenant persists, only cluster-side resources should be cleaned up (StorageClasses, CSI Secrets). The backend org and credentials must survive. A single "delete" action cannot distinguish these cases. +3. **Idempotent re-setup:** `osac-ensure-tenant-storage` is designed to be re-run safely (e.g., after cluster recreation), which requires it to be independent of backend provisioning. + +## 4.3 Data Model / Schema Changes + +### New CRD: TenantStorage + +```go +type TenantStorageSpec struct { + // +kubebuilder:validation:Required + // +kubebuilder:validation:MinLength=1 + TenantRef string `json:"tenantRef"` +} + +type TenantStorageStatus struct { + Phase TenantStoragePhaseType `json:"phase,omitempty"` + StorageClasses []ResolvedStorageClass `json:"storageClasses,omitempty"` + Conditions []metav1.Condition `json:"conditions,omitempty"` + Jobs []JobStatus `json:"jobs,omitempty"` +} +``` + +**Phase values:** `Progressing`, `Ready`, `Failed`, `Deleting` — same enum pattern as all other OSAC CRDs. + +**Condition types:** `StorageBackendReady` — indicates hub Secret exists for all configured tiers. + +**Reused types:** `ResolvedStorageClass` (from `tenant_types.go`), `JobStatus` (from `job_types.go`), `metav1.Condition`. No new shared types are introduced. + +**Print columns:** `Tenant` (`.spec.tenantRef`), `Phase` (`.status.phase`). [Locked: D6] + +### Removed from Tenant CRD + +| Field/Type | Location | +|-----------|----------| +| `StorageClasses []ResolvedStorageClass` | `TenantStatus` | +| `Jobs []JobStatus` | `TenantStatus` | +| `TenantConditionStorageClassReady` | condition type constant | +| `TenantReasonMultipleFound` | reason constant | +| Storage print columns | kubebuilder markers | +| `GetStatusJobs()` | method on TenantStatus | + +The `ResolvedStorageClass` type definition stays in `tenant_types.go` since it is a shared type. + +### Provisioning lifecycle type switch + +`GetJobsFromResource()` in `pkg/provisioning/provision_lifecycle.go` adds a `*v1alpha1.TenantStorage` case. The `*v1alpha1.Tenant` case is removed since Tenant no longer has a `Jobs` field. + +## 4.3 API Changes + +No API changes required. TenantStorage is an operator-internal CRD with no fulfillment-service API surface. No proto changes, no gRPC endpoints, no REST gateway routes. + +## 4.4 Scalability and Performance + +Minimal impact. The number of TenantStorage CRs equals the number of Tenants (1:1). The same AAP jobs run — just triggered by a different controller. The additional watch on TenantStorage CRs adds negligible memory overhead (one informer, same namespace filter as Tenant). + +StorageClass resolution (`getTenantStorageClasses`) runs on every reconcile but is a lightweight in-memory label filter over cached StorageClasses — no API server calls. + +## 4.5 Security Considerations + +The feature inherits the existing security model. Per-tenant hub Secrets remain in `osac-system` namespace with RBAC restricted to the operator service account. Admin credentials (VAST endpoint/password) remain ephemeral via AAP pod env vars — this behavior is unchanged. [PRD: NFR-1, NFR-2] + +The OSAC Storage Controller requires the same RBAC as the Tenant controller previously held for storage operations: get/list/watch on Secrets and StorageClasses, plus full CRUD on TenantStorage resources. RBAC markers are on the controller file; `make manifests` generates the ClusterRole. + +## 4.6 Failure Handling and Recovery + +All failure modes are inherited from the existing implementation: + +| Failure | Behavior | Recovery | +|---------|----------|----------| +| Hub Secret missing | Phase stays Progressing, AAP job triggered | Automatic retry via `RunProvisioningLifecycle` backoff | +| AAP job fails | Phase set to Failed, job status recorded | Re-reconcile triggers backoff with retry | +| AAP job times out | Poll continues at `StatusPollInterval` | Job eventually reaches terminal state | +| Hub Secret deleted after Ready | Secret watch triggers re-reconcile, `StorageBackendReady` condition set to False | Next reconcile re-triggers provisioning | +| Teardown job fails | Finalizer not removed, `BlockDeletionOnFailure` honored | Re-reconcile retries teardown | + +Idempotency is guaranteed by the provisioning lifecycle framework — `EvaluateAction()` checks config version hashes to avoid duplicate triggers. [Codebase: pkg/provisioning/provision_lifecycle.go] + +## 4.7 RBAC / Tenancy + +No RBAC or tenancy model changes. TenantStorage CRs live in the same tenant namespace as Tenant CRs, scoped by `OSAC_TENANT_NAMESPACE`. The owner reference from TenantStorage to Tenant ensures tenant isolation — a TenantStorage CR can only be created by the Tenant controller for its own tenant. + +The `osac.openshift.io/tenant` annotation is set on the TenantStorage CR, matching the parent Tenant name. StorageClass and Secret predicates filter by this annotation. + +## 4.8 Extensibility / Future-Proofing + +The controller is named "OSAC Storage Controller" (not "TenantStorage Controller") to accommodate future responsibilities without renaming. [Locked: D2] + +The TenantStorage spec is intentionally minimal (`tenantRef` only). When tier selection moves from env var to a StorageTier API, tier references can be added to the spec without breaking existing CRs — the field would be optional with a default behavior of "use all available tiers." + +CaaS Phase 2 (OSAC-1123) will trigger `osac-ensure-tenant-storage` from the ClusterOrder controller by reading TenantStorage status — the two-phase model is established by this design. The ClusterOrder controller checks `TenantStoragePhaseReady` before triggering Phase 2, ensuring Phase 1 is complete. + +# 5. Alternatives Considered + +## 5.1 Keep storage logic in Tenant controller, add feature flags + +Keep all storage provisioning in the Tenant controller and use a feature flag to enable/disable Phase 2. This avoids a new CRD and controller but perpetuates the coupling. Rejected because CaaS storage (OSAC-1123) needs the ClusterOrder controller to trigger Phase 2 independently, which requires a separate resource to read readiness from. Feature flags inside the Tenant controller would grow increasingly complex as delivery models diverge. + +## 5.2 Fulfillment-service creates TenantStorage CR + +Have the fulfillment-service create TenantStorage CRs via its Kubernetes reconciler instead of the Tenant controller. This would decouple TenantStorage creation from the operator entirely. Rejected because it adds a cross-component dependency for a simple creation step, the fulfillment-service would need CRD awareness and RBAC for TenantStorage, and the Tenant controller already has the tenant context needed. [Locked: D3] + +## 5.3 Embed storage state in a ConfigMap instead of a CRD + +Use a ConfigMap per tenant to store storage state instead of a custom resource. This avoids CRD registration but loses structured status, conditions, phase tracking, print columns, and `kubectl` integration. Rejected because TenantStorage benefits from the same status patterns as all other OSAC resources, and a CRD is the standard Kubernetes mechanism for operator-managed state. + +# 6. Observability and Monitoring + +No new observability changes. The OSAC Storage Controller logs at the same level as all other controllers (structured logging via `ctrllog`). TenantStorage status conditions and phase are observable via `kubectl get tenantstorages` with print columns showing Tenant and Phase. [Locked: D6] + +AAP job status is recorded in `status.jobs` and visible via `kubectl describe tenantstorage `, consistent with how all other OSAC controllers expose job history. + +# 7. Impact and Compatibility + +### Breaking changes + +- **Tenant CRD:** `StorageClasses`, `Jobs` fields, and `StorageClassReady` condition removed from status. External consumers reading these fields (none known) would break. The Tenant controller's `Ready` condition now means "namespace exists" rather than "namespace exists AND storage is ready." + +- **AAP job templates:** `osac-configure-tenant-storage` is replaced by four templates: `osac-create-tenant-storage`, `osac-ensure-tenant-storage`, `osac-cleanup-tenant-storage`, `osac-delete-tenant-storage`. AAP config-as-code must be updated alongside. [PRD: NFR-3] + +### Migration + +No data migration needed. On upgrade: +1. New TenantStorage CRD is registered (`make install` or Helm chart) +2. Operator deployment updated with both controller changes and new controller +3. AAP config-as-code updated with new job templates +4. Tenant controller creates TenantStorage CRs for existing tenants on next reconcile +5. OSAC Storage Controller picks up the new TenantStorage CRs and evaluates state + +Existing tenants with completed storage provisioning: the OSAC Storage Controller detects the hub Secret exists, resolves StorageClasses, and sets phase to Ready without re-triggering AAP jobs (no job history → `EvaluateAction` triggers, but `osac-create-tenant-storage` is idempotent and returns quickly if the backend org already exists). + +### Deployment order + +AAP config-as-code and operator must be deployed together. Deploying AAP changes first breaks the existing Tenant controller. Deploying operator changes first is safe — the controller will fail to find the new job templates and retry with backoff until AAP is updated. diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md new file mode 100644 index 000000000..8dfbbd800 --- /dev/null +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -0,0 +1,99 @@ +# Rework Tenant Storage Onboarding + +| Field | Value | +|-------------|---------| +| Author(s) | Zoltan Szabo, Akshay Nadkarni | +| Jira | https://redhat.atlassian.net/browse/OSAC-23 | +| Date | 2026-06-05 | + +## 1. Problem Statement + +Storage provisioning logic — Phase 1 backend setup, StorageClass resolution, AAP job tracking, and storage readiness — is embedded in the Tenant controller and the Tenant CR's status fields. This tight coupling means any change to storage onboarding risks breaking tenant lifecycle, and supporting different storage workflows per delivery model (VMaaS, CaaS, BMaaS) requires branching logic inside a single controller. Storage issues cannot be diagnosed independently from tenant status, and the storage working group cannot iterate on storage onboarding without coordinating every change with the core tenant lifecycle. + +## 2. Goals and Non-Goals + +### 2.1 Goals + +- Storage lifecycle operates independently from tenant lifecycle — storage provisioning, readiness tracking, and teardown are managed by a dedicated controller without affecting tenant state transitions. +- Per-tenant storage state (provisioned tiers, provider resources, readiness) is observable as a standalone Kubernetes resource via `kubectl get tenantstorages`. [Clarify: D6] +- The OSAC Storage Controller is the single entry point for all storage reconciliation, establishing the foundation for future storage capabilities (backend registration, CaaS cluster-side setup, resource creation storage). [Clarify: D1, D2] + +### 2.3 Non-Goals + +- StorageBackend CRD implementation (OSAC-917) [Clarify: D5] +- StorageTier CRD implementation (OSAC-1110) [Clarify: D5] +- CaaS Phase 2 trigger from ClusterOrder controller (OSAC-1123) [Clarify: D5] +- Backend registration automation [Clarify: D5] +- Prometheus metrics or fulfillment-service API visibility for storage state [Clarify: D6] +- Fulfillment-service proto or API changes +- Changes to the storage provider roles or VAST integration logic (only the playbook entry points change) + +## 3. Requirements + +### 3.1 Functional Requirements + +- **FR-1:** A TenantStorage CRD captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains `phase` (Progressing, Ready, Failed, Deleting), `storageClasses` (resolved StorageClass names with tier labels), `conditions` (structured status conditions), and `jobs` (AAP job tracking). [Clarify: D1] + +- **FR-2:** The OSAC Storage Controller watches TenantStorage CRs and provisions storage via AAP. On creation, the controller checks for the tenant's hub Secret — if absent, it triggers the `osac-create-tenant-storage` AAP job (Phase 1 backend setup). When the hub Secret exists, the controller resolves StorageClasses by tenant label and sets the phase to Ready. + +- **FR-3:** The Tenant controller creates a TenantStorage CR during tenant onboarding with an owner reference, so that TenantStorage is automatically deleted when the Tenant is deleted. [Clarify: D3] + +- **FR-4:** Storage-related fields are removed from the Tenant CRD: `StorageClasses` status field, `Jobs` status field, `StorageClassReady` condition type, and storage-related print columns. + +- **FR-5:** The ComputeInstance controller reads StorageClasses from the TenantStorage CR's status instead of the Tenant CR's status. + +- **FR-6:** Hub Secret readiness is gated on TenantStorage (via the `StorageBackendReady` condition), not on the Tenant CR. The Tenant CR reaches Ready when its namespace exists — storage readiness is the TenantStorage controller's concern. + +- **FR-7:** On deletion, the OSAC Storage Controller triggers the `osac-delete-tenant-storage` AAP job to clean up backend resources (VAST tenant, views, quotas) and the per-tenant hub Secret. + +- **FR-8:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage` (Phase 1 backend setup), `osac-ensure-tenant-storage` (Phase 2 cluster-side CSI + StorageClasses), `osac-cleanup-tenant-storage` (cluster-side resource removal), and `osac-delete-tenant-storage` (backend teardown). A corresponding `cleanup` action is added to the `storage_provider` role dispatcher. [Clarify: D4] + +- **FR-9:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to `Unmanaged`, consistent with all other OSAC controllers. + +### 3.2 Non-Functional Requirements + +- **NFR-1:** Admin credentials (VAST endpoint, username, password) are ephemeral — mounted as environment variables in the AAP automation pod, cleared after use, and never persisted to Kubernetes. + +- **NFR-2:** Per-tenant credentials are stored in a Secret in the `osac-system` namespace, managed exclusively by the TenantStorage controller. These Secrets are the handoff point to downstream workflows (VMaaS cluster-side setup, CaaS cluster-side setup). + +- **NFR-3:** The AAP playbook changes and operator changes must be deployed together. The `osac-create-tenant-storage` playbook replaces the previous combined playbook that executed both Phase 1 and Phase 2 — deploying the AAP changes without the operator changes breaks existing Tenant controller provisioning. + +## 4. Acceptance Criteria + +- [ ] TenantStorage CRD is defined with spec (`tenantRef`) and status (`phase`, `storageClasses`, `conditions`, `jobs`) +- [ ] OSAC Storage Controller reconciles TenantStorage CRs: triggers AAP provisioning when hub Secret is absent, sets phase to Ready when hub Secret exists and StorageClasses are resolved +- [ ] Tenant controller creates a TenantStorage CR with owner reference during onboarding +- [ ] Storage fields removed from Tenant CRD (`StorageClasses`, `Jobs`, `StorageClassReady` condition, storage print columns) +- [ ] ComputeInstance controller reads StorageClasses from TenantStorage status +- [ ] OSAC Storage Controller triggers AAP teardown on TenantStorage deletion and removes the finalizer after completion +- [ ] AAP playbooks split into four job templates: `osac-create-tenant-storage`, `osac-ensure-tenant-storage`, `osac-cleanup-tenant-storage`, `osac-delete-tenant-storage` +- [ ] `kubectl get tenantstorages` displays Tenant and Phase columns +- [ ] Unit tests pass for the new controller and updated Tenant/ComputeInstance controllers + +## 5. Assumptions + +- AAP is the only provisioning backend for v0.1 — no direct API provisioning path exists. +- VAST is the only storage provider for v0.1. +- Tier configuration via the `STORAGE_TIERS` environment variable is sufficient for v0.1; the StorageTier API is a separate effort. +- The TenantStorage CRD name can be changed in a future release when the broader storage controller scope materializes, without breaking external consumers (no external API contracts depend on this CRD name today). [Clarify: D2] + +## 6. Dependencies + +- **osac-aap** (OSAC-1145): AAP playbook split is delivered as part of this work. Coordinated merge required — AAP changes break backward compatibility if deployed without the operator changes. [Clarify: D4] +- **osac-operator provisioning framework**: The controller reuses the existing `provisioning.RunProvisioningLifecycle()` helper and `ProvisioningProvider` interface. No changes to the framework are required. + +## 7. Risks + +### 7.1 Breaking change in AAP playbook interface + +The previous combined playbook (`osac-configure-tenant-storage`) is replaced by four separate playbooks. The new `osac-create-tenant-storage` playbook executes Phase 1 only, whereas the previous playbook executed both Phase 1 and Phase 2. Deploying the AAP changes without the operator changes causes the existing Tenant controller to provision storage incompletely. + +- **Owner:** Zoltan Szabo +- **Mitigation:** Coordinated PR merge across osac-operator and osac-aap repos. PRs are linked in their descriptions. Deployment documentation specifies both components must be updated together. + +### 7.2 ComputeInstance controller migration + +The ComputeInstance controller's storage integration was wired up in May 2026 and is not yet widely used in production. Migrating it to read from TenantStorage instead of Tenant is low-risk but must be validated on the dev cluster. + +- **Owner:** Zoltan Szabo +- **Mitigation:** Unit tests cover the new code path. E2E validation on hypershift1 dev cluster before merge. From 6daddd05e76dc32da6e8928806353f5da104bf27 Mon Sep 17 00:00:00 2001 From: Zoltan Szabo Date: Mon, 8 Jun 2026 08:41:55 +0200 Subject: [PATCH 009/450] =?UTF-8?q?Remove=20design=20doc=20=E2=80=94=20PRD?= =?UTF-8?q?=20must=20be=20reviewed=20first?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PRD should be created, reviewed, and merged before the design doc. Design will come in a follow-up PR. Assisted-by: Claude Code Signed-off-by: Zoltan Szabo --- .../design.md | 258 ------------------ 1 file changed, 258 deletions(-) delete mode 100644 enhancements/tenant-storage-onboarding-OSAC-23/design.md diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/design.md b/enhancements/tenant-storage-onboarding-OSAC-23/design.md deleted file mode 100644 index e8524601e..000000000 --- a/enhancements/tenant-storage-onboarding-OSAC-23/design.md +++ /dev/null @@ -1,258 +0,0 @@ -# Rework Tenant Storage Onboarding — Design - -| Field | Value | -|-------------|------------------------------------------------------------| -| Author(s) | Zoltan Szabo, Akshay Nadkarni | -| Jira | [OSAC-23](https://redhat.atlassian.net/browse/OSAC-23) | -| PRD | [prd.md](prd.md) | -| Date | 2026-06-05 | - -# 1. Overview - -This design extracts storage provisioning logic from the Tenant controller into a dedicated OSAC Storage Controller with a new TenantStorage CRD. The Tenant controller creates a TenantStorage CR during onboarding; the OSAC Storage Controller handles Phase 1 backend setup, StorageClass resolution, and teardown independently. The AAP playbooks are split into four lifecycle actions to match. See the [PRD](prd.md) for requirements and motivation. - -# 2. Goals and Non-Goals - -## 2.1 Goals - -- Reuse the existing `provisioning.RunProvisioningLifecycle()` framework and `ProvisioningProvider` interface — no changes to the shared provisioning package. [Codebase: pkg/provisioning/provision_lifecycle.go] -- Follow the established controller pattern (Reconcile → handleUpdate/handleDelete, oldStatus compare, finalizer management). [Codebase: internal/controller/tenant_controller.go] -- Keep the TenantStorage CRD spec minimal (`tenantRef` only) to avoid coupling to tier schema that is still evolving. [Locked: D1] -- Name the controller broadly ("OSAC Storage Controller") so it can absorb future storage responsibilities (backend registration, resource creation storage) without renaming. [Locked: D2] - -## 2.2 Non-Goals - -- Modifying the shared provisioning framework (`pkg/provisioning/`) -- Adding a feedback controller for TenantStorage (no fulfillment-service integration needed for this CRD) -- Changes to the storage provider role logic (`setup`, `ensure_storage_class`, `teardown` tasks) — the same tasks run, only the playbook entry points, dispatcher, and job template names change - -# 3. Motivation / Background - -Storage provisioning currently runs inside the Tenant controller: it checks for hub Secrets, triggers AAP jobs, polls for completion, resolves StorageClasses by label, and tracks storage jobs — all interleaved with namespace creation, UDN reconciliation, and tenant status management. This coupling makes it difficult for the storage working group to iterate on storage onboarding without risking regressions in tenant lifecycle. It also blocks CaaS storage (OSAC-1123), which needs a two-phase model where Phase 1 runs at tenant onboarding and Phase 2 runs after the HyperShift cluster is ready — logic that doesn't belong in the Tenant controller. - -The extraction is mechanical: the same provisioning logic moves from one controller to another, the same AAP provider interface is used, and the same status patterns apply. The primary design decisions are about where to draw the boundary and how the two controllers communicate. - -# 4. Design - -## 4.1 Architecture - -The diagram below shows the component relationships after the extraction. Arrows indicate the direction of control flow or data dependency. - -```mermaid -flowchart TD - TC[Tenant Controller] -->|creates with owner ref| TS[TenantStorage CR] - SC[OSAC Storage Controller] -->|watches| TS - SC -->|provision: triggers| P1[osac-create-tenant-storage] - SC -->|deprovision: triggers| P4[osac-delete-tenant-storage] - SC -->|watches| SEC[Hub Secret] - SC -->|reads| STOCLASS[StorageClasses] - CI[ComputeInstance Controller] -->|reads storageClasses from| TS - P1 -->|creates| SEC - CO[ClusterOrder Controller] -.->|future: triggers| P2[osac-ensure-tenant-storage] - CO -.->|future: triggers| P3[osac-cleanup-tenant-storage] - P2 -->|creates| STOCLASS -``` - -The Tenant controller creates a TenantStorage CR with an owner reference during `handleUpdate`, immediately after ensuring the tenant namespace exists. The OSAC Storage Controller watches TenantStorage CRs and drives the provisioning lifecycle — triggering `osac-create-tenant-storage` (Phase 1) on provision and `osac-delete-tenant-storage` on deletion. The ComputeInstance controller looks up the TenantStorage CR by tenant name to get the resolved StorageClasses for AAP context injection. The owner reference ensures TenantStorage is garbage-collected when the Tenant is deleted. Dashed lines show future CaaS integration (OSAC-1123) where the ClusterOrder controller triggers Phase 2 (`osac-ensure-tenant-storage`) and cleanup. - -### Controller responsibilities after extraction - -| Controller | Responsibilities | -|-----------|-----------------| -| Tenant | Namespace creation, UDN reconciliation, TenantStorage CR creation | -| OSAC Storage | Hub Secret check, Phase 1 AAP provisioning, StorageClass resolution, phase/condition management, teardown | -| ComputeInstance | Read StorageClasses from TenantStorage status (instead of Tenant status) | - -### Reconciliation flow — OSAC Storage Controller - -`handleUpdate`: -1. Add finalizer (`osac.openshift.io/tenantstorage-finalizer`) -2. Look up hub Secret for tenant in `osac-system` namespace -3. If absent → call `provisioning.RunProvisioningLifecycle()` with `osac-create-tenant-storage` template -4. If present → resolve StorageClasses via `getTenantStorageClasses()`, set phase to Ready, set `StorageBackendReady` condition to True - -`handleDelete`: -1. Trigger `osac-delete-tenant-storage` AAP job via `TriggerDeprovisionJob()` -2. Poll until terminal -3. Remove finalizer - -This follows the same structure as the existing Tenant controller's storage handling — the logic moves, the pattern stays identical. [Codebase: internal/controller/tenant_controller.go] - -### Watches - -| Watch | Purpose | -|-------|---------| -| TenantStorage CRs (primary) | Standard controller watch | -| Secrets in tenant namespace | Re-reconcile when hub Secret appears/disappears (filtered by `secretTenantPredicate`) | -| StorageClasses | Re-reconcile when SCs with tenant label are created/deleted (filtered by `storageClassTenantPredicate`) | - -## 4.2 AAP Playbook Split - -The previous single playbook (`playbook_osac_configure_tenant_storage.yml`) executed both Phase 1 (backend setup) and Phase 2 (cluster-side CSI + StorageClasses) in a single run. This design splits it into four playbooks aligned with the storage lifecycle actions. [Locked: D4] - -### Playbook mapping - -| Before | After | Lifecycle Action | -|--------|-------|-----------------| -| `playbook_osac_configure_tenant_storage.yml` (Phase 1 + 2) | `playbook_osac_create_tenant_storage.yml` (Phase 1 only) | Provision backend: create VAST org, VIP pool, credentials, hub Secret | -| (embedded in configure) | `playbook_osac_ensure_tenant_storage.yml` (Phase 2 only) | Cluster-side setup: install CSI driver, create StorageClasses with retry | -| (none) | `playbook_osac_cleanup_tenant_storage.yml` | Cluster-side teardown: remove StorageClasses, VolumeSnapshotClasses, CSI Secret (handles unreachable clusters) | -| `playbook_osac_delete_tenant_storage.yml` | `playbook_osac_delete_tenant_storage.yml` | Backend teardown: delete VAST org, views, quotas, remove hub Secret | - -### Role changes - -The `storage_provider` role dispatcher (`storage_provider/tasks/main.yaml`) gains a `cleanup` action alongside the existing `setup`, `ensure_storage_class`, and `teardown` actions. The `vast_storage` role implements `cleanup.yaml` which removes Kubernetes resources by label (`osac.openshift.io/tenant`), handling unreachable target clusters gracefully (logs warning, continues). - -No changes to the core role logic for `setup`, `ensure_storage_class`, or `teardown` — the same tasks run, just triggered from separate playbook entry points. - -### AAP config-as-code - -Four job templates replace the previous two in `config_as_code/roles/aap/vars/controller.yml`: - -| Job Template | Playbook | Triggered By | -|-------------|----------|-------------| -| `osac-create-tenant-storage` | `playbook_osac_create_tenant_storage.yml` | OSAC Storage Controller (provision) | -| `osac-ensure-tenant-storage` | `playbook_osac_ensure_tenant_storage.yml` | ClusterOrder controller (future, OSAC-1123) | -| `osac-cleanup-tenant-storage` | `playbook_osac_cleanup_tenant_storage.yml` | OSAC Storage Controller (cluster cleanup before teardown) | -| `osac-delete-tenant-storage` | `playbook_osac_delete_tenant_storage.yml` | OSAC Storage Controller (deprovision) | - -The operator references these template names via environment variables: `OSAC_TENANT_STORAGE_AAP_PROVISION_TEMPLATE` (default: `osac-create-tenant-storage`) and `OSAC_TENANT_STORAGE_AAP_DEPROVISION_TEMPLATE` (default: `osac-delete-tenant-storage`). - -### Why four actions instead of two - -The previous two-playbook model (configure + delete) conflated backend provisioning with cluster-side setup. This creates three problems: - -1. **CaaS timing:** Phase 2 cannot run until the HyperShift cluster is ready, but Phase 1 should run at tenant onboarding — splitting them is required for CaaS support. -2. **Cluster teardown vs backend teardown:** When a tenant's cluster is destroyed but the tenant persists, only cluster-side resources should be cleaned up (StorageClasses, CSI Secrets). The backend org and credentials must survive. A single "delete" action cannot distinguish these cases. -3. **Idempotent re-setup:** `osac-ensure-tenant-storage` is designed to be re-run safely (e.g., after cluster recreation), which requires it to be independent of backend provisioning. - -## 4.3 Data Model / Schema Changes - -### New CRD: TenantStorage - -```go -type TenantStorageSpec struct { - // +kubebuilder:validation:Required - // +kubebuilder:validation:MinLength=1 - TenantRef string `json:"tenantRef"` -} - -type TenantStorageStatus struct { - Phase TenantStoragePhaseType `json:"phase,omitempty"` - StorageClasses []ResolvedStorageClass `json:"storageClasses,omitempty"` - Conditions []metav1.Condition `json:"conditions,omitempty"` - Jobs []JobStatus `json:"jobs,omitempty"` -} -``` - -**Phase values:** `Progressing`, `Ready`, `Failed`, `Deleting` — same enum pattern as all other OSAC CRDs. - -**Condition types:** `StorageBackendReady` — indicates hub Secret exists for all configured tiers. - -**Reused types:** `ResolvedStorageClass` (from `tenant_types.go`), `JobStatus` (from `job_types.go`), `metav1.Condition`. No new shared types are introduced. - -**Print columns:** `Tenant` (`.spec.tenantRef`), `Phase` (`.status.phase`). [Locked: D6] - -### Removed from Tenant CRD - -| Field/Type | Location | -|-----------|----------| -| `StorageClasses []ResolvedStorageClass` | `TenantStatus` | -| `Jobs []JobStatus` | `TenantStatus` | -| `TenantConditionStorageClassReady` | condition type constant | -| `TenantReasonMultipleFound` | reason constant | -| Storage print columns | kubebuilder markers | -| `GetStatusJobs()` | method on TenantStatus | - -The `ResolvedStorageClass` type definition stays in `tenant_types.go` since it is a shared type. - -### Provisioning lifecycle type switch - -`GetJobsFromResource()` in `pkg/provisioning/provision_lifecycle.go` adds a `*v1alpha1.TenantStorage` case. The `*v1alpha1.Tenant` case is removed since Tenant no longer has a `Jobs` field. - -## 4.3 API Changes - -No API changes required. TenantStorage is an operator-internal CRD with no fulfillment-service API surface. No proto changes, no gRPC endpoints, no REST gateway routes. - -## 4.4 Scalability and Performance - -Minimal impact. The number of TenantStorage CRs equals the number of Tenants (1:1). The same AAP jobs run — just triggered by a different controller. The additional watch on TenantStorage CRs adds negligible memory overhead (one informer, same namespace filter as Tenant). - -StorageClass resolution (`getTenantStorageClasses`) runs on every reconcile but is a lightweight in-memory label filter over cached StorageClasses — no API server calls. - -## 4.5 Security Considerations - -The feature inherits the existing security model. Per-tenant hub Secrets remain in `osac-system` namespace with RBAC restricted to the operator service account. Admin credentials (VAST endpoint/password) remain ephemeral via AAP pod env vars — this behavior is unchanged. [PRD: NFR-1, NFR-2] - -The OSAC Storage Controller requires the same RBAC as the Tenant controller previously held for storage operations: get/list/watch on Secrets and StorageClasses, plus full CRUD on TenantStorage resources. RBAC markers are on the controller file; `make manifests` generates the ClusterRole. - -## 4.6 Failure Handling and Recovery - -All failure modes are inherited from the existing implementation: - -| Failure | Behavior | Recovery | -|---------|----------|----------| -| Hub Secret missing | Phase stays Progressing, AAP job triggered | Automatic retry via `RunProvisioningLifecycle` backoff | -| AAP job fails | Phase set to Failed, job status recorded | Re-reconcile triggers backoff with retry | -| AAP job times out | Poll continues at `StatusPollInterval` | Job eventually reaches terminal state | -| Hub Secret deleted after Ready | Secret watch triggers re-reconcile, `StorageBackendReady` condition set to False | Next reconcile re-triggers provisioning | -| Teardown job fails | Finalizer not removed, `BlockDeletionOnFailure` honored | Re-reconcile retries teardown | - -Idempotency is guaranteed by the provisioning lifecycle framework — `EvaluateAction()` checks config version hashes to avoid duplicate triggers. [Codebase: pkg/provisioning/provision_lifecycle.go] - -## 4.7 RBAC / Tenancy - -No RBAC or tenancy model changes. TenantStorage CRs live in the same tenant namespace as Tenant CRs, scoped by `OSAC_TENANT_NAMESPACE`. The owner reference from TenantStorage to Tenant ensures tenant isolation — a TenantStorage CR can only be created by the Tenant controller for its own tenant. - -The `osac.openshift.io/tenant` annotation is set on the TenantStorage CR, matching the parent Tenant name. StorageClass and Secret predicates filter by this annotation. - -## 4.8 Extensibility / Future-Proofing - -The controller is named "OSAC Storage Controller" (not "TenantStorage Controller") to accommodate future responsibilities without renaming. [Locked: D2] - -The TenantStorage spec is intentionally minimal (`tenantRef` only). When tier selection moves from env var to a StorageTier API, tier references can be added to the spec without breaking existing CRs — the field would be optional with a default behavior of "use all available tiers." - -CaaS Phase 2 (OSAC-1123) will trigger `osac-ensure-tenant-storage` from the ClusterOrder controller by reading TenantStorage status — the two-phase model is established by this design. The ClusterOrder controller checks `TenantStoragePhaseReady` before triggering Phase 2, ensuring Phase 1 is complete. - -# 5. Alternatives Considered - -## 5.1 Keep storage logic in Tenant controller, add feature flags - -Keep all storage provisioning in the Tenant controller and use a feature flag to enable/disable Phase 2. This avoids a new CRD and controller but perpetuates the coupling. Rejected because CaaS storage (OSAC-1123) needs the ClusterOrder controller to trigger Phase 2 independently, which requires a separate resource to read readiness from. Feature flags inside the Tenant controller would grow increasingly complex as delivery models diverge. - -## 5.2 Fulfillment-service creates TenantStorage CR - -Have the fulfillment-service create TenantStorage CRs via its Kubernetes reconciler instead of the Tenant controller. This would decouple TenantStorage creation from the operator entirely. Rejected because it adds a cross-component dependency for a simple creation step, the fulfillment-service would need CRD awareness and RBAC for TenantStorage, and the Tenant controller already has the tenant context needed. [Locked: D3] - -## 5.3 Embed storage state in a ConfigMap instead of a CRD - -Use a ConfigMap per tenant to store storage state instead of a custom resource. This avoids CRD registration but loses structured status, conditions, phase tracking, print columns, and `kubectl` integration. Rejected because TenantStorage benefits from the same status patterns as all other OSAC resources, and a CRD is the standard Kubernetes mechanism for operator-managed state. - -# 6. Observability and Monitoring - -No new observability changes. The OSAC Storage Controller logs at the same level as all other controllers (structured logging via `ctrllog`). TenantStorage status conditions and phase are observable via `kubectl get tenantstorages` with print columns showing Tenant and Phase. [Locked: D6] - -AAP job status is recorded in `status.jobs` and visible via `kubectl describe tenantstorage `, consistent with how all other OSAC controllers expose job history. - -# 7. Impact and Compatibility - -### Breaking changes - -- **Tenant CRD:** `StorageClasses`, `Jobs` fields, and `StorageClassReady` condition removed from status. External consumers reading these fields (none known) would break. The Tenant controller's `Ready` condition now means "namespace exists" rather than "namespace exists AND storage is ready." - -- **AAP job templates:** `osac-configure-tenant-storage` is replaced by four templates: `osac-create-tenant-storage`, `osac-ensure-tenant-storage`, `osac-cleanup-tenant-storage`, `osac-delete-tenant-storage`. AAP config-as-code must be updated alongside. [PRD: NFR-3] - -### Migration - -No data migration needed. On upgrade: -1. New TenantStorage CRD is registered (`make install` or Helm chart) -2. Operator deployment updated with both controller changes and new controller -3. AAP config-as-code updated with new job templates -4. Tenant controller creates TenantStorage CRs for existing tenants on next reconcile -5. OSAC Storage Controller picks up the new TenantStorage CRs and evaluates state - -Existing tenants with completed storage provisioning: the OSAC Storage Controller detects the hub Secret exists, resolves StorageClasses, and sets phase to Ready without re-triggering AAP jobs (no job history → `EvaluateAction` triggers, but `osac-create-tenant-storage` is idempotent and returns quickly if the backend org already exists). - -### Deployment order - -AAP config-as-code and operator must be deployed together. Deploying AAP changes first breaks the existing Tenant controller. Deploying operator changes first is safe — the controller will fail to find the new job templates and retry with backoff until AAP is updated. From e67159d56d40f8a038449232cb6e4003cd34f935 Mon Sep 17 00:00:00 2001 From: Zoltan Szabo Date: Mon, 8 Jun 2026 09:39:50 +0200 Subject: [PATCH 010/450] OSAC-23: Rewrite PRD incorporating review feedback - Storage Controller watches Tenant CRs and creates TenantStorage (not Tenant controller) - Controller-driven deletion (no owner reference GC) - ClusterOrder watch included in v0.1 scope - Tenant CRD cleanup (remove storage fields) in scope - Phase 1/Phase 2 terminology defined explicitly - Remove internal workflow markers ([Clarify: D1] etc.) Assisted-by: Claude Code Signed-off-by: Zoltan Szabo --- .../tenant-storage-onboarding-OSAC-23/prd.md | 76 +++++++++---------- 1 file changed, 36 insertions(+), 40 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index 8dfbbd800..d86c66f66 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -4,96 +4,92 @@ |-------------|---------| | Author(s) | Zoltan Szabo, Akshay Nadkarni | | Jira | https://redhat.atlassian.net/browse/OSAC-23 | -| Date | 2026-06-05 | +| Date | 2026-06-08 | ## 1. Problem Statement -Storage provisioning logic — Phase 1 backend setup, StorageClass resolution, AAP job tracking, and storage readiness — is embedded in the Tenant controller and the Tenant CR's status fields. This tight coupling means any change to storage onboarding risks breaking tenant lifecycle, and supporting different storage workflows per delivery model (VMaaS, CaaS, BMaaS) requires branching logic inside a single controller. Storage issues cannot be diagnosed independently from tenant status, and the storage working group cannot iterate on storage onboarding without coordinating every change with the core tenant lifecycle. +Storage provisioning logic — backend setup, StorageClass resolution, AAP job tracking, and storage readiness — is embedded in the Tenant controller and the Tenant CR's status fields. Any change to storage onboarding risks breaking tenant lifecycle, and supporting different storage workflows per delivery model (VMaaS, CaaS, BMaaS) requires branching logic inside a single controller. Storage issues cannot be diagnosed independently from tenant status, and the storage working group cannot iterate on storage onboarding without coordinating every change with the core tenant lifecycle. ## 2. Goals and Non-Goals ### 2.1 Goals - Storage lifecycle operates independently from tenant lifecycle — storage provisioning, readiness tracking, and teardown are managed by a dedicated controller without affecting tenant state transitions. -- Per-tenant storage state (provisioned tiers, provider resources, readiness) is observable as a standalone Kubernetes resource via `kubectl get tenantstorages`. [Clarify: D6] -- The OSAC Storage Controller is the single entry point for all storage reconciliation, establishing the foundation for future storage capabilities (backend registration, CaaS cluster-side setup, resource creation storage). [Clarify: D1, D2] +- Per-tenant storage state (provisioned tiers, provider resources, readiness) is observable as a standalone Kubernetes resource via `kubectl get tenantstorage`. +- The OSAC Storage Controller is the single entry point for all storage reconciliation, establishing the foundation for future storage capabilities (backend registration, CaaS cluster-side setup, resource creation storage). ### 2.3 Non-Goals -- StorageBackend CRD implementation (OSAC-917) [Clarify: D5] -- StorageTier CRD implementation (OSAC-1110) [Clarify: D5] -- CaaS Phase 2 trigger from ClusterOrder controller (OSAC-1123) [Clarify: D5] -- Backend registration automation [Clarify: D5] -- Prometheus metrics or fulfillment-service API visibility for storage state [Clarify: D6] +- StorageBackend CRD implementation (OSAC-917) +- StorageTier CRD implementation (OSAC-1110) +- CaaS Phase 2 trigger logic from ClusterOrder (OSAC-1123) — the controller watches ClusterOrder CRs, but the trigger logic for cluster-side storage setup is a separate effort +- Backend registration automation +- Prometheus metrics or fulfillment-service API visibility for storage state - Fulfillment-service proto or API changes -- Changes to the storage provider roles or VAST integration logic (only the playbook entry points change) +- Changes to the storage provider role logic (setup, ensure_storage_class, teardown tasks) — only the playbook entry points and job template names change ## 3. Requirements +OSAC storage provisioning uses a two-phase model. Phase 1 (backend setup) runs at tenant onboarding: it creates the tenant's organization on the storage appliance, provisions VIP pools, credentials, per-tier views, and stores per-tenant credentials in a hub Secret. Phase 2 (cluster-side setup) runs when a target cluster is ready: it installs CSI drivers and creates StorageClasses on the cluster. This PRD covers the controller and CRD that manage Phase 1. Phase 2 triggering is out of scope (OSAC-1123). + ### 3.1 Functional Requirements -- **FR-1:** A TenantStorage CRD captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains `phase` (Progressing, Ready, Failed, Deleting), `storageClasses` (resolved StorageClass names with tier labels), `conditions` (structured status conditions), and `jobs` (AAP job tracking). [Clarify: D1] +- **FR-1:** A TenantStorage CRD captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains phase (Progressing, Ready, Failed, Deleting), resolved StorageClass names with tier labels, structured status conditions, and AAP job tracking. -- **FR-2:** The OSAC Storage Controller watches TenantStorage CRs and provisions storage via AAP. On creation, the controller checks for the tenant's hub Secret — if absent, it triggers the `osac-create-tenant-storage` AAP job (Phase 1 backend setup). When the hub Secret exists, the controller resolves StorageClasses by tenant label and sets the phase to Ready. +- **FR-2:** The OSAC Storage Controller watches Tenant CRs. When a Tenant reaches Ready (namespace exists), the controller creates a TenantStorage CR for that tenant and begins storage onboarding — checking for the tenant's hub Secret, triggering backend provisioning via AAP if absent, and resolving StorageClasses when the Secret exists. [Clarify: R1.Q1] -- **FR-3:** The Tenant controller creates a TenantStorage CR during tenant onboarding with an owner reference, so that TenantStorage is automatically deleted when the Tenant is deleted. [Clarify: D3] +- **FR-3:** The OSAC Storage Controller watches ClusterOrder CRs to prepare for future CaaS Phase 2 triggering. For v0.1, no trigger logic is implemented — only the watch and informer are set up. [Clarify: R1.Q2] -- **FR-4:** Storage-related fields are removed from the Tenant CRD: `StorageClasses` status field, `Jobs` status field, `StorageClassReady` condition type, and storage-related print columns. +- **FR-4:** Storage-related fields are removed from the Tenant CRD: StorageClasses status field, Jobs status field, StorageClassReady condition type, and storage-related print columns. [Clarify: R2.Q3] - **FR-5:** The ComputeInstance controller reads StorageClasses from the TenantStorage CR's status instead of the Tenant CR's status. -- **FR-6:** Hub Secret readiness is gated on TenantStorage (via the `StorageBackendReady` condition), not on the Tenant CR. The Tenant CR reaches Ready when its namespace exists — storage readiness is the TenantStorage controller's concern. +- **FR-6:** The Tenant controller has zero storage responsibilities. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. Storage readiness is entirely the OSAC Storage Controller's concern. -- **FR-7:** On deletion, the OSAC Storage Controller triggers the `osac-delete-tenant-storage` AAP job to clean up backend resources (VAST tenant, views, quotas) and the per-tenant hub Secret. +- **FR-7:** On Tenant deletion, the OSAC Storage Controller triggers backend teardown via AAP to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret, then deletes the TenantStorage CR. No owner reference is used — the controller explicitly watches Tenant deletion events. [Clarify: R1.Q3] -- **FR-8:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage` (Phase 1 backend setup), `osac-ensure-tenant-storage` (Phase 2 cluster-side CSI + StorageClasses), `osac-cleanup-tenant-storage` (cluster-side resource removal), and `osac-delete-tenant-storage` (backend teardown). A corresponding `cleanup` action is added to the `storage_provider` role dispatcher. [Clarify: D4] +- **FR-8:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage` (Phase 1 backend setup), `osac-ensure-tenant-storage` (Phase 2 cluster-side CSI + StorageClasses), `osac-cleanup-tenant-storage` (cluster-side resource removal), and `osac-delete-tenant-storage` (backend teardown). A corresponding cleanup action is added to the storage_provider role dispatcher. [Clarify: R1.Q4] -- **FR-9:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to `Unmanaged`, consistent with all other OSAC controllers. +- **FR-9:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to Unmanaged, consistent with all other OSAC controllers. ### 3.2 Non-Functional Requirements - **NFR-1:** Admin credentials (VAST endpoint, username, password) are ephemeral — mounted as environment variables in the AAP automation pod, cleared after use, and never persisted to Kubernetes. -- **NFR-2:** Per-tenant credentials are stored in a Secret in the `osac-system` namespace, managed exclusively by the TenantStorage controller. These Secrets are the handoff point to downstream workflows (VMaaS cluster-side setup, CaaS cluster-side setup). +- **NFR-2:** Per-tenant credentials are stored in a Secret in the osac-system namespace, managed exclusively by the OSAC Storage Controller. These Secrets are the handoff point to downstream workflows (VMaaS cluster-side setup, CaaS cluster-side setup). - **NFR-3:** The AAP playbook changes and operator changes must be deployed together. The `osac-create-tenant-storage` playbook replaces the previous combined playbook that executed both Phase 1 and Phase 2 — deploying the AAP changes without the operator changes breaks existing Tenant controller provisioning. ## 4. Acceptance Criteria -- [ ] TenantStorage CRD is defined with spec (`tenantRef`) and status (`phase`, `storageClasses`, `conditions`, `jobs`) -- [ ] OSAC Storage Controller reconciles TenantStorage CRs: triggers AAP provisioning when hub Secret is absent, sets phase to Ready when hub Secret exists and StorageClasses are resolved -- [ ] Tenant controller creates a TenantStorage CR with owner reference during onboarding -- [ ] Storage fields removed from Tenant CRD (`StorageClasses`, `Jobs`, `StorageClassReady` condition, storage print columns) +- [ ] TenantStorage CRD is defined with spec (tenantRef) and status (phase, storageClasses, conditions, jobs) +- [ ] OSAC Storage Controller watches Tenant CRs and creates TenantStorage when Tenant reaches Ready +- [ ] OSAC Storage Controller watches ClusterOrder CRs (informer set up, no trigger logic for v0.1) +- [ ] OSAC Storage Controller triggers AAP provisioning when hub Secret is absent, sets phase to Ready when Secret exists and StorageClasses are resolved +- [ ] Storage fields removed from Tenant CRD (StorageClasses, Jobs, StorageClassReady condition, storage print columns) +- [ ] Tenant controller has no storage-related logic or RBAC - [ ] ComputeInstance controller reads StorageClasses from TenantStorage status -- [ ] OSAC Storage Controller triggers AAP teardown on TenantStorage deletion and removes the finalizer after completion -- [ ] AAP playbooks split into four job templates: `osac-create-tenant-storage`, `osac-ensure-tenant-storage`, `osac-cleanup-tenant-storage`, `osac-delete-tenant-storage` -- [ ] `kubectl get tenantstorages` displays Tenant and Phase columns +- [ ] On Tenant deletion, OSAC Storage Controller triggers AAP teardown and deletes TenantStorage CR +- [ ] AAP playbooks split into four job templates: osac-create-tenant-storage, osac-ensure-tenant-storage, osac-cleanup-tenant-storage, osac-delete-tenant-storage +- [ ] `kubectl get tenantstorage` displays Tenant and Phase columns - [ ] Unit tests pass for the new controller and updated Tenant/ComputeInstance controllers ## 5. Assumptions - AAP is the only provisioning backend for v0.1 — no direct API provisioning path exists. - VAST is the only storage provider for v0.1. -- Tier configuration via the `STORAGE_TIERS` environment variable is sufficient for v0.1; the StorageTier API is a separate effort. -- The TenantStorage CRD name can be changed in a future release when the broader storage controller scope materializes, without breaking external consumers (no external API contracts depend on this CRD name today). [Clarify: D2] +- Tier configuration via the STORAGE_TIERS environment variable is sufficient for v0.1; the StorageTier API is a separate effort. ## 6. Dependencies -- **osac-aap** (OSAC-1145): AAP playbook split is delivered as part of this work. Coordinated merge required — AAP changes break backward compatibility if deployed without the operator changes. [Clarify: D4] -- **osac-operator provisioning framework**: The controller reuses the existing `provisioning.RunProvisioningLifecycle()` helper and `ProvisioningProvider` interface. No changes to the framework are required. +- **osac-aap** (OSAC-1145): AAP playbook split is delivered as part of this work. Coordinated merge required — AAP changes break backward compatibility if deployed without the operator changes. +- **osac-operator provisioning framework**: The controller reuses the existing `provisioning.RunProvisioningLifecycle()` helper and ProvisioningProvider interface. No changes to the framework are required. ## 7. Risks ### 7.1 Breaking change in AAP playbook interface -The previous combined playbook (`osac-configure-tenant-storage`) is replaced by four separate playbooks. The new `osac-create-tenant-storage` playbook executes Phase 1 only, whereas the previous playbook executed both Phase 1 and Phase 2. Deploying the AAP changes without the operator changes causes the existing Tenant controller to provision storage incompletely. - -- **Owner:** Zoltan Szabo -- **Mitigation:** Coordinated PR merge across osac-operator and osac-aap repos. PRs are linked in their descriptions. Deployment documentation specifies both components must be updated together. - -### 7.2 ComputeInstance controller migration - -The ComputeInstance controller's storage integration was wired up in May 2026 and is not yet widely used in production. Migrating it to read from TenantStorage instead of Tenant is low-risk but must be validated on the dev cluster. +The previous combined playbook (osac-configure-tenant-storage) is replaced by four separate playbooks. The new osac-create-tenant-storage playbook executes Phase 1 only, whereas the previous playbook executed both Phase 1 and Phase 2. Deploying the AAP changes without the operator changes causes the existing Tenant controller to provision storage incompletely. -- **Owner:** Zoltan Szabo -- **Mitigation:** Unit tests cover the new code path. E2E validation on hypershift1 dev cluster before merge. +- **Owner:** Storage WG +- **Mitigation:** Coordinated PR merge across osac-operator and osac-aap repos. PRs are linked in descriptions. Deployment documentation specifies both components must be updated together. From 3f2dd531c947e0a63f741bc7d429dc2883f973fe Mon Sep 17 00:00:00 2001 From: Zoltan Szabo Date: Tue, 9 Jun 2026 08:06:10 +0200 Subject: [PATCH 011/450] OSAC-23: Add StorageReady condition on Tenant CR Storage Controller sets StorageReady condition on the Tenant CR (and future ClusterOrder CR) so storage readiness is visible through the primary resources users interact with. The Tenant controller still runs no storage logic. Assisted-by: Claude Code Signed-off-by: Zoltan Szabo --- enhancements/tenant-storage-onboarding-OSAC-23/prd.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index d86c66f66..bf59e2b48 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -4,7 +4,7 @@ |-------------|---------| | Author(s) | Zoltan Szabo, Akshay Nadkarni | | Jira | https://redhat.atlassian.net/browse/OSAC-23 | -| Date | 2026-06-08 | +| Date | 2026-06-09 | ## 1. Problem Statement @@ -36,15 +36,15 @@ OSAC storage provisioning uses a two-phase model. Phase 1 (backend setup) runs a - **FR-1:** A TenantStorage CRD captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains phase (Progressing, Ready, Failed, Deleting), resolved StorageClass names with tier labels, structured status conditions, and AAP job tracking. -- **FR-2:** The OSAC Storage Controller watches Tenant CRs. When a Tenant reaches Ready (namespace exists), the controller creates a TenantStorage CR for that tenant and begins storage onboarding — checking for the tenant's hub Secret, triggering backend provisioning via AAP if absent, and resolving StorageClasses when the Secret exists. [Clarify: R1.Q1] +- **FR-2:** The OSAC Storage Controller watches Tenant CRs. When a Tenant reaches Ready (namespace exists), the controller creates a TenantStorage CR for that tenant and begins storage onboarding — checking for the tenant's hub Secret, triggering backend provisioning via AAP if absent, and resolving StorageClasses when the Secret exists. When storage onboarding completes, the controller sets a `StorageReady` condition on the Tenant CR so that storage readiness is visible to users through the primary Tenant resource. [Clarify: R1.Q1] - **FR-3:** The OSAC Storage Controller watches ClusterOrder CRs to prepare for future CaaS Phase 2 triggering. For v0.1, no trigger logic is implemented — only the watch and informer are set up. [Clarify: R1.Q2] -- **FR-4:** Storage-related fields are removed from the Tenant CRD: StorageClasses status field, Jobs status field, StorageClassReady condition type, and storage-related print columns. [Clarify: R2.Q3] +- **FR-4:** Storage-related fields are removed from the Tenant CRD: StorageClasses status field, Jobs status field, StorageClassReady condition type, and storage-related print columns. A new `StorageReady` condition type is added to the Tenant CRD, managed exclusively by the OSAC Storage Controller. [Clarify: R2.Q3] - **FR-5:** The ComputeInstance controller reads StorageClasses from the TenantStorage CR's status instead of the Tenant CR's status. -- **FR-6:** The Tenant controller has zero storage responsibilities. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. Storage readiness is entirely the OSAC Storage Controller's concern. +- **FR-6:** The Tenant controller does not run any storage logic. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. The OSAC Storage Controller is responsible for all storage operations and sets the `StorageReady` condition on the Tenant CR (and in the future, on ClusterOrder CR) so that storage readiness is visible through the resources users interact with. - **FR-7:** On Tenant deletion, the OSAC Storage Controller triggers backend teardown via AAP to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret, then deletes the TenantStorage CR. No owner reference is used — the controller explicitly watches Tenant deletion events. [Clarify: R1.Q3] @@ -67,7 +67,8 @@ OSAC storage provisioning uses a two-phase model. Phase 1 (backend setup) runs a - [ ] OSAC Storage Controller watches ClusterOrder CRs (informer set up, no trigger logic for v0.1) - [ ] OSAC Storage Controller triggers AAP provisioning when hub Secret is absent, sets phase to Ready when Secret exists and StorageClasses are resolved - [ ] Storage fields removed from Tenant CRD (StorageClasses, Jobs, StorageClassReady condition, storage print columns) -- [ ] Tenant controller has no storage-related logic or RBAC +- [ ] Tenant controller has no storage-related logic +- [ ] `StorageReady` condition is set on Tenant CR by the OSAC Storage Controller when storage onboarding completes - [ ] ComputeInstance controller reads StorageClasses from TenantStorage status - [ ] On Tenant deletion, OSAC Storage Controller triggers AAP teardown and deletes TenantStorage CR - [ ] AAP playbooks split into four job templates: osac-create-tenant-storage, osac-ensure-tenant-storage, osac-cleanup-tenant-storage, osac-delete-tenant-storage From e57fde1bd5ad8fd75ac0c4ac4e7af7254844fb3e Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Tue, 9 Jun 2026 16:47:32 -0400 Subject: [PATCH 012/450] OSAC-23: restructure PRD with stage separation and storage decoupling Reorganize requirements into Controller/API Foundation and Storage Onboarding Workflow sections to make the dependency explicit: storage must be extracted from the Tenant controller before onboarding workflows can run independently. Introduce a two-stage model for tenant storage provisioning. Stage 1 (backend setup) creates tenant credentials and sets StorageBackendReady on the Tenant. Stage 2 (cluster-side setup) discovers StorageClasses on target clusters. For VMaaS, both stages run at tenant onboarding. For CaaS, Stage 2 runs after ClusterOrder reaches Ready. Add per-cluster entries array to TenantStorage status so each cluster tracks its own storage readiness and delivery model. Add migration risk for existing tenants and open questions for delivery model detection and migration procedure. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- .../tenant-storage-onboarding-OSAC-23/prd.md | 115 ++++++++++++------ 1 file changed, 78 insertions(+), 37 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index bf59e2b48..c86b2e017 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -8,89 +8,130 @@ ## 1. Problem Statement -Storage provisioning logic — backend setup, StorageClass resolution, AAP job tracking, and storage readiness — is embedded in the Tenant controller and the Tenant CR's status fields. Any change to storage onboarding risks breaking tenant lifecycle, and supporting different storage workflows per delivery model (VMaaS, CaaS, BMaaS) requires branching logic inside a single controller. Storage issues cannot be diagnosed independently from tenant status, and the storage working group cannot iterate on storage onboarding without coordinating every change with the core tenant lifecycle. +Storage provisioning logic is embedded in the Tenant controller and the Tenant CR's status fields. This includes backend setup, StorageClass resolution, AAP job tracking, and storage readiness. The Tenant CR holds storage state (StorageClasses, jobs, storage conditions) that conceptually belongs to storage lifecycle, not tenant lifecycle. Any change to storage onboarding risks breaking tenant state transitions, and supporting different storage workflows per delivery model (VMaaS, CaaS, BMaaS) requires branching logic inside a single controller. Storage issues cannot be diagnosed independently from tenant status, and the storage working group cannot iterate on storage onboarding without coordinating every change with the core tenant lifecycle. ## 2. Goals and Non-Goals ### 2.1 Goals -- Storage lifecycle operates independently from tenant lifecycle — storage provisioning, readiness tracking, and teardown are managed by a dedicated controller without affecting tenant state transitions. +- Storage lifecycle operates independently from tenant lifecycle. Storage provisioning, readiness tracking, and teardown are managed by a dedicated controller without affecting tenant state transitions. - Per-tenant storage state (provisioned tiers, provider resources, readiness) is observable as a standalone Kubernetes resource via `kubectl get tenantstorage`. - The OSAC Storage Controller is the single entry point for all storage reconciliation, establishing the foundation for future storage capabilities (backend registration, CaaS cluster-side setup, resource creation storage). +- AAP storage playbooks are split into distinct lifecycle actions (create, ensure, cleanup, delete) so that each can be triggered independently by the controller. -### 2.3 Non-Goals +### 2.2 Non-Goals -- StorageBackend CRD implementation (OSAC-917) -- StorageTier CRD implementation (OSAC-1110) -- CaaS Phase 2 trigger logic from ClusterOrder (OSAC-1123) — the controller watches ClusterOrder CRs, but the trigger logic for cluster-side storage setup is a separate effort -- Backend registration automation -- Prometheus metrics or fulfillment-service API visibility for storage state -- Fulfillment-service proto or API changes -- Changes to the storage provider role logic (setup, ensure_storage_class, teardown tasks) — only the playbook entry points and job template names change +- StorageBackend API and registration automation +- StorageTier API +- CaaS Tenant Storage Setup +- VAST Support +- Fulfillment-service API or proto changes for storage state visibility +- Per-cluster storage resource on target clusters for tenant visibility. TenantStorage on the hub provides the CloudProviderAdmin view. A cluster-side resource for tenant admins to observe storage readiness on their own clusters is expected to follow in CaaS scope. ## 3. Requirements -OSAC storage provisioning uses a two-phase model. Phase 1 (backend setup) runs at tenant onboarding: it creates the tenant's organization on the storage appliance, provisions VIP pools, credentials, per-tier views, and stores per-tenant credentials in a hub Secret. Phase 2 (cluster-side setup) runs when a target cluster is ready: it installs CSI drivers and creates StorageClasses on the cluster. This PRD covers the controller and CRD that manage Phase 1. Phase 2 triggering is out of scope (OSAC-1123). +OSAC tenant storage provisioning uses a two-stage model. Stage 1 (backend setup) runs at tenant onboarding: it creates the tenant's organization on the storage appliance, provisions VIP pools, credentials, per-tier views, and stores per-tenant credentials in a hub Secret. Stage 2 (cluster-side setup) runs when a target cluster has storage available: it discovers StorageClasses and, in future, installs CSI drivers on the target cluster. For VMaaS, both stages can run at tenant onboarding because the cluster already exists. For CaaS, Stage 2 runs after the cluster is provisioned (ClusterOrder reaches Ready). This PRD covers the controller, API, and both stages. Stage 1 and Stage 2 must be implemented as separate operations in the controller to avoid coupling that would need to be untangled when CaaS support is added. -### 3.1 Functional Requirements +### 3.1 Controller and API Foundation -- **FR-1:** A TenantStorage CRD captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains phase (Progressing, Ready, Failed, Deleting), resolved StorageClass names with tier labels, structured status conditions, and AAP job tracking. +These requirements establish the OSAC Storage Controller and TenantStorage API as the new home for all storage state and logic, decoupled from the Tenant controller. -- **FR-2:** The OSAC Storage Controller watches Tenant CRs. When a Tenant reaches Ready (namespace exists), the controller creates a TenantStorage CR for that tenant and begins storage onboarding — checking for the tenant's hub Secret, triggering backend provisioning via AAP if absent, and resolving StorageClasses when the Secret exists. When storage onboarding completes, the controller sets a `StorageReady` condition on the Tenant CR so that storage readiness is visible to users through the primary Tenant resource. [Clarify: R1.Q1] +- **FR-1:** A TenantStorage API captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains phase (Progressing, Ready, Failed, Deleting), structured status conditions, AAP job tracking, and a per-cluster entries array. Each cluster entry tracks the cluster name, delivery model (VMaaS or CaaS), storage readiness, and resolved StorageClass names. For VMaaS, the cluster entry is populated at tenant onboarding. For CaaS, cluster entries are added when a ClusterOrder reaches Ready. -- **FR-3:** The OSAC Storage Controller watches ClusterOrder CRs to prepare for future CaaS Phase 2 triggering. For v0.1, no trigger logic is implemented — only the watch and informer are set up. [Clarify: R1.Q2] +- **FR-2:** Remove storage-related fields from the Tenant API: StorageClasses status field, Jobs status field, StorageClassReady condition type, and storage-related print columns. Add a new `StorageBackendReady` condition type to the Tenant API, managed exclusively by the OSAC Storage Controller. -- **FR-4:** Storage-related fields are removed from the Tenant CRD: StorageClasses status field, Jobs status field, StorageClassReady condition type, and storage-related print columns. A new `StorageReady` condition type is added to the Tenant CRD, managed exclusively by the OSAC Storage Controller. [Clarify: R2.Q3] +- **FR-3:** The Tenant controller does not run any storage logic. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. -- **FR-5:** The ComputeInstance controller reads StorageClasses from the TenantStorage CR's status instead of the Tenant CR's status. +- **FR-4:** The ComputeInstance controller reads StorageClasses from the TenantStorage status instead of the Tenant status. -- **FR-6:** The Tenant controller does not run any storage logic. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. The OSAC Storage Controller is responsible for all storage operations and sets the `StorageReady` condition on the Tenant CR (and in the future, on ClusterOrder CR) so that storage readiness is visible through the resources users interact with. +- **FR-5:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to Unmanaged, consistent with all other OSAC controllers. -- **FR-7:** On Tenant deletion, the OSAC Storage Controller triggers backend teardown via AAP to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret, then deletes the TenantStorage CR. No owner reference is used — the controller explicitly watches Tenant deletion events. [Clarify: R1.Q3] +### 3.2 Storage Onboarding Workflow -- **FR-8:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage` (Phase 1 backend setup), `osac-ensure-tenant-storage` (Phase 2 cluster-side CSI + StorageClasses), `osac-cleanup-tenant-storage` (cluster-side resource removal), and `osac-delete-tenant-storage` (backend teardown). A corresponding cleanup action is added to the storage_provider role dispatcher. [Clarify: R1.Q4] +These requirements define the onboarding, readiness, and teardown workflows that run on top of the foundation above. -- **FR-9:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to Unmanaged, consistent with all other OSAC controllers. +- **FR-6 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller creates a TenantStorage resource and begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage`) if absent. When the hub Secret exists, Stage 1 is complete. The controller sets the `StorageBackendReady` condition on the Tenant so that storage readiness is visible through the primary Tenant resource. -### 3.2 Non-Functional Requirements +- **FR-7 (Stage 2):** The OSAC Storage Controller discovers StorageClasses on the target cluster and populates the corresponding cluster entry in the TenantStorage status. Stage 2 is a separate operation from Stage 1. For VMaaS, Stage 2 runs at tenant onboarding because the cluster already exists and StorageClasses are manually pre-created. For CaaS, Stage 2 runs after ClusterOrder reaches Ready. Stage 2 does not automate StorageClass creation or CSI driver installation for v0.1. -- **NFR-1:** Admin credentials (VAST endpoint, username, password) are ephemeral — mounted as environment variables in the AAP automation pod, cleared after use, and never persisted to Kubernetes. +- **FR-8:** On Tenant deletion, the OSAC Storage Controller triggers backend teardown via AAP (`osac-delete-tenant-storage`) to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret, then deletes the TenantStorage resource. No owner reference is used. The controller explicitly watches Tenant deletion events. + +- **FR-9:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage` (Stage 1 backend setup), `osac-ensure-tenant-storage` (Stage 2 cluster-side StorageClasses), `osac-cleanup-tenant-storage` (cluster-side resource removal), and `osac-delete-tenant-storage` (backend teardown). + +- **FR-10:** The OSAC Storage Controller watches ClusterOrder resources. For VMaaS, this is not needed since Stage 2 runs at tenant onboarding. For CaaS, the watch enables Stage 2 triggering when ClusterOrder reaches Ready. Trigger logic is covered by CaaS Tenant Storage Setup. + +### 3.3 Non-Functional Requirements + +- **NFR-1:** Admin credentials (VAST endpoint, username, password) are ephemeral. They are mounted as environment variables in the AAP automation pod, cleared after use, and never persisted to Kubernetes. - **NFR-2:** Per-tenant credentials are stored in a Secret in the osac-system namespace, managed exclusively by the OSAC Storage Controller. These Secrets are the handoff point to downstream workflows (VMaaS cluster-side setup, CaaS cluster-side setup). -- **NFR-3:** The AAP playbook changes and operator changes must be deployed together. The `osac-create-tenant-storage` playbook replaces the previous combined playbook that executed both Phase 1 and Phase 2 — deploying the AAP changes without the operator changes breaks existing Tenant controller provisioning. +- **NFR-3:** The AAP playbook changes and operator changes must be deployed together. The `osac-create-tenant-storage` playbook replaces the previous combined playbook that executed both Stage 1 and Stage 2. Deploying the AAP changes without the operator changes breaks existing Tenant controller provisioning. ## 4. Acceptance Criteria -- [ ] TenantStorage CRD is defined with spec (tenantRef) and status (phase, storageClasses, conditions, jobs) -- [ ] OSAC Storage Controller watches Tenant CRs and creates TenantStorage when Tenant reaches Ready -- [ ] OSAC Storage Controller watches ClusterOrder CRs (informer set up, no trigger logic for v0.1) -- [ ] OSAC Storage Controller triggers AAP provisioning when hub Secret is absent, sets phase to Ready when Secret exists and StorageClasses are resolved -- [ ] Storage fields removed from Tenant CRD (StorageClasses, Jobs, StorageClassReady condition, storage print columns) +**Controller and API Foundation** +- [ ] TenantStorage API is defined with spec (`tenantRef`) and status (phase, conditions, AAP job tracking, per-cluster entries array) +- [ ] Storage fields removed from Tenant API (StorageClasses, Jobs, StorageClassReady condition, storage print columns) - [ ] Tenant controller has no storage-related logic -- [ ] `StorageReady` condition is set on Tenant CR by the OSAC Storage Controller when storage onboarding completes - [ ] ComputeInstance controller reads StorageClasses from TenantStorage status -- [ ] On Tenant deletion, OSAC Storage Controller triggers AAP teardown and deletes TenantStorage CR -- [ ] AAP playbooks split into four job templates: osac-create-tenant-storage, osac-ensure-tenant-storage, osac-cleanup-tenant-storage, osac-delete-tenant-storage - [ ] `kubectl get tenantstorage` displays Tenant and Phase columns +- [ ] OSAC Storage Controller skips reconciliation when `osac.openshift.io/management-state` is set to Unmanaged + +**Stage 1 (Backend Setup)** +- [ ] OSAC Storage Controller watches Tenant resources and creates TenantStorage when Tenant reaches Ready +- [ ] OSAC Storage Controller triggers AAP backend provisioning (`osac-create-tenant-storage`) when hub Secret is absent +- [ ] `StorageBackendReady` condition is set on Tenant by the OSAC Storage Controller when Stage 1 completes + +**Stage 2 (Cluster-Side Setup)** +- [ ] For VMaaS, OSAC Storage Controller discovers StorageClasses on the target cluster at tenant onboarding and populates the cluster entry in TenantStorage status +- [ ] ClusterOrder watch and informer are set up for CaaS Stage 2 triggering (trigger logic is covered by CaaS Tenant Storage Setup) + +**Teardown** +- [ ] On Tenant deletion, OSAC Storage Controller triggers AAP teardown (`osac-delete-tenant-storage`) and deletes TenantStorage resource + +**AAP Playbooks** +- [ ] AAP playbooks split into four job templates: `osac-create-tenant-storage`, `osac-ensure-tenant-storage`, `osac-cleanup-tenant-storage`, `osac-delete-tenant-storage` + +**Testing** - [ ] Unit tests pass for the new controller and updated Tenant/ComputeInstance controllers ## 5. Assumptions -- AAP is the only provisioning backend for v0.1 — no direct API provisioning path exists. +- AAP is the only provisioning backend for v0.1. No direct API provisioning path exists. - VAST is the only storage provider for v0.1. -- Tier configuration via the STORAGE_TIERS environment variable is sufficient for v0.1; the StorageTier API is a separate effort. +- Tier configuration via the STORAGE_TIERS environment variable is sufficient for this PRD. The StorageTier API is a separate effort. +- StorageClasses are manually created. The OSAC Storage Controller discovers them but does not create them. ## 6. Dependencies -- **osac-aap** (OSAC-1145): AAP playbook split is delivered as part of this work. Coordinated merge required — AAP changes break backward compatibility if deployed without the operator changes. -- **osac-operator provisioning framework**: The controller reuses the existing `provisioning.RunProvisioningLifecycle()` helper and ProvisioningProvider interface. No changes to the framework are required. +- **Tenant CR**: The OSAC Storage Controller depends on the Tenant CR lifecycle. Storage onboarding begins when a Tenant reaches Ready. Tenant deletion triggers storage teardown. +- **osac-operator and osac-aap coordinated deployment**: Changes span both repos and must be merged and deployed together. Deploying one without the other breaks tenant storage provisioning. ## 7. Risks ### 7.1 Breaking change in AAP playbook interface -The previous combined playbook (osac-configure-tenant-storage) is replaced by four separate playbooks. The new osac-create-tenant-storage playbook executes Phase 1 only, whereas the previous playbook executed both Phase 1 and Phase 2. Deploying the AAP changes without the operator changes causes the existing Tenant controller to provision storage incompletely. +The previous combined playbook (osac-configure-tenant-storage) is replaced by four separate playbooks. The new osac-create-tenant-storage playbook executes Stage 1 only, whereas the previous playbook executed both Stage 1 and Stage 2. Deploying the AAP changes without the operator changes causes the existing Tenant controller to provision storage incompletely. - **Owner:** Storage WG - **Mitigation:** Coordinated PR merge across osac-operator and osac-aap repos. PRs are linked in descriptions. Deployment documentation specifies both components must be updated together. + +### 7.2 Migration of existing tenants + +Existing tenants have storage state in the Tenant CR and no TenantStorage resource. When the new controller starts, it must create TenantStorage resources for existing tenants and populate them from current state, without re-triggering AAP provisioning. + +- **Owner:** Storage WG +- **Mitigation:** The controller detects existing tenants with storage already provisioned (hub Secret exists) and creates TenantStorage resources with the correct state. No reprovisioning is triggered. + +## 8. Open Questions + +### 8.1 How does the OSAC Storage Controller determine the delivery model (VMaaS vs CaaS) for a given cluster to decide when Stage 2 runs? + +- **Owner:** Storage WG +- **Impact:** FR-7 (Stage 2) and FR-10 (ClusterOrder watch) + +### 8.2 What is the exact migration procedure for existing tenants? + +- **Owner:** Storage WG +- **Impact:** Risk 7.2 (Migration of existing tenants) From b67de83c202d90779068f17eb466444010231af3 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Tue, 9 Jun 2026 16:53:11 -0400 Subject: [PATCH 013/450] OSAC-23: simplify PRD language Remove verbose phrasing, clarify the storage controller scope (clusters, compute instances, PVCs), and tighten wording throughout for readability. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- .../tenant-storage-onboarding-OSAC-23/prd.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index c86b2e017..1cac0bb2c 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -16,7 +16,7 @@ Storage provisioning logic is embedded in the Tenant controller and the Tenant C - Storage lifecycle operates independently from tenant lifecycle. Storage provisioning, readiness tracking, and teardown are managed by a dedicated controller without affecting tenant state transitions. - Per-tenant storage state (provisioned tiers, provider resources, readiness) is observable as a standalone Kubernetes resource via `kubectl get tenantstorage`. -- The OSAC Storage Controller is the single entry point for all storage reconciliation, establishing the foundation for future storage capabilities (backend registration, CaaS cluster-side setup, resource creation storage). +- The OSAC Storage Controller is the single entry point for all storage reconciliation, including storage for clusters, compute instances, and PVCs. - AAP storage playbooks are split into distinct lifecycle actions (create, ensure, cleanup, delete) so that each can be triggered independently by the controller. ### 2.2 Non-Goals @@ -26,7 +26,7 @@ Storage provisioning logic is embedded in the Tenant controller and the Tenant C - CaaS Tenant Storage Setup - VAST Support - Fulfillment-service API or proto changes for storage state visibility -- Per-cluster storage resource on target clusters for tenant visibility. TenantStorage on the hub provides the CloudProviderAdmin view. A cluster-side resource for tenant admins to observe storage readiness on their own clusters is expected to follow in CaaS scope. +- Per-cluster storage resource on target clusters for tenant visibility. TenantStorage on the hub provides the CloudProviderAdmin view. A cluster-side resource for tenant admin visibility will follow in CaaS scope. ## 3. Requirements @@ -36,7 +36,7 @@ OSAC tenant storage provisioning uses a two-stage model. Stage 1 (backend setup) These requirements establish the OSAC Storage Controller and TenantStorage API as the new home for all storage state and logic, decoupled from the Tenant controller. -- **FR-1:** A TenantStorage API captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains phase (Progressing, Ready, Failed, Deleting), structured status conditions, AAP job tracking, and a per-cluster entries array. Each cluster entry tracks the cluster name, delivery model (VMaaS or CaaS), storage readiness, and resolved StorageClass names. For VMaaS, the cluster entry is populated at tenant onboarding. For CaaS, cluster entries are added when a ClusterOrder reaches Ready. +- **FR-1:** A TenantStorage API captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains phase (Progressing, Ready, Failed, Deleting), status conditions, AAP job tracking, and a per-cluster entries array. Each cluster entry tracks the cluster name, delivery model (VMaaS or CaaS), storage readiness, and resolved StorageClass names. For VMaaS, the cluster entry is populated at tenant onboarding. For CaaS, cluster entries are added when a ClusterOrder reaches Ready. - **FR-2:** Remove storage-related fields from the Tenant API: StorageClasses status field, Jobs status field, StorageClassReady condition type, and storage-related print columns. Add a new `StorageBackendReady` condition type to the Tenant API, managed exclusively by the OSAC Storage Controller. @@ -50,9 +50,9 @@ These requirements establish the OSAC Storage Controller and TenantStorage API a These requirements define the onboarding, readiness, and teardown workflows that run on top of the foundation above. -- **FR-6 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller creates a TenantStorage resource and begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage`) if absent. When the hub Secret exists, Stage 1 is complete. The controller sets the `StorageBackendReady` condition on the Tenant so that storage readiness is visible through the primary Tenant resource. +- **FR-6 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller creates a TenantStorage resource and begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage`) if absent. When the hub Secret exists, Stage 1 is complete. The controller sets the `StorageBackendReady` condition on the Tenant so users can see storage readiness on the Tenant resource. -- **FR-7 (Stage 2):** The OSAC Storage Controller discovers StorageClasses on the target cluster and populates the corresponding cluster entry in the TenantStorage status. Stage 2 is a separate operation from Stage 1. For VMaaS, Stage 2 runs at tenant onboarding because the cluster already exists and StorageClasses are manually pre-created. For CaaS, Stage 2 runs after ClusterOrder reaches Ready. Stage 2 does not automate StorageClass creation or CSI driver installation for v0.1. +- **FR-7 (Stage 2):** The OSAC Storage Controller discovers StorageClasses on the target cluster and populates the corresponding cluster entry in the TenantStorage status. Stage 2 is a separate operation from Stage 1. For VMaaS, Stage 2 runs at tenant onboarding because the cluster already exists and StorageClasses are manually pre-created. For CaaS, Stage 2 runs after ClusterOrder reaches Ready. StorageClass creation and CSI driver installation are automated separately. - **FR-8:** On Tenant deletion, the OSAC Storage Controller triggers backend teardown via AAP (`osac-delete-tenant-storage`) to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret, then deletes the TenantStorage resource. No owner reference is used. The controller explicitly watches Tenant deletion events. @@ -64,7 +64,7 @@ These requirements define the onboarding, readiness, and teardown workflows that - **NFR-1:** Admin credentials (VAST endpoint, username, password) are ephemeral. They are mounted as environment variables in the AAP automation pod, cleared after use, and never persisted to Kubernetes. -- **NFR-2:** Per-tenant credentials are stored in a Secret in the osac-system namespace, managed exclusively by the OSAC Storage Controller. These Secrets are the handoff point to downstream workflows (VMaaS cluster-side setup, CaaS cluster-side setup). +- **NFR-2:** Per-tenant credentials are stored in a Secret in the osac-system namespace, managed exclusively by the OSAC Storage Controller. These Secrets are used by downstream workflows (VMaaS cluster-side setup, CaaS cluster-side setup). - **NFR-3:** The AAP playbook changes and operator changes must be deployed together. The `osac-create-tenant-storage` playbook replaces the previous combined playbook that executed both Stage 1 and Stage 2. Deploying the AAP changes without the operator changes breaks existing Tenant controller provisioning. From ed2e4b9f0a3c63a74b89abaf6e9520dfeb7ca789 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 10 Jun 2026 19:32:11 -0400 Subject: [PATCH 014/450] OSAC-23: adopt condition ownership, drop TenantStorage CRD The OSAC Storage Controller manages storage conditions and status fields directly on the Tenant CR instead of introducing a separate TenantStorage CRD. This follows the condition ownership pattern where multiple controllers manage distinct conditions on a shared resource. Key changes from the previous PRD revision: - StorageBackendReady and StorageClassReady conditions on Tenant CR - status.storageClasses and status.jobs ownership moves to storage controller - ComputeInstance controller unchanged (reads from Tenant as before) - Finalizer on Tenant CR for deletion ordering - AAP playbooks renamed: osac-create-tenant-storage-backend, osac-create-tenant-storage-class, osac-delete-tenant-storage-class, osac-delete-tenant-storage-backend - StorageClasses created by AAP (not manually), controller discovers them via osac.openshift.io/tenant label - VMaaS target cluster is remote or hub (statically configured) - Removed Risk 7.2 (migration): pre-GA, no existing tenants - Resolved all open questions - Added NFR for credential-safe logging - Added E2E testing acceptance criterion Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- .../tenant-storage-onboarding-OSAC-23/prd.md | 117 +++++++++--------- 1 file changed, 61 insertions(+), 56 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index 1cac0bb2c..7df53cd9d 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -4,104 +4,124 @@ |-------------|---------| | Author(s) | Zoltan Szabo, Akshay Nadkarni | | Jira | https://redhat.atlassian.net/browse/OSAC-23 | -| Date | 2026-06-09 | +| Date | 2026-06-10 | ## 1. Problem Statement -Storage provisioning logic is embedded in the Tenant controller and the Tenant CR's status fields. This includes backend setup, StorageClass resolution, AAP job tracking, and storage readiness. The Tenant CR holds storage state (StorageClasses, jobs, storage conditions) that conceptually belongs to storage lifecycle, not tenant lifecycle. Any change to storage onboarding risks breaking tenant state transitions, and supporting different storage workflows per delivery model (VMaaS, CaaS, BMaaS) requires branching logic inside a single controller. Storage issues cannot be diagnosed independently from tenant status, and the storage working group cannot iterate on storage onboarding without coordinating every change with the core tenant lifecycle. +Storage provisioning logic is embedded in the Tenant controller. Backend setup, StorageClass resolution, AAP job tracking, and storage readiness are all handled within a single controller alongside namespace creation and tenant lifecycle management. Any change to storage onboarding risks breaking tenant state transitions, and supporting different storage workflows per delivery model (VMaaS, CaaS, BMaaS) requires branching logic inside the Tenant controller. Storage issues cannot be diagnosed independently from tenant status, and the storage working group cannot iterate on storage onboarding without coordinating every change with the core tenant lifecycle. ## 2. Goals and Non-Goals ### 2.1 Goals -- Storage lifecycle operates independently from tenant lifecycle. Storage provisioning, readiness tracking, and teardown are managed by a dedicated controller without affecting tenant state transitions. -- Per-tenant storage state (provisioned tiers, provider resources, readiness) is observable as a standalone Kubernetes resource via `kubectl get tenantstorage`. -- The OSAC Storage Controller is the single entry point for all storage reconciliation, including storage for clusters, compute instances, and PVCs. -- AAP storage playbooks are split into distinct lifecycle actions (create, ensure, cleanup, delete) so that each can be triggered independently by the controller. +- Storage lifecycle operates independently from tenant lifecycle. Storage provisioning, readiness tracking, and teardown are managed by a dedicated OSAC Storage Controller without affecting tenant state transitions. +- The OSAC Storage Controller owns all storage-related conditions and status fields on the Tenant CR using the condition ownership pattern. The Tenant controller has zero storage responsibilities. +- The OSAC Storage Controller is the single entry point for all tenant storage reconciliation. +- AAP storage playbooks are split into distinct lifecycle actions (create-backend, create-class, delete-class, delete-backend) so that each can be triggered independently by the controller. +- Storage readiness is observable via `kubectl get tenant` print columns showing StorageBackendReady and StorageClassReady status. ### 2.2 Non-Goals - StorageBackend API and registration automation - StorageTier API -- CaaS Tenant Storage Setup -- VAST Support +- CaaS Tenant Storage Setup: Stage 2 trigger logic for CaaS clusters is a separate PRD +- VAST support for CaaS +- Per-cluster storage resource on target clusters for tenant admin visibility - Fulfillment-service API or proto changes for storage state visibility -- Per-cluster storage resource on target clusters for tenant visibility. TenantStorage on the hub provides the CloudProviderAdmin view. A cluster-side resource for tenant admin visibility will follow in CaaS scope. ## 3. Requirements -OSAC tenant storage provisioning uses a two-stage model. Stage 1 (backend setup) runs at tenant onboarding: it creates the tenant's organization on the storage appliance, provisions VIP pools, credentials, per-tier views, and stores per-tenant credentials in a hub Secret. Stage 2 (cluster-side setup) runs when a target cluster has storage available: it discovers StorageClasses and, in future, installs CSI drivers on the target cluster. For VMaaS, both stages can run at tenant onboarding because the cluster already exists. For CaaS, Stage 2 runs after the cluster is provisioned (ClusterOrder reaches Ready). This PRD covers the controller, API, and both stages. Stage 1 and Stage 2 must be implemented as separate operations in the controller to avoid coupling that would need to be untangled when CaaS support is added. +OSAC tenant storage provisioning uses a two-stage model: -### 3.1 Controller and API Foundation +- **Stage 1 (backend setup)** runs at tenant onboarding. It creates the tenant's organization on the storage appliance, provisions VIP pools, credentials, and per-tier views (each tier has its own view for a tenant), and stores per-tenant credentials in a hub Secret. +- **Stage 2 (cluster-side setup)** runs after Stage 1 completes. AAP playbooks install the CSI operator (if needed) and create per-tenant StorageClasses and CSI credentials on the VMaaS target cluster. -These requirements establish the OSAC Storage Controller and TenantStorage API as the new home for all storage state and logic, decoupled from the Tenant controller. +For VMaaS, both stages run at tenant onboarding. For CaaS, Stage 2 runs after the cluster is provisioned (ClusterOrder reaches Ready), covered by a separate PRD. -- **FR-1:** A TenantStorage API captures per-tenant storage state. The spec contains a `tenantRef` field referencing the owning Tenant by name. The status contains phase (Progressing, Ready, Failed, Deleting), status conditions, AAP job tracking, and a per-cluster entries array. Each cluster entry tracks the cluster name, delivery model (VMaaS or CaaS), storage readiness, and resolved StorageClass names. For VMaaS, the cluster entry is populated at tenant onboarding. For CaaS, cluster entries are added when a ClusterOrder reaches Ready. +This PRD covers the controller and both stages. Stage 1 and Stage 2 must be implemented as separate operations in the controller to avoid coupling that would need to be untangled when CaaS support is added. -- **FR-2:** Remove storage-related fields from the Tenant API: StorageClasses status field, Jobs status field, StorageClassReady condition type, and storage-related print columns. Add a new `StorageBackendReady` condition type to the Tenant API, managed exclusively by the OSAC Storage Controller. +### 3.1 Controller Foundation -- **FR-3:** The Tenant controller does not run any storage logic. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. +These requirements establish the OSAC Storage Controller as the owner of all storage-related state on the Tenant CR. -- **FR-4:** The ComputeInstance controller reads StorageClasses from the TenantStorage status instead of the Tenant status. +- **FR-1:** The OSAC Storage Controller owns the following fields on the Tenant CR: + - `StorageBackendReady` condition: indicates whether the tenant's storage backend is provisioned (organization on the storage appliance, VIP pools, credentials in hub Secret, per-tier views). Set to True when Stage 1 completes successfully, False on failure with a descriptive reason. + - `StorageClassReady` condition: indicates whether StorageClasses and CSI drivers are installed on the VMaaS target cluster for this tenant. The VMaaS target cluster is either a remote cluster or the hub cluster itself. Set to True when Stage 2 completes during tenant storage onboarding. + - `status.storageClasses`: list of the tenant's resolved StorageClass names. + - `status.jobs`: AAP job tracking for in-flight storage operations. -- **FR-5:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to Unmanaged, consistent with all other OSAC controllers. + `kubectl get tenant` displays StorageBackendReady and StorageClassReady status via additional print columns. + +- **FR-2:** The Tenant controller does not run any storage logic. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. All storage-related conditions and status fields are managed exclusively by the OSAC Storage Controller. + +- **FR-3:** The ComputeInstance controller continues to read `status.storageClasses` from the Tenant CR. No changes are required to the ComputeInstance controller. + +- **FR-4:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to Unmanaged, consistent with all other OSAC controllers. ### 3.2 Storage Onboarding Workflow -These requirements define the onboarding, readiness, and teardown workflows that run on top of the foundation above. +These requirements define the onboarding, readiness, and teardown workflows managed by the OSAC Storage Controller. -- **FR-6 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller creates a TenantStorage resource and begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage`) if absent. When the hub Secret exists, Stage 1 is complete. The controller sets the `StorageBackendReady` condition on the Tenant so users can see storage readiness on the Tenant resource. +- **FR-5 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage-backend`) if absent. When the hub Secret exists, Stage 1 is complete and the controller sets `StorageBackendReady=True` on the Tenant. If the AAP job fails, the controller sets `StorageBackendReady=False` with a reason reflecting the failure and retries with backoff. -- **FR-7 (Stage 2):** The OSAC Storage Controller discovers StorageClasses on the target cluster and populates the corresponding cluster entry in the TenantStorage status. Stage 2 is a separate operation from Stage 1. For VMaaS, Stage 2 runs at tenant onboarding because the cluster already exists and StorageClasses are manually pre-created. For CaaS, Stage 2 runs after ClusterOrder reaches Ready. StorageClass creation and CSI driver installation are automated separately. +- **FR-6 (Stage 2):** After Stage 1 completes, the OSAC Storage Controller triggers cluster-side setup via AAP (`osac-create-tenant-storage-class`) to install the CSI operator and create per-tenant StorageClasses on the VMaaS target cluster (remote or hub). The controller discovers installed StorageClasses using the `osac.openshift.io/tenant` label and populates `status.storageClasses` on the Tenant. When discovery confirms all expected StorageClasses are present, the controller sets `StorageClassReady=True` on the Tenant. -- **FR-8:** On Tenant deletion, the OSAC Storage Controller triggers backend teardown via AAP (`osac-delete-tenant-storage`) to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret, then deletes the TenantStorage resource. No owner reference is used. The controller explicitly watches Tenant deletion events. +- **FR-7:** The OSAC Storage Controller places a finalizer on the Tenant CR to block deletion until storage teardown completes. On Tenant deletion, the teardown sequence is: first, trigger cluster-side cleanup via AAP (`osac-delete-tenant-storage-class`) to remove StorageClasses, CSI Secrets, and VolumeSnapshotClasses; then, trigger backend teardown via AAP (`osac-delete-tenant-storage-backend`) to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret. The finalizer is removed only after both steps complete successfully. If teardown fails, the finalizer remains in place and the Tenant stays in Terminating until the issue is resolved. -- **FR-9:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage` (Stage 1 backend setup), `osac-ensure-tenant-storage` (Stage 2 cluster-side StorageClasses), `osac-cleanup-tenant-storage` (cluster-side resource removal), and `osac-delete-tenant-storage` (backend teardown). +- **FR-8:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage-backend` (Stage 1 backend setup), `osac-create-tenant-storage-class` (Stage 2 cluster-side StorageClasses and CSI), `osac-delete-tenant-storage-class` (cluster-side resource removal during deletion), and `osac-delete-tenant-storage-backend` (backend teardown). -- **FR-10:** The OSAC Storage Controller watches ClusterOrder resources. For VMaaS, this is not needed since Stage 2 runs at tenant onboarding. For CaaS, the watch enables Stage 2 triggering when ClusterOrder reaches Ready. Trigger logic is covered by CaaS Tenant Storage Setup. +- **FR-9:** The OSAC Storage Controller establishes a watch on ClusterOrder resources to prepare for CaaS support. No action is taken on ClusterOrder events in this scope. CaaS Stage 2 trigger logic is defined in a separate PRD. ### 3.3 Non-Functional Requirements - **NFR-1:** Admin credentials (VAST endpoint, username, password) are ephemeral. They are mounted as environment variables in the AAP automation pod, cleared after use, and never persisted to Kubernetes. -- **NFR-2:** Per-tenant credentials are stored in a Secret in the osac-system namespace, managed exclusively by the OSAC Storage Controller. These Secrets are used by downstream workflows (VMaaS cluster-side setup, CaaS cluster-side setup). +- **NFR-2:** Per-tenant credentials are stored in a Secret in the osac-system namespace, managed exclusively by the OSAC Storage Controller. These Secrets are used by downstream cluster-side setup workflows. + +- **NFR-3:** The AAP playbook changes and operator changes must be deployed together. The `osac-create-tenant-storage-backend` playbook replaces the previous combined playbook that executed both Stage 1 and Stage 2. Deploying the AAP changes without the operator changes breaks existing Tenant controller provisioning. -- **NFR-3:** The AAP playbook changes and operator changes must be deployed together. The `osac-create-tenant-storage` playbook replaces the previous combined playbook that executed both Stage 1 and Stage 2. Deploying the AAP changes without the operator changes breaks existing Tenant controller provisioning. +- **NFR-4:** Controller logs and Kubernetes events must not expose credentials, Secret contents, or sensitive AAP parameters during any storage operation (provisioning, teardown, or failure recovery). ## 4. Acceptance Criteria -**Controller and API Foundation** -- [ ] TenantStorage API is defined with spec (`tenantRef`) and status (phase, conditions, AAP job tracking, per-cluster entries array) -- [ ] Storage fields removed from Tenant API (StorageClasses, Jobs, StorageClassReady condition, storage print columns) +**Controller Foundation** +- [ ] OSAC Storage Controller owns `StorageBackendReady` and `StorageClassReady` conditions on the Tenant CR +- [ ] OSAC Storage Controller manages `status.storageClasses` and `status.jobs` on the Tenant CR - [ ] Tenant controller has no storage-related logic -- [ ] ComputeInstance controller reads StorageClasses from TenantStorage status -- [ ] `kubectl get tenantstorage` displays Tenant and Phase columns +- [ ] ComputeInstance controller reads StorageClasses from Tenant status (no change required) +- [ ] `kubectl get tenant` displays StorageBackendReady and StorageClassReady columns - [ ] OSAC Storage Controller skips reconciliation when `osac.openshift.io/management-state` is set to Unmanaged **Stage 1 (Backend Setup)** -- [ ] OSAC Storage Controller watches Tenant resources and creates TenantStorage when Tenant reaches Ready -- [ ] OSAC Storage Controller triggers AAP backend provisioning (`osac-create-tenant-storage`) when hub Secret is absent -- [ ] `StorageBackendReady` condition is set on Tenant by the OSAC Storage Controller when Stage 1 completes +- [ ] OSAC Storage Controller watches Tenant resources and begins storage onboarding when Tenant reaches Ready +- [ ] OSAC Storage Controller triggers AAP backend provisioning (`osac-create-tenant-storage-backend`) when hub Secret is absent +- [ ] `StorageBackendReady` condition is set to True on Tenant when Stage 1 completes +- [ ] `StorageBackendReady` condition is set to False with reason on AAP job failure, controller retries with backoff **Stage 2 (Cluster-Side Setup)** -- [ ] For VMaaS, OSAC Storage Controller discovers StorageClasses on the target cluster at tenant onboarding and populates the cluster entry in TenantStorage status -- [ ] ClusterOrder watch and informer are set up for CaaS Stage 2 triggering (trigger logic is covered by CaaS Tenant Storage Setup) +- [ ] After Stage 1 completes, OSAC Storage Controller triggers cluster-side setup (`osac-create-tenant-storage-class`) on the VMaaS target cluster (remote or hub) +- [ ] OSAC Storage Controller discovers StorageClasses using `osac.openshift.io/tenant` label and populates `status.storageClasses` +- [ ] `StorageClassReady` condition is set to True when StorageClasses are confirmed on the VMaaS target cluster +- [ ] ClusterOrder watch is established (no action taken on events in this scope) **Teardown** -- [ ] On Tenant deletion, OSAC Storage Controller triggers AAP teardown (`osac-delete-tenant-storage`) and deletes TenantStorage resource +- [ ] OSAC Storage Controller places a finalizer on the Tenant CR to block deletion until teardown completes +- [ ] On Tenant deletion, controller triggers cluster-side cleanup (`osac-delete-tenant-storage-class`) then backend teardown (`osac-delete-tenant-storage-backend`) +- [ ] Finalizer is removed only after both cleanup steps complete successfully **AAP Playbooks** -- [ ] AAP playbooks split into four job templates: `osac-create-tenant-storage`, `osac-ensure-tenant-storage`, `osac-cleanup-tenant-storage`, `osac-delete-tenant-storage` +- [ ] AAP playbooks split into four job templates: `osac-create-tenant-storage-backend`, `osac-create-tenant-storage-class`, `osac-delete-tenant-storage-class`, `osac-delete-tenant-storage-backend` **Testing** -- [ ] Unit tests pass for the new controller and updated Tenant/ComputeInstance controllers +- [ ] Unit tests pass for the new OSAC Storage Controller and updated Tenant controller +- [ ] E2E tests validate the full storage onboarding lifecycle (Stage 1 and Stage 2) against a VAST appliance ## 5. Assumptions - AAP is the only provisioning backend for v0.1. No direct API provisioning path exists. - VAST is the only storage provider for v0.1. - Tier configuration via the STORAGE_TIERS environment variable is sufficient for this PRD. The StorageTier API is a separate effort. -- StorageClasses are manually created. The OSAC Storage Controller discovers them but does not create them. +- StorageClasses are created by AAP playbooks (`osac-create-tenant-storage-class`). The OSAC Storage Controller discovers them using the `osac.openshift.io/tenant` label but does not create them directly. In development environments without access to a VAST appliance, StorageClasses may need to be created manually for testing. ## 6. Dependencies @@ -112,26 +132,11 @@ These requirements define the onboarding, readiness, and teardown workflows that ### 7.1 Breaking change in AAP playbook interface -The previous combined playbook (osac-configure-tenant-storage) is replaced by four separate playbooks. The new osac-create-tenant-storage playbook executes Stage 1 only, whereas the previous playbook executed both Stage 1 and Stage 2. Deploying the AAP changes without the operator changes causes the existing Tenant controller to provision storage incompletely. +The previous combined playbook (osac-configure-tenant-storage) is replaced by four separate playbooks. The new osac-create-tenant-storage-backend playbook executes Stage 1 only, whereas the previous playbook executed both Stage 1 and Stage 2. Deploying the AAP changes without the operator changes causes the existing Tenant controller to provision storage incompletely. - **Owner:** Storage WG - **Mitigation:** Coordinated PR merge across osac-operator and osac-aap repos. PRs are linked in descriptions. Deployment documentation specifies both components must be updated together. -### 7.2 Migration of existing tenants - -Existing tenants have storage state in the Tenant CR and no TenantStorage resource. When the new controller starts, it must create TenantStorage resources for existing tenants and populate them from current state, without re-triggering AAP provisioning. - -- **Owner:** Storage WG -- **Mitigation:** The controller detects existing tenants with storage already provisioned (hub Secret exists) and creates TenantStorage resources with the correct state. No reprovisioning is triggered. - ## 8. Open Questions -### 8.1 How does the OSAC Storage Controller determine the delivery model (VMaaS vs CaaS) for a given cluster to decide when Stage 2 runs? - -- **Owner:** Storage WG -- **Impact:** FR-7 (Stage 2) and FR-10 (ClusterOrder watch) - -### 8.2 What is the exact migration procedure for existing tenants? - -- **Owner:** Storage WG -- **Impact:** Risk 7.2 (Migration of existing tenants) +None at this time. From 5644e1c347cdf8302d29cf820fc7cfede67e5941 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 10 Jun 2026 22:12:07 -0400 Subject: [PATCH 015/450] OSAC-23: add tier resolution note and Secret validation wording Clarify that status.storageClasses uses the same tier resolution algorithm (tenant-specific with Default fallback) previously owned by the Tenant controller. Tighten FR-5 to require the hub Secret exists and is valid before marking StorageBackendReady=True. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- enhancements/tenant-storage-onboarding-OSAC-23/prd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index 7df53cd9d..06837460e 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -47,7 +47,7 @@ These requirements establish the OSAC Storage Controller as the owner of all sto - **FR-1:** The OSAC Storage Controller owns the following fields on the Tenant CR: - `StorageBackendReady` condition: indicates whether the tenant's storage backend is provisioned (organization on the storage appliance, VIP pools, credentials in hub Secret, per-tier views). Set to True when Stage 1 completes successfully, False on failure with a descriptive reason. - `StorageClassReady` condition: indicates whether StorageClasses and CSI drivers are installed on the VMaaS target cluster for this tenant. The VMaaS target cluster is either a remote cluster or the hub cluster itself. Set to True when Stage 2 completes during tenant storage onboarding. - - `status.storageClasses`: list of the tenant's resolved StorageClass names. + - `status.storageClasses`: list of the tenant's resolved StorageClass names. The storage controller uses the same tier resolution algorithm (tenant-specific entry with Default fallback) previously owned by the Tenant controller. The detailed algorithm is specified in the design document. - `status.jobs`: AAP job tracking for in-flight storage operations. `kubectl get tenant` displays StorageBackendReady and StorageClassReady status via additional print columns. @@ -62,7 +62,7 @@ These requirements establish the OSAC Storage Controller as the owner of all sto These requirements define the onboarding, readiness, and teardown workflows managed by the OSAC Storage Controller. -- **FR-5 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage-backend`) if absent. When the hub Secret exists, Stage 1 is complete and the controller sets `StorageBackendReady=True` on the Tenant. If the AAP job fails, the controller sets `StorageBackendReady=False` with a reason reflecting the failure and retries with backoff. +- **FR-5 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage-backend`) if absent. When the hub Secret exists and is valid, Stage 1 is complete and the controller sets `StorageBackendReady=True` on the Tenant. If the AAP job fails, the controller sets `StorageBackendReady=False` with a reason reflecting the failure and retries with backoff. - **FR-6 (Stage 2):** After Stage 1 completes, the OSAC Storage Controller triggers cluster-side setup via AAP (`osac-create-tenant-storage-class`) to install the CSI operator and create per-tenant StorageClasses on the VMaaS target cluster (remote or hub). The controller discovers installed StorageClasses using the `osac.openshift.io/tenant` label and populates `status.storageClasses` on the Tenant. When discovery confirms all expected StorageClasses are present, the controller sets `StorageClassReady=True` on the Tenant. From 3a50bb71a55e255c6055236ed990186a101b2273 Mon Sep 17 00:00:00 2001 From: Zoltan Szabo Date: Fri, 12 Jun 2026 10:03:02 +0200 Subject: [PATCH 016/450] OSAC-23: Address Avishay's v3 review feedback - Remove Risk 7.1 (pre-GA, breaking changes are expected) - Move API field names from FR-1 to design doc scope - Fix NFR-1: admin credentials are in a pre-configured Secret, not "never persisted to Kubernetes" Assisted-by: Claude Code Signed-off-by: Zoltan Szabo --- .../tenant-storage-onboarding-OSAC-23/prd.md | 42 ++++++++----------- 1 file changed, 18 insertions(+), 24 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index 06837460e..9102e1c3b 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -18,7 +18,7 @@ Storage provisioning logic is embedded in the Tenant controller. Backend setup, - The OSAC Storage Controller owns all storage-related conditions and status fields on the Tenant CR using the condition ownership pattern. The Tenant controller has zero storage responsibilities. - The OSAC Storage Controller is the single entry point for all tenant storage reconciliation. - AAP storage playbooks are split into distinct lifecycle actions (create-backend, create-class, delete-class, delete-backend) so that each can be triggered independently by the controller. -- Storage readiness is observable via `kubectl get tenant` print columns showing StorageBackendReady and StorageClassReady status. +- Storage readiness is observable via `kubectl get tenant` print columns showing backend and class readiness status. ### 2.2 Non-Goals @@ -44,17 +44,17 @@ This PRD covers the controller and both stages. Stage 1 and Stage 2 must be impl These requirements establish the OSAC Storage Controller as the owner of all storage-related state on the Tenant CR. -- **FR-1:** The OSAC Storage Controller owns the following fields on the Tenant CR: - - `StorageBackendReady` condition: indicates whether the tenant's storage backend is provisioned (organization on the storage appliance, VIP pools, credentials in hub Secret, per-tier views). Set to True when Stage 1 completes successfully, False on failure with a descriptive reason. - - `StorageClassReady` condition: indicates whether StorageClasses and CSI drivers are installed on the VMaaS target cluster for this tenant. The VMaaS target cluster is either a remote cluster or the hub cluster itself. Set to True when Stage 2 completes during tenant storage onboarding. - - `status.storageClasses`: list of the tenant's resolved StorageClass names. The storage controller uses the same tier resolution algorithm (tenant-specific entry with Default fallback) previously owned by the Tenant controller. The detailed algorithm is specified in the design document. - - `status.jobs`: AAP job tracking for in-flight storage operations. +- **FR-1:** The OSAC Storage Controller owns all storage-related conditions and status on the Tenant CR: + - A condition indicating whether the tenant's storage backend is provisioned (organization on the storage appliance, VIP pools, credentials in hub Secret, per-tier views). Set to True when Stage 1 completes successfully, False on failure with a descriptive reason. + - A condition indicating whether StorageClasses and CSI drivers are installed on the VMaaS target cluster for this tenant. Set to True when Stage 2 completes during tenant storage onboarding. + - The list of the tenant's resolved StorageClass names, using the same tier resolution algorithm (tenant-specific entry with Default fallback) previously owned by the Tenant controller. + - AAP job tracking for in-flight storage operations. - `kubectl get tenant` displays StorageBackendReady and StorageClassReady status via additional print columns. + Storage readiness is observable via `kubectl get tenant`. Exact field names and API details are specified in the design document. - **FR-2:** The Tenant controller does not run any storage logic. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. All storage-related conditions and status fields are managed exclusively by the OSAC Storage Controller. -- **FR-3:** The ComputeInstance controller continues to read `status.storageClasses` from the Tenant CR. No changes are required to the ComputeInstance controller. +- **FR-3:** The ComputeInstance controller continues to read the tenant's resolved StorageClasses from the Tenant CR. No changes are required to the ComputeInstance controller. - **FR-4:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to Unmanaged, consistent with all other OSAC controllers. @@ -62,9 +62,9 @@ These requirements establish the OSAC Storage Controller as the owner of all sto These requirements define the onboarding, readiness, and teardown workflows managed by the OSAC Storage Controller. -- **FR-5 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage-backend`) if absent. When the hub Secret exists and is valid, Stage 1 is complete and the controller sets `StorageBackendReady=True` on the Tenant. If the AAP job fails, the controller sets `StorageBackendReady=False` with a reason reflecting the failure and retries with backoff. +- **FR-5 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage-backend`) if absent. When the hub Secret exists and is valid, Stage 1 is complete and the controller sets the backend readiness condition to True on the Tenant. If the AAP job fails, the condition is set to False with a reason reflecting the failure and the controller retries with backoff. -- **FR-6 (Stage 2):** After Stage 1 completes, the OSAC Storage Controller triggers cluster-side setup via AAP (`osac-create-tenant-storage-class`) to install the CSI operator and create per-tenant StorageClasses on the VMaaS target cluster (remote or hub). The controller discovers installed StorageClasses using the `osac.openshift.io/tenant` label and populates `status.storageClasses` on the Tenant. When discovery confirms all expected StorageClasses are present, the controller sets `StorageClassReady=True` on the Tenant. +- **FR-6 (Stage 2):** After Stage 1 completes, the OSAC Storage Controller triggers cluster-side setup via AAP (`osac-create-tenant-storage-class`) to install the CSI operator and create per-tenant StorageClasses on the VMaaS target cluster (remote or hub). The controller discovers installed StorageClasses using the `osac.openshift.io/tenant` label and populates the resolved StorageClass list on the Tenant. When discovery confirms all expected StorageClasses are present, the controller sets the class readiness condition to True on the Tenant. - **FR-7:** The OSAC Storage Controller places a finalizer on the Tenant CR to block deletion until storage teardown completes. On Tenant deletion, the teardown sequence is: first, trigger cluster-side cleanup via AAP (`osac-delete-tenant-storage-class`) to remove StorageClasses, CSI Secrets, and VolumeSnapshotClasses; then, trigger backend teardown via AAP (`osac-delete-tenant-storage-backend`) to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret. The finalizer is removed only after both steps complete successfully. If teardown fails, the finalizer remains in place and the Tenant stays in Terminating until the issue is resolved. @@ -74,7 +74,7 @@ These requirements define the onboarding, readiness, and teardown workflows mana ### 3.3 Non-Functional Requirements -- **NFR-1:** Admin credentials (VAST endpoint, username, password) are ephemeral. They are mounted as environment variables in the AAP automation pod, cleared after use, and never persisted to Kubernetes. +- **NFR-1:** Admin credentials (VAST endpoint, username, password) are mounted as environment variables in the AAP automation pod from a pre-configured Secret, cleared from playbook memory after use, and never accessed by the controller. - **NFR-2:** Per-tenant credentials are stored in a Secret in the osac-system namespace, managed exclusively by the OSAC Storage Controller. These Secrets are used by downstream cluster-side setup workflows. @@ -85,23 +85,22 @@ These requirements define the onboarding, readiness, and teardown workflows mana ## 4. Acceptance Criteria **Controller Foundation** -- [ ] OSAC Storage Controller owns `StorageBackendReady` and `StorageClassReady` conditions on the Tenant CR -- [ ] OSAC Storage Controller manages `status.storageClasses` and `status.jobs` on the Tenant CR +- [ ] OSAC Storage Controller owns storage-related conditions and status fields on the Tenant CR - [ ] Tenant controller has no storage-related logic - [ ] ComputeInstance controller reads StorageClasses from Tenant status (no change required) -- [ ] `kubectl get tenant` displays StorageBackendReady and StorageClassReady columns +- [ ] `kubectl get tenant` displays storage readiness columns - [ ] OSAC Storage Controller skips reconciliation when `osac.openshift.io/management-state` is set to Unmanaged **Stage 1 (Backend Setup)** - [ ] OSAC Storage Controller watches Tenant resources and begins storage onboarding when Tenant reaches Ready - [ ] OSAC Storage Controller triggers AAP backend provisioning (`osac-create-tenant-storage-backend`) when hub Secret is absent -- [ ] `StorageBackendReady` condition is set to True on Tenant when Stage 1 completes -- [ ] `StorageBackendReady` condition is set to False with reason on AAP job failure, controller retries with backoff +- [ ] Backend readiness condition is set to True on Tenant when Stage 1 completes +- [ ] Backend readiness condition is set to False with reason on AAP job failure, controller retries with backoff **Stage 2 (Cluster-Side Setup)** - [ ] After Stage 1 completes, OSAC Storage Controller triggers cluster-side setup (`osac-create-tenant-storage-class`) on the VMaaS target cluster (remote or hub) -- [ ] OSAC Storage Controller discovers StorageClasses using `osac.openshift.io/tenant` label and populates `status.storageClasses` -- [ ] `StorageClassReady` condition is set to True when StorageClasses are confirmed on the VMaaS target cluster +- [ ] OSAC Storage Controller discovers StorageClasses using `osac.openshift.io/tenant` label and populates the resolved StorageClass list +- [ ] Class readiness condition is set to True when StorageClasses are confirmed on the VMaaS target cluster - [ ] ClusterOrder watch is established (no action taken on events in this scope) **Teardown** @@ -130,12 +129,7 @@ These requirements define the onboarding, readiness, and teardown workflows mana ## 7. Risks -### 7.1 Breaking change in AAP playbook interface - -The previous combined playbook (osac-configure-tenant-storage) is replaced by four separate playbooks. The new osac-create-tenant-storage-backend playbook executes Stage 1 only, whereas the previous playbook executed both Stage 1 and Stage 2. Deploying the AAP changes without the operator changes causes the existing Tenant controller to provision storage incompletely. - -- **Owner:** Storage WG -- **Mitigation:** Coordinated PR merge across osac-operator and osac-aap repos. PRs are linked in descriptions. Deployment documentation specifies both components must be updated together. +None. The project is pre-GA — breaking changes are expected and acceptable. ## 8. Open Questions From bc25a6c188b2f5d8b30e98a37d04f012bfde0588 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Fri, 12 Jun 2026 11:12:58 -0400 Subject: [PATCH 017/450] OSAC-23: add multi-backend and per-cluster tracking to FR-1 FR-1 now requires per-backend detail (name, provider, status, error) and per-cluster detail (cluster name, readiness, reason) in the Tenant status. Clarifies that status.jobs is a shared field used by multiple controllers, not owned by the storage controller. For this PRD, cluster storage readiness covers the VMaaS target cluster only. CaaS clusters are covered by a separate PRD. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- enhancements/tenant-storage-onboarding-OSAC-23/prd.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index 9102e1c3b..e64f06591 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -45,12 +45,12 @@ This PRD covers the controller and both stages. Stage 1 and Stage 2 must be impl These requirements establish the OSAC Storage Controller as the owner of all storage-related state on the Tenant CR. - **FR-1:** The OSAC Storage Controller owns all storage-related conditions and status on the Tenant CR: - - A condition indicating whether the tenant's storage backend is provisioned (organization on the storage appliance, VIP pools, credentials in hub Secret, per-tier views). Set to True when Stage 1 completes successfully, False on failure with a descriptive reason. - - A condition indicating whether StorageClasses and CSI drivers are installed on the VMaaS target cluster for this tenant. Set to True when Stage 2 completes during tenant storage onboarding. + - Backend readiness: tracks whether per-tenant resources are provisioned on each configured storage backend. Set to True when Stage 1 completes successfully for all backends, False on failure with a descriptive reason. Per-backend detail (name, provider, status, error) is available in the Tenant status. + - Cluster storage readiness: tracks whether StorageClasses and CSI drivers are installed on each cluster associated with the tenant. For this PRD, this covers the VMaaS target cluster only. Per-cluster detail (cluster name, readiness, reason) is available in the Tenant status. - The list of the tenant's resolved StorageClass names, using the same tier resolution algorithm (tenant-specific entry with Default fallback) previously owned by the Tenant controller. - - AAP job tracking for in-flight storage operations. + - AAP job tracking for in-flight storage operations, written to the shared `status.jobs` field on the Tenant CR (also used by other controllers such as networking). - Storage readiness is observable via `kubectl get tenant`. Exact field names and API details are specified in the design document. + Storage readiness is observable via `kubectl get tenant`. Exact condition names, status structure, and API details are specified in the design document. - **FR-2:** The Tenant controller does not run any storage logic. It manages namespace creation, UDN reconciliation, and tenant lifecycle only. All storage-related conditions and status fields are managed exclusively by the OSAC Storage Controller. From f1e568a32a81142c29a9b62f24e5bb836f8d299b Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Fri, 12 Jun 2026 11:40:47 -0400 Subject: [PATCH 018/450] OSAC-23: rename cluster-side playbooks to cluster-storage Rename osac-create-tenant-storage-class to osac-create-tenant-cluster-storage and osac-delete-tenant-storage-class to osac-delete-tenant-cluster-storage. "cluster-storage" better describes the scope (CSI operator, CSI Secrets, StorageClasses) than "storage-class" which only captures one artifact. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- .../tenant-storage-onboarding-OSAC-23/prd.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index e64f06591..850645d2c 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -17,7 +17,7 @@ Storage provisioning logic is embedded in the Tenant controller. Backend setup, - Storage lifecycle operates independently from tenant lifecycle. Storage provisioning, readiness tracking, and teardown are managed by a dedicated OSAC Storage Controller without affecting tenant state transitions. - The OSAC Storage Controller owns all storage-related conditions and status fields on the Tenant CR using the condition ownership pattern. The Tenant controller has zero storage responsibilities. - The OSAC Storage Controller is the single entry point for all tenant storage reconciliation. -- AAP storage playbooks are split into distinct lifecycle actions (create-backend, create-class, delete-class, delete-backend) so that each can be triggered independently by the controller. +- AAP storage playbooks are split into distinct lifecycle actions (create-storage-backend, create-cluster-storage, delete-cluster-storage, delete-storage-backend) so that each can be triggered independently by the controller. - Storage readiness is observable via `kubectl get tenant` print columns showing backend and class readiness status. ### 2.2 Non-Goals @@ -64,11 +64,11 @@ These requirements define the onboarding, readiness, and teardown workflows mana - **FR-5 (Stage 1):** The OSAC Storage Controller watches Tenant resources. When a Tenant reaches Ready (namespace exists), the controller begins backend setup: checking for the tenant's hub Secret and triggering backend provisioning via AAP (`osac-create-tenant-storage-backend`) if absent. When the hub Secret exists and is valid, Stage 1 is complete and the controller sets the backend readiness condition to True on the Tenant. If the AAP job fails, the condition is set to False with a reason reflecting the failure and the controller retries with backoff. -- **FR-6 (Stage 2):** After Stage 1 completes, the OSAC Storage Controller triggers cluster-side setup via AAP (`osac-create-tenant-storage-class`) to install the CSI operator and create per-tenant StorageClasses on the VMaaS target cluster (remote or hub). The controller discovers installed StorageClasses using the `osac.openshift.io/tenant` label and populates the resolved StorageClass list on the Tenant. When discovery confirms all expected StorageClasses are present, the controller sets the class readiness condition to True on the Tenant. +- **FR-6 (Stage 2):** After Stage 1 completes, the OSAC Storage Controller triggers cluster-side setup via AAP (`osac-create-tenant-cluster-storage`) to install the CSI operator and create per-tenant StorageClasses on the VMaaS target cluster (remote or hub). The controller discovers installed StorageClasses using the `osac.openshift.io/tenant` label and populates the resolved StorageClass list on the Tenant. When discovery confirms all expected StorageClasses are present, the controller sets the class readiness condition to True on the Tenant. -- **FR-7:** The OSAC Storage Controller places a finalizer on the Tenant CR to block deletion until storage teardown completes. On Tenant deletion, the teardown sequence is: first, trigger cluster-side cleanup via AAP (`osac-delete-tenant-storage-class`) to remove StorageClasses, CSI Secrets, and VolumeSnapshotClasses; then, trigger backend teardown via AAP (`osac-delete-tenant-storage-backend`) to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret. The finalizer is removed only after both steps complete successfully. If teardown fails, the finalizer remains in place and the Tenant stays in Terminating until the issue is resolved. +- **FR-7:** The OSAC Storage Controller places a finalizer on the Tenant CR to block deletion until storage teardown completes. On Tenant deletion, the teardown sequence is: first, trigger cluster-side cleanup via AAP (`osac-delete-tenant-cluster-storage`) to remove StorageClasses, CSI Secrets, and VolumeSnapshotClasses; then, trigger backend teardown via AAP (`osac-delete-tenant-storage-backend`) to clean up storage provider resources (VAST tenant, views, quotas) and the per-tenant hub Secret. The finalizer is removed only after both steps complete successfully. If teardown fails, the finalizer remains in place and the Tenant stays in Terminating until the issue is resolved. -- **FR-8:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage-backend` (Stage 1 backend setup), `osac-create-tenant-storage-class` (Stage 2 cluster-side StorageClasses and CSI), `osac-delete-tenant-storage-class` (cluster-side resource removal during deletion), and `osac-delete-tenant-storage-backend` (backend teardown). +- **FR-8:** AAP playbooks are split into four lifecycle actions: `osac-create-tenant-storage-backend` (Stage 1 backend setup), `osac-create-tenant-cluster-storage` (Stage 2 cluster-side StorageClasses and CSI), `osac-delete-tenant-cluster-storage` (cluster-side resource removal during deletion), and `osac-delete-tenant-storage-backend` (backend teardown). - **FR-9:** The OSAC Storage Controller establishes a watch on ClusterOrder resources to prepare for CaaS support. No action is taken on ClusterOrder events in this scope. CaaS Stage 2 trigger logic is defined in a separate PRD. @@ -98,18 +98,18 @@ These requirements define the onboarding, readiness, and teardown workflows mana - [ ] Backend readiness condition is set to False with reason on AAP job failure, controller retries with backoff **Stage 2 (Cluster-Side Setup)** -- [ ] After Stage 1 completes, OSAC Storage Controller triggers cluster-side setup (`osac-create-tenant-storage-class`) on the VMaaS target cluster (remote or hub) +- [ ] After Stage 1 completes, OSAC Storage Controller triggers cluster-side setup (`osac-create-tenant-cluster-storage`) on the VMaaS target cluster (remote or hub) - [ ] OSAC Storage Controller discovers StorageClasses using `osac.openshift.io/tenant` label and populates the resolved StorageClass list - [ ] Class readiness condition is set to True when StorageClasses are confirmed on the VMaaS target cluster - [ ] ClusterOrder watch is established (no action taken on events in this scope) **Teardown** - [ ] OSAC Storage Controller places a finalizer on the Tenant CR to block deletion until teardown completes -- [ ] On Tenant deletion, controller triggers cluster-side cleanup (`osac-delete-tenant-storage-class`) then backend teardown (`osac-delete-tenant-storage-backend`) +- [ ] On Tenant deletion, controller triggers cluster-side cleanup (`osac-delete-tenant-cluster-storage`) then backend teardown (`osac-delete-tenant-storage-backend`) - [ ] Finalizer is removed only after both cleanup steps complete successfully **AAP Playbooks** -- [ ] AAP playbooks split into four job templates: `osac-create-tenant-storage-backend`, `osac-create-tenant-storage-class`, `osac-delete-tenant-storage-class`, `osac-delete-tenant-storage-backend` +- [ ] AAP playbooks split into four job templates: `osac-create-tenant-storage-backend`, `osac-create-tenant-cluster-storage`, `osac-delete-tenant-cluster-storage`, `osac-delete-tenant-storage-backend` **Testing** - [ ] Unit tests pass for the new OSAC Storage Controller and updated Tenant controller @@ -120,7 +120,7 @@ These requirements define the onboarding, readiness, and teardown workflows mana - AAP is the only provisioning backend for v0.1. No direct API provisioning path exists. - VAST is the only storage provider for v0.1. - Tier configuration via the STORAGE_TIERS environment variable is sufficient for this PRD. The StorageTier API is a separate effort. -- StorageClasses are created by AAP playbooks (`osac-create-tenant-storage-class`). The OSAC Storage Controller discovers them using the `osac.openshift.io/tenant` label but does not create them directly. In development environments without access to a VAST appliance, StorageClasses may need to be created manually for testing. +- StorageClasses are created by AAP playbooks (`osac-create-tenant-cluster-storage`). The OSAC Storage Controller discovers them using the `osac.openshift.io/tenant` label but does not create them directly. In development environments without access to a VAST appliance, StorageClasses may need to be created manually for testing. ## 6. Dependencies From b184fe6fee476fd2f92aa5c26e93f11f0b19c386 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Sun, 14 Jun 2026 20:02:41 -0400 Subject: [PATCH 019/450] OSAC-23: add tier workflows to non-goals, scope Unmanaged skip Add "Tier addition and removal workflows" to non-goals per Avishay's feedback. Scope the management-state Unmanaged check to steady-state reconciliation only so that finalizer and deletion handling still run, preventing Tenants from being stuck in Terminating. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- enhancements/tenant-storage-onboarding-OSAC-23/prd.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md index 850645d2c..e6b8068d0 100644 --- a/enhancements/tenant-storage-onboarding-OSAC-23/prd.md +++ b/enhancements/tenant-storage-onboarding-OSAC-23/prd.md @@ -24,6 +24,7 @@ Storage provisioning logic is embedded in the Tenant controller. Backend setup, - StorageBackend API and registration automation - StorageTier API +- Tier addition and removal workflows - CaaS Tenant Storage Setup: Stage 2 trigger logic for CaaS clusters is a separate PRD - VAST support for CaaS - Per-cluster storage resource on target clusters for tenant admin visibility @@ -56,7 +57,7 @@ These requirements establish the OSAC Storage Controller as the owner of all sto - **FR-3:** The ComputeInstance controller continues to read the tenant's resolved StorageClasses from the Tenant CR. No changes are required to the ComputeInstance controller. -- **FR-4:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips reconciliation when set to Unmanaged, consistent with all other OSAC controllers. +- **FR-4:** The OSAC Storage Controller checks the `osac.openshift.io/management-state` annotation and skips steady-state reconciliation when set to Unmanaged. Finalizer and deletion handling must still run to prevent Tenants from being stuck in Terminating. ### 3.2 Storage Onboarding Workflow @@ -89,7 +90,7 @@ These requirements define the onboarding, readiness, and teardown workflows mana - [ ] Tenant controller has no storage-related logic - [ ] ComputeInstance controller reads StorageClasses from Tenant status (no change required) - [ ] `kubectl get tenant` displays storage readiness columns -- [ ] OSAC Storage Controller skips reconciliation when `osac.openshift.io/management-state` is set to Unmanaged +- [ ] OSAC Storage Controller skips steady-state reconciliation when `osac.openshift.io/management-state` is set to Unmanaged (finalizer and deletion handling still run) **Stage 1 (Backend Setup)** - [ ] OSAC Storage Controller watches Tenant resources and begins storage onboarding when Tenant reaches Ready From c36a17f6325ec22a39a582b7befced990b87671e Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Mon, 15 Jun 2026 20:14:36 +0300 Subject: [PATCH 020/450] OSAC-1111: Add StorageBackend enhancement proposal (#51) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * OSAC-1111: Add StorageBackend enhancement proposal Signed-off-by: Roy Golan Assisted-by: Claude Code * OSAC-1111: Address EP review findings - Add README.md index with YAML frontmatter and stub section headers - Clarify PENDING→READY state transition: manual via Signal RPC (Phase 1), not READY-on-create (NetworkClass) or full reconciler (PublicIPPool) - Explain Signal RPC divergence from NetworkClass (status payload needed because fulfillment-service has no direct knowledge of array state) - Fix description field: mark as `optional string` in proto appendix - Document soft-delete name reuse as intentional (tiers reference by ID) - Fix copyright year: 2025 → 2026 - Resolve Open Question 1: hub = management cluster in OSAC HCP model - Standardize "management cluster" terminology throughout design.md - Fix tracking-link to issues.redhat.com Assisted-by: Claude Code Signed-off-by: Roy Golan * remove design.md, it will come later Signed-off-by: Roy Golan * OSAC-1111: Rework PRD to structured requirement format Restructure the StorageBackend PRD from narrative format to the numbered section template with FR-N/NFR-N requirement IDs, acceptance criteria checkboxes, and structured risks/assumptions/dependencies sections. Assisted-by: Claude Code Signed-off-by: Roy Golan * OSAC-1111: Remove implementation details from PRD Reframe requirements in terms of product capabilities rather than implementation choices (PostgreSQL, FieldMask, buf lint, SQL indexes). Addresses review feedback to keep the PRD scoped to product requirements. Assisted-by: Claude Code Signed-off-by: Roy Golan * OSAC-1111: Address review feedback round 2 - Remove Risk 7.1 (proto appendix divergence) — coding concern, not product requirement - Clarify CSI drivers and StorageClasses install on target clusters (VMaaS/CaaS), not management cluster - Renumber remaining risks Assisted-by: Claude Code Signed-off-by: Roy Golan * OSAC-1111: Clarify FR-4 and FR-8 per review feedback - FR-4: Note that pagination/filtering/ordering follows the established OSAC List API pattern used by all existing entities - FR-8: Define each state meaning (UNSPECIFIED, PENDING, READY, FAILED) and explain why Phase 1 transitions are manual (no reconciler) Assisted-by: Claude Code Signed-off-by: Roy Golan * OSAC-1111: Simplify state lifecycle to UNSPECIFIED and READY only Remove PENDING and FAILED states — they will be introduced when reconciliation or health probing is added in a future phase. Backends are created with initial state READY. Assisted-by: Claude Code Signed-off-by: Roy Golan * OSAC-1111: Remove UNSPECIFIED state, keep READY only Assisted-by: Claude Code Signed-off-by: Roy Golan --------- Signed-off-by: Roy Golan --- enhancements/storage-backend-osac-1111/prd.md | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 enhancements/storage-backend-osac-1111/prd.md diff --git a/enhancements/storage-backend-osac-1111/prd.md b/enhancements/storage-backend-osac-1111/prd.md new file mode 100644 index 000000000..591efbb54 --- /dev/null +++ b/enhancements/storage-backend-osac-1111/prd.md @@ -0,0 +1,98 @@ +# StorageBackend API + +| Field | Value | +|-------------|---------| +| Author(s) | Roy Golan rgolan@redhat.com | +| Jira | https://redhat.atlassian.net/browse/OSAC-1111 | +| Date | 2026-06-04 | + +## 1. Problem Statement + +OSAC has no API-managed inventory of storage backends. Storage configuration flows through Ansible Automation Platform extra vars (`VAST_ENDPOINT`, `VAST_USERNAME`, `VAST_PASSWORD`), making backends opaque to the OSAC data model. Cloud Provider Admins cannot discover registered backends, rotate credentials without redeploying the fulfillment-service, or inspect backend operational status through the OSAC API. This blocks the platform from composing tiered storage offerings (StorageTier, OSAC-1110) because there is no registered infrastructure to reference. + +Three prior enhancements addressed tenant-to-StorageClass resolution (`tenant-specific-storageclasses`, `tenant-storage-tiers`) but left the infrastructure layer — which backends exist, their endpoints, their credentials, and their health — invisible. + +## 2. Goals and Non-Goals + +### 2.1 Goals + +- Cloud Provider Admins can register, list, update, and decommission storage backends through the OSAC private gRPC API without modifying environment variables or Ansible playbooks. +- Cloud Provider Admins can rotate storage backend credentials by updating the `credentials_ref` field, without redeploying the fulfillment-service. +- Cloud Provider Admins can inspect backend operational status (state, model, firmware version) through the API to verify availability and troubleshoot connectivity. +- The StorageBackend entity follows the same DB-backed private API pattern as NetworkClass and PublicIPPool, maintaining consistency across OSAC infrastructure entities. +- The StorageBackend entity provides the foundation for future StorageTier composition (OSAC-1110), where tiers reference registered backends. + +### 2.3 Non-Goals + +- Full StorageTier design — OSAC-1110 is referenced in the design document's appendix (entity model + reconciliation flow only). Full design is a separate EP. +- Provider-specific integration beyond VAST — the `provider` field is generic ("vast", "ceph", "pure"). Provider-specific logic (model auto-detection, credential validation) is implemented incrementally, VAST first. +- Storage observability and monitoring — metrics, health checks, and capacity tracking are covered by the "OSAC Storage Observability" roadmap item. +- Automatic backend discovery — backends are explicitly registered by Cloud Provider Admins. No auto-discovery from infrastructure management systems. +- StorageBackend as a Kubernetes CRD — the entity is managed through the OSAC private API, not as a cluster-scoped Kubernetes resource. Reconciliation is triggered by Tenant onboarding, not by backend registration. +- CSI proxy or volume-level interception — intercepting CSI `CreateVolume` calls to inject OSAC metadata is a separate future effort (see `.planning/storage/csi-proxy-architecture.md`). + +## 3. Requirements + +### 3.1 Functional Requirements + +- **FR-1:** The fulfillment-service must expose a `StorageBackends` gRPC service under `osac.private.v1` with Create, Get, List, Update, and Delete RPCs. +- **FR-2:** All CRUD RPCs must include HTTP annotations for REST access via grpc-gateway (POST, GET, GET, PATCH, DELETE). +- **FR-3:** `CreateStorageBackend` must accept provider type, management endpoint (host + optional port), credentials reference (Kubernetes Secret), and optional description. The backend must be created with initial state `READY`. +- **FR-4:** `ListStorageBackends` must support pagination (`offset`/`limit`), CEL-based filtering, and SQL-like ordering. This follows the established OSAC List API pattern used by all existing entities (NetworkClass, Clusters, ComputeInstances, Roles, etc.). +- **FR-5:** `UpdateStorageBackend` must support partial updates (only specified fields are modified) and optimistic concurrency control to prevent conflicting writes. +- **FR-6:** `DeleteStorageBackend` must perform a soft delete. Deleted backends must be excluded from List results but preserved for audit and future references from StorageTier. +- **FR-7:** Cloud Provider Admins must be able to update backend operational metadata (model, firmware_version) and status message via the standard `Update` RPC. No Signal RPC is needed — StorageBackend has no reconciler or controller. +- **FR-8:** Backend state must include `READY` (backend registered and available). Additional states will be introduced as needed when reconciliation or health probing capabilities are added in future phases. [User] +- **FR-9:** Backend names must be unique among active (non-deleted) backends, allowing name reuse after decommission. +- **FR-10:** The `credentials_ref` field must reference a Kubernetes Secret. The reference format (namespace-scoped `namespace/secret-name` or implicit namespace) is determined during implementation. This credential storage mechanism is temporary — it will be revisited when OSAC core establishes a unified credential management pattern. + +### 3.2 Non-Functional Requirements + +- **NFR-1:** The StorageBackend entity must track creation and modification timestamps for auditability. +- **NFR-2:** Provider-specific credential schemas are not validated by the StorageBackend API. The credential content is opaque; validation is the responsibility of the provider-specific integration layer. + +## 4. Acceptance Criteria + +- [ ] `CreateStorageBackend` creates a backend with state `READY` and returns the created object with a generated ID. +- [ ] `GetStorageBackend` retrieves a backend by ID with all fields populated (including status). +- [ ] `ListStorageBackends` returns paginated results, supports filtering by field values (e.g., by provider), and excludes soft-deleted records. +- [ ] `UpdateStorageBackend` applies partial updates without modifying unspecified fields. +- [ ] `UpdateStorageBackend` rejects concurrent conflicting writes. +- [ ] `DeleteStorageBackend` soft-deletes the backend. Subsequent List calls exclude the deleted backend. +- [ ] All CRUD RPCs are accessible via both gRPC and REST endpoints. +- [ ] Integration tests cover the full CRUD lifecycle, pagination, filtering, and concurrency control. + +## 5. Assumptions + +- The management cluster (hub) is the same cluster that runs the fulfillment-service and creates HostedClusters via Hosted Control Planes. The terms "hub" (ACM terminology) and "management cluster" refer to the same cluster. +- StorageClasses are associated with tenants and tiers by naming convention, not by Kubernetes labels. The prior EP label-based approach (`osac.openshift.io/tenant`, `osac.openshift.io/storage-tier`) is superseded. +- The `credentials_ref` Secret is created and managed by the Cloud Provider Admin outside of the StorageBackend API. The API stores only the reference, not the credential content. + +## 6. Dependencies + +- **StorageTier (OSAC-1110):** StorageTier entities will reference StorageBackend by ID. StorageTier design depends on StorageBackend but not vice versa. StorageBackend can be implemented and deployed independently. +- **Tenant onboarding reconciliation:** Future reconciliation flow (resolve tier → backend(s) → install CSI driver on target cluster → create tenant-scoped StorageClass on target cluster) depends on both StorageBackend and StorageTier being implemented. CSI drivers and StorageClasses are installed on target clusters (VMaaS/CaaS), not the management cluster. Not in scope for this work. + +## 7. Risks + +### 7.1 Multi-provider maintenance burden + +- **Owner:** Storage architect +- **Mitigation:** Start with VAST as the reference implementation. Establish a provider abstraction pattern (interface or strategy pattern for backend validation and model detection) and document the provider integration process for future contributors. + +### 7.2 Soft-delete name reuse ambiguity + +- **Owner:** Storage architect +- **Mitigation:** Name reuse after soft deletion is allowed (FR-9). If a future StorageTier references the decommissioned backend by ID (not name), there is no conflict. Document this behavior in the API reference. + +## 8. Open Questions + +### 8.1 Should credential rotation trigger re-validation? + +- **Owner:** Storage architect +- **Impact:** FR-3, FR-8. When `credentials_ref` is updated via `UpdateStorageBackend`, should the backend state change to indicate re-validation is needed? Deferred until additional states (e.g., `PENDING`) are introduced in a future phase. + +### 8.2 What is the credentials_ref format? + +- **Owner:** Storage architect, fulfillment-service maintainers +- **Impact:** FR-10. The reference format — namespace-scoped (`namespace/secret-name`) vs. implicit namespace (`secret-name` assuming `osac-system`) — has security implications for cross-namespace access. A decision is needed before implementation. From 71d3ca8c79934380f66f5f7800e50a337738e7ff Mon Sep 17 00:00:00 2001 From: Avishay Traeger Date: Wed, 17 Jun 2026 14:01:38 +0300 Subject: [PATCH 021/450] Add multi-tenant organization architecture with hierarchical project permissions (#38) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Update organizations & authentication enhancement Key updates: - Clarify multi-cluster scope (infrastructure clusters, not CaaS) - Add project-scoped resource access pattern (project switcher) - Defer cross-project views to future enhancement - Answer deletion lifecycle semantics (block by default, optional cascade) - Define hierarchical project permissions policy - Add Keycloak Authorization Services integration details (local RPT validation) - Clarify project permission model and resource listing - Align scope examples with OSAC resources (CREATE_COMPUTE_INSTANCE) - Note that specific scopes are implementation details - Update last-updated date to 2026-05-06 Addresses feedback from Crystal Chun and Juan Antonio Hernandez Fernandez on multi-cluster architecture, project permissions, and resource visibility. Co-Authored-By: Claude Sonnet 4.5 Signed-off-by: Avishay Traeger * Address CodeRabbit review comments - Split long summary paragraph into 5 focused paragraphs for readability - Clarify that local RPT validation is the recommended approach (not token introspection) - Add reference link to detailed RPT validation section Co-Authored-By: Claude Sonnet 4.5 Signed-off-by: Avishay Traeger * Fix CodeRabbit review findings - Fix HTTP code block formatting (add Content-Type header, use http syntax) - Clarify hierarchical permission propagation is consistent across document - Expand RPT validation to include full JWT validation details (JWKS/JWS, claims, time checks) Co-Authored-By: Claude Sonnet 4.5 Signed-off-by: Avishay Traeger * Address CrystalChun's review comments - Clarify hierarchical permissions: MANAGE_PROJECT and VIEW_PROJECT cascade from parent to child projects, but resource-specific permissions do not (line 248) - Clarify that the 'default' project does not grant automatic access to all org users - permissions must be explicitly granted (line 165) Co-Authored-By: Claude Sonnet 4.5 Signed-off-by: Avishay Traeger * Address mhrivnak's scalability concerns - Add implementation note about on-demand OpenShift Project creation to avoid N×M scaling issues - Note that database queries need efficient indexing on project_id and organization_id - Clarify that OpenShift Projects can be created on-demand or upfront as implementation choice Co-Authored-By: Claude Sonnet 4.5 Signed-off-by: Avishay Traeger --------- Signed-off-by: Avishay Traeger Co-authored-by: Claude Sonnet 4.5 --- enhancements/organizations/README.md | 246 ++++++++++++++++++++++++--- 1 file changed, 220 insertions(+), 26 deletions(-) diff --git a/enhancements/organizations/README.md b/enhancements/organizations/README.md index d1470d429..154f79c0d 100644 --- a/enhancements/organizations/README.md +++ b/enhancements/organizations/README.md @@ -3,7 +3,7 @@ title: osac-organizations-and-authentication authors: - Avishay Traeger creation-date: 2025-12-21 -last-updated: 2025-12-21 +last-updated: 2026-05-06 tracking-link: - TBD see-also: @@ -17,7 +17,15 @@ This enhancement describes the organization model, authentication flows, and adm ## Summary -This enhancement introduces a comprehensive multi-tenant organization model for OSAC that enables Cloud Provider Admins to create and manage Organizations, each with their own identity providers. Organizations can contain multiple Projects (with support for nested Projects) for additional resource organization and isolation. Each Organization must configure an external identity provider (LDAP/AD/OIDC/SAML) for user authentication. The system provides built-in OSAC break-glass accounts with limited privileges (IdP and role management only) for emergency access when the IdP is unavailable. Full admin roles (Cloud Provider Admin, Tenant Admin) are defined in the IdP. The architecture follows the Sovereign Gateway Pattern, integrating with Keycloak (managed by the user) for identity management and native role assignment, Gateway API with Kuadrant (which includes Authorino for authorization and Limitador for rate limiting) for declarative, infrastructure-level API protection, and integration with OpenShift for resource isolation. The OSAC Console and CLI authenticate directly with Keycloak using standard OIDC Authorization Code flows. Keycloak Protocol Mappers normalize upstream IdP claims into the standard OSAC Token schema. Kuadrant operates at the Gateway layer via AuthPolicy resources, validating Keycloak-issued tokens and enforcing authorization policies before requests reach the Fulfillment Service. Roles are managed natively in Keycloak and included in tokens, eliminating the need for separate role mapping storage. Authorino provides extensible authorization policies supporting RBAC (initially), with the ability to extend to OPA for advanced policies, SpiceDB for ReBAC, and integration with external systems (e.g., cost-management/billing) for future policy decisions. +This enhancement introduces a comprehensive multi-tenant organization model for OSAC that enables Cloud Provider Admins to create and manage Organizations, each with their own identity providers. Organizations can contain multiple Projects (with support for nested Projects) for additional resource organization and isolation. + +Each Organization must configure an external identity provider (LDAP/AD/OIDC/SAML) for user authentication. The system provides built-in OSAC break-glass accounts with limited privileges (IdP and role management only) for emergency access when the IdP is unavailable. Full admin roles (Cloud Provider Admin, Tenant Admin) are defined in the IdP. Keycloak (managed by the user) handles identity management and native role assignment. + +The authorization model uses a two-tier approach: Organization-level roles are managed as Keycloak roles, while Project-level permissions leverage Keycloak Authorization Services. Each Project is modeled as a Keycloak Authorization Resource with fine-grained scoped permissions (e.g., CREATE_COMPUTE_INSTANCE, VIEW_PROJECT, MANAGE_PROJECT), enabling true multi-project tenancy where different users have different access to different Projects within the same Organization. Keycloak Protocol Mappers normalize upstream IdP claims into the standard OSAC Token schema. + +The architecture follows the Sovereign Gateway Pattern, integrating Gateway API with Kuadrant (which includes Authorino for authorization and Limitador for rate limiting) for declarative, infrastructure-level API protection. Kuadrant operates at the Gateway layer via AuthPolicy resources, validating Keycloak-issued tokens (using local RPT validation for performance) and enforcing authorization policies (including project-level permissions via Keycloak Authorization Services) before requests reach the Fulfillment Service. + +A single OSAC instance with a single API entry point and Keycloak installation can manage multiple OpenShift infrastructure clusters, reconciling Project resources across all clusters. The OSAC Console and CLI authenticate directly with Keycloak using standard OIDC Authorization Code flows. ## Motivation @@ -83,11 +91,13 @@ Keycloak is an open-source identity and access management platform that connects The OSAC Console and CLI authenticate directly with Keycloak using standard OIDC Authorization Code flows. Users are redirected to Keycloak, which handles the authentication flow with the organization's configured Identity Provider. Upstream Identity Provider claims (groups, roles) are mapped into the standard OSAC Token schema using Keycloak Protocol Mappers, ensuring the OSAC platform receives a normalized token without requiring code changes. Keycloak issues OIDC tokens directly to clients; OSAC does not wrap or proxy these tokens. +**Keycloak Authorization Services** provide fine-grained, resource-level permissions for Projects. Each Project is modeled as a Keycloak Authorization Resource with a unique identifier. Keycloak Authorization supports scoped permissions (e.g., `CREATE_COMPUTE_INSTANCE`, `VIEW_PROJECT`, `MANAGE_PROJECT`) that control what operations users can perform on specific Projects. This enables true multi-project tenancy where different users can have different access levels to different Projects within the same Organization. Authorino queries Keycloak Authorization Services at request time to evaluate: "Can user X perform action Y on Project Z?" This authorization happens at the Gateway layer before requests reach the Fulfillment Service. + ### Kuadrant Kuadrant is an open source API security platform that provides comprehensive API protection including authentication, authorization, and rate limiting. Kuadrant integrates Authorino for fine-grained authorization policies and Limitador for rate limiting. This enhancement uses Kuadrant to protect OSAC APIs at the infrastructure layer via Gateway API and AuthPolicy resources: -- **Authorino**: Validates Keycloak-issued OIDC tokens and enforces authorization policies at the Gateway layer before requests reach the Fulfillment Service. Authorino validates token signatures using Keycloak's public keys (retrieved from Keycloak's JWKS endpoint), extracts identity claims and OSAC roles from tokens, and evaluates authorization policies. Initially, Authorino will use RBAC policies based on OSAC roles from Keycloak tokens. Authorino's extensible architecture supports: +- **Authorino**: Validates Keycloak-issued OIDC tokens and enforces authorization policies at the Gateway layer before requests reach the Fulfillment Service. Authorino validates token signatures using Keycloak's public keys (retrieved from Keycloak's JWKS endpoint), extracts identity claims and OSAC roles from tokens, and evaluates authorization policies. For Organization-level access, Authorino uses RBAC policies based on OSAC roles from Keycloak tokens. For Project-level access, Authorino integrates with Keycloak Authorization Services to query resource permissions at request time, checking: "Can user X perform action Y on Project Z?" Authorino's extensible architecture also supports: - OPA (Open Policy Agent) for advanced policy evaluation and complex authorization logic - SpiceDB for Relationship-Based Access Control (ReBAC) when hierarchical permissions are needed - External authorization services for integration with cost-management/billing systems to deny access based on quota or billing status @@ -95,7 +105,7 @@ Kuadrant is an open source API security platform that provides comprehensive API - **Limitador**: Provides rate limiting capabilities to protect OSAC APIs from abuse and ensure fair resource usage across organizations and projects. -Kuadrant operates at the Gateway layer (Envoy Proxy) via AuthPolicy resources. When a request arrives at the OSAC Gateway, Kuadrant intercepts it before it reaches the Fulfillment Service. Authorino validates the Keycloak token signature and claims, then injects verified user context into HTTP headers (e.g., `X-Remote-User`, `X-Organization`, `X-Roles`) that the Fulfillment Service consumes. This declarative, infrastructure-level enforcement ensures consistent authorization behavior across all OSAC APIs while providing extensibility for future policy requirements. +Kuadrant operates at the Gateway layer (Envoy Proxy) via AuthPolicy resources. When a request arrives at the OSAC Gateway, Kuadrant intercepts it before it reaches the Fulfillment Service. Authorino validates the Keycloak token signature and claims, queries Keycloak Authorization Services for project-level permissions (if the request targets a specific Project), then injects verified user context into HTTP headers (e.g., `X-Remote-User`, `X-Organization`, `X-Project`, `X-Roles`) that the Fulfillment Service consumes. This declarative, infrastructure-level enforcement ensures consistent authorization behavior across all OSAC APIs while providing extensibility for future policy requirements. ### Workflow Description @@ -152,7 +162,7 @@ This flow uses standard OIDC protocol endpoints; OSAC does not provide custom to - Creation of a Keycloak realm for the organization (via Keycloak Admin API) - Creation of standard OSAC roles in the Keycloak realm (tenant-admin, tenant-reader, tenant-user, idp-manager) - Creation of a break-glass account for the organization directly in Keycloak (via Keycloak Admin API) - Keycloak is the single source of truth for break-glass accounts - - Creation of a "default" Project in the Organization + - Creation of a "default" Project in the Organization (users do NOT automatically have access to this project - permissions must be explicitly granted via Keycloak Authorization Services) 4. The organization creation response includes the break-glass account credentials (username and initial password). @@ -187,8 +197,11 @@ This flow uses standard OIDC protocol endpoints; OSAC does not provide custom to 2. Tenant Admin calls `POST /api/fulfillment/v1/organizations/{name}/projects` with project details (name, description, metadata, optional parent_project for nested projects). 3. OSAC creates the Project within the Organization, which triggers: - - Creation of an OpenShift Project (e.g., `osac-org--project-`) - - If a parent project is specified, the project is created as a nested project with appropriate hierarchical permissions + - Creation of a Project record in the database with hierarchy information (parent-child relationships) + - Creation of a Keycloak Authorization Resource representing the Project (e.g., `PROJECT-acme-web-app`) + - **Implementation note**: To avoid N×M scalability issues with many projects and clusters, OpenShift Project creation can be deferred until resources are first deployed to a specific cluster (on-demand creation) rather than pre-creating in all managed clusters upfront + - OpenShift Projects include owner reference or annotations to represent the parent-child relationship for nested projects + - If new clusters are added later, reconciliation creates OpenShift Projects as needed when resources are deployed - Project is ready to contain resources (VMs, Networks, Volumes, etc.) or child projects 4. Tenant Admin can then: @@ -197,15 +210,64 @@ This flow uses standard OIDC protocol endpoints; OSAC does not provide custom to - Update project metadata via `PATCH /api/fulfillment/v1/organizations/{name}/projects/{name}` - Delete project via `DELETE /api/fulfillment/v1/organizations/{name}/projects/{name}` (with proper teardown, including nested projects if any) - List nested projects via `GET /api/fulfillment/v1/organizations/{name}/projects/{parent_name}/projects` + - Assign project permissions to users/groups via Keycloak Authorization Services 5. Tenant Users can now provision resources within the Project (or nested Project) using Tenant APIs. +#### Tenant User Workflow: Accessing Projects and Creating Resources + +1. Tenant User authenticates using the organization's IdP, receiving an OIDC token from Keycloak. + +2. Tenant User specifies the project context in API requests. The project is included in the request body or as a path parameter: + ```http + POST /api/fulfillment/v1/compute_instances + Content-Type: application/json + + { + "organization": "acme", + "project": "web-app", + "name": "my-vm", + "image": "rhel-9", + ... + } + ``` + +3. The request hits the OSAC Gateway (Envoy Proxy). + +4. Authorino (via Kuadrant) validates the request: + - Validates the OIDC token signature and claims + - Extracts organization and project from the request + - Queries Keycloak Authorization Services: "Does this user have `CREATE_COMPUTE_INSTANCE` permission on Project resource `PROJECT-acme-web-app`?" + - If yes, injects verified context into HTTP headers (`X-Remote-User`, `X-Organization`, `X-Project`, `X-Roles`, `X-Permissions`) + - If no, returns 403 Forbidden + +5. Fulfillment Service receives the request with verified context and creates the ComputeInstance in the appropriate OpenShift Project. + +**Project Visibility and Permissions:** +- Users only see Projects they have permissions on when listing projects +- Different users can have different permissions on different Projects within the same Organization +- Project permissions are scoped (e.g., `VIEW_PROJECT`, `CREATE_COMPUTE_INSTANCE`, `MANAGE_PROJECT`) +- **Hierarchical permissions**: Users with `MANAGE_PROJECT` or `VIEW_PROJECT` permission on a parent project automatically receive the same permission on all child projects. Resource-specific permissions (e.g., `CREATE_COMPUTE_INSTANCE`) do NOT cascade - these must be granted explicitly per project. See [Hierarchical Project Permissions](#hierarchical-project-permissions) for details. +- Project hierarchy is stored in the database and enforced through Keycloak Authorization policies + ### API Extensions This enhancement introduces new API endpoints following the fulfillment API pattern (`/api/fulfillment/v1/{resource}`). The APIs will be defined using Protocol Buffers in the fulfillment-api repository, following the same patterns as existing fulfillment services. **Authentication**: Authentication is handled natively by the OIDC Protocol and Keycloak's standard endpoints. The OSAC Console and CLI authenticate directly with Keycloak using standard OIDC Authorization Code flows. Users obtain tokens from Keycloak's standard endpoints (`/realms/{realm}/protocol/openid-connect/token`, `/realms/{realm}/protocol/openid-connect/userinfo`, `/realms/{realm}/protocol/openid-connect/certs`). OSAC does not provide custom authentication endpoints; all authentication flows use standard OIDC protocol endpoints provided by Keycloak. +**Project Context**: For Tenant APIs that operate on resources within Projects (e.g., ComputeInstances, VirtualNetworks), the project context is specified in the request body. The `organization` and `project` fields identify which Project the resource belongs to. Authorino extracts these fields, queries Keycloak Authorization Services to verify the user has the required permission on that Project resource, and then injects the verified context into HTTP headers for the Fulfillment Service. Example: +```json +POST /api/fulfillment/v1/compute_instances +{ + "organization": "acme", + "project": "web-app", + "name": "my-vm", + "image": "rhel-9", + ... +} +``` + **Cloud Provider Admin APIs** (`/api/fulfillment/v1`): - Organization CRUD operations: - `POST /api/fulfillment/v1/organizations` - Create Organization @@ -230,13 +292,17 @@ This enhancement introduces new API endpoints following the fulfillment API patt - `GET /api/fulfillment/v1/organizations/{name}/identity_provider/events` - Get IdP events (reads from Keycloak) - Project CRUD operations: - `POST /api/fulfillment/v1/organizations/{name}/projects` - Create a Project within the Organization (supports nested projects via `parent_project` field) - - `GET /api/fulfillment/v1/organizations/{name}/projects` - List Projects in the Organization (supports filtering by parent project) + - `GET /api/fulfillment/v1/organizations/{name}/projects` - List Projects in the Organization (supports filtering by parent project; returns only projects the user has VIEW_PROJECT permission on) - `GET /api/fulfillment/v1/organizations/{name}/projects/{name}` - Get Project details (includes parent project information for nested projects) - `PATCH /api/fulfillment/v1/organizations/{name}/projects/{name}` - Update Project metadata - `DELETE /api/fulfillment/v1/organizations/{name}/projects/{name}` - Delete Project (with proper teardown, including nested projects) - `GET /api/fulfillment/v1/organizations/{name}/projects/{parent_name}/projects` - List nested Projects under a parent Project -- Role assignment management (roles are pre-created in Keycloak realms): - - `GET /api/fulfillment/v1/organizations/{name}/roles` - List available roles in Keycloak realm +- Project permission management (permissions managed via Keycloak Authorization Services): + - `POST /api/fulfillment/v1/organizations/{name}/projects/{project_name}/permissions` - Grant permissions to users/groups on a Project + - `GET /api/fulfillment/v1/organizations/{name}/projects/{project_name}/permissions` - List permissions for a Project + - `DELETE /api/fulfillment/v1/organizations/{name}/projects/{project_name}/permissions/{permission_id}` - Revoke permissions +- Role assignment management (organization-level roles are pre-created in Keycloak realms): + - `GET /api/fulfillment/v1/organizations/{name}/roles` - List available organization-level roles in Keycloak realm - `GET /api/fulfillment/v1/organizations/{name}/roles/{name}` - Get role details - `POST /api/fulfillment/v1/organizations/{name}/groups/{group_identifier}/roles` - Assign roles to a group (group identifier from IdP; must be URL-escaped) - `GET /api/fulfillment/v1/organizations/{name}/groups` - List groups and their assigned roles (group information retrieved from IdP via Keycloak) @@ -245,24 +311,41 @@ This enhancement introduces new API endpoints following the fulfillment API patt ### Role Model and Scoping +OSAC uses a two-tier permission model: Organization-level roles (managed as Keycloak roles) and Project-level permissions (managed via Keycloak Authorization Services). + **System-Level Roles** (assigned only in `System` Organization): - `cloud-provider-admin`: Full control over all Organizations in OSAC - `cloud-provider-reader`: Read-only access to all Organizations **Organization-Level Roles** (assigned in tenant Organizations): -- `tenant-admin`: Full control within a single Organization -- `tenant-reader`: Read-only access within a single Organization -- `tenant-user`: Standard user access within a single Organization +- `tenant-admin`: Full control within a single Organization (including all Projects) +- `tenant-reader`: Read-only access within a single Organization (to all Projects the user has VIEW_PROJECT permission on) +- `tenant-user`: Standard user access within a single Organization (to Projects the user has been granted access to) **Break-Glass Role** (assigned to break-glass accounts): - `idp-manager`: Limited role that can manage IdP configuration (view, update, test, view health) and assign roles to groups/users in Keycloak for the organization. Cannot create/delete organizations, manage users (except for role assignment), create projects, or perform any other administrative actions. -**Role Assignment Constraints**: +**Project-Level Permissions** (managed via Keycloak Authorization Services): +- Project permissions provide fine-grained, resource-level access control for individual Projects +- Each Project is a Keycloak Authorization Resource with scoped permissions +- Example scopes include: + - `VIEW_PROJECT` - View project details and status + - `MANAGE_PROJECT` - Update project metadata, delete project, manage permissions + - `CREATE_COMPUTE_INSTANCE` - Create ComputeInstances in the project + - `DELETE_COMPUTE_INSTANCE` - Delete ComputeInstances in the project + - Additional scopes for other OSAC resource types (VirtualNetwork, Subnet, SecurityGroup, etc.) +- **Note**: The specific set of scopes and their granularity (e.g., per-resource-type vs. generic CRUD scopes) will be defined during implementation based on OSAC's resource model and operational requirements. The examples above illustrate the concept. +- Project permissions are assigned to users or groups independently of organization-level roles +- Users with `tenant-admin` role automatically have all permissions on all Projects in their Organization +- Users with `tenant-user` role have permissions only on Projects they've been explicitly granted access to + +**Role and Permission Assignment Constraints**: - System-level roles can only be assigned in the `System` Organization - Organization-level roles can only be assigned in tenant Organizations +- Project-level permissions can only be assigned within the context of a specific Project - Break-glass accounts have the `idp-manager` role and cannot be assigned other roles - Cloud Provider Admins (authenticated via IdP) and break-glass accounts can manage System Organization roles in Keycloak -- Tenant Admins and break-glass accounts can manage organization-level roles in Keycloak within their Organization +- Tenant Admins and break-glass accounts can manage organization-level roles and project-level permissions in Keycloak within their Organization - Authorization policies enforce these constraints to prevent privilege escalation ### Implementation Details/Notes/Constraints @@ -293,6 +376,38 @@ This enhancement introduces new API endpoints following the fulfillment API patt - Roles appear in token claims: `realm_access.roles` for realm roles, or `resource_access.{client}.roles` for client roles - No separate role mapping storage is needed - Keycloak is the source of truth for roles +**Keycloak Authorization Services for Project Permissions** +- Keycloak Authorization Services (UMA - User-Managed Access) provide fine-grained, resource-level permissions for Projects +- Each Project is modeled as a Keycloak Authorization Resource with a unique identifier (e.g., `PROJECT-acme-web-app`) +- Projects have a foreign key relationship to their parent Organization in the OSAC database +- Scoped permissions control what operations users can perform on specific Projects (examples): + - `VIEW_PROJECT` - List project details, view project status + - `MANAGE_PROJECT` - Update project metadata, delete project + - `CREATE_COMPUTE_INSTANCE` - Create ComputeInstances in the project + - `DELETE_COMPUTE_INSTANCE` - Delete ComputeInstances in the project + - Additional scopes as needed for other OSAC resource types +- The specific set of scopes will be defined during implementation +- Authorino validates Keycloak Authorization at request time by decoding RPT tokens locally (recommended for performance) or via UMA endpoints: "Can user X perform action Y on resource Z?" See the [Keycloak Authorization Services Integration](#keycloak-authorization-services-integration) section for details on the local RPT validation approach. +- Authorization policies can be defined in Keycloak to handle hierarchical permissions (see [Hierarchical Project Permissions](#hierarchical-project-permissions) for details on which permissions cascade from parent to child projects) +- Project visibility is controlled through permissions - users only see Projects they have at least `VIEW_PROJECT` permission on +- This enables true multi-project tenancy where different users have different access to different Projects within the same Organization +- No separate authorization storage is needed - Keycloak Authorization Services is the source of truth for project permissions + +**Project-Scoped Resource Access** +- All resource operations (list, create, update, delete) are scoped to a specific Project context +- Users specify the project in the API request (either in the request body or as a path/query parameter) +- Authorino validates that the user has the required permission on the specified Project before allowing the request to proceed +- The Fulfillment Service filters resources to only those within the specified Project +- **Implementation note**: Database queries must use efficient indexing on `project_id` and `organization_id` columns to ensure good performance when filtering resources +- This provides clear context and simplifies authorization - each request explicitly states which Project it operates on +- The OSAC Console uses a project switcher (similar to OpenShift) where users select their current project context +- The CLI requires explicit project specification via flags or configuration + +**Future Enhancement: Cross-Project Views** +- Cross-project resource listing (e.g., "show all VMs across all my accessible projects") is deferred to a future enhancement +- When implemented, this would require querying Keycloak Authorization Services to retrieve the list of Projects the user has access to, then aggregating resources across those Projects +- Initial implementation focuses on single-project scoping for simplicity and clarity + **OSAC Gateway (Envoy Proxy)** - Receives all incoming API requests at the infrastructure layer - Routes requests to appropriate Fulfillment Services based on Gateway API routing rules @@ -321,9 +436,17 @@ This enhancement introduces new API endpoints following the fulfillment API patt - Invoke Fulfillment/Infrastructure layer to realize changes **OpenShift Integration** -- Each Project within an Organization maps to an OpenShift Project (e.g., `osac-org--project-`) +- A single OSAC instance with a single API entry point and Keycloak installation can manage multiple OpenShift infrastructure clusters +- **Important**: This enhancement covers OSAC's infrastructure clusters where OSAC creates Projects for resource isolation. User-facing Cluster-as-a-Service (CaaS) authentication and management is out of scope for this enhancement and will be addressed separately. +- Each Project within an Organization maps to an OpenShift Project in every managed infrastructure cluster (e.g., `osac-org--project-`) - Project names are unique within an Organization, so the OpenShift Project name format is sufficient for all projects (including nested projects) +- Due to OpenShift's name length limits, the full project hierarchy is not encoded in the OpenShift Project name +- Parent-child relationships are stored in the database and represented on OpenShift Projects via owner references or annotations +- OpenShift Projects can be created on-demand when resources are first deployed to a cluster, avoiding N×M scaling issues with many projects and clusters +- Alternatively, OpenShift Projects can be pre-created in all managed clusters at Project creation time for simpler implementation +- If new clusters are added later, reconciliation creates OpenShift Projects as needed (either immediately or on-demand depending on implementation choice) - Organizations do not directly map to OpenShift Projects; they are logical containers for Projects +- OSAC's authentication to infrastructure clusters, credential management (kubeconfig storage), and ensuring continued API access are implementation details deferred to infrastructure management design #### Authentication Boundaries @@ -371,24 +494,32 @@ Break-glass account management uses Keycloak as the single source of truth. - OSAC reads IdP configuration from Keycloak via Admin REST API or from token claims - The Keycloak instance itself (installation, upgrade, backup, etc.) is managed by the user -#### Authorization Policy Extensibility +#### Authorization Policy Architecture + +Kuadrant's Authorino component provides multi-layered authorization capabilities: -Kuadrant's Authorino component provides extensible authorization capabilities to support evolving policy requirements: +**Organization-Level Authorization (RBAC):** +- Authorization policies use role-based access control (RBAC) based on OSAC organization-level roles from Keycloak tokens +- Policies evaluate user roles and organization context to permit or deny API requests +- Roles include `cloud-provider-admin`, `tenant-admin`, `tenant-reader`, `tenant-user`, `idp-manager` -**Initial Implementation (RBAC):** -- Authorization policies use role-based access control (RBAC) based on OSAC roles from Keycloak tokens -- Policies evaluate user roles, organization context, and project context to permit or deny API requests +**Project-Level Authorization (Keycloak Authorization Services):** +- Each Project is modeled as a Keycloak Authorization Resource +- Fine-grained scoped permissions control what operations users can perform on specific Projects +- Authorino queries Keycloak Authorization Services at request time to evaluate permissions +- Supports hierarchical permission policies where MANAGE_PROJECT and VIEW_PROJECT permissions automatically cascade from parent to child projects (see [Hierarchical Project Permissions](#hierarchical-project-permissions)) +- Enables true multi-project tenancy with per-project access control **Future Extensibility:** - **OPA Integration**: Authorino can integrate with Open Policy Agent (OPA) for advanced policy evaluation, complex authorization logic, and policy-as-code workflows -- **SpiceDB Integration**: Authorino can integrate with SpiceDB for Relationship-Based Access Control (ReBAC), enabling hierarchical permissions and relationship-based authorization decisions +- **SpiceDB Integration**: Authorino can integrate with SpiceDB for Relationship-Based Access Control (ReBAC), enabling more complex relationship-based authorization decisions beyond Keycloak's capabilities - **External Authorization Services**: Authorino supports external authorization services, enabling integration with: - Cost-management/billing systems to deny access based on quota exhaustion or billing status - Compliance systems for policy-based access control - Custom authorization logic as requirements evolve - **Rate Limiting**: Limitador provides rate limiting capabilities that can be configured per organization, project, user, or API endpoint -This extensibility ensures OSAC can adapt to future authorization requirements without requiring architectural changes. +This architecture provides comprehensive authorization from day one while ensuring OSAC can adapt to future requirements without architectural changes. ### Risks and Mitigations @@ -459,11 +590,74 @@ An alternative approach would be to create OpenShift Users for each OSAC user an The proposed approach maintains clear separation of concerns and allows OSAC to implement its own authorization model independent of OpenShift RBAC. -## Open Questions +## Deletion Lifecycle Semantics + +**Project Deletion:** +Following industry best practices from Kubernetes, OpenStack, and AWS, OSAC implements a safe deletion policy: + +1. **Default behavior: Block deletion if Project contains resources** + - DELETE request returns an error (e.g., 409 Conflict) with a list of resources that must be deleted first + - Prevents accidental data loss and orphaned resources + - User must explicitly delete all resources (ComputeInstances, VirtualNetworks, etc.) before deleting the Project + +2. **Optional cascade deletion with explicit confirmation** + - Add `cascade=true` query parameter to DELETE request + - Requires explicit user acknowledgment in API call or CLI flag + - OSAC deletes all resources within the Project in dependency order, then deletes the Project itself + - Returns a detailed list of deleted resources + +3. **Automatic cleanup of metadata** + - Keycloak Authorization Resources are automatically deleted when Project is deleted (these are just permission metadata, not infrastructure) + - OpenShift Projects are reconciled for deletion across all managed clusters + - Database records are cleaned up in a transaction + +4. **Nested project handling** + - Parent projects cannot be deleted if they contain child projects + - User must delete child projects first (or use cascade deletion) + - Cascade deletion on a parent project deletes all descendants in tree order + +**Organization Deletion:** +Similar semantics apply to Organization deletion: +1. Block deletion if Organization contains Projects (default) +2. Optional cascade deletion with `cascade=true` deletes all Projects and their resources +3. Automatic cleanup: Keycloak realm, break-glass accounts, and Authorization Resources + +## Hierarchical Project Permissions + +Parent project administrators automatically receive permissions on child projects via Keycloak Authorization policies. This matches typical organizational hierarchies and simplifies management: + +- Users with `MANAGE_PROJECT` permission on a parent project automatically have `MANAGE_PROJECT` on all child projects +- Users with `VIEW_PROJECT` on a parent can view all child projects +- Resource-specific permissions (e.g., `CREATE_COMPUTE_INSTANCE`) on parent projects do NOT automatically cascade to children - these must be granted explicitly per project +- This policy is implemented via Keycloak Authorization policies and is consistent across all Organizations + +## Keycloak Authorization Services Integration + +Authorino integrates with Keycloak Authorization Services using a hybrid approach that optimizes for performance: + +**Token Validation Approach:** +- **RPT tokens**: Authorino performs full JWT validation by verifying the JWS signature against Keycloak's JWKS endpoint, validating mandatory claims (iss, aud, exp, nbf), and performing time checks (exp > now and nbf ≤ now). After JWT validation succeeds, Authorino decodes the token locally and checks the `permissions` claim to evaluate authorization. No remote introspection call is needed - validation is fast and local. +- **Standard Keycloak tokens**: When a user presents a regular Keycloak access token (not an RPT), Authorino negotiates a User-Managed Access (UMA) ticket with Keycloak and exchanges it for an RPT containing the user's permissions. + +**Performance and Caching Strategy:** +- **Client-side caching**: Once Authorino obtains an RPT, it returns the RPT to the API consumer for use in subsequent requests. This skips new UMA ticket negotiations and enables fast local validation. +- **RPT validity**: RPTs are short-lived tokens (typically 5-15 minutes, configurable in Keycloak). Clients should cache and reuse RPTs until expiration. +- **Server-side caching**: Keycloak's internal caching (resource cache, policy cache) significantly impacts UMA ticket negotiation performance. Proper Keycloak cache configuration is critical for production deployments. +- **Initial negotiation cost**: The first request using a standard token incurs UMA ticket negotiation latency (can be 100ms-2000ms depending on the number of resources and policies). Subsequent requests with the RPT are fast (local JWT validation). + +**API Calls to Keycloak:** +1. Protection Resource API - lookup resource ID from request path +2. Permission endpoint - negotiate UMA ticket for user's requested resource/scope +3. Token endpoint - exchange UMA ticket for RPT containing permissions +4. No token introspection calls needed - RPT validation is local + +**Recommendation**: Use local RPT validation (not token introspection) as documented in [Authorino's Keycloak Authorization Services guide](https://docs.kuadrant.io/1.2.x/authorino/docs/user-guides/keycloak-authorization-services/). Ensure Keycloak is properly configured with appropriate cache sizes and high availability for production workloads. + +## Future Enhancements -1. IdP health monitoring and alerting implementation details are deferred to a future improvement. Should this be a separate enhancement or part of a monitoring enhancement? +1. **IdP health monitoring and alerting**: Implementation details for proactive IdP health monitoring, alerting, and failover mechanisms will be addressed in a future monitoring-focused enhancement. -2. Organization and Project lifecycle semantics (especially deletion) need further definition. What happens to resources when an organization or project is deleted? What is the teardown process? Should project deletion be blocked if it contains resources? +2. **Cross-project resource views**: The initial implementation requires users to specify a single Project context for all resource operations. Future enhancements may add cross-project listing capabilities (e.g., "show all VMs across all my accessible projects", `--all-projects` flag in CLI) for dashboard and monitoring use cases. ## Test Plan From 68021e9efeb92a6c38ac107886a86c9b91934322 Mon Sep 17 00:00:00 2001 From: Akshay Nadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 17 Jun 2026 08:14:08 -0400 Subject: [PATCH 022/450] Design: Rework Tenant Storage Onboarding (OSAC-23) (#58) * OSAC-23: add design document for tenant storage onboarding Introduces a dedicated OSAC Storage Controller that extracts all storage provisioning logic from the Tenant controller. The controller manages a two-stage onboarding workflow (backend setup, then cluster-side StorageClass installation) and a two-step ordered teardown, using condition ownership on the Tenant CR. Key design decisions: - Two conditions on Tenant CR: StorageBackendReady, ClusterStorageReady - Two ProvisioningProvider instances (backend + cluster-storage) - Tenant Phase=Ready decoupled from storage readiness - Four AAP playbook lifecycle actions - Per-backend and per-cluster status detail structs Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> * OSAC-23: use annotation-based ClusterOrder watch mapping ClusterOrder and Tenant live in different namespaces, so ownerReferences cannot be used. Replace EnqueueRequestForOwner with EnqueueRequestsFromMapFunc that reads the osac.openshift.io/tenant annotation from ClusterOrder to enqueue the correct Tenant. This is consistent with how ComputeInstance and all other OSAC resources associate with Tenants via the same annotation, set by the fulfillment-service at CR creation time. Also incorporates review feedback: management state scoping (FR-4 steady-state only), label-based hub Secret lookup, no-auto-retry on failure, operator-first deployment order, priority=1 print columns, per-tenant management-state in support procedures, and tier addition workflows as non-goal. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> * OSAC-23: add extension points for multi-backend and tier source Frame the reconciliation loop around two pluggable sources (backend list and tier source) so the controller logic can evolve from single VAST backend to multiple backends (OSAC-1111) and from STORAGE_TIERS env var to StorageTier CRs (OSAC-1110) without rewriting the core loop or changing the status schema. Clarify that v0.1 storageBackends and clusterStorage lists each have a single entry, with the list structures anticipating multi-backend and multi-cluster support. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> * OSAC-23: update last-updated date to 2026-06-15 Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> * OSAC-23: clarify implicit stage derivation in reconciliation steps Add parenthetical notes to make the stage sequencing explicit for code reviewers: Secret absence indicates Stage 1 has not run, StorageBackendReady=True indicates Stage 1 is complete. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> * OSAC-23: tighten design document prose Remove source markers, cut redundant sentences, shorten failure table, simplify alternatives and boilerplate sections, remove inline Go comments from self-documenting structs. Preserves all technical content in a more concise form. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> * OSAC-23: rename design document from README.md to design.md Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --------- Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- .../design.md | 530 ++++++++++++++++++ 1 file changed, 530 insertions(+) create mode 100644 enhancements/tenant-storage-onboarding-OSAC-23/design.md diff --git a/enhancements/tenant-storage-onboarding-OSAC-23/design.md b/enhancements/tenant-storage-onboarding-OSAC-23/design.md new file mode 100644 index 000000000..c2d21c9bc --- /dev/null +++ b/enhancements/tenant-storage-onboarding-OSAC-23/design.md @@ -0,0 +1,530 @@ +--- +title: rework-tenant-storage-onboarding +authors: + - zszabo@redhat.com + - anadkarn@redhat.com +creation-date: 2026-06-12 +last-updated: 2026-06-15 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-23 +prd: + - "prd.md" +see-also: + - "/enhancements/tenant-storage-onboarding-OSAC-23/prd.md" +replaces: + - N/A +superseded-by: + - N/A +--- + +# Rework Tenant Storage Onboarding + +## Summary + +Extract all storage provisioning logic from the Tenant controller into a dedicated OSAC Storage Controller. The new controller owns storage conditions and status on the Tenant CR, manages a two-stage onboarding workflow (backend setup, then cluster-side StorageClass installation), and performs ordered two-step teardown. Four independent AAP playbooks replace the current combined playbooks. See [PRD](prd.md) for detailed requirements. + +## Motivation + +Storage provisioning is embedded in the Tenant controller alongside namespace creation, UDN reconciliation, and tenant lifecycle management. The controller handles StorageClass resolution, AAP job triggering, job polling, and the `StorageClassReady` condition. Any change to storage risks breaking tenant state transitions, and supporting different workflows per delivery model (VMaaS, CaaS, BMaaS) requires branching logic inside the Tenant controller. + +The AAP integration compounds the coupling: a single playbook executes both backend setup and cluster-side StorageClass creation in sequence, making it impossible to trigger or retry each stage independently. CaaS clusters need Stage 2 to run after ClusterOrder reaches Ready, which requires independent stage control. + +This design moves storage into a dedicated controller. The Tenant controller becomes namespace and UDN only, with `Phase=Ready` based solely on namespace readiness. Storage readiness is tracked via two new conditions (`StorageBackendReady` and `ClusterStorageReady`) owned by the storage controller. + +### Goals + +- Reuse the existing reconciliation pattern and `ProvisioningProvider` interface without modifying `pkg/provisioning/`. +- Keep Stage 1 and Stage 2 as independent operations so CaaS support can be added without modifying the storage controller's core logic. +- Maintain backward compatibility for the ComputeInstance controller, which reads `status.storageClasses` from the Tenant CR. +- Support per-backend and per-cluster status detail for observability via `kubectl get tenant -o wide`. + +### Non-Goals + +- StorageBackend API and registration automation (OSAC-1111). +- StorageTier API (OSAC-1110). +- Tier addition and removal workflows. +- CaaS Stage 2 trigger logic (OSAC-1123, separate PRD). +- VAST support for CaaS (OSAC-1122). +- Per-cluster storage resource on target clusters for tenant admin visibility. +- Fulfillment-service API or proto changes for storage state visibility. + +## Proposal + +Three changes: + +1. **New OSAC Storage Controller** watches Tenant CRs and manages the full storage lifecycle. It uses two `ProvisioningProvider` instances (backend and cluster-storage) and sets two conditions on the Tenant CR: `StorageBackendReady` and `ClusterStorageReady`. + +2. **Tenant controller stripped of storage logic.** It retains namespace creation, UDN reconciliation, and `NamespaceReady`. `Phase=Ready` means namespace is ready; storage conditions are not consulted. + +3. **AAP playbooks split** from two combined playbooks into four lifecycle actions: `osac-create-tenant-storage-backend`, `osac-create-tenant-cluster-storage`, `osac-delete-tenant-cluster-storage`, `osac-delete-tenant-storage-backend`. + +### Workflow Description + +#### Actors + +- **Platform Admin**: Configures storage backends, deploys osac-operator and osac-aap, creates Tenant CRs. +- **Storage Controller**: Reconciles storage state on Tenant CRs. +- **AAP**: Executes storage provisioning and teardown playbooks. + +#### Onboarding Workflow + +Starting state: Tenant CR exists, Tenant controller has set `Phase=Ready` (namespace exists, UDN reconciled). + +```mermaid +sequenceDiagram + participant TC as Tenant Controller + participant SC as Storage Controller + participant AAP as AAP + participant Hub as Hub Cluster + participant Target as VMaaS Target + + TC->>TC: Create namespace, reconcile UDN + TC->>TC: Set Phase=Ready, NamespaceReady=True + + SC->>SC: Watch detects Tenant with Phase=Ready + SC->>Hub: Check hub Secret by label (osac.openshift.io/tenant) + alt Hub Secret missing + SC->>AAP: Trigger osac-create-tenant-storage-backend + AAP->>Hub: Create VAST tenant, views, quotas, manager user + AAP->>Hub: Write hub Secret with per-tenant credentials + SC->>SC: Poll job until Succeeded + end + SC->>SC: Set StorageBackendReady=True + + SC->>AAP: Trigger osac-create-tenant-cluster-storage + AAP->>Target: Install CSI operator (if needed) + AAP->>Target: Create per-tenant StorageClasses and CSI Secret + SC->>SC: Poll job until Succeeded + SC->>Target: Discover StorageClasses by osac.openshift.io/tenant label + SC->>SC: Populate status.storageClasses with resolved list + SC->>SC: Set ClusterStorageReady=True +``` + +On detecting a ready Tenant, the storage controller checks for a hub Secret in `OSAC_STORAGE_CONFIG_NAMESPACE` with label `osac.openshift.io/tenant=`. If absent, it triggers `osac-create-tenant-storage-backend` and polls until the job completes. On success, it sets `StorageBackendReady=True`. + +After Stage 1, it triggers `osac-create-tenant-cluster-storage` on the VMaaS target cluster, polls until completion, discovers installed StorageClasses by label, populates `status.storageClasses`, and sets `ClusterStorageReady=True`. + +#### Teardown Workflow + +Starting state: Tenant CR is being deleted (DeletionTimestamp set). + +```mermaid +sequenceDiagram + participant SC as Storage Controller + participant AAP as AAP + participant Target as VMaaS Target + participant Hub as Hub Cluster + + SC->>SC: Finalizer blocks deletion + SC->>AAP: Trigger osac-delete-tenant-cluster-storage + AAP->>Target: Delete StorageClasses, VolumeSnapshotClasses, CSI Secrets + SC->>SC: Poll job until Succeeded + + SC->>AAP: Trigger osac-delete-tenant-storage-backend + AAP->>Hub: Delete VAST tenant, views, quotas, manager user + AAP->>Hub: Delete hub Secret + SC->>SC: Poll job until Succeeded + + SC->>SC: Remove storage finalizer +``` + +The storage controller places its own finalizer (`osac.openshift.io/storage`) on the Tenant CR. Teardown runs cluster-side cleanup first, then backend teardown. The finalizer is removed only after both steps succeed. If either fails, the Tenant stays in Terminating until the issue is resolved. + +#### Error Handling + +- **Stage 1 or 2 AAP job failure**: The relevant condition is set to `False` with a reason reflecting the failure. The controller does not auto-retry; it waits for an external trigger (Tenant CR update, Secret or StorageClass watch event) to avoid infinite retry loops. This matches the pattern used by all other OSAC controllers. +- **Teardown failure**: The finalizer remains. The controller retries on the next reconciliation trigger. The Tenant stays in Terminating and `kubectl get tenant` shows the failure reason. + +### API Extensions + +The Tenant CRD gains new conditions (`StorageBackendReady`, `ClusterStorageReady`), new status fields (`storageBackends`, `clusterStorage`), and a new finalizer (`osac.openshift.io/storage`). The existing `StorageClassReady` condition is removed. No new CRDs. + +The storage finalizer coexists with the existing `osac.openshift.io/tenant` finalizer. Each controller manages its own independently. + +If the storage controller is down, Tenants still reach `Phase=Ready` (namespace + UDN are independent). Storage conditions remain stale until the controller recovers. Tenant deletion is blocked if the storage finalizer is present. + +### Implementation Details/Notes/Constraints + +#### Tenant CRD Type Changes + +New status structs for per-backend and per-cluster detail: + +```go +type StorageBackendStatus struct { + Name string `json:"name"` + Provider string `json:"provider"` + Ready bool `json:"ready"` + Message string `json:"message,omitempty"` +} + +type ClusterStorageStatus struct { + ClusterName string `json:"clusterName"` + Ready bool `json:"ready"` + Reason string `json:"reason,omitempty"` +} +``` + +For v0.1, `storageBackends` has one entry (one VAST appliance) and `clusterStorage` has one entry (VMaaS target). The list structures anticipate multi-backend (OSAC-1111) and multi-cluster (OSAC-1123) support without schema changes. + +Updated TenantStatus: + +```go +type TenantStatus struct { + Phase TenantPhaseType `json:"phase,omitempty"` + Namespace string `json:"namespace,omitempty"` + StorageClasses []ResolvedStorageClass `json:"storageClasses,omitempty"` + Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"` + Jobs []JobStatus `json:"jobs,omitempty"` + + // +listType=map + // +listMapKey=name + StorageBackends []StorageBackendStatus `json:"storageBackends,omitempty"` + + // +listType=map + // +listMapKey=clusterName + ClusterStorage []ClusterStorageStatus `json:"clusterStorage,omitempty"` +} +``` + +New condition types and reasons: + +```go +const ( + TenantConditionStorageBackendReady TenantConditionType = "StorageBackendReady" + TenantConditionClusterStorageReady TenantConditionType = "ClusterStorageReady" +) + +const ( + TenantReasonProvisioning = "Provisioning" + TenantReasonProvisionFailed = "ProvisionFailed" + TenantReasonDeprovisioning = "Deprovisioning" + TenantReasonDeprovisionFailed = "DeprovisionFailed" + TenantReasonSecretNotFound = "SecretNotFound" + TenantReasonTenantNotReady = "TenantNotReady" +) +``` + +The existing `TenantConditionStorageClassReady` is removed. `TenantConditionNamespaceReady` is unchanged. + +Print columns: + +```go +// +kubebuilder:printcolumn:name="Tenant Namespace",type=string,JSONPath=`.status.namespace` +// +kubebuilder:printcolumn:name="Storage Classes",type=string,JSONPath=`.status.storageClasses[*].name` +// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase` +// +kubebuilder:printcolumn:name="Backend Ready",type=string,JSONPath=`.status.conditions[?(@.type=="StorageBackendReady")].status`,priority=1 +// +kubebuilder:printcolumn:name="Cluster Storage",type=string,JSONPath=`.status.conditions[?(@.type=="ClusterStorageReady")].status`,priority=1 +``` + +Storage conditions use `priority=1` (visible with `-o wide`), keeping the default output compact. + +#### Storage Controller Architecture + +New controller in `internal/controller/storage_controller.go` with two `ProvisioningProvider` instances: + +- `backendProvider`: templates `osac-create-tenant-storage-backend` / `osac-delete-tenant-storage-backend` +- `clusterStorageProvider`: templates `osac-create-tenant-cluster-storage` / `osac-delete-tenant-cluster-storage` + +```go +type StorageReconciler struct { + client.Client + Scheme *runtime.Scheme + Recorder events.EventRecorder + mgr mcmanager.Manager + targetCluster mc.ClusterName + storageNamespace string + BackendProvider provisioning.ProvisioningProvider + ClusterStorageProvider provisioning.ProvisioningProvider + StatusPollInterval time.Duration + MaxJobHistory int +} +``` + +Env vars for AAP templates: `OSAC_STORAGE_BACKEND_AAP_PROVISION_TEMPLATE`, `OSAC_STORAGE_BACKEND_AAP_DEPROVISION_TEMPLATE`, `OSAC_STORAGE_CLUSTER_AAP_PROVISION_TEMPLATE`, `OSAC_STORAGE_CLUSTER_AAP_DEPROVISION_TEMPLATE`. + +#### Reconciliation State Machine + +```mermaid +stateDiagram-v2 + [*] --> WaitForTenant: Tenant exists + WaitForTenant --> Stage1: Tenant Phase=Ready + WaitForTenant --> WaitForTenant: Tenant not Ready + + Stage1 --> Stage1Polling: Backend AAP job triggered + Stage1 --> Stage2: Hub Secret exists (Stage 1 already done) + Stage1Polling --> Stage1: Job failed (wait for external trigger) + Stage1Polling --> Stage2: Job succeeded + + Stage2 --> Stage2Polling: Cluster storage AAP job triggered + Stage2Polling --> Stage2: Job failed (wait for external trigger) + Stage2Polling --> StorageReady: Job succeeded + SCs discovered + + StorageReady --> [*]: Steady state + + WaitForTenant --> Teardown: DeletionTimestamp set + Stage1 --> Teardown: DeletionTimestamp set + Stage2 --> Teardown: DeletionTimestamp set + StorageReady --> Teardown: DeletionTimestamp set + + Teardown --> TeardownCluster: Delete cluster storage + TeardownCluster --> TeardownBackend: Cluster cleanup done + TeardownBackend --> [*]: Backend teardown done, remove finalizer +``` + +The controller derives its current stage from conditions and hub Secret state (level-triggered): + +1. **Tenant not Ready**: Both conditions set to `False` with reason `TenantNotReady`. Requeue. +2. **Stage 1 (Backend)**: Iterate over the backend list. For each backend, query Secrets by `osac.openshift.io/tenant=` label. If no Secret exists (Stage 1 has not run for that backend), trigger `backendProvider.TriggerProvision()` and poll. On success, update `status.storageBackends`. Set `StorageBackendReady=True` when all backends are ready. +3. **Stage 2 (Cluster Storage)**: If `StorageBackendReady=True` (Stage 1 complete), iterate over the cluster list. For each cluster, trigger `clusterStorageProvider.TriggerProvision()` and poll. Discover StorageClasses using the tier source. On success, update `status.clusterStorage` and `status.storageClasses`. Set `ClusterStorageReady=True` when all clusters are ready. +4. **Deletion**: Ordered teardown per cluster then per backend. Remove finalizer only after all complete. + +#### Extension Points + +The reconciliation is structured around two pluggable sources: + +**Backend list:** Which backends to provision for a tenant. For v0.1, a single implicit backend from the AAP `storage-operations-ig` configuration. When the StorageBackend API (OSAC-1111) is available, this becomes a query against StorageBackend resources. The iteration loop and `status.storageBackends` array stay the same; only the list source changes. + +**Tier source:** Which tiers to expect during StorageClass resolution. For v0.1, tiers come from the `STORAGE_TIERS` environment variable. When the StorageTier API (OSAC-1110) is available, this becomes a query against StorageTier CRs. The resolution algorithm and `status.storageClasses` array stay the same; only the tier source changes. + +#### Tier Resolution Algorithm + +Moved from the Tenant controller without modification: + +1. List StorageClasses on the target cluster with label `osac.openshift.io/tenant={tenantName}`. +2. List StorageClasses with label `osac.openshift.io/tenant=Default`. +3. Group by `osac.openshift.io/storage-tier` label. +4. Per tier: use tenant-specific SC if exactly one exists; fall back to shared Default if exactly one exists; emit warning event on duplicates. +5. Populate `status.storageClasses` with the resolved list. + +Tier label validation (`^[a-z0-9]([a-z0-9._-]*[a-z0-9])?$`) is preserved. + +#### Tenant Controller Simplification + +**Removed:** `getTenantStorageClasses()`, `groupByTier()`, `handleStorageProvisioning()`, `pollProvisionJob()`, `needsProvisionJob()`, `StorageClassReady` condition, `status.StorageClasses` population, StorageClass watch, `ProvisioningProvider` field, `storagev1` import. + +**Retained:** Namespace creation/verification, UDN reconciliation, `NamespaceReady` condition, `osac.openshift.io/tenant` finalizer, phase management. + +`Phase=Ready` is set when `NamespaceReady=True`. Storage conditions are not consulted. + +#### Controller Registration + +```go +func setupStorageController(mgr mcmanager.Manager, maxJobHistory int) error { + backendProvider, backendPollInterval, err := createAAPProviderFromEnv( + "OSAC_STORAGE_BACKEND_AAP_PROVISION_TEMPLATE", + "OSAC_STORAGE_BACKEND_AAP_DEPROVISION_TEMPLATE", + ) + if err != nil { + return fmt.Errorf("backend provider: %w", err) + } + + clusterStorageProvider, clusterPollInterval, err := createAAPProviderFromEnv( + "OSAC_STORAGE_CLUSTER_AAP_PROVISION_TEMPLATE", + "OSAC_STORAGE_CLUSTER_AAP_DEPROVISION_TEMPLATE", + ) + if err != nil { + return fmt.Errorf("cluster storage provider: %w", err) + } + + return NewStorageReconciler( + mgr, + storageNamespace, + targetCluster, + backendProvider, + clusterStorageProvider, + max(backendPollInterval, clusterPollInterval), + maxJobHistory, + ).SetupWithManager(mgr) +} +``` + +Gated by `OSAC_ENABLE_STORAGE_CONTROLLER` / `--enable-storage-controller`. + +#### AAP Playbook Split + +| Current Playbook | New Playbooks | Dispatcher Action | +|---|---|---| +| `playbook_osac_configure_tenant_storage.yml` | `playbook_osac_create_tenant_storage_backend.yml` | `setup` | +| | `playbook_osac_create_tenant_cluster_storage.yml` | `ensure_storage_class` | +| `playbook_osac_delete_tenant_storage.yml` | `playbook_osac_delete_tenant_cluster_storage.yml` | `teardown_cluster_storage` | +| | `playbook_osac_delete_tenant_storage_backend.yml` | `teardown_backend` | + +The dispatcher role already supports `setup` and `ensure_storage_class`. For teardown, the existing `teardown` action is replaced with `teardown_cluster_storage` and `teardown_backend`. PR #338 (OSAC-1145) covers this split. + +#### ClusterOrder Watch + +The storage controller watches ClusterOrder resources to prepare for CaaS support (FR-9). No reconciliation action is taken on ClusterOrder events in this scope, but the watch and tenant mapping are wired now. + +ClusterOrders carry an `osac.openshift.io/tenant` annotation set by the fulfillment-service at creation time, the same annotation used by ComputeInstance and all other OSAC resources. Since ClusterOrders and Tenants live in different namespaces, `ownerReferences` cannot be used. The watch uses `EnqueueRequestsFromMapFunc` to map ClusterOrder events to the owning Tenant: + +```go +func (r *StorageReconciler) SetupWithManager(mgr mcmanager.Manager) error { + return mcbuilder.ControllerManagedBy(mgr). + For(&v1alpha1.Tenant{}, /* ... */). + Watches(&v1alpha1.ClusterOrder{}, + handler.EnqueueRequestsFromMapFunc(r.mapClusterOrderToTenant)). + Watches(&storagev1.StorageClass{}, /* ... enqueue Tenant by label */). + Complete(r) +} + +func (r *StorageReconciler) mapClusterOrderToTenant(_ context.Context, obj client.Object) []reconcile.Request { + tenantName := obj.GetAnnotations()[osacTenantAnnotation] + if tenantName == "" { + return nil + } + return []reconcile.Request{{ + NamespacedName: types.NamespacedName{ + Name: tenantName, + Namespace: r.tenantNamespace, + }, + }} +} +``` + +When OSAC-1123 adds CaaS Stage 2 trigger logic, only the reconcile handler needs to change; the watch plumbing is already in place. + +### Security Considerations + +The storage controller inherits the existing OSAC security model. + +**Credentials** are handled in three tiers, none of which the storage controller accesses directly: +- Admin credentials (VAST endpoint, username, password) live in the `storage-operations-ig` Secret, mounted as env vars in the AAP pod, cleared from playbook memory after use. +- Per-tenant credentials live in hub Secrets (`vast-tenant-config-{tenant_name}`) in `OSAC_STORAGE_CONFIG_NAMESPACE`. The storage controller checks their existence (Stage 1 gate) but never reads their contents. +- CSI credentials (`vast-csi-{tenant_name}`) are created by AAP in the tenant namespace with per-tenant manager credentials (not admin credentials). + +The fulfillment-service has no role in storage credentials. It only sets the `osac.openshift.io/tenant` annotation when creating CRs. + +**Logging**: the storage controller must not log Secret contents, credential values, or sensitive AAP parameters. Log messages reference Secret names and job IDs only. Kubernetes events must not include credential values. + +**Tenant isolation**: StorageClass discovery uses label-based filtering (`osac.openshift.io/tenant={tenantName}`). OPA policies continue to enforce isolation at runtime. + +### Failure Handling and Recovery + +| Failure Mode | Behavior | User Sees | Recovery | +|---|---|---|---| +| Stage 1 AAP fails | `StorageBackendReady=False`, waits for external trigger | `Backend Ready: False` in `-o wide` | Fix AAP issue, touch Tenant CR or wait for Secret event | +| Stage 2 AAP fails | `ClusterStorageReady=False`, waits for external trigger | `Cluster Storage: False` | Fix AAP/cluster issue, touch Tenant CR or wait for SC event | +| Hub Secret deleted | Secret watch triggers re-reconcile, `StorageBackendReady=False` | Transient `False` | Automatic: Stage 1 re-runs | +| StorageClasses deleted | SC watch triggers re-reconcile, `ClusterStorageReady=False` | `Cluster Storage: False` | Automatic: Stage 2 re-runs | +| Teardown fails | Finalizer remains, Tenant stuck in `Terminating` | Events show failure | Fix issue, controller retries on next reconcile | +| Storage controller down | Conditions stale, Tenant lifecycle unaffected | Stale storage conditions | Restart controller, it reconciles all Tenants | +| Status conflict | controller-runtime retries automatically | No impact | Automatic (optimistic concurrency) | + +All operations are idempotent. AAP playbooks check for existing resources before creating and handle already-deleted resources gracefully. On restart, the controller reads current state (conditions, Secrets, StorageClasses) and resumes from the appropriate stage. + +### RBAC / Tenancy + +Storage controller RBAC on hub cluster: + +```go +// +kubebuilder:rbac:groups=osac.openshift.io,resources=tenants,verbs=get;list;watch;update;patch +// +kubebuilder:rbac:groups=osac.openshift.io,resources=tenants/status,verbs=get;update;patch +// +kubebuilder:rbac:groups=osac.openshift.io,resources=tenants/finalizers,verbs=update +// +kubebuilder:rbac:groups=osac.openshift.io,resources=clusterorders,verbs=get;list;watch +// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch +// +kubebuilder:rbac:groups=events.k8s.io,resources=events,verbs=create;patch +``` + +On the target cluster: `storage.k8s.io/storageclasses: get;list;watch`. + +The Tenant controller's StorageClass RBAC is removed. No tenancy model changes. + +### Observability and Monitoring + +**Events (Normal):** `StorageBackendProvisioned`, `ClusterStorageProvisioned`, `StorageTeardownComplete`. + +**Events (Warning):** `StorageProvisionFailed` (includes AAP job ID and error), `StorageTeardownFailed`, `DuplicateStorageClass` (preserved from current controller). + +**Logs:** Structured key-value pairs via `ctrllog` at each stage transition. No credentials in logs. + +**Metrics:** No new metrics. Existing controller-runtime metrics (reconciliation duration, queue depth, errors) apply automatically. + +### Risks and Mitigations + +| Risk | Impact | Mitigation | +|---|---|---| +| Two controllers writing Tenant status concurrently | Low (transient delays) | Optimistic concurrency + DeepCopy-compare-update pattern | +| Operator and AAP deployed out of sync | High (storage provisioning breaks) | Operator-first deployment order; failures are transient and self-heal once AAP catches up | + +### Drawbacks + +The main cost is operational: two controllers manage the same CR's status, requiring understanding of which controller owns which conditions. This is justified because the storage lifecycle is genuinely independent (different failure modes, different AAP integrations), and embedding CaaS support in the Tenant controller would require branching logic that couples delivery model decisions to tenant lifecycle. Condition ownership is a well-established Kubernetes pattern. + +## Alternatives (Not Implemented) + +### TenantStorage CRD + +Separate CRD for per-tenant storage state. Clean ownership boundary, but adds a resource that must be created, watched, and garbage-collected. ComputeInstance would need to read StorageClasses from TenantStorage instead of Tenant, breaking the existing API. Rejected during PRD review in favor of condition ownership. + +### Single ProvisioningProvider with Template Switching + +Extend the `ProvisioningProvider` interface to accept a template name on each call. Simpler configuration, but breaks the interface contract used by all other controllers. The two-provider approach keeps each provider independently configurable. + +### Keep Storage Logic in Tenant Controller + +Add feature flags for CaaS vs VMaaS. No new controller, but CaaS requires ClusterOrder events to trigger the Tenant controller, coupling cluster provisioning to tenant lifecycle. Storage issues cannot be diagnosed independently. + +## Test Plan + +### Unit Tests + +Using mock `ProvisioningProvider`, following existing controller test patterns: + +- **Stage 1:** Tenant not ready (skip), no hub Secret (trigger provision), Secret exists (skip to Stage 2), job failure (condition False). +- **Stage 2:** Stage 1 incomplete (wait), Stage 1 complete + no SCs (trigger provision), SCs discovered (condition True), job failure. +- **Tier resolution:** Tenant-specific SC, Default fallback, duplicate detection, missing tier. +- **Teardown:** Ordered two-step, partial failure, finalizer removal only after both complete. +- **Management state:** `Unmanaged` skips steady-state reconciliation; finalizer and deletion handling still run (FR-4). +- **Tenant controller:** Phase=Ready with NamespaceReady only, no storage logic. + +### Integration Tests + +envtest with both controllers running simultaneously: + +- Condition updates propagate correctly. +- Finalizer coexistence (tenant + storage). +- Status conflict resolution. + +### E2E Tests + +Full lifecycle against a VAST appliance: + +- Tenant creation triggers Stage 1 then Stage 2. +- Tenant deletion triggers ordered teardown. +- StorageClasses usable by ComputeInstance. + +## Graduation Criteria + +To be defined when targeting a release. Expected: Dev Preview, Tech Preview, GA. + +## Upgrade / Downgrade Strategy + +Pre-GA change, no migration needed. `StorageClassReady` is removed and replaced by `StorageBackendReady` and `ClusterStorageReady`. + +**Deployment order:** Operator first, then AAP. If the operator is deployed before AAP, the storage controller calls templates that don't exist yet; AAP returns "template not found", the controller records a Failed job, and retries once AAP is updated. If AAP is deployed first, the old Tenant controller calls the old template names which no longer exist, breaking storage for all tenants. Operator-first is safer because failures are transient and self-heal. + +## Version Skew Strategy + +During upgrade, the old Tenant controller writes `StorageClassReady` while the new storage controller writes `StorageBackendReady`/`ClusterStorageReady`. No conflict. The orphaned `StorageClassReady` condition can be cleaned up manually after the upgrade. + +No fulfillment-service version skew concern (the storage controller does not use the gRPC API). + +## Support Procedures + +**Detecting failures:** +- `kubectl get tenant -o wide` shows `Backend Ready` and `Cluster Storage` columns. +- `kubectl describe tenant ` shows conditions with reasons and AAP job IDs. +- Controller logs: `kubectl logs -n osac-system deployment/osac-operator-controller-manager | grep "storage"`. + +**Disabling the storage controller:** +- Set `OSAC_ENABLE_STORAGE_CONTROLLER=false`. New tenants won't get storage. Tenant deletion is blocked if the storage finalizer is present (remove manually). + +**Disabling storage for a single tenant:** +- Set `osac.openshift.io/management-state: Unmanaged` on the Tenant CR. Steady-state reconciliation is skipped; finalizer and deletion handling still run so the Tenant can be deleted cleanly. + +**Re-enabling:** +- Remove the flag or annotation. The controller reconciles all Tenants on next run. All operations are idempotent. + +## Infrastructure Needed + +None. From bf58dc33f4940796d7de72a4229b8ba3f8c8215a Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Tue, 16 Jun 2026 15:54:39 +0300 Subject: [PATCH 023/450] OSAC-1111: Add StorageBackend API design document Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 429 ++++++++++++++++++ 1 file changed, 429 insertions(+) create mode 100644 enhancements/storage-backend-osac-1111/design.md diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md new file mode 100644 index 000000000..d1d0b3517 --- /dev/null +++ b/enhancements/storage-backend-osac-1111/design.md @@ -0,0 +1,429 @@ +--- +title: storage-backend-api +authors: + - rgolan@redhat.com +creation-date: 2026-06-15 +last-updated: 2026-06-15 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-1111 +prd: + - "prd.md" +see-also: + - "/enhancements/storage-tier (OSAC-1110, future)" +replaces: + - N/A +superseded-by: + - N/A +--- + +# StorageBackend API + +## Summary + +This enhancement adds a `StorageBackend` entity to the fulfillment-service, providing a private gRPC API (CRUD) and a public read-only API for managing storage array registrations. StorageBackend follows the NetworkClass pattern: DB-backed with no Kubernetes CRD and no reconciler. See [PRD](prd.md) for detailed requirements. + +## Motivation + +OSAC manages networking infrastructure through first-class API entities (NetworkClass, VirtualNetwork, Subnet) but has no equivalent for storage. Storage backend configuration is embedded in Ansible extra vars (`VAST_ENDPOINT`, `VAST_USERNAME`, `VAST_PASSWORD`), making backends invisible to the OSAC data model. Cloud Provider Admins cannot discover registered backends, rotate credentials, or inspect backend health through the API. + +This gap blocks two capabilities: (1) composing tiered storage offerings where StorageTier entities (OSAC-1110) reference registered backends by ID, and (2) credential rotation without redeploying the fulfillment-service. Three prior enhancements addressed tenant-to-StorageClass resolution but left the infrastructure layer unmanaged. + +StorageBackend introduces the infrastructure registration layer. It stores the provider type, management endpoint, and a reference to a Kubernetes Secret containing credentials. The entity is platform-scoped (not tenant-scoped), managed exclusively by Cloud Provider Admins through the private API, and exposed read-only through the public API with credential references stripped. + +### Goals + +- Reuse the NetworkClass DB-backed server pattern (generic server, generic DAO, builder construction) with no reconciler or CRD. [Codebase: internal/servers/private_network_classes_server.go] +- Expose both private (full CRUD) and public (read-only) API layers following the established dual-API pattern. +- Store credentials as opaque Kubernetes Secret references — no credential content touches the fulfillment-service database. +- Keep the StorageBackend schema provider-agnostic so that non-VAST backends (Ceph, Pure) use the same entity without schema changes. +- Provide the foundation entity for StorageTier (OSAC-1110) composition without introducing StorageTier dependencies. + +### Non-Goals + +- Kubernetes CRD for StorageBackend — the entity is DB-backed only, like NetworkClass. +- Reconciler or controller registration — no backend provisioning occurs on create; the initial state is `READY`. +- Provider-specific credential validation — credential content is opaque per NFR-2. +- CSI driver installation or StorageClass creation on target clusters — these are part of the tenant onboarding reconciliation flow, which depends on both StorageBackend and StorageTier. +- StorageTier or StorageTierBackend entities — covered by OSAC-1110. + +## Proposal + +StorageBackend is a new entity in the fulfillment-service with two API surfaces: + +1. **Private API** (`osac.private.v1.StorageBackends`): Full CRUD (Create, Get, List, Update, Delete). Used by Cloud Provider Admins. No Signal RPC — StorageBackend has no reconciler or controller, so there is nothing to signal. + +2. **Public API** (`osac.public.v1.StorageBackends`): Read-only (List, Get). Delegates to the private server with field mapping. The `credentials_ref` field is excluded from the public API via `AddIgnoredFields()`. + +The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables. A unique partial index on `name` enforces FR-9 (name uniqueness among active records). + +The generic server's `setPayload()` switch statement requires a new case for `StorageBackend`, and the Event proto requires a new `storage_backend` payload field. + +### Workflow Description + +**Actors:** +- Cloud Provider Admin — registers and manages storage backends via the private API +- Tenant — discovers available backends (provider, endpoint, status) via the public API; cannot see credentials + +**Starting state:** The fulfillment-service is deployed with no StorageBackend records. Storage configuration exists only in Ansible extra vars. + +**Registration flow:** + +```mermaid +sequenceDiagram + participant Admin as Cloud Provider Admin + participant K8s as Kubernetes API + participant FS as fulfillment-service (private) + participant DB as PostgreSQL + + Admin->>K8s: Create Secret with storage credentials + Admin->>FS: CreateStorageBackend(provider, endpoint, credentials_ref, description) + FS->>FS: Validate required fields (provider, endpoint.host, credentials_ref) + FS->>FS: Set state = READY, clear caller-provided ID + FS->>DB: Insert into storage_backends (JSONB data column) + DB-->>FS: Return with generated UUID + FS-->>Admin: StorageBackend with id, state=READY +``` + +The diagram shows the registration path. The admin first creates a Kubernetes Secret containing provider-specific credentials, then registers the backend via the private API. The fulfillment-service validates required fields, sets the initial state to `READY`, generates a UUID, and persists the entity. + +**Credential rotation:** The admin calls `UpdateStorageBackend` with a new `credentials_ref` pointing to a rotated Secret. No redeployment is required. + +**Decommission:** The admin calls `DeleteStorageBackend`. The backend is soft-deleted: excluded from List results but preserved in the `archived_storage_backends` table for audit. Name reuse is allowed after soft deletion. + +**Error cases:** +- Duplicate name on active backend: Create returns `ALREADY_EXISTS` (enforced by the unique partial index). +- Missing required fields (provider, endpoint.host, credentials_ref): Create/Update returns `INVALID_ARGUMENT`. +- Update with stale version and `lock=true`: returns `FAILED_PRECONDITION` (optimistic concurrency control). +- Delete of non-existent backend: returns `NOT_FOUND`. + +### API Extensions + +This enhancement adds the following API extensions: + +- **Private gRPC service** `osac.private.v1.StorageBackends` — CRUD, registered in the gRPC server startup alongside `NetworkClasses`. +- **Public gRPC service** `osac.public.v1.StorageBackends` — List + Get only, registered alongside the public `NetworkClasses` server. +- **Event payload field** `storage_backend` added to `osac.private.v1.Event` — enables event-driven notifications for StorageBackend changes. + +No CRDs, webhooks, or finalizers are introduced. No existing resources are modified. If the fulfillment-service is unavailable, StorageBackend operations fail but no other OSAC resources are affected — StorageBackend has no controller loop that could leave resources in an inconsistent state. + +### Implementation Details/Notes/Constraints + +#### Proto Schema — Private API + +**`storage_backend_type.proto`** (`osac.private.v1`): + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `id` | `string` | Generated | UUID assigned by the DAO on create | +| `metadata` | `Metadata` | Yes | Standard OSAC metadata (name, labels, annotations, version) | +| `provider` | `string` | Yes | Storage provider identifier (e.g., `"vast"`, `"ceph"`, `"pure"`) | +| `description` | `string` | No | Human-readable description of the backend | +| `endpoint` | `StorageBackendEndpoint` | Yes | Management endpoint for the storage array | +| `credentials_ref` | `string` | Yes | Reference to a Kubernetes Secret (`namespace/secret-name` or `secret-name`). **Temporary** — the credential storage mechanism will be revisited when OSAC core establishes a unified credential management pattern. | +| `status` | `StorageBackendStatus` | System | Operational status, set on create | + +**`StorageBackendEndpoint` message:** + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `host` | `string` | Yes | Hostname or IP of the storage management API | +| `port` | `int32` | No | Port number; provider-specific default if omitted | + +**`StorageBackendStatus` message:** + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `state` | `StorageBackendState` | Yes | Current lifecycle state | +| `message` | `string` | No | Human-readable status detail | +| `model` | `string` | No | Storage array model (e.g., `"VAST C-100"`) | +| `firmware_version` | `string` | No | Reported firmware version | + +**`StorageBackendState` enum:** + +| Value | Number | Description | +|-------|--------|-------------| +| `STORAGE_BACKEND_STATE_UNSPECIFIED` | 0 | Default zero value | +| `STORAGE_BACKEND_STATE_READY` | 1 | Backend registered and available | + +The enum starts with `READY` as the only operational state. Additional states (e.g., `PENDING`, `FAILED`, `DEGRADED`) are added when health probing or reconciliation capabilities are introduced. The `UNSPECIFIED` sentinel follows proto3 convention. + +**`storage_backends_service.proto`** (`osac.private.v1`): + +The service defines five RPCs following the NetworkClass service pattern (without Signal, since StorageBackend has no reconciler): + +| RPC | HTTP Method | Path | Notes | +|-----|-------------|------|-------| +| `List` | `GET` | `/api/private/v1/storage_backends` | Pagination, CEL filter, ordering | +| `Get` | `GET` | `/api/private/v1/storage_backends/{id}` | `response_body: "object"` | +| `Create` | `POST` | `/api/private/v1/storage_backends` | `body: "object"`, `response_body: "object"` | +| `Update` | `PATCH` | `/api/private/v1/storage_backends/{object.id}` | `body: "object"`, `response_body: "object"` | +| `Delete` | `DELETE` | `/api/private/v1/storage_backends/{id}` | | + +No Signal RPC is defined. Signal exists on other entities to wake up a reconciler when external state changes — StorageBackend has no reconciler, so Signal would have no consumer. Status fields (model, firmware_version) can be updated via the standard `Update` RPC if needed in the future. + +#### Proto Schema — Public API + +**`storage_backend_type.proto`** (`osac.public.v1`): + +The public message mirrors the private message with these differences: +- `credentials_ref` is excluded entirely (not present in the message definition) +- `endpoint.host` is included (tenants need to know which array serves their storage) +- `status` is included (tenants need to see backend availability) +- All fields are effectively `OUTPUT_ONLY` since the public API is read-only + +**`storage_backends_service.proto`** (`osac.public.v1`): + +| RPC | HTTP Method | Path | +|-----|-------------|------| +| `List` | `GET` | `/api/fulfillment/v1/storage_backends` | +| `Get` | `GET` | `/api/fulfillment/v1/storage_backends/{id}` | + +No Create, Update, or Delete RPCs in the public API. + +#### Server Implementation + +**`private_storage_backends_server.go`**: Follows the `PrivateNetworkClassesServer` pattern: + +- Struct wraps `GenericServer[*privatev1.StorageBackend]` +- Builder pattern: `NewPrivateStorageBackendsServer()` returns a builder with `SetLogger`, `SetNotifier`, `SetAttributionLogic`, `SetTenancyLogic`, `SetMetricsRegisterer`, `Build()` +- `Create`: validates required fields (`provider`, `endpoint.host`, `credentials_ref`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID +- `Update`: fetches existing object, merges via `applyStorageBackendUpdate`, validates immutability of `provider` field, delegates to generic server +- `Delete`: delegates directly to generic server (soft delete) + +**Immutable fields on update:** `provider` is immutable after creation. Changing the storage provider would invalidate the endpoint, credentials, and any downstream StorageTier references. + +**`storage_backends_server.go`** (public): Follows the `NetworkClassesServer` pattern: + +- Composes a `PrivateStorageBackendsServer` delegate + `GenericMapper` pair +- `inMapper` is not needed (no write RPCs on public API), but the pattern requires it for consistency with the builder; unused mapper fields can be nil +- `outMapper` maps private StorageBackend to public StorageBackend, skipping `credentials_ref` +- Exposes only `List` and `Get` + +#### Database Migration + +Migration file: `54_create_storage_backends_tables.up.sql` (next available sequence number) + +The migration creates two tables following the standard schema: + +```sql +create table storage_backends ( + id text not null primary key, + name text not null default '', + creation_timestamp timestamp with time zone not null default now(), + deletion_timestamp timestamp with time zone not null default 'epoch', + finalizers text[] not null default '{}', + creators text[] not null default '{}', + tenants text[] not null default '{}', + labels jsonb not null default '{}'::jsonb, + annotations jsonb not null default '{}'::jsonb, + data jsonb not null +); + +create table archived_storage_backends ( + id text not null, + name text not null default '', + creation_timestamp timestamp with time zone not null, + deletion_timestamp timestamp with time zone not null, + archival_timestamp timestamp with time zone not null default now(), + creators text[] not null default '{}', + tenants text[] not null default '{}', + labels jsonb not null default '{}'::jsonb, + annotations jsonb not null default '{}'::jsonb, + data jsonb not null +); +``` + +Indexes: + +```sql +create index storage_backends_by_name on storage_backends (name); +create index storage_backends_by_owner on storage_backends using gin (creators); +create index storage_backends_by_tenant on storage_backends using gin (tenants); +create index storage_backends_by_label on storage_backends using gin (labels); +``` + +Name uniqueness among active backends (FR-9): + +```sql +create unique index storage_backends_unique_active_name + on storage_backends (name) + where deletion_timestamp = 'epoch'; +``` + +This partial index permits name reuse after soft deletion while preventing duplicate names among active records. The DAO's unique violation error maps to gRPC `ALREADY_EXISTS`. + +#### Generic Server Changes + +**`generic_server.go`** — `setPayload()` switch: Add a case for `*privatev1.StorageBackend` that calls `event.SetStorageBackend(object)`. + +**`event_type.proto`** — Add a `StorageBackend storage_backend` field to the `Event` message's oneof payload (next available field number). + +#### gRPC Server Registration + +In `start_grpc_server_cmd.go`, add registration blocks for both servers following the NetworkClass pattern: + +```go +// Public StorageBackends server +storageBackendsServer, err := servers.NewStorageBackendsServer(). + SetLogger(logger). + SetNotifier(notifier). + SetAttributionLogic(attributionLogic). + SetTenancyLogic(tenancyLogic). + SetMetricsRegisterer(metricsRegisterer). + Build() +publicv1.RegisterStorageBackendsServer(grpcServer, storageBackendsServer) + +// Private StorageBackends server +privateStorageBackendsServer, err := servers.NewPrivateStorageBackendsServer(). + SetLogger(logger). + SetNotifier(notifier). + SetAttributionLogic(attributionLogic). + SetTenancyLogic(tenancyLogic). + SetMetricsRegisterer(metricsRegisterer). + Build() +privatev1.RegisterStorageBackendsServer(grpcServer, privateStorageBackendsServer) +``` + +### Security Considerations + +StorageBackend inherits the existing fulfillment-service security model: + +- **Authentication:** JWT validation via the gRPC interceptor chain. No changes to the authentication flow. +- **Authorization:** OPA policies control access to private vs. public API endpoints. Cloud Provider Admins have access to private CRUD. Tenants have access to public List + Get only. +- **Credential isolation:** The `credentials_ref` field stores a Secret reference string, not credential content. The fulfillment-service never reads, decrypts, or forwards the Secret's data. The public API excludes `credentials_ref` entirely — tenants cannot discover Secret names. This mechanism is temporary — it will be revisited when OSAC core establishes a unified credential management pattern across all entities. +- **Input validation:** The server validates required fields (`provider`, `endpoint.host`, `credentials_ref`) and provider immutability. The `endpoint.host` field is a string stored as-is; no DNS resolution or connection attempt is made during registration. The `credentials_ref` format is validated syntactically (non-empty string) but not resolved against the Kubernetes API. + +No new authentication, authorization, or encryption mechanisms are introduced. + +### Failure Handling and Recovery + +| Failure Mode | Behavior | User Observation | Recovery | +|---|---|---|---| +| Database unavailable during Create | gRPC interceptor rolls back the transaction | `UNAVAILABLE` error | Retry the request; the operation is idempotent (name uniqueness check prevents duplicates) | +| Duplicate active name on Create | Unique partial index violation | `ALREADY_EXISTS` with the conflicting name | Choose a different name or delete the existing backend first | +| Stale version on Update with `lock=true` | DAO rejects the write | `FAILED_PRECONDITION` | Re-fetch the current version and retry | +All operations are transactional (wrapped by the gRPC database transaction interceptor) and idempotent. There is no reconciliation loop that could leave resources in a partially updated state. + +### RBAC / Tenancy + +StorageBackend is platform-scoped, not tenant-scoped. It follows the same access model as NetworkClass: + +- **Private API:** Accessible to Cloud Provider Admins. OPA policies enforce role-based access. No Signal RPC. The `tenants` column in the database is empty (platform-scoped resources have no tenant owner). +- **Public API:** Accessible to all authenticated users (tenants). List and Get return all active backends (no tenant filtering). The `credentials_ref` field is stripped by the public server's output mapper. + +No `osac.openshift.io/tenant` or `osac.openshift.io/owner-reference` annotations are set on StorageBackend. This is consistent with NetworkClass, which is also platform-scoped. Tenant isolation applies at the StorageTier level (OSAC-1110), where tiers are assigned to tenants. + +### Observability and Monitoring + +No new observability changes. Existing monitoring mechanisms apply: + +- The gRPC Prometheus interceptor records request counts, latencies, and error rates for all StorageBackend RPCs automatically (same as all other gRPC services). +- PostgreSQL NOTIFY events for StorageBackend CRUD operations are published through the existing event infrastructure. +- Structured logging via `slog` in the server implementation covers validation failures, Signal updates, and DAO errors. + +### Risks and Mitigations + +**Credential reference format ambiguity:** The `credentials_ref` format (namespace-scoped `namespace/secret-name` vs. implicit namespace) is deferred to implementation. Choosing the wrong format creates a breaking API change later. + +*Mitigation:* The `credentials_ref` is an opaque string. Namespace-scoped format (`namespace/secret-name`) is the safer default because it avoids ambiguity when backends span namespaces. The format is documented in the proto field comments and validated syntactically (must contain `/`). + +**Soft-delete name reuse and StorageTier references:** When a backend is soft-deleted and its name is reused for a new backend, a StorageTier referencing the old backend by ID continues to resolve correctly (references use ID, not name). However, confusion may arise if operators expect name-based lookup to return the latest backend. + +*Mitigation:* StorageTier references backends by ID. The API documentation explicitly states that List excludes soft-deleted backends and that name reuse is permitted after decommission. + +### Drawbacks + +**Platform-scoped entity with credentials.** StorageBackend stores a reference to a Kubernetes Secret. Although the fulfillment-service never reads the Secret, the reference itself (namespace + name) is sensitive metadata. The public API strips it, but the private API exposes it to all Cloud Provider Admins regardless of which backend they manage. In a multi-admin scenario, this provides full visibility of all credential Secret names. This is consistent with NetworkClass, where all admins see all configuration, but worth noting as the attack surface grows with the number of backends. + +## Alternatives (Not Implemented) + +### Kubernetes CRD instead of DB-backed entity + +StorageBackend could be a Kubernetes Custom Resource (like VirtualNetwork or Subnet) with a reconciler. + +*Pros:* Native kubectl access, standard RBAC, built-in watch semantics, controller reconciliation loop for health checks. + +*Cons:* Requires CRD definition in osac-operator, reconciler registration, cross-process coordination between the operator and fulfillment-service. NetworkClass established the DB-backed pattern for platform infrastructure entities, and deviating from it fragments the architecture without clear benefit — StorageBackend has no cluster-side state to reconcile. + +*Rejection:* The DB-backed pattern is simpler, already proven for this use case, and aligns with the NetworkClass precedent. A CRD can be introduced later if reconciliation requirements emerge. + +### Credentials stored in the database + +Instead of referencing a Kubernetes Secret, the fulfillment-service could store credentials directly in the JSONB `data` column (encrypted at rest). + +*Pros:* Self-contained entity — no dependency on Kubernetes Secret management. Simpler operational model for credential rotation. + +*Cons:* Credential content in PostgreSQL requires application-level encryption, key rotation, and access auditing. The fulfillment-service becomes a credential store, expanding its attack surface. Kubernetes Secrets already provide encryption at rest (via etcd encryption), RBAC-controlled access, and integration with external secret managers (e.g., Vault via External Secrets Operator). + +*Rejection:* Storing credential references delegates security responsibility to the Kubernetes Secret infrastructure, which is purpose-built for this function. However, this is a temporary approach — all existing OSAC entities currently store credentials inline. The credential storage mechanism will be revisited when OSAC core establishes a unified pattern, and StorageBackend will adopt whatever pattern is chosen. + +## Open Questions + +1. **credentials_ref format and longevity.** Should the format be `namespace/secret-name` (explicit namespace) or `secret-name` (implicit `osac-system` namespace)? Explicit namespace is more flexible but requires the admin to know the Secret's namespace. Implicit namespace is simpler but constrains all credential Secrets to a single namespace. Note: existing OSAC entities (break_glass_credentials, identity_provider, user, cluster_template, hub) all store credentials inline in the JSONB data column. The `credentials_ref` approach is a temporary divergence — the credential storage mechanism will be revisited when OSAC core establishes a unified pattern. + +## Test Plan + +Testing follows the fulfillment-service's established Ginkgo v2 test patterns: + +**Unit/integration tests** (co-located in `internal/servers/`): + +- `private_storage_backends_server_test.go`: + - CRUD lifecycle: Create with all fields, Get by ID, List with pagination, Update with field mask, Delete (soft-delete) + - Validation: missing `provider`, missing `endpoint.host`, missing `credentials_ref` return `INVALID_ARGUMENT` + - Immutability: Update with changed `provider` returns `INVALID_ARGUMENT` + - Name uniqueness: Create with duplicate active name returns `ALREADY_EXISTS`; Create after soft-delete of same name succeeds + - Optimistic locking: Update with stale version and `lock=true` returns `FAILED_PRECONDITION` + - CEL filtering: List with `this.provider == "vast"` returns matching backends + - Ordering: List with `metadata.name asc` returns sorted results +- `storage_backends_server_test.go` (public API): + - List and Get delegate correctly to private server + - `credentials_ref` is absent from public responses + - Create/Update/Delete are not exposed + +**Integration tests** (`it/`): StorageBackend CRUD via gRPC and REST endpoints against a kind cluster deployment, covering authentication and authorization. + +**E2E tests** (deferred to StorageTier integration): End-to-end tests covering the StorageBackend -> StorageTier -> tenant onboarding flow are scoped to OSAC-1110. + +## Graduation Criteria + +Graduation criteria will be defined when targeting a release. Expected stages: Dev Preview -> Tech Preview -> GA based on production deployment feedback. + +For initial merge: + +- All integration tests pass +- `buf lint` passes with no warnings +- Private and public API endpoints are accessible via both gRPC and REST +- `credentials_ref` is confirmed absent from public API responses + +## Upgrade / Downgrade Strategy + +This is a new API with no upgrade impact. The database migration (`54_create_storage_backends_tables.up.sql`) is additive — it creates new tables and indexes without modifying existing schema. + +**Downgrade:** Requires deleting all StorageBackend records and then reverting the database migration (dropping the `storage_backends` and `archived_storage_backends` tables). No other OSAC entities reference StorageBackend in this phase, so deletion has no cascading effects. + +## Version Skew Strategy + +StorageBackend exists entirely within the fulfillment-service. There is no CRD in osac-operator and no cross-component API dependency. Version skew between fulfillment-service and osac-operator does not affect StorageBackend. + +When StorageTier (OSAC-1110) is implemented, it will reference StorageBackend by ID. At that point, the StorageTier implementation must handle the case where a referenced StorageBackend does not exist (fulfillment-service upgraded with StorageTier before StorageBackend records are created, or downgrade removes StorageBackend). This is scoped to OSAC-1110. + +## Support Procedures + +**Failure detection:** + +- Missing or inaccessible backends: `grpc_server_handled_total{grpc_service="osac.private.v1.StorageBackends", grpc_code!="OK"}` metric shows elevated error rates. +- Database migration failure: `fulfillment-service` fails to start with migration error in logs. + +**Disabling the feature:** + +StorageBackend can be disabled by removing the server registration from `start_grpc_server_cmd.go` and reverting the database migration. Consequences: + +- No impact on cluster health — StorageBackend has no reconciler or controller. +- No impact on existing workloads — no OSAC entity currently references StorageBackend. +- New StorageBackend API calls return `UNIMPLEMENTED`. + +**Recovery:** Re-enable by restoring the server registration and re-running the migration. Data in the `storage_backends` table is preserved if only the server registration was removed. Consistency is maintained because all operations are transactional. + +## Infrastructure Needed + +None. From 84a262f585716adbcac13e556039205c7fbb64b8 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Tue, 16 Jun 2026 16:04:26 +0300 Subject: [PATCH 024/450] =?UTF-8?q?OSAC-1111:=20revise=20=E2=80=94=20inlin?= =?UTF-8?q?e=20credentials,=20remove=20Signal=20RPC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace credentials_ref (K8s Secret reference) with inline credentials (username/password) stored in JSONB, consistent with all existing OSAC entities - Remove Signal RPC — no reconciler, no consumer - Update PRD to match (FR-3, FR-10, open questions) Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 67 ++++++++++--------- enhancements/storage-backend-osac-1111/prd.md | 15 ++--- 2 files changed, 40 insertions(+), 42 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index d1d0b3517..b6d9f18da 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -28,13 +28,13 @@ OSAC manages networking infrastructure through first-class API entities (Network This gap blocks two capabilities: (1) composing tiered storage offerings where StorageTier entities (OSAC-1110) reference registered backends by ID, and (2) credential rotation without redeploying the fulfillment-service. Three prior enhancements addressed tenant-to-StorageClass resolution but left the infrastructure layer unmanaged. -StorageBackend introduces the infrastructure registration layer. It stores the provider type, management endpoint, and a reference to a Kubernetes Secret containing credentials. The entity is platform-scoped (not tenant-scoped), managed exclusively by Cloud Provider Admins through the private API, and exposed read-only through the public API with credential references stripped. +StorageBackend introduces the infrastructure registration layer. It stores the provider type, management endpoint, and credentials. The entity is platform-scoped (not tenant-scoped), managed exclusively by Cloud Provider Admins through the private API, and exposed read-only through the public API with credentials stripped. ### Goals - Reuse the NetworkClass DB-backed server pattern (generic server, generic DAO, builder construction) with no reconciler or CRD. [Codebase: internal/servers/private_network_classes_server.go] - Expose both private (full CRUD) and public (read-only) API layers following the established dual-API pattern. -- Store credentials as opaque Kubernetes Secret references — no credential content touches the fulfillment-service database. +- Store credentials inline in the JSONB `data` column, consistent with all existing OSAC entities (break_glass_credentials, identity_provider, user, cluster_template, hub). - Keep the StorageBackend schema provider-agnostic so that non-VAST backends (Ceph, Pure) use the same entity without schema changes. - Provide the foundation entity for StorageTier (OSAC-1110) composition without introducing StorageTier dependencies. @@ -52,7 +52,7 @@ StorageBackend is a new entity in the fulfillment-service with two API surfaces: 1. **Private API** (`osac.private.v1.StorageBackends`): Full CRUD (Create, Get, List, Update, Delete). Used by Cloud Provider Admins. No Signal RPC — StorageBackend has no reconciler or controller, so there is nothing to signal. -2. **Public API** (`osac.public.v1.StorageBackends`): Read-only (List, Get). Delegates to the private server with field mapping. The `credentials_ref` field is excluded from the public API via `AddIgnoredFields()`. +2. **Public API** (`osac.public.v1.StorageBackends`): Read-only (List, Get). Delegates to the private server with field mapping. Credential fields are excluded from the public API via `AddIgnoredFields()`. The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables. A unique partial index on `name` enforces FR-9 (name uniqueness among active records). @@ -71,28 +71,26 @@ The generic server's `setPayload()` switch statement requires a new case for `St ```mermaid sequenceDiagram participant Admin as Cloud Provider Admin - participant K8s as Kubernetes API participant FS as fulfillment-service (private) participant DB as PostgreSQL - Admin->>K8s: Create Secret with storage credentials - Admin->>FS: CreateStorageBackend(provider, endpoint, credentials_ref, description) - FS->>FS: Validate required fields (provider, endpoint.host, credentials_ref) + Admin->>FS: CreateStorageBackend(provider, endpoint, credentials, description) + FS->>FS: Validate required fields (provider, endpoint.host, credentials) FS->>FS: Set state = READY, clear caller-provided ID FS->>DB: Insert into storage_backends (JSONB data column) DB-->>FS: Return with generated UUID FS-->>Admin: StorageBackend with id, state=READY ``` -The diagram shows the registration path. The admin first creates a Kubernetes Secret containing provider-specific credentials, then registers the backend via the private API. The fulfillment-service validates required fields, sets the initial state to `READY`, generates a UUID, and persists the entity. +The diagram shows the registration path. The admin registers the backend via the private API with credentials provided inline. The fulfillment-service validates required fields, sets the initial state to `READY`, generates a UUID, and persists the entity. Credentials are stored in the JSONB `data` column, consistent with how all other OSAC entities store credentials. -**Credential rotation:** The admin calls `UpdateStorageBackend` with a new `credentials_ref` pointing to a rotated Secret. No redeployment is required. +**Credential rotation:** The admin calls `UpdateStorageBackend` with new `credentials` fields. No redeployment is required. **Decommission:** The admin calls `DeleteStorageBackend`. The backend is soft-deleted: excluded from List results but preserved in the `archived_storage_backends` table for audit. Name reuse is allowed after soft deletion. **Error cases:** - Duplicate name on active backend: Create returns `ALREADY_EXISTS` (enforced by the unique partial index). -- Missing required fields (provider, endpoint.host, credentials_ref): Create/Update returns `INVALID_ARGUMENT`. +- Missing required fields (provider, endpoint.host, credentials): Create/Update returns `INVALID_ARGUMENT`. - Update with stale version and `lock=true`: returns `FAILED_PRECONDITION` (optimistic concurrency control). - Delete of non-existent backend: returns `NOT_FOUND`. @@ -119,7 +117,7 @@ No CRDs, webhooks, or finalizers are introduced. No existing resources are modif | `provider` | `string` | Yes | Storage provider identifier (e.g., `"vast"`, `"ceph"`, `"pure"`) | | `description` | `string` | No | Human-readable description of the backend | | `endpoint` | `StorageBackendEndpoint` | Yes | Management endpoint for the storage array | -| `credentials_ref` | `string` | Yes | Reference to a Kubernetes Secret (`namespace/secret-name` or `secret-name`). **Temporary** — the credential storage mechanism will be revisited when OSAC core establishes a unified credential management pattern. | +| `credentials` | `StorageBackendCredentials` | Yes | Provider-specific credentials stored inline, consistent with existing OSAC credential patterns | | `status` | `StorageBackendStatus` | System | Operational status, set on create | **`StorageBackendEndpoint` message:** @@ -129,6 +127,15 @@ No CRDs, webhooks, or finalizers are introduced. No existing resources are modif | `host` | `string` | Yes | Hostname or IP of the storage management API | | `port` | `int32` | No | Port number; provider-specific default if omitted | +**`StorageBackendCredentials` message:** + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `username` | `string` | Yes | Username for storage management API authentication | +| `password` | `string` | Yes | Password for storage management API authentication | + +Credentials are stored inline in the JSONB `data` column, following the same pattern as `break_glass_credentials.password`, `identity_provider.client_secret`, `user.password`, `cluster_template.pull_secret`, and `hub.kubeconfig`. The credential content is opaque to the fulfillment-service — no provider-specific validation is performed (NFR-2). + **`StorageBackendStatus` message:** | Field | Type | Required | Description | @@ -166,7 +173,7 @@ No Signal RPC is defined. Signal exists on other entities to wake up a reconcile **`storage_backend_type.proto`** (`osac.public.v1`): The public message mirrors the private message with these differences: -- `credentials_ref` is excluded entirely (not present in the message definition) +- `credentials` is excluded entirely (not present in the message definition) - `endpoint.host` is included (tenants need to know which array serves their storage) - `status` is included (tenants need to see backend availability) - All fields are effectively `OUTPUT_ONLY` since the public API is read-only @@ -186,7 +193,7 @@ No Create, Update, or Delete RPCs in the public API. - Struct wraps `GenericServer[*privatev1.StorageBackend]` - Builder pattern: `NewPrivateStorageBackendsServer()` returns a builder with `SetLogger`, `SetNotifier`, `SetAttributionLogic`, `SetTenancyLogic`, `SetMetricsRegisterer`, `Build()` -- `Create`: validates required fields (`provider`, `endpoint.host`, `credentials_ref`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID +- `Create`: validates required fields (`provider`, `endpoint.host`, `credentials.username`, `credentials.password`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID - `Update`: fetches existing object, merges via `applyStorageBackendUpdate`, validates immutability of `provider` field, delegates to generic server - `Delete`: delegates directly to generic server (soft delete) @@ -196,7 +203,7 @@ No Create, Update, or Delete RPCs in the public API. - Composes a `PrivateStorageBackendsServer` delegate + `GenericMapper` pair - `inMapper` is not needed (no write RPCs on public API), but the pattern requires it for consistency with the builder; unused mapper fields can be nil -- `outMapper` maps private StorageBackend to public StorageBackend, skipping `credentials_ref` +- `outMapper` maps private StorageBackend to public StorageBackend, skipping `credentials` - Exposes only `List` and `Get` #### Database Migration @@ -290,8 +297,8 @@ StorageBackend inherits the existing fulfillment-service security model: - **Authentication:** JWT validation via the gRPC interceptor chain. No changes to the authentication flow. - **Authorization:** OPA policies control access to private vs. public API endpoints. Cloud Provider Admins have access to private CRUD. Tenants have access to public List + Get only. -- **Credential isolation:** The `credentials_ref` field stores a Secret reference string, not credential content. The fulfillment-service never reads, decrypts, or forwards the Secret's data. The public API excludes `credentials_ref` entirely — tenants cannot discover Secret names. This mechanism is temporary — it will be revisited when OSAC core establishes a unified credential management pattern across all entities. -- **Input validation:** The server validates required fields (`provider`, `endpoint.host`, `credentials_ref`) and provider immutability. The `endpoint.host` field is a string stored as-is; no DNS resolution or connection attempt is made during registration. The `credentials_ref` format is validated syntactically (non-empty string) but not resolved against the Kubernetes API. +- **Credential storage:** Credentials are stored inline in the JSONB `data` column, consistent with all existing OSAC entities. The public API excludes `credentials` entirely — tenants cannot see credential fields. This follows the same pattern as `break_glass_credentials`, `identity_provider`, `user`, `cluster_template`, and `hub`. +- **Input validation:** The server validates required fields (`provider`, `endpoint.host`, `credentials.username`, `credentials.password`) and provider immutability. The `endpoint.host` field is a string stored as-is; no DNS resolution or connection attempt is made during registration. No new authentication, authorization, or encryption mechanisms are introduced. @@ -309,7 +316,7 @@ All operations are transactional (wrapped by the gRPC database transaction inter StorageBackend is platform-scoped, not tenant-scoped. It follows the same access model as NetworkClass: - **Private API:** Accessible to Cloud Provider Admins. OPA policies enforce role-based access. No Signal RPC. The `tenants` column in the database is empty (platform-scoped resources have no tenant owner). -- **Public API:** Accessible to all authenticated users (tenants). List and Get return all active backends (no tenant filtering). The `credentials_ref` field is stripped by the public server's output mapper. +- **Public API:** Accessible to all authenticated users (tenants). List and Get return all active backends (no tenant filtering). Credential fields are stripped by the public server's output mapper. No `osac.openshift.io/tenant` or `osac.openshift.io/owner-reference` annotations are set on StorageBackend. This is consistent with NetworkClass, which is also platform-scoped. Tenant isolation applies at the StorageTier level (OSAC-1110), where tiers are assigned to tenants. @@ -319,21 +326,17 @@ No new observability changes. Existing monitoring mechanisms apply: - The gRPC Prometheus interceptor records request counts, latencies, and error rates for all StorageBackend RPCs automatically (same as all other gRPC services). - PostgreSQL NOTIFY events for StorageBackend CRUD operations are published through the existing event infrastructure. -- Structured logging via `slog` in the server implementation covers validation failures, Signal updates, and DAO errors. +- Structured logging via `slog` in the server implementation covers validation failures and DAO errors. ### Risks and Mitigations -**Credential reference format ambiguity:** The `credentials_ref` format (namespace-scoped `namespace/secret-name` vs. implicit namespace) is deferred to implementation. Choosing the wrong format creates a breaking API change later. - -*Mitigation:* The `credentials_ref` is an opaque string. Namespace-scoped format (`namespace/secret-name`) is the safer default because it avoids ambiguity when backends span namespaces. The format is documented in the proto field comments and validated syntactically (must contain `/`). - **Soft-delete name reuse and StorageTier references:** When a backend is soft-deleted and its name is reused for a new backend, a StorageTier referencing the old backend by ID continues to resolve correctly (references use ID, not name). However, confusion may arise if operators expect name-based lookup to return the latest backend. *Mitigation:* StorageTier references backends by ID. The API documentation explicitly states that List excludes soft-deleted backends and that name reuse is permitted after decommission. ### Drawbacks -**Platform-scoped entity with credentials.** StorageBackend stores a reference to a Kubernetes Secret. Although the fulfillment-service never reads the Secret, the reference itself (namespace + name) is sensitive metadata. The public API strips it, but the private API exposes it to all Cloud Provider Admins regardless of which backend they manage. In a multi-admin scenario, this provides full visibility of all credential Secret names. This is consistent with NetworkClass, where all admins see all configuration, but worth noting as the attack surface grows with the number of backends. +**Platform-scoped entity with credentials in the database.** StorageBackend stores credentials inline in the JSONB `data` column, which means credential content (username/password) lives in PostgreSQL. This is consistent with all existing OSAC entities that store credentials (break_glass_credentials, identity_provider, user, cluster_template, hub), but worth noting as the attack surface grows with the number of backends. The public API strips credentials, but the private API exposes them to all Cloud Provider Admins regardless of which backend they manage. ## Alternatives (Not Implemented) @@ -347,19 +350,19 @@ StorageBackend could be a Kubernetes Custom Resource (like VirtualNetwork or Sub *Rejection:* The DB-backed pattern is simpler, already proven for this use case, and aligns with the NetworkClass precedent. A CRD can be introduced later if reconciliation requirements emerge. -### Credentials stored in the database +### Credentials as Kubernetes Secret reference -Instead of referencing a Kubernetes Secret, the fulfillment-service could store credentials directly in the JSONB `data` column (encrypted at rest). +Instead of storing credentials inline, the fulfillment-service could store a `credentials_ref` string referencing a Kubernetes Secret. -*Pros:* Self-contained entity — no dependency on Kubernetes Secret management. Simpler operational model for credential rotation. +*Pros:* Credential content never touches the database. Delegates security to the Kubernetes Secret infrastructure (encryption at rest, RBAC, integration with external secret managers like Vault via External Secrets Operator). -*Cons:* Credential content in PostgreSQL requires application-level encryption, key rotation, and access auditing. The fulfillment-service becomes a credential store, expanding its attack surface. Kubernetes Secrets already provide encryption at rest (via etcd encryption), RBAC-controlled access, and integration with external secret managers (e.g., Vault via External Secrets Operator). +*Cons:* Introduces a new credential pattern that no existing OSAC entity uses — all current entities (break_glass_credentials, identity_provider, user, cluster_template, hub) store credentials inline. Adds operational complexity (Secret must exist, namespace resolution, lifecycle management). Creates an inconsistency in the codebase. -*Rejection:* Storing credential references delegates security responsibility to the Kubernetes Secret infrastructure, which is purpose-built for this function. However, this is a temporary approach — all existing OSAC entities currently store credentials inline. The credential storage mechanism will be revisited when OSAC core establishes a unified pattern, and StorageBackend will adopt whatever pattern is chosen. +*Rejection:* Consistency with existing OSAC patterns outweighs the security benefit for phase 1. All other entities store credentials inline, and introducing a divergent pattern without an OSAC-core decision on unified credential management would create unnecessary complexity. This can be revisited when OSAC establishes a platform-wide credential storage strategy. ## Open Questions -1. **credentials_ref format and longevity.** Should the format be `namespace/secret-name` (explicit namespace) or `secret-name` (implicit `osac-system` namespace)? Explicit namespace is more flexible but requires the admin to know the Secret's namespace. Implicit namespace is simpler but constrains all credential Secrets to a single namespace. Note: existing OSAC entities (break_glass_credentials, identity_provider, user, cluster_template, hub) all store credentials inline in the JSONB data column. The `credentials_ref` approach is a temporary divergence — the credential storage mechanism will be revisited when OSAC core establishes a unified pattern. +None — all prior open questions have been resolved. Credentials are stored inline (matching existing OSAC patterns), and Signal RPC has been removed (no reconciler). ## Test Plan @@ -369,7 +372,7 @@ Testing follows the fulfillment-service's established Ginkgo v2 test patterns: - `private_storage_backends_server_test.go`: - CRUD lifecycle: Create with all fields, Get by ID, List with pagination, Update with field mask, Delete (soft-delete) - - Validation: missing `provider`, missing `endpoint.host`, missing `credentials_ref` return `INVALID_ARGUMENT` + - Validation: missing `provider`, missing `endpoint.host`, missing `credentials` return `INVALID_ARGUMENT` - Immutability: Update with changed `provider` returns `INVALID_ARGUMENT` - Name uniqueness: Create with duplicate active name returns `ALREADY_EXISTS`; Create after soft-delete of same name succeeds - Optimistic locking: Update with stale version and `lock=true` returns `FAILED_PRECONDITION` @@ -377,7 +380,7 @@ Testing follows the fulfillment-service's established Ginkgo v2 test patterns: - Ordering: List with `metadata.name asc` returns sorted results - `storage_backends_server_test.go` (public API): - List and Get delegate correctly to private server - - `credentials_ref` is absent from public responses + - `credentials` is absent from public responses - Create/Update/Delete are not exposed **Integration tests** (`it/`): StorageBackend CRUD via gRPC and REST endpoints against a kind cluster deployment, covering authentication and authorization. @@ -393,7 +396,7 @@ For initial merge: - All integration tests pass - `buf lint` passes with no warnings - Private and public API endpoints are accessible via both gRPC and REST -- `credentials_ref` is confirmed absent from public API responses +- `credentials` is confirmed absent from public API responses ## Upgrade / Downgrade Strategy diff --git a/enhancements/storage-backend-osac-1111/prd.md b/enhancements/storage-backend-osac-1111/prd.md index 591efbb54..2caf74422 100644 --- a/enhancements/storage-backend-osac-1111/prd.md +++ b/enhancements/storage-backend-osac-1111/prd.md @@ -17,7 +17,7 @@ Three prior enhancements addressed tenant-to-StorageClass resolution (`tenant-sp ### 2.1 Goals - Cloud Provider Admins can register, list, update, and decommission storage backends through the OSAC private gRPC API without modifying environment variables or Ansible playbooks. -- Cloud Provider Admins can rotate storage backend credentials by updating the `credentials_ref` field, without redeploying the fulfillment-service. +- Cloud Provider Admins can rotate storage backend credentials by updating the credential fields, without redeploying the fulfillment-service. - Cloud Provider Admins can inspect backend operational status (state, model, firmware version) through the API to verify availability and troubleshoot connectivity. - The StorageBackend entity follows the same DB-backed private API pattern as NetworkClass and PublicIPPool, maintaining consistency across OSAC infrastructure entities. - The StorageBackend entity provides the foundation for future StorageTier composition (OSAC-1110), where tiers reference registered backends. @@ -37,14 +37,14 @@ Three prior enhancements addressed tenant-to-StorageClass resolution (`tenant-sp - **FR-1:** The fulfillment-service must expose a `StorageBackends` gRPC service under `osac.private.v1` with Create, Get, List, Update, and Delete RPCs. - **FR-2:** All CRUD RPCs must include HTTP annotations for REST access via grpc-gateway (POST, GET, GET, PATCH, DELETE). -- **FR-3:** `CreateStorageBackend` must accept provider type, management endpoint (host + optional port), credentials reference (Kubernetes Secret), and optional description. The backend must be created with initial state `READY`. +- **FR-3:** `CreateStorageBackend` must accept provider type, management endpoint (host + optional port), credentials (username and password), and optional description. The backend must be created with initial state `READY`. - **FR-4:** `ListStorageBackends` must support pagination (`offset`/`limit`), CEL-based filtering, and SQL-like ordering. This follows the established OSAC List API pattern used by all existing entities (NetworkClass, Clusters, ComputeInstances, Roles, etc.). - **FR-5:** `UpdateStorageBackend` must support partial updates (only specified fields are modified) and optimistic concurrency control to prevent conflicting writes. - **FR-6:** `DeleteStorageBackend` must perform a soft delete. Deleted backends must be excluded from List results but preserved for audit and future references from StorageTier. - **FR-7:** Cloud Provider Admins must be able to update backend operational metadata (model, firmware_version) and status message via the standard `Update` RPC. No Signal RPC is needed — StorageBackend has no reconciler or controller. - **FR-8:** Backend state must include `READY` (backend registered and available). Additional states will be introduced as needed when reconciliation or health probing capabilities are added in future phases. [User] - **FR-9:** Backend names must be unique among active (non-deleted) backends, allowing name reuse after decommission. -- **FR-10:** The `credentials_ref` field must reference a Kubernetes Secret. The reference format (namespace-scoped `namespace/secret-name` or implicit namespace) is determined during implementation. This credential storage mechanism is temporary — it will be revisited when OSAC core establishes a unified credential management pattern. +- **FR-10:** Credentials (username, password) must be stored inline in the StorageBackend entity, consistent with how all existing OSAC entities store credentials (break_glass_credentials, identity_provider, user, cluster_template, hub). Credentials must be excluded from the public API. ### 3.2 Non-Functional Requirements @@ -66,7 +66,7 @@ Three prior enhancements addressed tenant-to-StorageClass resolution (`tenant-sp - The management cluster (hub) is the same cluster that runs the fulfillment-service and creates HostedClusters via Hosted Control Planes. The terms "hub" (ACM terminology) and "management cluster" refer to the same cluster. - StorageClasses are associated with tenants and tiers by naming convention, not by Kubernetes labels. The prior EP label-based approach (`osac.openshift.io/tenant`, `osac.openshift.io/storage-tier`) is superseded. -- The `credentials_ref` Secret is created and managed by the Cloud Provider Admin outside of the StorageBackend API. The API stores only the reference, not the credential content. +- Credentials are provided by the Cloud Provider Admin during backend registration and stored inline in the database, consistent with all existing OSAC credential patterns. ## 6. Dependencies @@ -90,9 +90,4 @@ Three prior enhancements addressed tenant-to-StorageClass resolution (`tenant-sp ### 8.1 Should credential rotation trigger re-validation? - **Owner:** Storage architect -- **Impact:** FR-3, FR-8. When `credentials_ref` is updated via `UpdateStorageBackend`, should the backend state change to indicate re-validation is needed? Deferred until additional states (e.g., `PENDING`) are introduced in a future phase. - -### 8.2 What is the credentials_ref format? - -- **Owner:** Storage architect, fulfillment-service maintainers -- **Impact:** FR-10. The reference format — namespace-scoped (`namespace/secret-name`) vs. implicit namespace (`secret-name` assuming `osac-system`) — has security implications for cross-namespace access. A decision is needed before implementation. +- **Impact:** FR-3, FR-8. When credentials are updated via `UpdateStorageBackend`, should the backend state change to indicate re-validation is needed? Deferred until additional states (e.g., `PENDING`) are introduced in a future phase. From ce921f8a315d9349ba76bae8f9f1ad4db08a5fb7 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Tue, 16 Jun 2026 17:35:03 +0300 Subject: [PATCH 025/450] =?UTF-8?q?OSAC-1111:=20revise=20=E2=80=94=20remov?= =?UTF-8?q?e=20public=20API,=20private-only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tenants don't need direct access to StorageBackend — they interact with storage through StorageTier (OSAC-1110). Removed public API proto, server, tests, and all references. Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 67 ++++--------------- 1 file changed, 12 insertions(+), 55 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index b6d9f18da..028f41c53 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -28,12 +28,12 @@ OSAC manages networking infrastructure through first-class API entities (Network This gap blocks two capabilities: (1) composing tiered storage offerings where StorageTier entities (OSAC-1110) reference registered backends by ID, and (2) credential rotation without redeploying the fulfillment-service. Three prior enhancements addressed tenant-to-StorageClass resolution but left the infrastructure layer unmanaged. -StorageBackend introduces the infrastructure registration layer. It stores the provider type, management endpoint, and credentials. The entity is platform-scoped (not tenant-scoped), managed exclusively by Cloud Provider Admins through the private API, and exposed read-only through the public API with credentials stripped. +StorageBackend introduces the infrastructure registration layer. It stores the provider type, management endpoint, and credentials. The entity is platform-scoped (not tenant-scoped), managed exclusively by Cloud Provider Admins through the private API. There is no public API — tenants do not need visibility into storage backends; they interact with storage through StorageTier (OSAC-1110). ### Goals - Reuse the NetworkClass DB-backed server pattern (generic server, generic DAO, builder construction) with no reconciler or CRD. [Codebase: internal/servers/private_network_classes_server.go] -- Expose both private (full CRUD) and public (read-only) API layers following the established dual-API pattern. +- Expose a private CRUD API only — no public API. Tenants interact with storage through StorageTier (OSAC-1110), not directly with backends. - Store credentials inline in the JSONB `data` column, consistent with all existing OSAC entities (break_glass_credentials, identity_provider, user, cluster_template, hub). - Keep the StorageBackend schema provider-agnostic so that non-VAST backends (Ceph, Pure) use the same entity without schema changes. - Provide the foundation entity for StorageTier (OSAC-1110) composition without introducing StorageTier dependencies. @@ -50,9 +50,7 @@ StorageBackend introduces the infrastructure registration layer. It stores the p StorageBackend is a new entity in the fulfillment-service with two API surfaces: -1. **Private API** (`osac.private.v1.StorageBackends`): Full CRUD (Create, Get, List, Update, Delete). Used by Cloud Provider Admins. No Signal RPC — StorageBackend has no reconciler or controller, so there is nothing to signal. - -2. **Public API** (`osac.public.v1.StorageBackends`): Read-only (List, Get). Delegates to the private server with field mapping. Credential fields are excluded from the public API via `AddIgnoredFields()`. +**Private API** (`osac.private.v1.StorageBackends`): Full CRUD (Create, Get, List, Update, Delete). Used by Cloud Provider Admins and internal systems (e.g., future tenant onboarding controller that needs credentials to pass to AAP). No Signal RPC — StorageBackend has no reconciler or controller. No public API — tenants interact with storage through StorageTier, not directly with backends. The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables. A unique partial index on `name` enforces FR-9 (name uniqueness among active records). @@ -62,7 +60,7 @@ The generic server's `setPayload()` switch statement requires a new case for `St **Actors:** - Cloud Provider Admin — registers and manages storage backends via the private API -- Tenant — discovers available backends (provider, endpoint, status) via the public API; cannot see credentials +- Tenant — no direct access to StorageBackend; interacts with storage through StorageTier (OSAC-1110) **Starting state:** The fulfillment-service is deployed with no StorageBackend records. Storage configuration exists only in Ansible extra vars. @@ -99,9 +97,10 @@ The diagram shows the registration path. The admin registers the backend via the This enhancement adds the following API extensions: - **Private gRPC service** `osac.private.v1.StorageBackends` — CRUD, registered in the gRPC server startup alongside `NetworkClasses`. -- **Public gRPC service** `osac.public.v1.StorageBackends` — List + Get only, registered alongside the public `NetworkClasses` server. - **Event payload field** `storage_backend` added to `osac.private.v1.Event` — enables event-driven notifications for StorageBackend changes. +No public API is registered. Tenants do not need direct access to storage backends. + No CRDs, webhooks, or finalizers are introduced. No existing resources are modified. If the fulfillment-service is unavailable, StorageBackend operations fail but no other OSAC resources are affected — StorageBackend has no controller loop that could leave resources in an inconsistent state. ### Implementation Details/Notes/Constraints @@ -168,25 +167,6 @@ The service defines five RPCs following the NetworkClass service pattern (withou No Signal RPC is defined. Signal exists on other entities to wake up a reconciler when external state changes — StorageBackend has no reconciler, so Signal would have no consumer. Status fields (model, firmware_version) can be updated via the standard `Update` RPC if needed in the future. -#### Proto Schema — Public API - -**`storage_backend_type.proto`** (`osac.public.v1`): - -The public message mirrors the private message with these differences: -- `credentials` is excluded entirely (not present in the message definition) -- `endpoint.host` is included (tenants need to know which array serves their storage) -- `status` is included (tenants need to see backend availability) -- All fields are effectively `OUTPUT_ONLY` since the public API is read-only - -**`storage_backends_service.proto`** (`osac.public.v1`): - -| RPC | HTTP Method | Path | -|-----|-------------|------| -| `List` | `GET` | `/api/fulfillment/v1/storage_backends` | -| `Get` | `GET` | `/api/fulfillment/v1/storage_backends/{id}` | - -No Create, Update, or Delete RPCs in the public API. - #### Server Implementation **`private_storage_backends_server.go`**: Follows the `PrivateNetworkClassesServer` pattern: @@ -199,13 +179,6 @@ No Create, Update, or Delete RPCs in the public API. **Immutable fields on update:** `provider` is immutable after creation. Changing the storage provider would invalidate the endpoint, credentials, and any downstream StorageTier references. -**`storage_backends_server.go`** (public): Follows the `NetworkClassesServer` pattern: - -- Composes a `PrivateStorageBackendsServer` delegate + `GenericMapper` pair -- `inMapper` is not needed (no write RPCs on public API), but the pattern requires it for consistency with the builder; unused mapper fields can be nil -- `outMapper` maps private StorageBackend to public StorageBackend, skipping `credentials` -- Exposes only `List` and `Get` - #### Database Migration Migration file: `54_create_storage_backends_tables.up.sql` (next available sequence number) @@ -270,16 +243,6 @@ This partial index permits name reuse after soft deletion while preventing dupli In `start_grpc_server_cmd.go`, add registration blocks for both servers following the NetworkClass pattern: ```go -// Public StorageBackends server -storageBackendsServer, err := servers.NewStorageBackendsServer(). - SetLogger(logger). - SetNotifier(notifier). - SetAttributionLogic(attributionLogic). - SetTenancyLogic(tenancyLogic). - SetMetricsRegisterer(metricsRegisterer). - Build() -publicv1.RegisterStorageBackendsServer(grpcServer, storageBackendsServer) - // Private StorageBackends server privateStorageBackendsServer, err := servers.NewPrivateStorageBackendsServer(). SetLogger(logger). @@ -296,8 +259,8 @@ privatev1.RegisterStorageBackendsServer(grpcServer, privateStorageBackendsServer StorageBackend inherits the existing fulfillment-service security model: - **Authentication:** JWT validation via the gRPC interceptor chain. No changes to the authentication flow. -- **Authorization:** OPA policies control access to private vs. public API endpoints. Cloud Provider Admins have access to private CRUD. Tenants have access to public List + Get only. -- **Credential storage:** Credentials are stored inline in the JSONB `data` column, consistent with all existing OSAC entities. The public API excludes `credentials` entirely — tenants cannot see credential fields. This follows the same pattern as `break_glass_credentials`, `identity_provider`, `user`, `cluster_template`, and `hub`. +- **Authorization:** OPA policies control access to the private API endpoint. Only Cloud Provider Admins have access. There is no public API — tenants cannot access StorageBackend at all. +- **Credential storage:** Credentials are stored inline in the JSONB `data` column, consistent with all existing OSAC entities (`break_glass_credentials`, `identity_provider`, `user`, `cluster_template`, `hub`). Since there is no public API, credentials are never exposed to tenants. - **Input validation:** The server validates required fields (`provider`, `endpoint.host`, `credentials.username`, `credentials.password`) and provider immutability. The `endpoint.host` field is a string stored as-is; no DNS resolution or connection attempt is made during registration. No new authentication, authorization, or encryption mechanisms are introduced. @@ -315,8 +278,8 @@ All operations are transactional (wrapped by the gRPC database transaction inter StorageBackend is platform-scoped, not tenant-scoped. It follows the same access model as NetworkClass: -- **Private API:** Accessible to Cloud Provider Admins. OPA policies enforce role-based access. No Signal RPC. The `tenants` column in the database is empty (platform-scoped resources have no tenant owner). -- **Public API:** Accessible to all authenticated users (tenants). List and Get return all active backends (no tenant filtering). Credential fields are stripped by the public server's output mapper. +- **Private API:** Accessible to Cloud Provider Admins and internal systems (osac-operator). OPA policies enforce role-based access. The `tenants` column in the database is empty (platform-scoped resources have no tenant owner). +- **No public API.** Tenants interact with storage through StorageTier (OSAC-1110), not directly with backends. No `osac.openshift.io/tenant` or `osac.openshift.io/owner-reference` annotations are set on StorageBackend. This is consistent with NetworkClass, which is also platform-scoped. Tenant isolation applies at the StorageTier level (OSAC-1110), where tiers are assigned to tenants. @@ -336,7 +299,7 @@ No new observability changes. Existing monitoring mechanisms apply: ### Drawbacks -**Platform-scoped entity with credentials in the database.** StorageBackend stores credentials inline in the JSONB `data` column, which means credential content (username/password) lives in PostgreSQL. This is consistent with all existing OSAC entities that store credentials (break_glass_credentials, identity_provider, user, cluster_template, hub), but worth noting as the attack surface grows with the number of backends. The public API strips credentials, but the private API exposes them to all Cloud Provider Admins regardless of which backend they manage. +**Platform-scoped entity with credentials in the database.** StorageBackend stores credentials inline in the JSONB `data` column, which means credential content (username/password) lives in PostgreSQL. This is consistent with all existing OSAC entities that store credentials (break_glass_credentials, identity_provider, user, cluster_template, hub), but worth noting as the attack surface grows with the number of backends. The private API exposes credentials to all Cloud Provider Admins regardless of which backend they manage. ## Alternatives (Not Implemented) @@ -378,11 +341,6 @@ Testing follows the fulfillment-service's established Ginkgo v2 test patterns: - Optimistic locking: Update with stale version and `lock=true` returns `FAILED_PRECONDITION` - CEL filtering: List with `this.provider == "vast"` returns matching backends - Ordering: List with `metadata.name asc` returns sorted results -- `storage_backends_server_test.go` (public API): - - List and Get delegate correctly to private server - - `credentials` is absent from public responses - - Create/Update/Delete are not exposed - **Integration tests** (`it/`): StorageBackend CRUD via gRPC and REST endpoints against a kind cluster deployment, covering authentication and authorization. **E2E tests** (deferred to StorageTier integration): End-to-end tests covering the StorageBackend -> StorageTier -> tenant onboarding flow are scoped to OSAC-1110. @@ -395,8 +353,7 @@ For initial merge: - All integration tests pass - `buf lint` passes with no warnings -- Private and public API endpoints are accessible via both gRPC and REST -- `credentials` is confirmed absent from public API responses +- Private API endpoints are accessible via both gRPC and REST ## Upgrade / Downgrade Strategy From 99abd8b63f89fa8cc5265cecf0074c2a5c9ea503 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Tue, 16 Jun 2026 17:46:00 +0300 Subject: [PATCH 026/450] OSAC-1111: enforce metadata.name as required, immutable, and DNS-validated Add name validation constraints to StorageBackend design: - metadata.name is required on Create (must be non-empty) - metadata.name is immutable on Update (cannot be changed after creation) - DNS label format enforced by generic server (RFC 1035) - Uniqueness already covered by the partial index (FR-9) Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index 028f41c53..f33eeeab0 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -73,14 +73,14 @@ sequenceDiagram participant DB as PostgreSQL Admin->>FS: CreateStorageBackend(provider, endpoint, credentials, description) - FS->>FS: Validate required fields (provider, endpoint.host, credentials) + FS->>FS: Validate required fields (metadata.name, provider, endpoint.host, credentials) FS->>FS: Set state = READY, clear caller-provided ID FS->>DB: Insert into storage_backends (JSONB data column) DB-->>FS: Return with generated UUID FS-->>Admin: StorageBackend with id, state=READY ``` -The diagram shows the registration path. The admin registers the backend via the private API with credentials provided inline. The fulfillment-service validates required fields, sets the initial state to `READY`, generates a UUID, and persists the entity. Credentials are stored in the JSONB `data` column, consistent with how all other OSAC entities store credentials. +The diagram shows the registration path. The admin registers the backend via the private API with credentials provided inline. The fulfillment-service validates required fields (including `metadata.name`, which must be a valid DNS label per RFC 1035), sets the initial state to `READY`, generates a UUID, and persists the entity. Credentials are stored in the JSONB `data` column, consistent with how all other OSAC entities store credentials. **Credential rotation:** The admin calls `UpdateStorageBackend` with new `credentials` fields. No redeployment is required. @@ -88,7 +88,8 @@ The diagram shows the registration path. The admin registers the backend via the **Error cases:** - Duplicate name on active backend: Create returns `ALREADY_EXISTS` (enforced by the unique partial index). -- Missing required fields (provider, endpoint.host, credentials): Create/Update returns `INVALID_ARGUMENT`. +- Missing required fields (metadata.name, provider, endpoint.host, credentials): Create returns `INVALID_ARGUMENT`. +- Name change on Update: returns `INVALID_ARGUMENT` (`metadata.name` is immutable after creation). - Update with stale version and `lock=true`: returns `FAILED_PRECONDITION` (optimistic concurrency control). - Delete of non-existent backend: returns `NOT_FOUND`. @@ -173,11 +174,11 @@ No Signal RPC is defined. Signal exists on other entities to wake up a reconcile - Struct wraps `GenericServer[*privatev1.StorageBackend]` - Builder pattern: `NewPrivateStorageBackendsServer()` returns a builder with `SetLogger`, `SetNotifier`, `SetAttributionLogic`, `SetTenancyLogic`, `SetMetricsRegisterer`, `Build()` -- `Create`: validates required fields (`provider`, `endpoint.host`, `credentials.username`, `credentials.password`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID -- `Update`: fetches existing object, merges via `applyStorageBackendUpdate`, validates immutability of `provider` field, delegates to generic server +- `Create`: validates required fields (`metadata.name`, `provider`, `endpoint.host`, `credentials.username`, `credentials.password`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID. The generic server's `validateName()` enforces RFC 1035 DNS label format (1–63 chars, lowercase alphanumeric + hyphens, no leading/trailing hyphens). Name uniqueness among active backends is enforced by the `storage_backends_unique_active_name` partial index. +- `Update`: fetches existing object, merges via `applyStorageBackendUpdate`, validates immutability of `provider` and `metadata.name` fields, delegates to generic server - `Delete`: delegates directly to generic server (soft delete) -**Immutable fields on update:** `provider` is immutable after creation. Changing the storage provider would invalidate the endpoint, credentials, and any downstream StorageTier references. +**Immutable fields on update:** `provider` and `metadata.name` are immutable after creation. Changing the storage provider would invalidate the endpoint, credentials, and any downstream StorageTier references. Name immutability ensures stable human-readable identifiers for operational use and prevents confusion when backends are referenced by name in logs and configurations. #### Database Migration @@ -261,7 +262,7 @@ StorageBackend inherits the existing fulfillment-service security model: - **Authentication:** JWT validation via the gRPC interceptor chain. No changes to the authentication flow. - **Authorization:** OPA policies control access to the private API endpoint. Only Cloud Provider Admins have access. There is no public API — tenants cannot access StorageBackend at all. - **Credential storage:** Credentials are stored inline in the JSONB `data` column, consistent with all existing OSAC entities (`break_glass_credentials`, `identity_provider`, `user`, `cluster_template`, `hub`). Since there is no public API, credentials are never exposed to tenants. -- **Input validation:** The server validates required fields (`provider`, `endpoint.host`, `credentials.username`, `credentials.password`) and provider immutability. The `endpoint.host` field is a string stored as-is; no DNS resolution or connection attempt is made during registration. +- **Input validation:** The server validates required fields (`metadata.name`, `provider`, `endpoint.host`, `credentials.username`, `credentials.password`), DNS label format for `metadata.name` (RFC 1035, enforced by the generic server), and immutability of `provider` and `metadata.name` on update. The `endpoint.host` field is a string stored as-is; no DNS resolution or connection attempt is made during registration. No new authentication, authorization, or encryption mechanisms are introduced. @@ -335,8 +336,8 @@ Testing follows the fulfillment-service's established Ginkgo v2 test patterns: - `private_storage_backends_server_test.go`: - CRUD lifecycle: Create with all fields, Get by ID, List with pagination, Update with field mask, Delete (soft-delete) - - Validation: missing `provider`, missing `endpoint.host`, missing `credentials` return `INVALID_ARGUMENT` - - Immutability: Update with changed `provider` returns `INVALID_ARGUMENT` + - Validation: missing `metadata.name`, missing `provider`, missing `endpoint.host`, missing `credentials` return `INVALID_ARGUMENT`; invalid DNS label name returns `INVALID_ARGUMENT` + - Immutability: Update with changed `provider` returns `INVALID_ARGUMENT`; Update with changed `metadata.name` returns `INVALID_ARGUMENT` - Name uniqueness: Create with duplicate active name returns `ALREADY_EXISTS`; Create after soft-delete of same name succeeds - Optimistic locking: Update with stale version and `lock=true` returns `FAILED_PRECONDITION` - CEL filtering: List with `this.provider == "vast"` returns matching backends From 93f3a13b9f18fb4b714d006a1cac9de8d9a324fc Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Tue, 16 Jun 2026 17:48:29 +0300 Subject: [PATCH 027/450] OSAC-1111: simplify endpoint to a single string field Replace StorageBackendEndpoint message (host + port) with a plain string field that accepts a URL or host:port. Simpler and more flexible. Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index f33eeeab0..fa5c8682a 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -73,7 +73,7 @@ sequenceDiagram participant DB as PostgreSQL Admin->>FS: CreateStorageBackend(provider, endpoint, credentials, description) - FS->>FS: Validate required fields (metadata.name, provider, endpoint.host, credentials) + FS->>FS: Validate required fields (metadata.name, provider, endpoint, credentials) FS->>FS: Set state = READY, clear caller-provided ID FS->>DB: Insert into storage_backends (JSONB data column) DB-->>FS: Return with generated UUID @@ -88,7 +88,7 @@ The diagram shows the registration path. The admin registers the backend via the **Error cases:** - Duplicate name on active backend: Create returns `ALREADY_EXISTS` (enforced by the unique partial index). -- Missing required fields (metadata.name, provider, endpoint.host, credentials): Create returns `INVALID_ARGUMENT`. +- Missing required fields (metadata.name, provider, endpoint, credentials): Create returns `INVALID_ARGUMENT`. - Name change on Update: returns `INVALID_ARGUMENT` (`metadata.name` is immutable after creation). - Update with stale version and `lock=true`: returns `FAILED_PRECONDITION` (optimistic concurrency control). - Delete of non-existent backend: returns `NOT_FOUND`. @@ -116,17 +116,10 @@ No CRDs, webhooks, or finalizers are introduced. No existing resources are modif | `metadata` | `Metadata` | Yes | Standard OSAC metadata (name, labels, annotations, version) | | `provider` | `string` | Yes | Storage provider identifier (e.g., `"vast"`, `"ceph"`, `"pure"`) | | `description` | `string` | No | Human-readable description of the backend | -| `endpoint` | `StorageBackendEndpoint` | Yes | Management endpoint for the storage array | +| `endpoint` | `string` | Yes | Management endpoint for the storage array (URL or `host:port`) | | `credentials` | `StorageBackendCredentials` | Yes | Provider-specific credentials stored inline, consistent with existing OSAC credential patterns | | `status` | `StorageBackendStatus` | System | Operational status, set on create | -**`StorageBackendEndpoint` message:** - -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `host` | `string` | Yes | Hostname or IP of the storage management API | -| `port` | `int32` | No | Port number; provider-specific default if omitted | - **`StorageBackendCredentials` message:** | Field | Type | Required | Description | @@ -174,7 +167,7 @@ No Signal RPC is defined. Signal exists on other entities to wake up a reconcile - Struct wraps `GenericServer[*privatev1.StorageBackend]` - Builder pattern: `NewPrivateStorageBackendsServer()` returns a builder with `SetLogger`, `SetNotifier`, `SetAttributionLogic`, `SetTenancyLogic`, `SetMetricsRegisterer`, `Build()` -- `Create`: validates required fields (`metadata.name`, `provider`, `endpoint.host`, `credentials.username`, `credentials.password`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID. The generic server's `validateName()` enforces RFC 1035 DNS label format (1–63 chars, lowercase alphanumeric + hyphens, no leading/trailing hyphens). Name uniqueness among active backends is enforced by the `storage_backends_unique_active_name` partial index. +- `Create`: validates required fields (`metadata.name`, `provider`, `endpoint`, `credentials.username`, `credentials.password`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID. The generic server's `validateName()` enforces RFC 1035 DNS label format (1–63 chars, lowercase alphanumeric + hyphens, no leading/trailing hyphens). Name uniqueness among active backends is enforced by the `storage_backends_unique_active_name` partial index. - `Update`: fetches existing object, merges via `applyStorageBackendUpdate`, validates immutability of `provider` and `metadata.name` fields, delegates to generic server - `Delete`: delegates directly to generic server (soft delete) @@ -262,7 +255,7 @@ StorageBackend inherits the existing fulfillment-service security model: - **Authentication:** JWT validation via the gRPC interceptor chain. No changes to the authentication flow. - **Authorization:** OPA policies control access to the private API endpoint. Only Cloud Provider Admins have access. There is no public API — tenants cannot access StorageBackend at all. - **Credential storage:** Credentials are stored inline in the JSONB `data` column, consistent with all existing OSAC entities (`break_glass_credentials`, `identity_provider`, `user`, `cluster_template`, `hub`). Since there is no public API, credentials are never exposed to tenants. -- **Input validation:** The server validates required fields (`metadata.name`, `provider`, `endpoint.host`, `credentials.username`, `credentials.password`), DNS label format for `metadata.name` (RFC 1035, enforced by the generic server), and immutability of `provider` and `metadata.name` on update. The `endpoint.host` field is a string stored as-is; no DNS resolution or connection attempt is made during registration. +- **Input validation:** The server validates required fields (`metadata.name`, `provider`, `endpoint`, `credentials.username`, `credentials.password`), DNS label format for `metadata.name` (RFC 1035, enforced by the generic server), and immutability of `provider` and `metadata.name` on update. The `endpoint` field is an opaque string (URL or `host:port`) stored as-is; no DNS resolution or connection attempt is made during registration. No new authentication, authorization, or encryption mechanisms are introduced. @@ -336,7 +329,7 @@ Testing follows the fulfillment-service's established Ginkgo v2 test patterns: - `private_storage_backends_server_test.go`: - CRUD lifecycle: Create with all fields, Get by ID, List with pagination, Update with field mask, Delete (soft-delete) - - Validation: missing `metadata.name`, missing `provider`, missing `endpoint.host`, missing `credentials` return `INVALID_ARGUMENT`; invalid DNS label name returns `INVALID_ARGUMENT` + - Validation: missing `metadata.name`, missing `provider`, missing `endpoint`, missing `credentials` return `INVALID_ARGUMENT`; invalid DNS label name returns `INVALID_ARGUMENT` - Immutability: Update with changed `provider` returns `INVALID_ARGUMENT`; Update with changed `metadata.name` returns `INVALID_ARGUMENT` - Name uniqueness: Create with duplicate active name returns `ALREADY_EXISTS`; Create after soft-delete of same name succeeds - Optimistic locking: Update with stale version and `lock=true` returns `FAILED_PRECONDITION` From 22333f84f2998bc92490cd19aa00eecd3abdb8fc Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Tue, 16 Jun 2026 18:29:47 +0300 Subject: [PATCH 028/450] OSAC-1111: mark deletion strategy as open question (soft-delete vs hard-delete) The generic DAO uses soft-delete by default. Switching to hard-delete requires DAO extension or custom server logic. Mark as OQ-1 for review. Also adds referential integrity check: delete is rejected if any StorageTier references the backend. Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 28 +++++++++++++------ 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index fa5c8682a..d7fcb20ab 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -52,7 +52,7 @@ StorageBackend is a new entity in the fulfillment-service with two API surfaces: **Private API** (`osac.private.v1.StorageBackends`): Full CRUD (Create, Get, List, Update, Delete). Used by Cloud Provider Admins and internal systems (e.g., future tenant onboarding controller that needs credentials to pass to AAP). No Signal RPC — StorageBackend has no reconciler or controller. No public API — tenants interact with storage through StorageTier, not directly with backends. -The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables. A unique partial index on `name` enforces FR-9 (name uniqueness among active records). +The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables. A unique partial index on `name` enforces FR-9 (name uniqueness among active records). **[OPEN: The deletion strategy (soft-delete vs hard-delete) is under review — see Open Questions.]** The generic server's `setPayload()` switch statement requires a new case for `StorageBackend`, and the Event proto requires a new `storage_backend` payload field. @@ -84,7 +84,7 @@ The diagram shows the registration path. The admin registers the backend via the **Credential rotation:** The admin calls `UpdateStorageBackend` with new `credentials` fields. No redeployment is required. -**Decommission:** The admin calls `DeleteStorageBackend`. The backend is soft-deleted: excluded from List results but preserved in the `archived_storage_backends` table for audit. Name reuse is allowed after soft deletion. +**Decommission:** The admin calls `DeleteStorageBackend`. The backend is soft-deleted using the standard DAO pattern (sets `deletion_timestamp`, archives the record). Delete is rejected with `FAILED_PRECONDITION` if any StorageTier references the backend — the admin must remove all tier associations first. **[OPEN: The deletion strategy (soft-delete vs hard-delete) is under review — see Open Questions.]** **Error cases:** - Duplicate name on active backend: Create returns `ALREADY_EXISTS` (enforced by the unique partial index). @@ -169,7 +169,7 @@ No Signal RPC is defined. Signal exists on other entities to wake up a reconcile - Builder pattern: `NewPrivateStorageBackendsServer()` returns a builder with `SetLogger`, `SetNotifier`, `SetAttributionLogic`, `SetTenancyLogic`, `SetMetricsRegisterer`, `Build()` - `Create`: validates required fields (`metadata.name`, `provider`, `endpoint`, `credentials.username`, `credentials.password`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID. The generic server's `validateName()` enforces RFC 1035 DNS label format (1–63 chars, lowercase alphanumeric + hyphens, no leading/trailing hyphens). Name uniqueness among active backends is enforced by the `storage_backends_unique_active_name` partial index. - `Update`: fetches existing object, merges via `applyStorageBackendUpdate`, validates immutability of `provider` and `metadata.name` fields, delegates to generic server -- `Delete`: delegates directly to generic server (soft delete) +- `Delete`: checks for referencing StorageTier records; rejects with `FAILED_PRECONDITION` if any exist, otherwise delegates to generic server (soft delete via standard DAO pattern). **[OPEN: soft-delete vs hard-delete — see Open Questions]** **Immutable fields on update:** `provider` and `metadata.name` are immutable after creation. Changing the storage provider would invalidate the endpoint, credentials, and any downstream StorageTier references. Name immutability ensures stable human-readable identifiers for operational use and prevents confusion when backends are referenced by name in logs and configurations. @@ -224,7 +224,7 @@ create unique index storage_backends_unique_active_name where deletion_timestamp = 'epoch'; ``` -This partial index permits name reuse after soft deletion while preventing duplicate names among active records. The DAO's unique violation error maps to gRPC `ALREADY_EXISTS`. +This partial index permits name reuse after soft deletion while preventing duplicate names among active records. The DAO's unique violation error maps to gRPC `ALREADY_EXISTS`. **[OPEN: If hard-delete is chosen, this becomes a plain unique index without the partial condition.]** #### Generic Server Changes @@ -265,6 +265,7 @@ No new authentication, authorization, or encryption mechanisms are introduced. |---|---|---|---| | Database unavailable during Create | gRPC interceptor rolls back the transaction | `UNAVAILABLE` error | Retry the request; the operation is idempotent (name uniqueness check prevents duplicates) | | Duplicate active name on Create | Unique partial index violation | `ALREADY_EXISTS` with the conflicting name | Choose a different name or delete the existing backend first | +| Delete with referencing StorageTiers | Server checks for references before delete | `FAILED_PRECONDITION` listing the referencing tiers | Remove all StorageTier associations first, then retry | | Stale version on Update with `lock=true` | DAO rejects the write | `FAILED_PRECONDITION` | Re-fetch the current version and retry | All operations are transactional (wrapped by the gRPC database transaction interceptor) and idempotent. There is no reconciliation loop that could leave resources in a partially updated state. @@ -287,9 +288,9 @@ No new observability changes. Existing monitoring mechanisms apply: ### Risks and Mitigations -**Soft-delete name reuse and StorageTier references:** When a backend is soft-deleted and its name is reused for a new backend, a StorageTier referencing the old backend by ID continues to resolve correctly (references use ID, not name). However, confusion may arise if operators expect name-based lookup to return the latest backend. +**Referential integrity on delete:** Delete is rejected if any StorageTier references the backend. This prevents orphaned tier-to-backend references but requires the admin to manually remove all tier associations before decommissioning a backend. -*Mitigation:* StorageTier references backends by ID. The API documentation explicitly states that List excludes soft-deleted backends and that name reuse is permitted after decommission. +*Mitigation:* The `FAILED_PRECONDITION` error message lists the referencing StorageTier IDs so the admin knows exactly what to clean up. ### Drawbacks @@ -319,7 +320,15 @@ Instead of storing credentials inline, the fulfillment-service could store a `cr ## Open Questions -None — all prior open questions have been resolved. Credentials are stored inline (matching existing OSAC patterns), and Signal RPC has been removed (no reconciler). +**OQ-1: Soft-delete vs hard-delete for StorageBackend decommission.** + +The generic DAO uses soft-delete by default (sets `deletion_timestamp`, archives the record to `archived_storage_backends`). All existing OSAC entities follow this pattern. However, StorageBackend may not need the archived record — audit can be handled by the existing observability infrastructure (structured logging, PostgreSQL NOTIFY events), and referential integrity (rejecting delete when StorageTiers reference the backend) prevents orphaned references. + +Options: +- **Keep soft-delete (current DAO default):** No DAO changes needed. Archived records are preserved. Name reuse is allowed via the partial index. Follows the established pattern. +- **Switch to hard-delete:** Requires extending the generic DAO with a hard-delete option or implementing custom delete logic in the StorageBackend server. Simpler schema (no archived table, plain unique index on name). Aligns with AWS/GCP patterns where infrastructure resources are hard-deleted and audit relies on external logs. + +**Owner:** Storage architect. **Impact:** Database schema, DAO usage, FR-6, FR-9. ## Test Plan @@ -328,10 +337,11 @@ Testing follows the fulfillment-service's established Ginkgo v2 test patterns: **Unit/integration tests** (co-located in `internal/servers/`): - `private_storage_backends_server_test.go`: - - CRUD lifecycle: Create with all fields, Get by ID, List with pagination, Update with field mask, Delete (soft-delete) + - CRUD lifecycle: Create with all fields, Get by ID, List with pagination, Update with field mask, Delete - Validation: missing `metadata.name`, missing `provider`, missing `endpoint`, missing `credentials` return `INVALID_ARGUMENT`; invalid DNS label name returns `INVALID_ARGUMENT` - Immutability: Update with changed `provider` returns `INVALID_ARGUMENT`; Update with changed `metadata.name` returns `INVALID_ARGUMENT` - - Name uniqueness: Create with duplicate active name returns `ALREADY_EXISTS`; Create after soft-delete of same name succeeds + - Name uniqueness: Create with duplicate active name returns `ALREADY_EXISTS`; Create after delete of same name succeeds + - Referential integrity: Delete with referencing StorageTier returns `FAILED_PRECONDITION`; Delete without references succeeds - Optimistic locking: Update with stale version and `lock=true` returns `FAILED_PRECONDITION` - CEL filtering: List with `this.provider == "vast"` returns matching backends - Ordering: List with `metadata.name asc` returns sorted results From 6f192d0cfdc6de8645f6440b4e8b2ea1f2f70172 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Wed, 17 Jun 2026 09:32:25 +0300 Subject: [PATCH 029/450] OSAC-1111: propose soft-delete + optional purge flag for deletion strategy OQ-1 resolution: soft-delete by default (standard DAO), hard-delete when purge=true on the request. Both modes enforce referential integrity (reject if StorageTiers reference the backend). Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 25 +++++++++++-------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index d7fcb20ab..ed89f19d2 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -52,7 +52,7 @@ StorageBackend is a new entity in the fulfillment-service with two API surfaces: **Private API** (`osac.private.v1.StorageBackends`): Full CRUD (Create, Get, List, Update, Delete). Used by Cloud Provider Admins and internal systems (e.g., future tenant onboarding controller that needs credentials to pass to AAP). No Signal RPC — StorageBackend has no reconciler or controller. No public API — tenants interact with storage through StorageTier, not directly with backends. -The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables. A unique partial index on `name` enforces FR-9 (name uniqueness among active records). **[OPEN: The deletion strategy (soft-delete vs hard-delete) is under review — see Open Questions.]** +The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables. A unique partial index on `name` enforces FR-9 (name uniqueness among active records). **[OPEN: see OQ-1]** The generic server's `setPayload()` switch statement requires a new case for `StorageBackend`, and the Event proto requires a new `storage_backend` payload field. @@ -84,7 +84,7 @@ The diagram shows the registration path. The admin registers the backend via the **Credential rotation:** The admin calls `UpdateStorageBackend` with new `credentials` fields. No redeployment is required. -**Decommission:** The admin calls `DeleteStorageBackend`. The backend is soft-deleted using the standard DAO pattern (sets `deletion_timestamp`, archives the record). Delete is rejected with `FAILED_PRECONDITION` if any StorageTier references the backend — the admin must remove all tier associations first. **[OPEN: The deletion strategy (soft-delete vs hard-delete) is under review — see Open Questions.]** +**Decommission:** The admin calls `DeleteStorageBackend`. By default the backend is soft-deleted (standard DAO pattern — sets `deletion_timestamp`, archives the record). When `purge=true` is set on the request, the backend is permanently removed from the database. Both modes reject the delete with `FAILED_PRECONDITION` if any StorageTier references the backend — the admin must remove all tier associations first. **[OPEN: see OQ-1]** **Error cases:** - Duplicate name on active backend: Create returns `ALREADY_EXISTS` (enforced by the unique partial index). @@ -169,7 +169,7 @@ No Signal RPC is defined. Signal exists on other entities to wake up a reconcile - Builder pattern: `NewPrivateStorageBackendsServer()` returns a builder with `SetLogger`, `SetNotifier`, `SetAttributionLogic`, `SetTenancyLogic`, `SetMetricsRegisterer`, `Build()` - `Create`: validates required fields (`metadata.name`, `provider`, `endpoint`, `credentials.username`, `credentials.password`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID. The generic server's `validateName()` enforces RFC 1035 DNS label format (1–63 chars, lowercase alphanumeric + hyphens, no leading/trailing hyphens). Name uniqueness among active backends is enforced by the `storage_backends_unique_active_name` partial index. - `Update`: fetches existing object, merges via `applyStorageBackendUpdate`, validates immutability of `provider` and `metadata.name` fields, delegates to generic server -- `Delete`: checks for referencing StorageTier records; rejects with `FAILED_PRECONDITION` if any exist, otherwise delegates to generic server (soft delete via standard DAO pattern). **[OPEN: soft-delete vs hard-delete — see Open Questions]** +- `Delete`: checks for referencing StorageTier records; rejects with `FAILED_PRECONDITION` if any exist. Default: soft-delete via standard DAO. When `purge=true`: hard-delete, permanently removing the record. **[OPEN: see OQ-1]** **Immutable fields on update:** `provider` and `metadata.name` are immutable after creation. Changing the storage provider would invalidate the endpoint, credentials, and any downstream StorageTier references. Name immutability ensures stable human-readable identifiers for operational use and prevents confusion when backends are referenced by name in logs and configurations. @@ -224,7 +224,7 @@ create unique index storage_backends_unique_active_name where deletion_timestamp = 'epoch'; ``` -This partial index permits name reuse after soft deletion while preventing duplicate names among active records. The DAO's unique violation error maps to gRPC `ALREADY_EXISTS`. **[OPEN: If hard-delete is chosen, this becomes a plain unique index without the partial condition.]** +This partial index permits name reuse after soft deletion while preventing duplicate names among active records. The DAO's unique violation error maps to gRPC `ALREADY_EXISTS`. **[OPEN: see OQ-1 — the partial condition is needed for soft-delete; hard-delete via `purge` removes the row entirely so the partial index still works correctly.]** #### Generic Server Changes @@ -316,19 +316,22 @@ Instead of storing credentials inline, the fulfillment-service could store a `cr *Cons:* Introduces a new credential pattern that no existing OSAC entity uses — all current entities (break_glass_credentials, identity_provider, user, cluster_template, hub) store credentials inline. Adds operational complexity (Secret must exist, namespace resolution, lifecycle management). Creates an inconsistency in the codebase. -*Rejection:* Consistency with existing OSAC patterns outweighs the security benefit for phase 1. All other entities store credentials inline, and introducing a divergent pattern without an OSAC-core decision on unified credential management would create unnecessary complexity. This can be revisited when OSAC establishes a platform-wide credential storage strategy. +*Rejection:* OSAC core are aware of the need for a credentials solution for the system(this is not a specific to storage backends) - see this ticket https://redhat.atlassian.net/browse/OSAC-1567 . Also, consistency with existing OSAC patterns outweighs the security benefit for phase 1. All other entities store credentials inline, and introducing a divergent pattern without an OSAC-core decision on unified credential management would create unnecessary complexity. This can be revisited when OSAC establishes a platform-wide credential storage strategy. ## Open Questions -**OQ-1: Soft-delete vs hard-delete for StorageBackend decommission.** +**OQ-1: Deletion strategy — soft-delete by default, optional hard-delete with `purge` flag.** -The generic DAO uses soft-delete by default (sets `deletion_timestamp`, archives the record to `archived_storage_backends`). All existing OSAC entities follow this pattern. However, StorageBackend may not need the archived record — audit can be handled by the existing observability infrastructure (structured logging, PostgreSQL NOTIFY events), and referential integrity (rejecting delete when StorageTiers reference the backend) prevents orphaned references. +**Proposed resolution:** Support both modes. `DeleteStorageBackend` performs a soft-delete by default (standard DAO pattern — sets `deletion_timestamp`, archives to `archived_storage_backends`). When `purge=true` is set on the request, the backend is permanently removed from the database. -Options: -- **Keep soft-delete (current DAO default):** No DAO changes needed. Archived records are preserved. Name reuse is allowed via the partial index. Follows the established pattern. -- **Switch to hard-delete:** Requires extending the generic DAO with a hard-delete option or implementing custom delete logic in the StorageBackend server. Simpler schema (no archived table, plain unique index on name). Aligns with AWS/GCP patterns where infrastructure resources are hard-deleted and audit relies on external logs. +Both modes enforce referential integrity — delete is rejected with `FAILED_PRECONDITION` if any StorageTier references the backend, regardless of soft vs hard delete. -**Owner:** Storage architect. **Impact:** Database schema, DAO usage, FR-6, FR-9. +This requires: +- A `bool purge` field on the `DeleteStorageBackendRequest` message. +- Extending the generic DAO or adding custom delete logic in `PrivateStorageBackendsServer.Delete()` to bypass the archive step when `purge=true`. +- The `archived_storage_backends` table and partial unique index are still needed for the soft-delete path. + +**Owner:** Storage architect. **Impact:** Proto schema (`DeleteStorageBackendRequest`), DAO usage, FR-6. ## Test Plan From 5ae1584433d51f7f4a7f730ace8712389af7aa5f Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Wed, 17 Jun 2026 09:57:45 +0300 Subject: [PATCH 030/450] OSAC-1111: model decommission as state lifecycle, not archive Replace soft-delete/archive as user-facing decommission with state-based lifecycle: - READY: operational (phase 1) - MAINTENANCE: temporary unavailability, reversible (phase 0.2) - DECOMMISSIONED: terminal, backend retired but visible (phase 0.2) Delete RPC permanently removes the record. In phase 0.2, only allowed on DECOMMISSIONED backends with no StorageTier references. The archived table remains as DAO infrastructure but is not user-facing. Resolves OQ-1. Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 50 ++++++++++++------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index ed89f19d2..e94dd698b 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -52,7 +52,7 @@ StorageBackend is a new entity in the fulfillment-service with two API surfaces: **Private API** (`osac.private.v1.StorageBackends`): Full CRUD (Create, Get, List, Update, Delete). Used by Cloud Provider Admins and internal systems (e.g., future tenant onboarding controller that needs credentials to pass to AAP). No Signal RPC — StorageBackend has no reconciler or controller. No public API — tenants interact with storage through StorageTier, not directly with backends. -The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables. A unique partial index on `name` enforces FR-9 (name uniqueness among active records). **[OPEN: see OQ-1]** +The database schema follows the standard JSONB `data` column pattern with `storage_backends` and `archived_storage_backends` tables (the archived table is required by the generic DAO, not user-facing). A unique partial index on `name` enforces FR-9 (name uniqueness among active records). The generic server's `setPayload()` switch statement requires a new case for `StorageBackend`, and the Event proto requires a new `storage_backend` payload field. @@ -84,7 +84,9 @@ The diagram shows the registration path. The admin registers the backend via the **Credential rotation:** The admin calls `UpdateStorageBackend` with new `credentials` fields. No redeployment is required. -**Decommission:** The admin calls `DeleteStorageBackend`. By default the backend is soft-deleted (standard DAO pattern — sets `deletion_timestamp`, archives the record). When `purge=true` is set on the request, the backend is permanently removed from the database. Both modes reject the delete with `FAILED_PRECONDITION` if any StorageTier references the backend — the admin must remove all tier associations first. **[OPEN: see OQ-1]** +**Decommission (phase 0.2):** The admin transitions the backend to `DECOMMISSIONED` state via `UpdateStorageBackend`. The record remains in the database and is visible via Get and List, but the backend is no longer available for new StorageTier associations. This is a terminal state — the backend cannot return to `READY`. An intermediate `MAINTENANCE` state is available for temporary unavailability (e.g., firmware upgrade) and can be reversed back to `READY`. + +**Permanent removal:** The admin calls `DeleteStorageBackend`. Delete is only allowed on `DECOMMISSIONED` backends (phase 0.2) or any backend with no StorageTier references (phase 1). The DAO performs the standard archive-and-delete internally, but the archived table is an infrastructure detail, not a user-facing feature. **Error cases:** - Duplicate name on active backend: Create returns `ALREADY_EXISTS` (enforced by the unique partial index). @@ -144,8 +146,22 @@ Credentials are stored inline in the JSONB `data` column, following the same pat |-------|--------|-------------| | `STORAGE_BACKEND_STATE_UNSPECIFIED` | 0 | Default zero value | | `STORAGE_BACKEND_STATE_READY` | 1 | Backend registered and available | +| `STORAGE_BACKEND_STATE_MAINTENANCE` | 2 | Backend temporarily unavailable (reversible — can return to `READY`) | +| `STORAGE_BACKEND_STATE_DECOMMISSIONED` | 3 | Backend permanently retired (terminal — cannot return to `READY`) | + +**Phase 1** implements only `READY`. The `MAINTENANCE` and `DECOMMISSIONED` states are defined in the proto enum from the start but rejected by the server until phase 0.2. + +**State transitions (phase 0.2):** + +``` +READY ⇄ MAINTENANCE (reversible) +READY → DECOMMISSIONED (terminal, one-way) +MAINTENANCE → DECOMMISSIONED (terminal, one-way) +``` + +`DECOMMISSIONED` replaces the traditional soft-delete pattern — the record stays in the database and remains visible via Get and List (filterable by state), but the backend is no longer available for new StorageTier associations. Delete (permanent removal) is only allowed on `DECOMMISSIONED` backends with no remaining StorageTier references. -The enum starts with `READY` as the only operational state. Additional states (e.g., `PENDING`, `FAILED`, `DEGRADED`) are added when health probing or reconciliation capabilities are introduced. The `UNSPECIFIED` sentinel follows proto3 convention. +The `UNSPECIFIED` sentinel follows proto3 convention. **`storage_backends_service.proto`** (`osac.private.v1`): @@ -169,10 +185,12 @@ No Signal RPC is defined. Signal exists on other entities to wake up a reconcile - Builder pattern: `NewPrivateStorageBackendsServer()` returns a builder with `SetLogger`, `SetNotifier`, `SetAttributionLogic`, `SetTenancyLogic`, `SetMetricsRegisterer`, `Build()` - `Create`: validates required fields (`metadata.name`, `provider`, `endpoint`, `credentials.username`, `credentials.password`), sets state to `STORAGE_BACKEND_STATE_READY`, clears caller-provided ID. The generic server's `validateName()` enforces RFC 1035 DNS label format (1–63 chars, lowercase alphanumeric + hyphens, no leading/trailing hyphens). Name uniqueness among active backends is enforced by the `storage_backends_unique_active_name` partial index. - `Update`: fetches existing object, merges via `applyStorageBackendUpdate`, validates immutability of `provider` and `metadata.name` fields, delegates to generic server -- `Delete`: checks for referencing StorageTier records; rejects with `FAILED_PRECONDITION` if any exist. Default: soft-delete via standard DAO. When `purge=true`: hard-delete, permanently removing the record. **[OPEN: see OQ-1]** +- `Delete`: checks for referencing StorageTier records; rejects with `FAILED_PRECONDITION` if any exist. In phase 0.2, also rejects unless the backend is in `DECOMMISSIONED` state. Delegates to the generic server (standard DAO archive-and-delete) **Immutable fields on update:** `provider` and `metadata.name` are immutable after creation. Changing the storage provider would invalidate the endpoint, credentials, and any downstream StorageTier references. Name immutability ensures stable human-readable identifiers for operational use and prevents confusion when backends are referenced by name in logs and configurations. +**State transition validation (phase 0.2):** The server enforces the state machine — `READY ⇄ MAINTENANCE` (reversible), `READY → DECOMMISSIONED` and `MAINTENANCE → DECOMMISSIONED` (terminal). Any other transition returns `INVALID_ARGUMENT`. + #### Database Migration Migration file: `54_create_storage_backends_tables.up.sql` (next available sequence number) @@ -224,7 +242,7 @@ create unique index storage_backends_unique_active_name where deletion_timestamp = 'epoch'; ``` -This partial index permits name reuse after soft deletion while preventing duplicate names among active records. The DAO's unique violation error maps to gRPC `ALREADY_EXISTS`. **[OPEN: see OQ-1 — the partial condition is needed for soft-delete; hard-delete via `purge` removes the row entirely so the partial index still works correctly.]** +This partial index permits name reuse after deletion while preventing duplicate names among active records. The DAO's unique violation error maps to gRPC `ALREADY_EXISTS`. #### Generic Server Changes @@ -288,9 +306,9 @@ No new observability changes. Existing monitoring mechanisms apply: ### Risks and Mitigations -**Referential integrity on delete:** Delete is rejected if any StorageTier references the backend. This prevents orphaned tier-to-backend references but requires the admin to manually remove all tier associations before decommissioning a backend. +**Referential integrity on delete:** Delete is rejected if any StorageTier references the backend. This prevents orphaned tier-to-backend references but requires the admin to decommission the backend and remove all tier associations before permanent removal. -*Mitigation:* The `FAILED_PRECONDITION` error message lists the referencing StorageTier IDs so the admin knows exactly what to clean up. +*Mitigation:* The `FAILED_PRECONDITION` error message lists the referencing StorageTier IDs so the admin knows exactly what to clean up. The `DECOMMISSIONED` state (phase 0.2) provides a safe intermediate step — the backend is visible but no longer available for new associations. ### Drawbacks @@ -316,22 +334,18 @@ Instead of storing credentials inline, the fulfillment-service could store a `cr *Cons:* Introduces a new credential pattern that no existing OSAC entity uses — all current entities (break_glass_credentials, identity_provider, user, cluster_template, hub) store credentials inline. Adds operational complexity (Secret must exist, namespace resolution, lifecycle management). Creates an inconsistency in the codebase. -*Rejection:* OSAC core are aware of the need for a credentials solution for the system(this is not a specific to storage backends) - see this ticket https://redhat.atlassian.net/browse/OSAC-1567 . Also, consistency with existing OSAC patterns outweighs the security benefit for phase 1. All other entities store credentials inline, and introducing a divergent pattern without an OSAC-core decision on unified credential management would create unnecessary complexity. This can be revisited when OSAC establishes a platform-wide credential storage strategy. +*Rejection:* Consistency with existing OSAC patterns outweighs the security benefit for phase 1. All other entities store credentials inline, and introducing a divergent pattern without an OSAC-core decision on unified credential management would create unnecessary complexity. This can be revisited when OSAC establishes a platform-wide credential storage strategy. ## Open Questions -**OQ-1: Deletion strategy — soft-delete by default, optional hard-delete with `purge` flag.** - -**Proposed resolution:** Support both modes. `DeleteStorageBackend` performs a soft-delete by default (standard DAO pattern — sets `deletion_timestamp`, archives to `archived_storage_backends`). When `purge=true` is set on the request, the backend is permanently removed from the database. - -Both modes enforce referential integrity — delete is rejected with `FAILED_PRECONDITION` if any StorageTier references the backend, regardless of soft vs hard delete. +**OQ-1 (resolved): Deletion strategy — state-based lifecycle replaces soft-delete.** -This requires: -- A `bool purge` field on the `DeleteStorageBackendRequest` message. -- Extending the generic DAO or adding custom delete logic in `PrivateStorageBackendsServer.Delete()` to bypass the archive step when `purge=true`. -- The `archived_storage_backends` table and partial unique index are still needed for the soft-delete path. +Decommissioning is modeled as a state transition rather than the DAO's archive pattern: +- `MAINTENANCE` (phase 0.2): temporary unavailability, reversible to `READY`. +- `DECOMMISSIONED` (phase 0.2): terminal state, backend is retired but remains visible via Get/List. +- `Delete` RPC: permanent removal via the standard DAO (archive-and-delete). In phase 0.2, only allowed on `DECOMMISSIONED` backends with no StorageTier references. In phase 1, allowed on any backend with no StorageTier references. -**Owner:** Storage architect. **Impact:** Proto schema (`DeleteStorageBackendRequest`), DAO usage, FR-6. +The `archived_storage_backends` table is required by the generic DAO infrastructure but is not exposed as a user-facing feature. ## Test Plan From 8adf158ea9be60c4c24deee6e87a995c486c69cd Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Wed, 17 Jun 2026 13:26:10 +0300 Subject: [PATCH 031/450] OSAC-1111: address review feedback (comments 1, 2, 3, 7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fix summary: remove stale "public read-only API" reference - Fix proposal: "a private API surface" not "two API surfaces" - Remove storage_backends_by_tenant index (platform-scoped, no tenants) - Add phase 1→0.2 behavioral change and downgrade notes for state lifecycle tightening Assisted-by: Claude Code Signed-off-by: Roy Golan --- enhancements/storage-backend-osac-1111/design.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index e94dd698b..997a10497 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -20,7 +20,7 @@ superseded-by: ## Summary -This enhancement adds a `StorageBackend` entity to the fulfillment-service, providing a private gRPC API (CRUD) and a public read-only API for managing storage array registrations. StorageBackend follows the NetworkClass pattern: DB-backed with no Kubernetes CRD and no reconciler. See [PRD](prd.md) for detailed requirements. +This enhancement adds a `StorageBackend` entity to the fulfillment-service, providing a private gRPC API (CRUD) for managing storage array registrations. StorageBackend follows the NetworkClass pattern: DB-backed with no Kubernetes CRD and no reconciler. See [PRD](prd.md) for detailed requirements. ## Motivation @@ -48,7 +48,7 @@ StorageBackend introduces the infrastructure registration layer. It stores the p ## Proposal -StorageBackend is a new entity in the fulfillment-service with two API surfaces: +StorageBackend is a new entity in the fulfillment-service with a private API surface: **Private API** (`osac.private.v1.StorageBackends`): Full CRUD (Create, Get, List, Update, Delete). Used by Cloud Provider Admins and internal systems (e.g., future tenant onboarding controller that needs credentials to pass to AAP). No Signal RPC — StorageBackend has no reconciler or controller. No public API — tenants interact with storage through StorageTier, not directly with backends. @@ -230,7 +230,6 @@ Indexes: ```sql create index storage_backends_by_name on storage_backends (name); create index storage_backends_by_owner on storage_backends using gin (creators); -create index storage_backends_by_tenant on storage_backends using gin (tenants); create index storage_backends_by_label on storage_backends using gin (labels); ``` @@ -380,7 +379,11 @@ For initial merge: This is a new API with no upgrade impact. The database migration (`54_create_storage_backends_tables.up.sql`) is additive — it creates new tables and indexes without modifying existing schema. -**Downgrade:** Requires deleting all StorageBackend records and then reverting the database migration (dropping the `storage_backends` and `archived_storage_backends` tables). No other OSAC entities reference StorageBackend in this phase, so deletion has no cascading effects. +**Phase 1 → Phase 0.2 behavioral change:** In phase 1, `Delete` is allowed on any backend with no StorageTier references. In phase 0.2, `Delete` is restricted to backends in `DECOMMISSIONED` state. Admins who previously deleted backends directly must first transition them to `DECOMMISSIONED`. This is a deliberate tightening — document in release notes. + +**Downgrade (phase 0.2 → phase 1):** Any backends in `MAINTENANCE` or `DECOMMISSIONED` state must be transitioned back to `READY` or deleted before downgrading, since phase 1 only recognizes the `READY` state. + +**Downgrade (phase 1 → removal):** Requires deleting all StorageBackend records and then reverting the database migration (dropping the `storage_backends` and `archived_storage_backends` tables). No other OSAC entities reference StorageBackend in this phase, so deletion has no cascading effects. ## Version Skew Strategy From fbc21aa891f8c45a019b9656f6ec29254c193433 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Wed, 17 Jun 2026 13:39:27 +0300 Subject: [PATCH 032/450] OSAC-1111: address review feedback round 2 (comments 5, 6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove model and firmware_version from StorageBackendStatus — deferred until state reconciliation is implemented - Fix schema to use current column pattern: singular creator/tenant (not legacy creators/tenants arrays), add version column - Add creator and tenant B-tree indexes (standard pattern) Assisted-by: Claude Code Signed-off-by: Roy Golan --- .../storage-backend-osac-1111/design.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index 997a10497..fe1f242a3 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -137,8 +137,6 @@ Credentials are stored inline in the JSONB `data` column, following the same pat |-------|------|----------|-------------| | `state` | `StorageBackendState` | Yes | Current lifecycle state | | `message` | `string` | No | Human-readable status detail | -| `model` | `string` | No | Storage array model (e.g., `"VAST C-100"`) | -| `firmware_version` | `string` | No | Reported firmware version | **`StorageBackendState` enum:** @@ -175,7 +173,7 @@ The service defines five RPCs following the NetworkClass service pattern (withou | `Update` | `PATCH` | `/api/private/v1/storage_backends/{object.id}` | `body: "object"`, `response_body: "object"` | | `Delete` | `DELETE` | `/api/private/v1/storage_backends/{id}` | | -No Signal RPC is defined. Signal exists on other entities to wake up a reconciler when external state changes — StorageBackend has no reconciler, so Signal would have no consumer. Status fields (model, firmware_version) can be updated via the standard `Update` RPC if needed in the future. +No Signal RPC is defined. Signal exists on other entities to wake up a reconciler when external state changes — StorageBackend has no reconciler, so Signal would have no consumer. #### Server Implementation @@ -204,11 +202,12 @@ create table storage_backends ( creation_timestamp timestamp with time zone not null default now(), deletion_timestamp timestamp with time zone not null default 'epoch', finalizers text[] not null default '{}', - creators text[] not null default '{}', - tenants text[] not null default '{}', + creator text not null default '', + tenant text not null default '', labels jsonb not null default '{}'::jsonb, annotations jsonb not null default '{}'::jsonb, - data jsonb not null + data jsonb not null, + version integer not null default 0 ); create table archived_storage_backends ( @@ -217,11 +216,12 @@ create table archived_storage_backends ( creation_timestamp timestamp with time zone not null, deletion_timestamp timestamp with time zone not null, archival_timestamp timestamp with time zone not null default now(), - creators text[] not null default '{}', - tenants text[] not null default '{}', + creator text not null default '', + tenant text not null default '', labels jsonb not null default '{}'::jsonb, annotations jsonb not null default '{}'::jsonb, - data jsonb not null + data jsonb not null, + version integer not null default 0 ); ``` @@ -229,7 +229,8 @@ Indexes: ```sql create index storage_backends_by_name on storage_backends (name); -create index storage_backends_by_owner on storage_backends using gin (creators); +create index storage_backends_by_creator on storage_backends (creator); +create index storage_backends_by_tenant on storage_backends (tenant); create index storage_backends_by_label on storage_backends using gin (labels); ``` From ba8bc97007772ecd88885b9f742de9c9f222c903 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Wed, 17 Jun 2026 13:40:22 +0300 Subject: [PATCH 033/450] OSAC-1111: remove tenant index (platform-scoped, always empty) Assisted-by: Claude Code Signed-off-by: Roy Golan --- enhancements/storage-backend-osac-1111/design.md | 1 - 1 file changed, 1 deletion(-) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index fe1f242a3..1d2110be3 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -230,7 +230,6 @@ Indexes: ```sql create index storage_backends_by_name on storage_backends (name); create index storage_backends_by_creator on storage_backends (creator); -create index storage_backends_by_tenant on storage_backends (tenant); create index storage_backends_by_label on storage_backends using gin (labels); ``` From 54abf1933ed57d213f3e49521610ac164c3af11c Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Wed, 17 Jun 2026 15:09:30 +0300 Subject: [PATCH 034/450] docs(OSAC-1111): add intra-component version skew handling for state enum values Address coderabbit review comment: document how phase 1 instances handle unknown state enum values (MAINTENANCE, DECOMMISSIONED) created by phase 0.2 during rolling upgrades. Proto3 preserves unknown enum values as integers, so data integrity is maintained. Assisted-by: Claude Code Signed-off-by: Roy Golan --- enhancements/storage-backend-osac-1111/design.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/enhancements/storage-backend-osac-1111/design.md b/enhancements/storage-backend-osac-1111/design.md index 1d2110be3..f8a52a711 100644 --- a/enhancements/storage-backend-osac-1111/design.md +++ b/enhancements/storage-backend-osac-1111/design.md @@ -389,6 +389,8 @@ This is a new API with no upgrade impact. The database migration (`54_create_sto StorageBackend exists entirely within the fulfillment-service. There is no CRD in osac-operator and no cross-component API dependency. Version skew between fulfillment-service and osac-operator does not affect StorageBackend. +**Intra-component skew (rolling upgrades):** During a rolling upgrade from phase 1 to phase 0.2, a phase 1 instance may read backends from PostgreSQL with state values it does not recognize (`MAINTENANCE` = 2, `DECOMMISSIONED` = 3). Proto3 preserves unknown enum values as their integer representation, so the data is not corrupted. The phase 1 instance will serve these backends with the numeric state value in API responses. This is acceptable for the brief duration of a rolling upgrade. Downgrades from phase 0.2 to phase 1 require all backends in `MAINTENANCE` or `DECOMMISSIONED` state to be transitioned back to `READY` or deleted before the rollback (see Upgrade / Downgrade Strategy above). + When StorageTier (OSAC-1110) is implemented, it will reference StorageBackend by ID. At that point, the StorageTier implementation must handle the case where a referenced StorageBackend does not exist (fulfillment-service upgraded with StorageTier before StorageBackend records are created, or downgrade removes StorageBackend). This is scoped to OSAC-1110. ## Support Procedures From 37242924b9cc1ef28a61dd75ea77b0d8db7d8c36 Mon Sep 17 00:00:00 2001 From: Dan Manor Date: Wed, 17 Jun 2026 16:53:23 +0300 Subject: [PATCH 035/450] OSAC-1029: unified networking API for VMaaS, CaaS, and BMaaS (#50) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces a redesigned networking architecture that generalizes the OSAC Networking API across all three service types with infrastructure-agnostic subnets where VMs participate in the fabric. Key design decisions: - NetworkClass with fabricManager + optional k8sManager (two fields) - ExternalIP (renamed from PublicIP) — external to VN, not internet-routable - Fabric manager handles all physical networking as a single product - K8s manager bridges OVN overlay to fabric (VMs become part of fabric) - Uniform dispatcher — no VM-vs-BM distinction for ExternalIP or NATGateway - Fabric-only security enforcement (no separate k8s ACL) - BM multi-interface selection via template discovery - CaaS prerequisite ordering handled by template, not API mandate Signed-off-by: Dan Manor --- enhancements/unified-networking-prd/README.md | 402 ++++++++ enhancements/unified-networking/README.md | 915 ++++++++++++++++++ 2 files changed, 1317 insertions(+) create mode 100644 enhancements/unified-networking-prd/README.md create mode 100644 enhancements/unified-networking/README.md diff --git a/enhancements/unified-networking-prd/README.md b/enhancements/unified-networking-prd/README.md new file mode 100644 index 000000000..83455c3c0 --- /dev/null +++ b/enhancements/unified-networking-prd/README.md @@ -0,0 +1,402 @@ +--- +title: Unified Networking Requirements for VMaaS, CaaS, and BMaaS +authors: + - dmanor@redhat.com +creation-date: 2026-06-03 +last-updated: 2026-06-10 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-1029 +see-also: + - Unified Networking Design: /enhancements/unified-networking + - Original Networking API: /enhancements/networking + - BareMetal Instance API: /enhancements/baremetal-instance-api + - Three-Layer Networking Model: https://docs.google.com/document/d/1MwBjpmYoZoUN3PVjeIRZ2Y6mBuf0lu1uvTtN6XXPPTM +replaces: + - N/A +superseded-by: + - N/A +--- + +# Unified Networking Requirements for VMaaS, CaaS, and BMaaS + +## Summary + +The OSAC Networking API must serve as a foundational service across all three +OSAC service types — VMaaS, CaaS, and BMaaS — with a single, consistent +resource model. This document defines the requirements, identifies gaps in the +current design, and establishes acceptance criteria. The technical design that +fulfills these requirements is described in a companion enhancement: +[Unified Networking Design](/enhancements/unified-networking). + +## Terminology + +This section defines key terms used throughout this document. + +- **Tenant**: An organization or user consuming OSAC services. Tenants create + and manage their own networking resources (VirtualNetworks, Subnets, + SecurityGroups, ExternalIPs) and place workloads on them. + +- **Provider**: The cloud administrator who deploys and configures OSAC + infrastructure. Providers define regions, install networking managers, + configure NetworkClasses, and manage ExternalIPPools. Tenants do not see + provider-level configuration. + +- **Region**: A management boundary representing a deployment location (e.g., + a data center). Networking resources are scoped to a region. Each region has + its own networking infrastructure and manager configuration. + +- **Service Types**: The three workload types OSAC supports: + - **VMaaS** (Virtual Machine as a Service): Provisions virtual machines. + The API resource is `ComputeInstance`. + - **CaaS** (Cluster as a Service): Provisions managed clusters. The API + resource is `Cluster`. + - **BMaaS** (Bare Metal as a Service): Provisions physical bare-metal + servers. The API resource is `BaremetalInstance` (defined in the + [BareMetal Instance API enhancement](/enhancements/baremetal-instance-api)). + +- **VirtualNetwork**: A tenant's isolated network environment with its own + address space (CIDR). Analogous to an AWS VPC or Azure VNet. Scoped to a + region. + +- **Subnet**: A subdivision of a VirtualNetwork's IP address space. Resources + are attached to subnets to receive IP addresses and network connectivity. + +- **SecurityGroup**: A stateful firewall controlling inbound and outbound + traffic for resources. Rules specify allowed protocols, ports, and + source/destination addresses. + +- **ExternalIPPool**: A provider-defined pool of IP addresses that are + routable outside the VirtualNetwork. "External" means external to the VN — + not necessarily internet-routable (see gap #8). + +- **ExternalIP**: An IP address allocated from an ExternalIPPool. Persists + independently of the resources it's attached to. + +- **ExternalIPAttachment**: The binding between an ExternalIP and a target + resource for inbound traffic (DNAT). + +- **NATGateway**: Optionally provides a dedicated outbound NAT (SNAT) for + resources in a VirtualNetwork, giving them a known, stable source IP for + egress traffic. Without a NATGateway, resources may still have default + egress but without a controlled source identity. + +- **NetworkClass**: A provider-configured resource that defines how networking + is implemented in a region. Specifies which fabric manager and K8s manager + handle networking for the region. In the current design, tenants select it + when creating a VirtualNetwork (this is one of the gaps — see #2). + +- **Fabric Manager**: A single product (e.g., Netris, Neutron) that manages + all physical networking for a region: tenant isolation, ACLs, IP + allocation, DNAT, SNAT. The physical fabric is one infrastructure — one + controller manages it all. + +- **K8s Manager**: Handles everything needed to make VMs part of the fabric: + creates the K8s overlay and bridges it to the fabric segment. Only needed + for regions that host VMs. + +- **Fabric**: The physical network infrastructure — switches, routers, + gateways — that connects bare-metal servers and provides external + connectivity. In this design, VMs also participate in the fabric through + a K8s manager that bridges the OVN overlay to the physical network. + +## Problem Statement + +The original [Networking API enhancement](/enhancements/networking) was designed +with VMaaS (ComputeInstance) as the only consumer, explicitly listing CaaS and +BMaaS as non-goals. As OSAC grows and new teams onboard, this limitation forces +each service type to implement networking independently: + +- **CaaS** manages networking entirely through fabric-specific Ansible roles, + bypassing the OSAC API. Tenants ordering a Cluster have no way to specify + which VirtualNetwork or Subnet their cluster nodes should use. +- **BMaaS** calls inventory backends directly for network configuration, + bypassing the OSAC API. Tenants ordering a BaremetalInstance have no + networking integration at all (deferred in + the [BareMetal Instance API enhancement](/enhancements/baremetal-instance-api)). +- **Tenants** have no unified way to manage networking across service types. + A tenant running VMs, clusters, and bare-metal servers must use three + different networking models. +- **Providers** cannot swap network managers without API changes. Adding a + new fabric manager or changing the one for a region requires modifying the + fulfillment service and operator. + +The result is fragmented networking with no consistency, no reuse, and no +tenant-facing abstraction. + +## Gaps in the Current Design + +### 1. CaaS and BMaaS have no networking API + +The Networking API only supports ComputeInstance (VMaaS). The Cluster resource +has no `network_attachments` field — there is no way for a tenant to specify +which VirtualNetwork, Subnet, or SecurityGroup a cluster's nodes should use. +A tenant cannot place two clusters in the same VirtualNetwork to share an +address space, or isolate clusters in separate VirtualNetworks — the networking +is entirely opaque and managed ad-hoc by the CaaS template role. + +The same applies to BMaaS. BaremetalInstance (defined in +the [BareMetal Instance API enhancement](/enhancements/baremetal-instance-api)) +explicitly defers networking integration. A tenant cannot specify +which Subnet a bare-metal server should be placed on, cannot apply +SecurityGroups, and cannot share a VirtualNetwork between bare-metal servers +and other resources. Both service types build ad-hoc networking outside the +API. + +### 2. Tenants must choose networking backends + +NetworkClass is modeled after Kubernetes StorageClass — tenants select it when +creating a VirtualNetwork. But unlike StorageClass (where "fast" vs "cheap" is +a meaningful tenant choice about capability), NetworkClass exposes network +backend implementation details ("udn-net" vs "phys-net") that tenants should +not need to understand. The provider's infrastructure determines the backend, +not the tenant's preference. + +### 3. No manager capability discovery or registration + +There is no registry of which networking managers are installed or what each +supports. A K8s manager like `cudn_localnet` handles VM overlay and bridging +but not IP allocation or ACLs. A fabric manager like Netris handles +everything on the physical side. The system has no way to know this — there +is no machine-readable declaration of manager capabilities, and no validation +that a manager is assigned to a role it can handle. + +### 4. ExternalIPAttachment only supports VMs + +The ExternalIPAttachment target is a `oneof` with only `compute_instance`. +CaaS needs ExternalIPs for cluster API server and ingress endpoints (two +separate IPs for two different purposes on the same Cluster). BMaaS needs +ExternalIPs for bare-metal servers. Neither can use the existing +ExternalIPAttachment. + +### 5. Ingress and egress are not clearly separated + +ExternalIPAttachment is described as "routes traffic to the resource" — +ambiguous about whether it handles inbound traffic only or is bidirectional. +NATGateway is described as "outbound NAT" but the relationship between the +two is undefined. If a resource has both an ExternalIPAttachment and a +NATGateway, which takes precedence for egress? The current implementation is +ingress-only, but this is not documented. + +### 6. VMs are not part of the fabric + +VMs running on OpenShift use OVN (User Defined Networks) for isolation. Their +IP addresses exist only within the OVN overlay and are not visible on the +physical fabric. When a fabric manager needs to perform DNAT to route +external traffic to a VM, it cannot reach the VM's OVN-internal IP directly. +A K8s manager (e.g., CUDN with LocalNet) is needed to bridge VMs to the +fabric. The current design does not address this, and there is no way for a +provider to configure which bridging mechanism to use. + +### 7. VMs and bare metal cannot share a network + +VMs use OVN for isolation — a software-defined overlay on the OpenShift +cluster. Bare-metal servers use physical VLANs configured on switches in the +fabric. These are fundamentally different L2 domains. A K8s manager using +LocalNet mode can bridge OVN to the physical fabric, making VMs first-class +participants alongside BM servers. The current design does not address how +VMs and bare-metal servers coexist in the same region, whether they can share +a VirtualNetwork, or how traffic flows between them. + +### 8. Air-gapped environments not considered + +ExternalIPPool and ExternalIP must work in air-gapped deployments where there +are no internet-routable IPs. Tenants still need the same API primitives (IP +allocation, inbound DNAT, outbound SNAT) for data-center-internal external +access. For example, CaaS cluster worker nodes reach the API server via +hairpin NAT through an ExternalIP regardless of whether that IP is +internet-routable. "External" means external to the VirtualNetwork, not +internet-routable. + +### 9. CaaS has unique prerequisite ordering + +Cluster worker nodes reach the hosted control plane API server via hairpin +NAT — egress traffic SNATs to an ExternalIP, then DNATs back in through +another ExternalIP to reach the API server. This means ExternalIPs and +NATGateway must exist before the cluster is provisioned, not after. The +current design assumes ExternalIPs are attached post-creation (as they are +for VMs), which does not work for CaaS. + +## User Stories + +### Tenant Stories (All Services) + +- As a tenant, I want to create isolated VirtualNetworks and Subnets for my + workloads without choosing a networking backend +- As a tenant, I want to define SecurityGroups to control traffic to and + from my resources +- As a tenant, I want to allocate ExternalIPs and attach them to my VMs, + clusters, or bare-metal servers for inbound access +- As a tenant, I want to create a NATGateway for outbound access from my + VirtualNetwork + +### CaaS-Specific Stories + +- As a tenant, I want to place my cluster's worker nodes on a Subnet in my + VirtualNetwork +- As a tenant, I want to attach ExternalIPs to my cluster's API server and + ingress endpoints before provisioning +- As a tenant, I want my cluster to work in air-gapped environments using + data-center-routable IPs + +### BMaaS-Specific Stories + +- As a tenant, I want to place my BaremetalInstance on Subnets in my + VirtualNetwork +- As a tenant, I want to see the available physical interfaces on a bare-metal + template so I can decide how to attach networks +- As a tenant, I want to attach different physical interfaces of my + BaremetalInstance to different Subnets (e.g., data interface to a data + subnet, management interface to a management subnet) +- As a tenant, I want to attach an ExternalIP to my bare-metal server for + inbound access + +### Provider Stories + +- As a provider, I want to configure a fabric manager and K8s manager per + region without exposing implementation details to tenants +- As a provider, I want to register new managers without modifying the API + or the operator +- As a provider, I want to add new networking managers by deploying a + ConfigMap and an Ansible role +- As a provider, I want to be able to provision ExternalIP pools for tenants + +## Non-Goals + +- VPC Peering / cross-VN communication (separate enhancement) +- DNS API for tenant-managed DNS zones (separate enhancement) +- Advanced per-physical-interface configuration for BaremetalInstance (NIC + bonding, VLAN trunking, etc. — basic per-interface subnet attachment is + supported via the `interface` field on NetworkAttachment) +- Load Balancer API +- Internet Gateway API +- Quota enforcement for networking resources + +## Requirements + +### Core Networking + +#### R1: Network isolation and connectivity + +VirtualNetworks must provide tenant isolation. Subnets within a VirtualNetwork +must provide L2 and L3 connectivity. These guarantees must hold regardless of +the physical location of the resource or the infrastructure it runs on. The +fabric is the single source of truth for isolation across all resource types. + +**Acceptance criteria:** +- Resources in different VirtualNetworks cannot communicate (full isolation) +- Resources in the same Subnet are in the same L2 broadcast domain +- Resources in different Subnets within the same VirtualNetwork can + communicate via Layer 3 routing +- SecurityGroups control which traffic is permitted within these boundaries + — enforced by the fabric for all resource types +- Bare-metal servers in the same Subnet are in the same broadcast domain + regardless of their physical location (rack, switch) +- VMs in the same Subnet are in the same broadcast domain regardless of + which hypervisor host or hosting cluster they run on + +#### R2: Infrastructure-agnostic subnets + +The same subnet must be able to host VMs, BM servers, and cluster nodes. +The tenant does not declare the resource type when creating a VirtualNetwork +or Subnet. Multiple hosting clusters per region are supported — VMs on +different hosting clusters share the same subnet via the fabric. + +**Acceptance criteria:** +- VMs participate in the fabric via the K8s manager — once bridged, VMs + are reachable from the fabric at their subnet IP +- The dispatcher provisions both K8s overlay and fabric segments for each + subnet +- Any resource type (ComputeInstance, Cluster, BaremetalInstance) can be + placed on any subnet +- The fabric manager handles VMs uniformly alongside BM servers — no + intermediary bridge needed per ExternalIP or SecurityGroup operation +- SecurityGroup enforcement for VMs is handled by the fabric, not by a + separate K8s-level ACL + +#### R3: Uniform networking across all service types + +All three service types (VMaaS, CaaS, BMaaS) must consume the networking API +using the same resource model: VirtualNetwork, Subnet, SecurityGroup, +ExternalIPPool, ExternalIP, ExternalIPAttachment, NATGateway. + +**Acceptance criteria:** +- ComputeInstance, Cluster, and BaremetalInstance all have a + `network_attachments` field using the same `NetworkAttachment` message +- ExternalIPAttachment supports all three as targets +- The tenant workflow for creating networking resources is identical + regardless of service type + +### External Access + +#### R4: ExternalIP is external to the VirtualNetwork + +"External" means external to the VirtualNetwork — OSAC does not prescribe +whether the IPs are internet-routable, intranet-only, or data-center-local. +The provider defines the pools; the API is the same regardless. + +**Acceptance criteria:** +- ExternalIP semantics do not depend on internet reachability +- The API and workflow are identical for all deployment topologies + (air-gapped, internet-connected, intranet-only) +- CaaS clusters can provision using any routable ExternalIPs for API server + and ingress + +#### R5: Clear ingress/egress separation + +The API must clearly separate inbound and outbound external access. + +**Acceptance criteria:** +- ExternalIPAttachment handles inbound traffic only (DNAT) +- NATGateway handles outbound SNAT only — it is optional and provides a + dedicated egress identity, not a prerequisite for basic connectivity +- The fabric manager handles both DNAT and SNAT uniformly for all resource + types — VMs, BM servers, and cluster nodes are all on the fabric + +### Provider Architecture + +#### R6: Pluggable managers with transparent selection + +Providers configure which fabric manager and K8s manager handle networking +for a region. Tenants never choose networking managers — the system selects +them based on the provider's NetworkClass configuration. + +**Acceptance criteria:** +- NetworkClass is not exposed in the tenant API +- The fabric manager handles all physical networking (isolation, ACLs, IP + allocation, DNAT, SNAT) as a single product +- The K8s manager handles VM-to-fabric bridging as a single product +- Managers are self-registering via ConfigMaps deployed with the OSAC + installation +- The system validates that a manager supports its assigned role +- A new manager can be added by deploying a ConfigMap and an Ansible role — + no API or operator changes needed + +### Resource-Specific + +#### R7: Per-interface network attachment for bare metal + +Bare-metal servers have multiple physical interfaces. Tenants must be able to +attach different interfaces to different Subnets based on the interface +descriptions provided by the template. + +**Acceptance criteria:** +- `BaremetalInstanceTemplate` describes available interfaces (name, + description) +- `NetworkAttachment` includes an optional `interface` field that references + a named interface from the template +- Multiple `network_attachments` are supported — one per physical interface +- The same interface cannot appear in multiple attachments +- All referenced subnets must belong to the same VirtualNetwork + +## Success Metrics + +- All three service types consume the same networking API (R3) +- No service type bypasses the networking API for network configuration (R3) +- A new manager can be added without modifying existing managers or the API (R6) +- Tenant experience is uniform across service types — same resources, same + workflow, same CLI patterns (R3, R4) + +## Technical Design + +The technical design fulfilling these requirements is described in: +[Unified Networking Design](/enhancements/unified-networking) diff --git a/enhancements/unified-networking/README.md b/enhancements/unified-networking/README.md new file mode 100644 index 000000000..47a78fb1b --- /dev/null +++ b/enhancements/unified-networking/README.md @@ -0,0 +1,915 @@ +--- +title: Unified Networking API for VMaaS, CaaS, and BMaaS +authors: + - dmanor@redhat.com +creation-date: 2026-06-03 +last-updated: 2026-06-10 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-1029 +see-also: + - Requirements (PRD): /enhancements/unified-networking-prd + - Networking API: /enhancements/networking + - BareMetal Instance API: /enhancements/baremetal-instance-api + - Three-Layer Networking Model: https://docs.google.com/document/d/1MwBjpmYoZoUN3PVjeIRZ2Y6mBuf0lu1uvTtN6XXPPTM +replaces: + - /enhancements/networking +superseded-by: + - N/A +--- + +# Unified Networking API for VMaaS, CaaS, and BMaaS + +## Summary + +This enhancement describes the technical design for the OSAC unified +networking architecture. For the problem statement, gaps analysis, and +requirements, see the companion +[Requirements Document (PRD)](/enhancements/unified-networking-prd). + +OSAC runs VMs on OpenShift using KubeVirt, which encapsulates each VM in a +pod. Pod networking is managed by OVN-Kubernetes, meaning VMs live inside an +OVN overlay that is not directly visible on the physical fabric. The core +premise of this design is that **VMs are part of the fabric**. Through a +[K8s manager](#how-vms-join-the-fabric) that bridges the OVN overlay to the +physical network, VMs become first-class participants in the fabric alongside +bare-metal servers and cluster nodes. Once on the fabric, all resource types +are treated uniformly — the fabric manager handles isolation, security, IP +allocation, DNAT, and SNAT for everything. + +The design introduces: + +- **NetworkClass** with two fields: `fabricManager` (handles all physical + networking) and optional `k8sManager` (bridges VMs to the fabric) +- **Infrastructure-agnostic subnets** where the same subnet can host VMs, + BM servers, and cluster nodes +- **ExternalIP** (renamed from PublicIP) to clarify that addresses are + external to the VirtualNetwork, not necessarily internet-routable +- **Uniform API** where the same networking resources (VirtualNetwork, + Subnet, SecurityGroup, ExternalIP, ExternalIPAttachment, NATGateway) + serve VMaaS, CaaS, and BMaaS identically + +The BMaaS integration is based on the `BaremetalInstance` resource defined in +the [BareMetal Instance API enhancement](/enhancements/baremetal-instance-api), +which provides a per-server resource aligned with ComputeInstance. + +For user stories, goals, and non-goals, see the +[Requirements Document (PRD)](/enhancements/unified-networking-prd). + +## Proposal + +### NetworkClass + +NetworkClass is the provider-level CRD that defines which managers handle +networking for a region. Tenants never interact with it. One NetworkClass +per region. + +#### Two Managers + +OSAC networking is handled by two managers: + +- **Fabric Manager** — a single product (e.g., Netris, Neutron) that manages + all physical networking: tenant isolation, ACLs, IP allocation, DNAT, SNAT. + The physical fabric is one infrastructure — one controller manages it all. + +- **K8s Manager** (optional) — handles everything needed to make VMs part of + the fabric: creates the K8s overlay (e.g., CUDN with LocalNet) and bridges + it to the fabric segment. Only needed for regions that host VMs. Once VMs + are on the fabric, the fabric manager handles them identically to + bare-metal servers. + +#### Why Two Managers? + +The fabric is one product. You cannot have Netris handling isolation and +Neutron handling ACLs on the same switches — splitting into per-action +drivers does not reflect how physical networking works. A single +`fabricManager` field captures this reality. + +The K8s side is a separate concern: it bridges the OVN overlay to the +physical fabric. The mechanism depends on the deployment — see +[How VMs Join the Fabric](#how-vms-join-the-fabric) for the available +options. The goal is always the same: make VMs part of the fabric. A single +`k8sManager` field captures this. + +Once VMs are on the fabric, the fabric manager handles everything for all +resource types uniformly. There is no VM-vs-BM distinction for security, +ExternalIP, DNAT, or SNAT. + +#### NetworkClass Examples + +**Netris + CUDN (VMs and BM):** + +```yaml +apiVersion: osac.openshift.io/v1alpha1 +kind: NetworkClass +metadata: + name: moc-region-1 +spec: + region: moc-region-1 + fabricManager: netris + k8sManager: cudn_localnet +status: + capabilities: + addressFamily: dualStack +``` + +**Neutron + CUDN (VMs and BM):** + +```yaml +apiVersion: osac.openshift.io/v1alpha1 +kind: NetworkClass +metadata: + name: bos-region-1 +spec: + region: bos-region-1 + fabricManager: neutron + k8sManager: cudn_localnet +status: + capabilities: + addressFamily: ipv4 +``` + +**BM-only region (no VMs):** + +```yaml +apiVersion: osac.openshift.io/v1alpha1 +kind: NetworkClass +metadata: + name: gpu-region-1 +spec: + region: gpu-region-1 + fabricManager: netris +status: + capabilities: + addressFamily: ipv4 +``` + +#### Capabilities + +Capabilities are **inferred from the assigned managers** and published in +the NetworkClass status — the provider does not set them manually. The +operator computes the intersection of capabilities declared by the fabric +manager and k8sManager ConfigMaps and populates `status.capabilities` +automatically. + +If the provider needs to restrict a capability that the managers support +(e.g., disable IPv6 in a region even though the fabric manager supports +it), they can set `spec.disableCapabilities`: + +```yaml +spec: + fabricManager: netris + k8sManager: cudn_localnet + disableCapabilities: + - ipv6 +``` + +| Capability | Type | Meaning | +|-----------|------|---------| +| `addressFamily` | enum | `ipv4`, `ipv6`, or `dualStack` | +| `dpuSupport` | bool | DPU-accelerated networking available | + +The set of capabilities is defined by the operator and is fixed — adding a +new capability requires an operator update. Managers declare which +capabilities they support; they cannot define custom capabilities. + +#### Manager Registration (ConfigMap) + +Each manager ships a ConfigMap declaring its type and capabilities. These +ConfigMaps are deployed as part of the OSAC installation alongside the +manager's Ansible roles. + +**Fabric managers:** + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: fabric-manager-netris + namespace: osac + labels: + osac.openshift.io/network/fabric-manager: "true" +data: + name: netris + description: "Netris SDN — tenant isolation, ACL, IPAM, DNAT, SNAT" + capabilities: "addressFamily:ipv4" +``` + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: fabric-manager-neutron + namespace: osac + labels: + osac.openshift.io/network/fabric-manager: "true" +data: + name: neutron + description: "OpenStack Neutron — tenant isolation, IPAM, floating IPs" + capabilities: "addressFamily:ipv4" +``` + +**K8s managers:** + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: k8s-manager-cudn-localnet + namespace: osac + labels: + osac.openshift.io/network/k8s-manager: "true" +data: + name: cudn_localnet + description: "CUDN with LocalNet — bridges OVN overlay to physical fabric" + capabilities: "addressFamily:dualStack" +``` + +The operator discovers managers by listing ConfigMaps with the appropriate +labels. When a NetworkClass is created, the operator validates each manager +assignment against the corresponding ConfigMap. Adding a new manager means +deploying a new ConfigMap and Ansible role — no API or operator changes +needed. + +### How VMs Join the Fabric + +OSAC runs VMs on OpenShift using KubeVirt. Each VM is encapsulated in a pod +whose networking is managed by OVN-Kubernetes. By default, VM IP addresses +exist only within the OVN overlay and are not visible on the physical +fabric. The k8sManager bridges this overlay to the fabric so that VMs +become first-class fabric participants — reachable at their subnet IP from +any other resource on the same fabric segment. + +Several mechanisms can achieve this bridging. The k8sManager is pluggable — +different deployments use different mechanisms depending on their +infrastructure and requirements: + +**CUDN with LocalNet.** The k8sManager creates a ClusterUserDefinedNetwork +(CUDN) with LocalNet topology, mapping the OVN network directly to a +physical VLAN on the hosting cluster's trunk interface. VMs in this network +are bridged to the fabric at L2 — they share a broadcast domain with +bare-metal servers on the same VLAN. This is the simplest mechanism and +provides full L2 adjacency. + +**OVN EVPN.** OVN advertises VM routes to the fabric via BGP EVPN. The +fabric learns VM MAC/IP bindings and can route to them. VMs remain in the +OVN overlay but are reachable from the fabric at L3. This preserves OVN's +per-VM isolation on the same hypervisor while still making VMs fabric +participants. Note: OVN EVPN is not yet GA in OpenShift. + +**CUDN with VRF-lite.** The hosting cluster uses VRF (Virtual Routing and +Forwarding) instances to route between the OVN overlay and the fabric. Each +tenant VN maps to a VRF on the host, which peers with the fabric via BGP. +VMs are reachable from the fabric via L3 routing through the VRF. See the +[CUDN with VRF-lite setup guide](/docs/networking/setup-bpg-vrf-lite) for +a working lab example. + +**DPU-based bridging.** SmartNICs (DPUs) offload the OVN-to-fabric bridging +to hardware. The DPU handles packet encapsulation/decapsulation between OVN +and the physical network, providing line-rate bridging without host CPU +overhead. + +The choice of mechanism is transparent to tenants — it is configured by the +provider as part of the k8sManager installation. The networking API and +resource model are identical regardless of which mechanism is used. All that +matters is the contract: once the k8sManager has bridged a subnet, VMs on +that subnet are reachable from the fabric at their subnet IP. + +### Infrastructure-Agnostic Subnets + +VirtualNetwork and Subnet do not carry a scope or service field. Subnets are +infrastructure-agnostic — the dispatcher provisions both the fabric segment +and (if the region has a k8sManager) the K8s overlay for every subnet. Any +resource type can be placed on any subnet. + +At subnet creation, the dispatcher runs: + +1. **Fabric manager** — creates fabric segment (e.g., VLAN, VPC) +2. **K8s manager** (if present) — creates K8s overlay on each hosting + cluster in the region and bridges it to the fabric segment + +VMs are placed in the K8s overlay (which is bridged to the fabric), BM +servers and cluster nodes are placed directly on the fabric segment. The +fabric is the single source of truth for multi-tenancy and routing — all +resources, regardless of type, are on the fabric. + +### Dispatcher (Operator Composition Logic) + +The osac-operator acts as a **dispatcher**: when reconciling any networking +resource, it resolves the NetworkClass for the region and calls the +appropriate managers. Each manager corresponds to an Ansible role — the +dispatcher triggers the appropriate AAP playbook, passing the resource and +context as the event payload. + +| Operation | Managers called | +|-----------|----------------| +| VN create/delete | `fabricManager` | +| Subnet create/delete | `fabricManager` + `k8sManager` (per hosting cluster) | +| SecurityGroup create/delete | `fabricManager` | +| ExternalIP alloc/release | `fabricManager` | +| ExternalIPAttachment create/delete | `fabricManager` | +| NATGateway create/delete | `fabricManager` | + +Everything except subnet creation is handled by the fabric manager alone. +The k8sManager is only involved at subnet creation (to bridge the overlay) +— after that, VMs are on the fabric and the fabric manager handles them +like any other resource. + +### Resource Hierarchy + +```text +NetworkClass (per region, provider-only) + +VirtualNetwork (tenant-managed, infrastructure-agnostic) + ├── Subnet → fabricManager + k8sManager + ├── SecurityGroup → fabricManager + └── NATGateway → fabricManager + +ExternalIPPool (region-scoped, provider-managed) + └── ExternalIP (tenant-managed) → fabricManager + +ExternalIPAttachment (tenant-managed) + → fabricManager + references an ExternalIP and a target resource +``` + +### ExternalIPPool + +"External" in ExternalIPPool/ExternalIP means **external to the +VirtualNetwork** — not necessarily internet-routable. In air-gapped +environments, the provider creates pools with data-center-routable IPs. In +internet-connected environments, the pools contain internet-routable IPs. +The API and flow are identical regardless of the deployment topology. + +ExternalIPPools are provider-managed and region-scoped. The fabric manager +handles IP allocation — one pool serves all resource types. + +### End-to-End Flows + +This section shows how the unified networking API works from the tenant's +perspective. The flows are the same regardless of which fabric manager or +K8s manager the provider has deployed. Annotations mark what is **new** or +**changed** compared to the current design. + +#### Provider Setup + +1. Provider deploys hosting cluster(s) and fabric controller +2. Provider creates NetworkClass for the region (**new** — provider-only, + tenants never see it) +3. Provider creates ExternalIPPool (**renamed** from PublicIPPool): + +```bash +osac admin create externalippool \ + --region moc-region-1 \ + --cidrs 203.0.113.0/24 \ + --ip-family ipv4 \ + --name external-pool-1 +``` + +The fabric manager registers the IP range in its IPAM for allocation. + +#### Networking Setup (Same for All Resource Types) + +The tenant creates networking resources. This workflow is identical +regardless of whether the tenant plans to run VMs, clusters, or bare-metal +servers. + +**Create VirtualNetwork:** + +```bash +osac create virtualnetwork --region moc-region-1 --cidr 10.0.0.0/16 \ + --name my-net +``` + +The fabric manager creates an isolated tenant segment on the fabric. + +**Create Subnet:** + +```bash +osac create subnet --virtual-network my-net --cidr 10.0.1.0/24 \ + --name my-subnet +``` + +The fabric manager creates a fabric segment (e.g., VLAN) for the subnet. +If the region has a K8s manager, it also creates a K8s overlay on each +hosting cluster and bridges it to the fabric segment. After this step, VMs placed in the +overlay and BM servers with switch ports on the fabric segment are in the +same L2 domain. + +**Create SecurityGroup:** + +```bash +osac create security-group --virtual-network my-net --name my-sg \ + --ingress "protocol:tcp,port:443,source:0.0.0.0/0" +``` + +The fabric manager creates ACL rules on the fabric. + +#### Resource Creation (Differs by Type) + +The networking setup above is shared. Only the resource creation step +differs internally — the tenant CLI experience is the same for all types. + +**ComputeInstance (VM):** + +```bash +osac create computeinstance --template ocp_virt_vm \ + --network-attachment subnet=my-subnet,security-groups=my-sg \ + --name my-vm +``` + +VM is placed in the K8s overlay namespace on a hosting cluster. Because the +overlay is bridged to the fabric, the VM is directly on the fabric segment +and gets an IP from the subnet CIDR. + +**BaremetalInstance:** + +Bare-metal servers have multiple physical interfaces. The tenant discovers +available interfaces through BMaaS (the exact discovery mechanism — e.g., +host type metadata, template, or inventory query — is defined by the BMaaS +design). Given the interface identifiers, the tenant specifies which +interface to attach to which subnet. Each +`network_attachment` maps one physical interface to one subnet. If +`interface` is omitted, the fabric manager picks a default. + +Single interface (simple case): + +```bash +osac create baremetalinstance --template bcm_h100 \ + --network-attachment interface=data-0,subnet=my-subnet,security-groups=my-sg \ + --name my-server +``` + +Multiple interfaces (e.g., east-west traffic on one subnet, north-south +on another): + +```bash +osac create baremetalinstance --template bcm_h100 \ + --network-attachment interface=data-0,subnet=east-west-subnet,security-groups=my-sg \ + --network-attachment interface=data-1,subnet=north-south-subnet \ + --name my-server +``` + +The fabric manager configures each host switch port on the corresponding +fabric segment. Each interface gets an IP from its subnet's CIDR. + +Validation rules: +- All referenced subnets must belong to the same VirtualNetwork +- The same interface cannot appear in multiple attachments +- The `interface` must reference a valid interface identifier from BMaaS +- Multiple attachments without `interface` is invalid — if more than one + attachment is specified, each must have an explicit `interface` +- The number of attachments cannot exceed the number of available interfaces + +**Cluster:** + +```bash +osac create cluster --template ocp_4_17_small \ + --network-attachment subnet=my-subnet,security-groups=my-sg \ + --node-set workers=large,size=3 --name my-cluster +``` + +The template determines whether nodes are VMs or BM. BM nodes get switch +ports configured on the fabric segment. VM nodes are placed in the K8s +overlay. Either way, all nodes end up on the same fabric segment. + +For clusters with BM nodes, the nodes also have multiple physical +interfaces — the same reality as BaremetalInstance. The difference is that +the **template** handles the interface-to-subnet mapping, not the tenant. +The cluster template is created by the provider who knows the node hardware +(e.g., which interface is for data traffic, which is for management). The +tenant specifies which subnet(s) to use; the template maps them to the +correct physical interfaces for each node type. + +Different node sets can be placed on different subnets using the `node-set` +field (see [ClusterNetworkAttachment](#network-attachment-types)): + +```bash +osac create cluster --template ocp_4_17_small \ + --network-attachment node-set=compute,subnet=standard-subnet,security-groups=my-sg \ + --network-attachment node-set=gpu,subnet=gpu-subnet \ + --node-set compute=large,size=3 --node-set gpu=h100,size=2 \ + --name my-cluster +``` + +In all cases, the resource ends up on the fabric. The fabric manager sees +all resources equally — there is no VM-vs-BM distinction. + +#### External Access (Same for All Resource Types) + +Since all resources are on the fabric, external access operations are +uniform. There is no VM-vs-BM distinction (**changed** — the current design +has separate K8s-side steps for VMs). + +**Allocate ExternalIP:** (**renamed** from PublicIP) + +```bash +osac create externalip --pool external-pool-1 --name my-ip +``` + +The fabric manager allocates an IP from its IPAM (e.g., 203.0.113.45). + +**Attach for inbound access (DNAT):** + +```bash +# Attach to a VM +osac create externalipattachment --externalip my-ip \ + --compute-instance my-vm --name vm-att + +# Attach to a BM server (new target type) +osac create externalipattachment --externalip my-ip \ + --baremetal-instance my-server --name bm-att + +# Attach to a cluster API server (new target type + endpoint) +osac create externalipattachment --externalip my-ip \ + --cluster my-cluster --target-endpoint api --name api-att +``` + +The fabric manager creates a DNAT rule: external IP → resource's subnet IP. +Each resource (ComputeInstance, BaremetalInstance) is associated with one +subnet and has one fabric IP — the DNAT targets that IP directly. For +bare-metal servers with multiple interfaces, the ExternalIP is attached to +the resource, not to a specific interface — the fabric manager routes to +the resource's primary subnet IP. + +**Cluster ExternalIPAttachment flow:** + +For VMs and BM, the DNAT target is the resource's fabric IP — +straightforward. For clusters, the DNAT target is a service-level VIP +(API server or ingress) that is discovered during cluster provisioning. +The VIP allocation is decoupled from the networking layer: + +1. CaaS template provisions the cluster +2. Template provisions internal VIPs via MetalLB LoadBalancer Services: + - **API VIP**: MetalLB allocates an IP for the kube-apiserver Service + on the management cluster + - **Ingress VIP**: MetalLB allocates an IP for the ingress controller + Service on the hosted cluster (from the tenant's subnet) +3. Template writes VIPs to the ClusterOrder CR status +4. Feedback controller syncs VIPs to the Cluster object in the + fulfillment service as `api_endpoint` and `ingress_endpoint` fields +5. ExternalIPAttachment controller reads the VIP from the Cluster → + calls fabric manager to create DNAT: external IP → internal VIP +6. ExternalIPAttachment transitions to Ready + +The tenant can inspect the allocated VIPs: + +```bash +osac get cluster my-cluster -o yaml +# api_endpoint: 10.0.5.20 +# ingress_endpoint: 10.0.1.50 +``` + +ExternalIPAttachments can be created before or after the cluster. If +created before (Pending state), the controller activates them once the +cluster's endpoint VIPs are available. If created after, the DNAT rule +is configured immediately. + +**Enable outbound NAT (SNAT):** + +```bash +osac create externalip --pool external-pool-1 --name nat-ip +osac create natgateway --virtual-network my-net --externalip nat-ip \ + --name my-nat +``` + +The fabric manager creates a SNAT rule for the VN: all egress traffic from +the VN's CIDR is source-NATted to the ExternalIP. Applies to all resources +in the VN — VMs, BM servers, cluster nodes — since all are on the fabric. + +### API Extensions + +#### VirtualNetwork + +```protobuf +message VirtualNetworkSpec { + string region = 1; // required, immutable + string ipv4_cidr = 2; // optional, immutable + string ipv6_cidr = 3; // optional, immutable +} +``` + +No scope or service field — subnets are infrastructure-agnostic. + +#### Network Attachment Types + +Each resource type has its own network attachment message. The core fields +(`subnet`, `security_groups`) are shared, but each type adds +resource-specific fields. `network_attachments` are immutable after +resource creation — changing network attachment requires recreating the +resource. + +**ComputeNetworkAttachment** (for ComputeInstance): + +```protobuf +message ComputeNetworkAttachment { + string subnet = 1; // Subnet ID, required, immutable + repeated string security_groups = 2; // SecurityGroup IDs, optional, mutable +} +``` + +Each entry maps one virtual NIC to one subnet. Multiple entries create +a multi-homed VM. + +**BareMetalNetworkAttachment** (for BaremetalInstance): + +```protobuf +message BareMetalNetworkAttachment { + string subnet = 1; // Subnet ID, required, immutable + repeated string security_groups = 2; // SecurityGroup IDs, optional, mutable + string interface = 3; // optional, immutable: physical interface identifier from BMaaS +} +``` + +Each entry maps one physical interface to one subnet. The `interface` +field references an interface identifier provided by BMaaS. +If omitted, the fabric manager picks a default. See the +[Resource Creation](#resource-creation-differs-by-type) section for +interface discovery and multi-interface examples. + +**ClusterNetworkAttachment** (for Cluster): + +```protobuf +message ClusterNetworkAttachment { + string subnet = 1; // Subnet ID, required, immutable + repeated string security_groups = 2; // SecurityGroup IDs, optional, mutable + string node_set = 3; // optional, immutable: node set name from cluster spec +} +``` + +Each entry maps one node set to one subnet. If `node_set` is omitted, +all node sets use this subnet. Multiple entries place different node sets +on different subnets (e.g., GPU workers on a high-bandwidth subnet, +compute workers on a standard one). + +#### Resource Specs + +**ComputeInstance** (existing — field already exists, type changes): + +```protobuf +message ComputeInstanceSpec { + // ... existing fields ... + repeated ComputeNetworkAttachment network_attachments = 14; +} +``` + +**BaremetalInstance** (new — defined in the +[BareMetal Instance API enhancement](/enhancements/baremetal-instance-api)): + +```protobuf +message BaremetalInstanceSpec { + string template = 1; + optional string ssh_key = 2; + optional string user_data = 3; + optional BaremetalInstanceRunStrategy run_strategy = 4; + optional google.protobuf.Timestamp restart_requested_at = 5; + + // NEW: OSAC networking + repeated BareMetalNetworkAttachment network_attachments = 6; +} +``` + +**Cluster** (new): + +```protobuf +message ClusterSpec { + string template = 1; + map template_parameters = 2; + map node_sets = 3; + + // NEW: networking + repeated ClusterNetworkAttachment network_attachments = 4; +} +``` + +- Cluster-internal CNI (pod/service CIDRs) uses platform defaults. +- The cluster's template determines whether nodes are VMs or BM. Both + types are placed on the same subnet — VMs via the K8s overlay (already + bridged to the fabric), BM nodes directly on the fabric. + +The Cluster resource also gains two fields populated by the system +during provisioning: + +```protobuf +message ClusterStatus { + string api_endpoint = X; // set by CaaS template, internal API server VIP + string ingress_endpoint = Y; // set by CaaS template, internal ingress VIP +} +``` + +These are used by the ExternalIPAttachment controller as the DNAT backend +IP when the target is a cluster (see +[Cluster ExternalIPAttachment flow](#cluster-externalipattachment-flow)). + +#### ExternalIPAttachment — Inbound Traffic (DNAT) + +Handles **inbound traffic only**. Does not affect egress (that is +NATGateway's job). + +```protobuf +enum ExternalIPAttachmentEndpoint { + EXTERNAL_IP_ATTACHMENT_ENDPOINT_UNSPECIFIED = 0; + EXTERNAL_IP_ATTACHMENT_ENDPOINT_API = 1; // Cluster API server + EXTERNAL_IP_ATTACHMENT_ENDPOINT_INGRESS = 2; // Cluster ingress wildcard +} + +message ExternalIPAttachmentSpec { + string external_ip = 1; // required, immutable + + oneof target { + string compute_instance = 2; + string cluster = 3; + string baremetal_instance = 4; + } + ExternalIPAttachmentEndpoint target_endpoint = 5; + // Required when target=cluster; must be UNSPECIFIED otherwise. +} +``` + +All fields are immutable after creation. + +#### NATGateway — Outbound Traffic (SNAT) + +Handles **outbound traffic only**. + +```protobuf +message NATGatewaySpec { + string virtual_network = 1; // parent VN ID, required, immutable + string external_ip = 2; // required, immutable +} +``` + +An ExternalIP can only be used by one consumer (either an +ExternalIPAttachment or a NATGateway, not both). One NATGateway per +VirtualNetwork. NATGateway is optional — it provides a dedicated egress +identity. Without it, resources may still have default egress but without a +controlled source IP. + +All fields are immutable after creation. + +**Direction summary:** + +| Resource | Direction | Mechanism | +|----------|-----------|-----------| +| ExternalIPAttachment | Inbound (DNAT) | External IP → resource | +| NATGateway | Outbound (SNAT) | Resource → external IP | + +### Implementation Details + +#### NATGateway Scope + +One NATGateway per VirtualNetwork. All subnets in the VN use the gateway. +Per-subnet NAT association is a future enhancement. + +#### Multi-NIC Support + +ComputeInstance supports multiple `network_attachments` (virtual NICs). All +subnets must belong to the same VN. BaremetalInstance supports multiple +`network_attachments` with the `interface` field to map physical NICs to +subnets. Cluster supports multiple `network_attachments` with the +`node_set` field to place different node sets on different subnets. All +subnets must belong to the same VN across all resource types. + +For multi-subnet resources (VMs with multiple virtual NICs or bare-metal +servers with multiple physical interfaces), default gateway configuration +— which subnet provides the default route, how additional subnets are +configured without conflicting gateways — is an implementation detail of +the responsible manager (k8sManager for VMs, fabric manager for BM). + +#### Multiple Hosting Clusters Per Region + +Multiple hosting clusters are supported per region. At subnet creation, the +k8sManager creates a K8s overlay on each hosting cluster and bridges it to +the fabric segment. VMs on different hosting clusters share the same subnet +via the fabric. + +#### Cross-VN Communication + +VirtualNetworks are isolated. Cross-VN communication (VN Peering) is a +separate enhancement. + +#### DNS + +DNS is a service-integration concern, not part of the networking API. CaaS +template roles create DNS records. A DNS API is a separate enhancement. + +#### BM-Only Regions + +If a region's NetworkClass has no k8sManager, the region does not support +VMs. ComputeInstance creation is rejected if the target region has no +k8sManager — there is no K8s overlay to place the VM on. + +#### CIDR Overlap + +The operator validates that Subnet CIDRs do not overlap within a +VirtualNetwork at creation time. + +### Risks and Mitigations + +| Risk | Impact | Mitigation | +|------|--------|------------| +| Fabric manager complexity | One Ansible role handles all networking concerns | Clear interface contract per operation; tested independently per manager | +| K8s-to-fabric bridge failure | VMs unreachable from fabric | k8sManager validates bridge connectivity at subnet creation; subnet stays Pending until bridge is confirmed | +| CaaS prerequisite ordering | ExternalIPs may be needed before cluster | Pending state for attachments; template validates its own prerequisites | +| ExternalIPAttachment target validation | Target may not exist yet (CaaS) or may be deleted | Pending state for forward references; attachment tracks target lifecycle | +| CIDR overlap | Overlapping subnets cause routing ambiguity | Operator validates at creation time; rejected with clear error | + +### Drawbacks + +This design requires K8s-to-fabric connectivity in every deployment that +hosts VMs. The k8sManager must bridge the OVN overlay to the physical +fabric for VMs to participate. In BM-only deployments, the k8sManager is +not needed and the design reduces to fabric-manager-only. + +The trade-off is justified by infrastructure-agnostic subnets: any resource +type on any subnet, uniform security enforcement via the fabric, and no +per-resource-type dispatcher logic for ExternalIP or NATGateway. + +## Alternatives (Not Implemented) + +**Original NetworkClass model.** Tenants select a NetworkClass per VN. +Exposes implementation details. Not viable for multi-service support. + +**Per-action driver composition.** Separate drivers for each networking +concern (network, acl, ingress, egress, publicIP) with independent +registration and composition. Over-engineered — the fabric is one product, +and splitting it into per-action drivers does not reflect how physical +networking works. Also creates complexity in the dispatcher and validation. + +**Separate k8s ACL driver.** A dedicated k8s.acl driver (e.g., +NetworkPolicy) alongside fabric ACLs. Redundant — when VMs are on the +fabric, the fabric enforces security for all traffic including VM traffic. +Adding a k8s ACL layer creates dual enforcement with no clear benefit. + +**VN scope field (vm/bm).** Require tenants to declare what a network is +for at creation time. Makes subnets service-specific, prevents mixed +workloads, and leaks infrastructure details. + +**Lazy subnet provisioning.** Defer manager selection to resource placement +time. Creates ambiguous subnet state and complicates the tenant experience. + +## Resolved Questions + +1. **Infrastructure-agnostic subnets.** VMs participate in the fabric via + k8sManager. No scope/service field on VN. Any resource on any subnet. + +2. **Cluster endpoint types:** `api` and `ingress` — enum + `ExternalIPAttachmentEndpoint`. + +3. **ExternalIP ownership:** An ExternalIP can only be consumed by one + resource (either ExternalIPAttachment or NATGateway, not both). + +4. **One NATGateway per VN.** Multiple gateways are ambiguous. Per-subnet + NAT is a future enhancement. + +5. **ExternalIPPool shared.** The fabric manager handles allocation for all + resource types. One pool per region. + +6. **Multiple hosting clusters.** Subnet creation provisions K8s overlay + on each hosting cluster. VMs on different clusters share the subnet + via the fabric. + +7. **Internal IP pools.** Managed by managers with sensible defaults. Not + part of the tenant API or NetworkClass spec. + +8. **ExternalIP naming.** "External" means external to the VirtualNetwork — + not necessarily internet-routable. Applies equally to air-gapped and + internet-connected deployments. + +9. **network_attachments immutability.** Network attachments are immutable + after resource creation. Changing network attachment requires recreating + the resource. + +10. **Security enforcement.** The fabric is the single enforcement point + for SecurityGroups. No separate K8s-level ACL needed — VMs are on the + fabric. + +11. **Per-resource NetworkAttachment types.** Separate proto messages + (`ComputeNetworkAttachment`, `BareMetalNetworkAttachment`, + `ClusterNetworkAttachment`) instead of one shared type. Each resource + type has a different selector concept (virtual NIC, physical interface, + node set) — a shared type with optional fields would accumulate + dead weight per resource type. + +## Test Plan + +*Section to be completed when targeted at a release.* + +## Graduation Criteria + +*Section to be completed when targeted at a release.* + +## Upgrade / Downgrade Strategy + +*Section to be completed when targeted at a release.* + +## Version Skew Strategy + +*Section to be completed when targeted at a release.* + +## Support Procedures + +*Section to be completed when targeted at a release.* + +## Infrastructure Needed + +No additional infrastructure beyond existing OSAC components and managers. From 82e67c91191c46779abb79c55cce061afb841bb7 Mon Sep 17 00:00:00 2001 From: Dan Manor Date: Sun, 21 Jun 2026 10:51:47 +0300 Subject: [PATCH 036/450] OSAC-1029: add multi-NIC networking behavior to unified networking design (#64) Adds primary network attachment field and multi-NIC behavior inline in the unified networking design doc (no separate enhancement file): - primary field on ComputeNetworkAttachment and BareMetalNetworkAttachment - Default gateway, DHCP, ExternalIPAttachment, and NATGateway semantics for multi-homed resources - Validation rules for primary designation - Cluster multi-NIC deferred to CaaS template Removes the separate multi-nic-networking enhancement directory. Signed-off-by: Dan Manor --- enhancements/unified-networking/README.md | 56 +++++++++++++++++++---- 1 file changed, 47 insertions(+), 9 deletions(-) diff --git a/enhancements/unified-networking/README.md b/enhancements/unified-networking/README.md index 47a78fb1b..a89d01516 100644 --- a/enhancements/unified-networking/README.md +++ b/enhancements/unified-networking/README.md @@ -604,11 +604,13 @@ resource. message ComputeNetworkAttachment { string subnet = 1; // Subnet ID, required, immutable repeated string security_groups = 2; // SecurityGroup IDs, optional, mutable + bool primary = 3; // optional, immutable: designates default gateway } ``` Each entry maps one virtual NIC to one subnet. Multiple entries create -a multi-homed VM. +a multi-homed VM. See [Multi-NIC Behavior](#multi-nic-behavior) for +primary designation and default gateway semantics. **BareMetalNetworkAttachment** (for BaremetalInstance): @@ -617,14 +619,17 @@ message BareMetalNetworkAttachment { string subnet = 1; // Subnet ID, required, immutable repeated string security_groups = 2; // SecurityGroup IDs, optional, mutable string interface = 3; // optional, immutable: physical interface identifier from BMaaS + bool primary = 4; // optional, immutable: designates default gateway } ``` Each entry maps one physical interface to one subnet. The `interface` field references an interface identifier provided by BMaaS. -If omitted, the fabric manager picks a default. See the -[Resource Creation](#resource-creation-differs-by-type) section for -interface discovery and multi-interface examples. +If omitted, the fabric manager picks a default. Multiple entries create +a multi-homed BM server. See [Multi-NIC Behavior](#multi-nic-behavior) +for primary designation and default gateway semantics, and +[Resource Creation](#resource-creation-differs-by-type) for interface +discovery and multi-interface examples. **ClusterNetworkAttachment** (for Cluster): @@ -769,11 +774,44 @@ subnets. Cluster supports multiple `network_attachments` with the `node_set` field to place different node sets on different subnets. All subnets must belong to the same VN across all resource types. -For multi-subnet resources (VMs with multiple virtual NICs or bare-metal -servers with multiple physical interfaces), default gateway configuration -— which subnet provides the default route, how additional subnets are -configured without conflicting gateways — is an implementation detail of -the responsible manager (k8sManager for VMs, fabric manager for BM). +#### Multi-NIC Behavior + +When a resource has multiple network attachments, the tenant designates +one as **primary** via `primary: true` on the attachment. The primary +attachment determines: + +- Which subnet provides the **default gateway** for the resource +- Which subnet IP is used as the **DNAT target** for ExternalIPAttachment +- Which subnet IP is used as the **source** for NATGateway SNAT + +**Validation:** +- If only one attachment exists, it is primary by default +- If multiple attachments exist, exactly one must be marked `primary: true` +- If multiple attachments exist and none is marked primary, the request is + rejected +- `primary` is immutable after creation + +**IPAM and DHCP:** The responsible manager (k8sManager for VMs, fabric +manager for BM) configures DHCP per subnet based on the primary +designation: + +| Subnet role | DHCP provides | +|-------------|--------------| +| Primary | IP address + default gateway + DNS | +| Secondary | IP address + connected route only (no gateway) | + +This ensures the resource has exactly one default route. Secondary subnets +are reachable via directly connected routes. + +**ExternalIPAttachment:** When targeting a multi-homed resource, the fabric +manager creates a DNAT rule to the resource's primary subnet IP. The tenant +does not need to specify which interface — the primary designation +determines the target. + +**Cluster multi-NIC:** `ClusterNetworkAttachment` uses `node_set` rather +than per-NIC attachment. Multi-NIC for individual cluster nodes is handled +by the CaaS template (provider-configured). The `primary` field does not +apply to `ClusterNetworkAttachment`. #### Multiple Hosting Clusters Per Region From f1d9d210c594c28175925a0a0dfe4ff0b1855fe9 Mon Sep 17 00:00:00 2001 From: Daniel Erez Date: Sun, 21 Jun 2026 21:38:46 +0300 Subject: [PATCH 037/450] EP: Tenant On-boarding (OSAC-24) (#62) * Add Enhancement Proposal for OSAC-24: Tenant On-boarding Automation Two epics: Tenant CR Schema Extension (3 stories) and Tenant Controller Database Watch (4 stories). - Epic 1: Add spec.displayName and spec.emailDomains fields to Tenant CR - Epic 2: Extend Tenant controller to watch fulfillment database and auto-create/delete CRs - Author: Daniel Erez * Address CodeRabbit feedback: Clarify required field backfilling during legacy CR adoption Add FR-14 to document how the controller handles legacy Tenant CRs that are missing required spec fields (displayName, emailDomains). The controller will backfill these fields from the fulfillment database during the adoption process before adding the managed-by annotation. This addresses CodeRabbit's concern about the migration gap between required field definitions and the legacy adoption lifecycle. Assisted-by: Claude Code Signed-off-by: Daniel Erez --------- Signed-off-by: Daniel Erez --- enhancements/tenant-onboarding/README.md | 120 +++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 enhancements/tenant-onboarding/README.md diff --git a/enhancements/tenant-onboarding/README.md b/enhancements/tenant-onboarding/README.md new file mode 100644 index 000000000..64e746079 --- /dev/null +++ b/enhancements/tenant-onboarding/README.md @@ -0,0 +1,120 @@ +# Tenant On-boarding Automation + +| Field | Value | +|-------------|---------| +| **Status** | Proposed | +| **Feature** | OSAC-24 | +| **Jira** | https://redhat.atlassian.net/browse/OSAC-24 | +| **Authors** | Daniel Erez | +| **Date** | 2026-06-15 | + +## Summary + +Automate tenant onboarding by extending the Tenant CR schema and automating CR lifecycle management. When a new tenant is added to the fulfillment database, the Tenant controller in osac-operator will automatically create Tenant CRs on management hubs with tenant identity information (`displayName`, `emailDomains`), enabling component controllers (Storage, Networking, Authentication) to provision tenant-specific infrastructure. + +## Motivation + +Cloud Provider Admins create tenants via the fulfillment API, but there is no automated mechanism to create corresponding Tenant CRs on management hubs. Additionally, the Tenant CR lacks fields for tenant identity (`displayName`, `emailDomains`) that component controllers need for provisioning. + +## Proposal + +### Two-Part Enhancement + +1. **Schema Extension**: Add `spec.displayName` and `spec.emailDomains` fields to Tenant CR +2. **Lifecycle Automation**: Extend the existing Tenant controller to watch the fulfillment database and auto-create/delete Tenant CRs + +### Tenant CR Schema + +```yaml +apiVersion: osac.openshift.io/v1alpha1 +kind: Tenant +metadata: + name: acme-corp + namespace: default + annotations: + osac.openshift.io/managed-by: fulfillment-service +spec: + displayName: "ACME Corp" + emailDomains: + - acme.com + - acme.net +``` + +## Implementation + +### Epic 1: Tenant CR Schema Extension (Size: S, 3 stories) + +**Story 1.1: CRD Field Additions** +- Add `spec.displayName` and `spec.emailDomains` to `api/v1alpha1/tenant_types.go` +- Regenerate CRD manifests and Go types + +**Story 1.2: Validation Rules** +- Add kubebuilder validation markers +- Implement admission webhook for email domain validation + +**Story 1.3: Integration Tests** +- Test CR creation with valid/invalid spec fields + +### Epic 2: Tenant Controller Database Watch (Size: M, 4 stories) + +**Story 2.1: Database Watch Mechanism** +- Extend Tenant controller to watch fulfillment database (PostgreSQL LISTEN/NOTIFY or polling) +- Detect tenant INSERT/UPDATE/DELETE events +- **Security Model:** The controller uses read-only database credentials provided via Kubernetes Secret. Authentication, authorization, and tenant resource scoping remain the responsibility of the Fulfillment Service layer. The controller treats the database as an event source only and does not enforce access control or tenant isolation directly—those concerns are managed by the Fulfillment Service API + +**Story 2.2: CR CRUD Logic** +- Create Tenant CR when tenant added to DB +- Delete Tenant CR when tenant removed from DB +- Add `osac.openshift.io/managed-by: fulfillment-service` annotation + +**Story 2.3: Reconciliation Loop** +- Periodic reconciliation performs a full diff between the authoritative database state and live CRs in the cluster +- Recreate missing CRs that should exist based on DB state +- Delete orphaned CRs that no longer exist in the database (respecting `managed-by` annotation filter) +- Update CRs that exist in both but have diverged (e.g., displayName or emailDomains changed in DB) +- Skip CRs without `managed-by` annotation to avoid interfering with manually created Tenant CRs +- This ensures the cluster state fully converges to the database state of record, even after missed LISTEN/NOTIFY events during reconnects or controller restarts + +**Story 2.4: Integration Tests** +- Test full lifecycle: create tenant → CR appears → delete tenant → CR removed +- Test reconciliation recovery (manual CR deletion) + +## Requirements + +### Functional Requirements + +#### Tenant CR Specification +- **FR-1:** `spec.displayName` as required string field +- **FR-2:** `spec.emailDomains` as required list of strings field +- **FR-3:** Kubebuilder validation markers for field requirements +- **FR-4:** Admission webhook validates email domain format + +#### Tenant CR Lifecycle +- **FR-5:** Auto-create Tenant CR when tenant added to fulfillment database +- **FR-6:** Tenant CR resides in default namespace on each management hub +- **FR-7:** Auto-delete Tenant CR when tenant deleted from database +- **FR-8:** Recreate manually deleted CRs during reconciliation +- **FR-9:** Only manage CRs with `managed-by: fulfillment-service` annotation + +#### Legacy Tenant CR Adoption +- **FR-10:** Pre-existing Tenant CRs without the `managed-by` annotation are adopted during the initial reconciliation pass after controller upgrade +- **FR-11:** Adoption adds the `managed-by: fulfillment-service` annotation to existing CRs if they match a tenant in the database (matched by `metadata.name`) +- **FR-12:** After adoption, legacy tenants participate in full lifecycle management: updates, deletes, and reconciliation recovery +- **FR-13:** Tenant CRs that exist in the cluster but have no corresponding database entry are left unmanaged (no annotation added) to preserve manually created test/development tenants +- **FR-14:** During adoption, if a legacy Tenant CR is missing required fields (`spec.displayName` or `spec.emailDomains`), the controller backfills them from the fulfillment database before adding the `managed-by` annotation. If the corresponding tenant record is not found in the database or lacks the required fields, the CR is left unmanaged (no annotation added) + +## Dependencies + +- **osac-operator Tenant CRD and controller** (exists at `internal/controller/tenant_controller.go`) +- **Fulfillment Service database schema** (tenant name and email domains) + +## Open Questions + +### Q1: Database Watch Mechanism +Should the controller use PostgreSQL LISTEN/NOTIFY or polling? What is acceptable delay between DB insert and CR creation? + +### Q2: Email Domain Validation +Should webhook support subdomains? Internationalized domain names? DNS lookup validation? + +### Q3: Multi-Hub Distribution +Which management hubs should receive Tenant CRs? All hubs or tenant-specific selection? From d239afce53f82b39d0e7b3194f2a692f962a7be0 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Mon, 22 Jun 2026 15:21:34 +0300 Subject: [PATCH 038/450] OSAC-1110: add PRD for StorageTier API Assisted-by: Claude Code Signed-off-by: Roy Golan --- enhancements/storage-tier-OSAC-1110/prd.md | 94 ++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 enhancements/storage-tier-OSAC-1110/prd.md diff --git a/enhancements/storage-tier-OSAC-1110/prd.md b/enhancements/storage-tier-OSAC-1110/prd.md new file mode 100644 index 000000000..2cf1bd0d0 --- /dev/null +++ b/enhancements/storage-tier-OSAC-1110/prd.md @@ -0,0 +1,94 @@ +# StorageTier API + +| Field | Value | +|-------------|---------| +| Author(s) | Roy Golan rgolan@redhat.com | +| Jira | https://redhat.atlassian.net/browse/OSAC-1110 | +| Date | 2026-06-22 | + +## 1. Problem Statement + +OSAC has no API-managed catalog of storage tier offerings. Tier configuration flows through environment variables (`STORAGE_TIERS`) and Kubernetes label conventions on StorageClasses (`osac.openshift.io/storage-tier`). Neither mechanism exposes tiers through the OSAC API, captures QoS properties (IOPS limits, encryption), or establishes referential relationships with registered storage backends (StorageBackend, OSAC-1111). Without a tier abstraction, all tenants get the same storage characteristics, and downstream workflows (Tenant Storage Onboarding, Resource Creation) have no mechanism to determine which StorageClasses to create or what QoS policies to apply. + +StorageBackend (OSAC-1111) registers infrastructure — endpoints, credentials, provider type. The tenant-storage-tiers EP introduced label-based tier resolution — which StorageClass serves a given tier for a given tenant. The missing layer is a tier definition that binds a named offering to registered backends with QoS properties, enabling Cloud Provider Admins to compose differentiated storage offerings (e.g., fast, standard, archive) and making them queryable by internal services. + +## 2. Goals and Non-Goals + +### 2.1 Goals + +- Cloud Provider Admins can create, list, update, and delete storage tier offerings through the OSAC private gRPC API. +- Each StorageTier binds a named offering to one or more registered StorageBackends with per-backend protocol and provider-neutral QoS properties (bandwidth, quota, encryption), establishing a formal relationship between offerings and infrastructure. +- StorageTier replaces environment-variable-based tier configuration (`STORAGE_TIERS`) with API-managed definitions that are queryable, auditable, and validated against registered backends. +- Tier definitions are queryable by internal services (osac-operator, OSAC Storage Controller) to drive downstream workflows such as Tenant Storage Onboarding and Resource Creation. +- The StorageTier entity follows the same DB-backed private API pattern as StorageBackend and NetworkClass. +- StorageTier provides the referencing entity for tenant storage onboarding (OSAC-23), where the OSAC Storage Controller reads tier definitions to determine which backends and QoS parameters apply to a tenant. + +### 2.3 Non-Goals + +- Tenant-facing public StorageTier API — tenants see their assigned tiers through the Tenant CR status. A public read-only tier catalog may be added in a future enhancement. +- Automatic tier-to-tenant assignment during onboarding — managed by the OSAC Storage Controller (OSAC-23). +- CSI driver installation or StorageClass lifecycle on target clusters — part of the tenant storage onboarding flow (OSAC-23). +- Quota enforcement per tier. +- Per-tier capacity tracking or utilization monitoring — storage observability roadmap. +- StorageTier as a Kubernetes CRD — consistent with StorageBackend and NetworkClass. A CRD may be introduced in a future milestone. +- Advanced QoS policies and per-tenant entitlements — deferred to future milestones. +- Provider-specific QoS validation — QoS properties are stored as declared by the admin. Validation against provider capabilities is a future enhancement. +- CSI-level volume interception — today QoS enforcement flows through StorageClass parameters and provider-side policies (e.g., VAST QoS policies). A future CSI proxy layer (see `.planning/storage/csi-proxy-architecture.md`) could resolve QoS from the tier definition at volume-creation time, fully decoupling enforcement from Kubernetes StorageClass immutability. That architecture is out of scope for this work. + +## 3. Requirements + +### 3.1 Functional Requirements + +- **FR-1:** The fulfillment-service must expose a `StorageTiers` gRPC service under `osac.private.v1` with Create, Get, List, Update, and Delete RPCs. +- **FR-2:** All CRUD RPCs must include HTTP annotations for REST access via grpc-gateway (POST, GET, GET, PATCH, DELETE). +- **FR-3:** `CreateStorageTier` must accept a tier name, optional description, and one or more backend associations. Each backend association references a StorageBackend by ID, declares the storage protocol (`nfs` or `block`), and declares provider-neutral QoS properties: maximum read bandwidth (MB/s), maximum write bandwidth (MB/s), quota (bytes), and encryption enabled. The tier must be created with initial state `ACTIVE`. +- **FR-4:** `ListStorageTiers` must support pagination (`offset`/`limit`), CEL-based filtering, and SQL-like ordering, following the established OSAC List API pattern. +- **FR-5:** `UpdateStorageTier` must support partial updates — including backend association QoS properties — and optimistic concurrency control to prevent conflicting writes. QoS properties (e.g., IOPS limits) are mutable because the underlying storage provider policies (e.g., VAST QoS policies) can be updated in-place, and changes take effect for both existing and new volumes without StorageClass recreation. +- **FR-6:** `DeleteStorageTier` must permanently delete the tier. Delete must be rejected if any Tenant references the tier (referential integrity). +- **FR-7:** `CreateStorageTier` and `UpdateStorageTier` must validate that all referenced StorageBackend IDs exist. Referencing a non-existent backend must return an error. +- **FR-8:** Tier names must be unique among active (non-deleted) tiers, allowing name reuse after deletion. +- **FR-9:** StorageTier state must include `ACTIVE` (tier available for new tenant assignments). Additional states (`DEPRECATED` for retiring tiers with existing tenant assignments) are deferred to a later phase. +- **FR-10:** Deleting a StorageBackend (OSAC-1111) must be rejected if any StorageTier references it. This requirement is already captured in OSAC-1111 FR-6 and is restated here to confirm the bidirectional referential integrity contract. + +### 3.2 Non-Functional Requirements + +- **NFR-1:** The StorageTier entity must track creation and modification timestamps for auditability. +- **NFR-2:** QoS properties are provider-neutral: protocol (enum: `nfs`, `block`), max read bandwidth (integer, MB/s), max write bandwidth (integer, MB/s), quota (integer, bytes), and encryption (boolean). The design document maps these to provider-specific parameters (e.g., VAST QoS policy `static_limits`, VAST view quota). Provider-specific fields are not exposed in the StorageTier API. + +## 4. Acceptance Criteria + +- [ ] `CreateStorageTier` creates a tier with `ACTIVE` state and backend associations with protocol and QoS properties (bandwidth, quota, encryption), and returns the created object with a generated ID. +- [ ] `CreateStorageTier` rejects tiers that reference non-existent StorageBackends. +- [ ] `GetStorageTier` retrieves a tier by ID with all fields populated, including backend associations and their QoS properties. +- [ ] `ListStorageTiers` returns paginated results and supports filtering by field values (e.g., by state, by referenced backend). +- [ ] `UpdateStorageTier` applies partial updates — including QoS property changes on backend associations — without modifying unspecified fields. +- [ ] `UpdateStorageTier` rejects concurrent conflicting writes. +- [ ] `DeleteStorageTier` deletes the tier. Delete is rejected if any Tenant references the tier. +- [ ] `DeleteStorageBackend` (OSAC-1111) is rejected if any StorageTier references the backend. +- [ ] All CRUD RPCs are accessible via both gRPC and REST endpoints. +- [ ] Integration tests cover the full CRUD lifecycle, backend reference validation, pagination, filtering, and concurrency control. + +## 5. Assumptions + +- StorageTier is platform-scoped (not tenant-scoped), managed exclusively by Cloud Provider Admins. Tenant isolation applies at the tier-assignment level during tenant onboarding (OSAC-23), not at the tier-definition level. +- QoS properties are provider-neutral abstractions: protocol, max read/write bandwidth, quota, and encryption. The design document is responsible for mapping these to provider-specific parameters (e.g., VAST `qos_policy` name and `static_limits`). Additional properties (IOPS, latency targets, redundancy) can be added as proto fields without breaking changes. +- QoS property updates on a StorageTier propagate to the storage provider's policy (e.g., VAST QoS policy) and take effect for existing and new volumes. However, certain properties that are baked into Kubernetes StorageClass parameters (e.g., encryption settings, QoS policy *name*) require StorageClass recreation to take effect for new volumes — existing volumes are unaffected. A future CSI proxy layer could decouple QoS enforcement from StorageClass immutability entirely, but that is out of scope (see `.planning/storage/csi-proxy-architecture.md`). +- StorageTier is a catalog entity — it declares which backends can serve a tier and with what QoS properties. Backend selection for a specific tenant (which backend within the tier to provision on) is determined at the tier-to-tenant assignment layer (OSAC-23), not by the StorageTier API. +- StorageTier replaces the `STORAGE_TIERS` environment variable. Migration from env-var-based configuration to API-managed tiers is a one-time operation. + +## 6. Dependencies + +- **StorageBackend (OSAC-1111):** StorageTier references StorageBackend by ID. StorageBackend must be implemented first. Backend reference validation (FR-7) and referential integrity on backend deletion (FR-10) depend on the StorageBackend entity existing. +- **Tenant storage onboarding (OSAC-23):** The OSAC Storage Controller consumes StorageTier definitions when provisioning tenant storage. StorageTier can be implemented and deployed independently, but the full onboarding flow requires both. + +## 7. Risks + +### 7.1 QoS update propagation limits + +- **Owner:** Storage architect +- **Mitigation:** QoS limit changes (IOPS, bandwidth) propagate to the storage provider policy and take effect immediately. However, changes to properties embedded in Kubernetes StorageClass parameters (encryption settings, QoS policy name) require StorageClass recreation — only new volumes pick up the change. The OSAC Storage Controller (OSAC-23) must handle StorageClass recreation when the onboarding flow detects parameter drift. A future CSI proxy layer would eliminate this limitation by resolving QoS at volume-creation time from the tier definition rather than from static StorageClass parameters. + +### 7.2 Multi-backend tier complexity + +- **Owner:** Storage architect +- **Mitigation:** A tier can reference multiple backends, but StorageTier is a catalog — it declares which backends *can* serve this tier, not which backend a specific tenant *will* use. Backend selection for a tenant happens at the tier-to-tenant assignment layer (OSAC-23). This keeps the StorageTier API simple and pushes selection logic (affinity, capacity, admin preference) to the onboarding flow where tenant context is available. From eb49e9eeea46f11fb3f1a59445b790db36607f3c Mon Sep 17 00:00:00 2001 From: Menny Aboush Date: Tue, 23 Jun 2026 12:00:28 +0300 Subject: [PATCH 039/450] OSAC-1550: rename ssh_key to ssh_public_key in BareMetalInstance EP (#68) --- enhancements/baremetal-instance-api/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/enhancements/baremetal-instance-api/README.md b/enhancements/baremetal-instance-api/README.md index ec42a9820..820e5e5d0 100644 --- a/enhancements/baremetal-instance-api/README.md +++ b/enhancements/baremetal-instance-api/README.md @@ -87,7 +87,7 @@ Provisioning is driven by a chain of components: the fulfillment service creates ``` POST /api/fulfillment/v1/baremetal_instances ``` -4. The fulfillment service resolves the catalog item to a `templateID` and derives `templateParameters` from the `field_definitions`, then creates a `HostLease` CR in the management cluster with those values plus `ssh_key` and `user_data`; `BareMetalInstance.status.state` is set to `BARE_METAL_INSTANCE_STATE_PROVISIONING`. +4. The fulfillment service resolves the catalog item to a `templateID` and derives `templateParameters` from the `field_definitions`, then creates a `HostLease` CR in the management cluster with those values plus `ssh_public_key` and `user_data`; `BareMetalInstance.status.state` is set to `BARE_METAL_INSTANCE_STATE_PROVISIONING`. 5. The baremetal-fulfillment-operator picks up the `HostLease` and queries the inventory backend to find and assign a free host matching the requested host type and selector. 6. The baremetal-fulfillment-operator triggers `osac-aap` to run the host-level provisioning template (OS image, SSH key, user data) and updates the `HostLease` status on completion. 7. The osac-operator watches the `HostLease` CR and pushes status updates to the fulfillment service via the `Signal` RPC; the fulfillment service reflects this in `BareMetalInstance.status`. @@ -126,9 +126,9 @@ sequenceDiagram TU->>FS: GET /baremetal_instance_catalog_items FS-->>TU: list of available catalog items - TU->>FS: POST /baremetal_instances {catalog_item, ssh_key, ...} + TU->>FS: POST /baremetal_instances {catalog_item, ssh_public_key, ...} Note over FS: resolve catalog_item → templateID, apply field_definitions → templateParameters - FS->>MC: create HostLease CR (templateID, templateParameters, ssh_key, user_data) + FS->>MC: create HostLease CR (templateID, templateParameters, ssh_public_key, user_data) FS-->>TU: 201 Created {id, state: PROVISIONING} MC-->>BMF: watch: HostLease CR created (no ExternalHostID yet) @@ -188,7 +188,7 @@ sequenceDiagram - `PATCH /api/private/v1/baremetal_instance_catalog_items/{object.id}` - `DELETE /api/private/v1/baremetal_instance_catalog_items/{id}` -**PATCH semantics:** The `PATCH` endpoint supports partial updates to mutable fields only: `run_strategy` and `restart_requested_at`. The `catalog_item`, `ssh_key`, and `user_data` fields are immutable after creation; requests that attempt to modify them are rejected with `400 Bad Request`. A `FieldMask` is applied automatically from the fields present in the request body. +**PATCH semantics:** The `PATCH` endpoint supports partial updates to mutable fields only: `run_strategy` and `restart_requested_at`. The `catalog_item`, `ssh_public_key`, and `user_data` fields are immutable after creation; requests that attempt to modify them are rejected with `400 Bad Request`. A `FieldMask` is applied automatically from the fields present in the request body. ### Implementation Details/Notes/Constraints @@ -293,7 +293,7 @@ message BareMetalInstanceSpec { // SSH public key injected into the OS at provisioning time. Immutable after creation. // Must be a valid SSH public key in OpenSSH format (ssh-rsa, ssh-ed25519, etc.). // Invalid keys are rejected at create time with a 400 error. - optional string ssh_key = 2; + optional string ssh_public_key = 2; // User data (e.g. cloud-init). Passed to the OS at first boot. Immutable after creation. // Maximum size: 64 KB. @@ -431,7 +431,7 @@ The fulfillment-service, baremetal-fulfillment-operator, and osac-operator must ## Support Procedures -> **Log hygiene:** Before sharing or inspecting logs, operators must verify that `ssh_key`, `user_data`, and backend credentials (inventory API tokens, AAP credentials) are redacted. These values must never appear in fulfillment-service or operator logs; if they do, treat it as a security incident and rotate the affected credentials immediately. +> **Log hygiene:** Before sharing or inspecting logs, operators must verify that `ssh_public_key`, `user_data`, and backend credentials (inventory API tokens, AAP credentials) are redacted. These values must never appear in fulfillment-service or operator logs; if they do, treat it as a security incident and rotate the affected credentials immediately. **Symptom:** `BareMetalInstance` stuck in `BARE_METAL_INSTANCE_STATE_PROVISIONING`. **Diagnosis:** Check baremetal-fulfillment-operator logs for inventory or AAP errors. Check osac-aap job logs for playbook failures. Verify the `HostLease` CR exists and that `ExternalHostID` has been set (indicates host was found in inventory). From c853add36d9b959f7af9e0ea6a76d2fd849cfef4 Mon Sep 17 00:00:00 2001 From: Bat-Zion Rotman Date: Tue, 23 Jun 2026 12:21:47 +0300 Subject: [PATCH 040/450] PRD: Configuration wizard for cluster and VM resources (#57) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add PRD for OSAC-1421: Configuration wizard for cluster and VM resources Assisted-by: Claude Code * Fix trailing whitespace in PRD per pre-commit Assisted-by: Claude Code * Clarify non-goals as unsupported; remove admin credential validation item Assisted-by: Claude Code * Require Review to always show Basics fields (name, SSH key, pull secret) Assisted-by: Claude Code * Update enhancements/configuration-wizard-OSAC-1421/prd.md Co-authored-by: Avishay Traeger * Revise PRD: static wizard fields with catalog overlay Replace catalog-driven field discovery with a fixed per-resource-type field set. Catalog field_definitions override display, editability, defaults, required, and validation for matching paths. Add ComputeInstance networking pickers (virtual network, subnet, security groups) and resolve required-field semantics. Template parameters and BareMetal remain out of scope. Assisted-by: Claude Code * Adopt five-step wizard: Catalog → General → Compute → Networking → Review Assisted-by: Claude Code * Revise OSAC-1421 PRD: condensed spec and open decisions Streamline wizard requirements, document VM networking list APIs, read-only catalog overlay fields, field defaults, and shortened open-decision section for requiredness and additional disks. Assisted-by: Claude Code * Fix trailing newline in PRD per pre-commit Assisted-by: Claude Code * Revise OSAC-1421 PRD: rename, wizard steps, and review feedback. Move the PRD to cluster-and-vm-provisioning-wizard, add YAML frontmatter, renumber sections from §1, adopt Access/Configuration steps with image on Configuration, and ignore catalog field_definitions for network_attachments. Assisted-by: Claude Code * OSAC-1421: expand PRD with instance type, overlay, and wizard UX rules. Add VM instance type picker and OS family fields, open decisions for catalog overlay and cluster node_sets, Review mirroring wizard fields, Access step exempt from field_definitions, and validate-on-Next step navigation. Assisted-by: Claude Code Co-authored-by: Cursor * Fix trailing whitespace in PRD per pre-commit. Assisted-by: Claude Code Co-authored-by: Cursor * OSAC-1421: rename wizard Catalog step to Catalog Item. Assisted-by: Claude Code Co-authored-by: Cursor * OSAC-1421: reference OSAC-13 for spec.is_windows dependency. Assisted-by: Claude Code Co-authored-by: Cursor * resolve field_definitions vs API driven pickers question * OSAC-1421: rename wizard Access step to General Assisted-by: Claude Code Co-authored-by: Cursor --------- Co-authored-by: Avishay Traeger Co-authored-by: Cursor --- .../cluster-and-vm-provisioning-wizard/prd.md | 253 ++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 enhancements/cluster-and-vm-provisioning-wizard/prd.md diff --git a/enhancements/cluster-and-vm-provisioning-wizard/prd.md b/enhancements/cluster-and-vm-provisioning-wizard/prd.md new file mode 100644 index 000000000..116e5defa --- /dev/null +++ b/enhancements/cluster-and-vm-provisioning-wizard/prd.md @@ -0,0 +1,253 @@ +--- +title: Configuration Wizard for Cluster and VM Resources +authors: + - brotman@redhat.com +creation-date: 2026-06-14 +last-updated: 2026-06-24 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-1421 +see-also: + - Catalog Items: /enhancements/catalog-items + - VM Instance Types: /enhancements/vm-instance-types +replaces: + - N/A +superseded-by: + - N/A +--- + +# Configuration Wizard for Cluster and VM Resources + +## 1. Goals and Non-Goals + +### 1.1 Goals + +- Tenants provision VMs and clusters by selecting a catalog offering and completing a guided wizard with a **fixed field set per resource type** ([§2.1.1](#211-static-wizard-fields)). +- Both resource types use the same five steps: **Catalog Item → General → Configuration → Networking → Review** (submit from Review). **General** collects name and credentials; **Configuration** collects image/release, sizing, and platform parameters — not networking placement. +- Catalog `field_definitions` overlay matching static paths on **Configuration** and **Networking** steps only (not General, not picker-backed fields in v1) for **display name**, **editability**, **default**, and **validation_schema** — they do not add fields or payload paths ([§2.1.2](#212-catalog-overlay-and-defaults)). + +### 1.2 Non-Goals + +- **BareMetalInstance** provisioning (separate PRD) +- **Template parameters** +- **Multi-NIC** — wizard submits one `network_attachments` entry (one VN, one subnet, security groups); no add/remove NIC rows +- **Cluster pool add/remove** — tenants edit **size** only for template-defined `spec.node_sets` pools; no add/remove pools or `host_type` changes +- **`spec.additional_disks`** — wizard scope undecided ([§5](#5-open-decisions)); default: boot disk only + +## 2. Requirements + +### 2.1 Field model + +#### 2.1.1 Static wizard fields + +Fields are hardcoded per resource type, not discovered from `field_definitions`. **General** step fields always use wizard defaults (labels, editability, validation); catalog `field_definitions` are **ignored** on General ([§2.1.2](#212-catalog-overlay-and-defaults)). **Required** column: **?** = required vs optional not yet decided ([§5](#5-open-decisions)). + +**ComputeInstance** + + +| Step | Path | Label | Widget | Required | +| --------------- | ------------------------- | ---------------------------------------- | -------------------------------------- | -------- | +| General | `metadata.name` | Name | Text | Required | +| General | `spec.ssh_key` | SSH public key | Text (multiline) | ? | +| Configuration | `spec.image.source_ref` | VM image (OCI reference) | Text | Required | +| Configuration | `spec.is_windows` | OS family | Radio (`Linux`, `Windows`) | Required | +| Configuration | `spec.instance_type` | Instance type | Picker ([§2.1.5](#215-vm-instance-type-picker-api)) | Required | +| Configuration | `spec.user_data` | User data (cloud-init / Ignition) | Text (multiline) | Optional | +| Configuration | `spec.boot_disk.size_gib` | Boot disk size (GiB) | Number | ? | +| Configuration | `spec.run_strategy` | Run strategy | Select (`Always`, `Halted`) | Required | +| Networking | `spec.network_attachments` | Virtual network, subnet, security groups | Pickers ([§2.1.4](#214-vm-networking-picker-apis)) | Required | + +**Notes:** + +- **`spec.user_data`**: plain multiline string (cloud-init or Ignition); omit from payload when empty. Stored as Secret → KubeVirt `cloudInitNoCloud`. +- **`spec.image`**: wizard collects `source_ref` only; payload always sets `spec.image.source_type` to **`registry`**. Future: ComputeImage list picker ([OSAC-979](https://redhat.atlassian.net/browse/OSAC-979)). +- **`spec.is_windows`**: Configuration-step **OS family** radio — **Linux** → `is_windows: false`; **Windows** → `is_windows: true`. Maps to the optional boolean added in [fulfillment-service PR #734](https://github.com/osac-project/fulfillment-service/pull/734) ([OSAC-13](https://redhat.atlassian.net/browse/OSAC-13)); the reconciler maps this to CR `guestOSFamily` for AAP provisioning. Required on the wizard; default selection **Linux** when no catalog `default` ([§2.1.2](#212-catalog-overlay-and-defaults)). The wizard always sends an explicit value. +- **`spec.instance_type`**: Configuration-step **instance type** picker — tenant selects a named compute bundle (cores + memory) from [§2.1.5](#215-vm-instance-type-picker-api). Payload sends **`spec.instance_type` only** (instance type name); the wizard does **not** collect or send `spec.cores` or `spec.memory_gib` ([VM Instance Types EP](/enhancements/vm-instance-types), [fulfillment-service PR #735](https://github.com/osac-project/fulfillment-service/pull/735) / OSAC-1217). The API validates the name and state; the reconciler resolves cores/memory on the CR. Catalog `field_definitions` for this path are **ignored** in v1 ([§2.1.2](#212-catalog-overlay-and-defaults)). +- **Disks**: wizard collects `spec.boot_disk.size_gib` only unless [§5](#5-open-decisions) chooses `spec.additional_disks`. +- **Networking**: pickers assemble a single `spec.network_attachments` entry; raw JSON not shown. Catalog `field_definitions` for this path (including nested paths) are **ignored** in v1 ([§2.1.2](#212-catalog-overlay-and-defaults)). APIs: [§2.1.4](#214-vm-networking-picker-apis). + +**Cluster** + + +| Step | Path | Label | Widget | Required | +| --------------- | --------------------------- | ------------------------------------------------------------- | ------------------------------------ | -------- | +| General | `metadata.name` | Name | Text | Required | +| General | `spec.ssh_public_key` | SSH public key | Text (multiline) | ? | +| General | `spec.pull_secret` | Pull secret | Text (multiline, masked) | Required | +| Configuration | `spec.release_image` | OpenShift version (release image) | Text | Required | +| Configuration | `spec.node_sets` | Worker node pools | Table | Required | +| Networking | `spec.network.pod_cidr` | Pod network CIDR | Text | ? | +| Networking | `spec.network.service_cidr` | Service network CIDR | Text | ? | + +**Notes:** + +- **`spec.node_sets`**: after catalog selection, load `ClusterTemplates.Get` → one table row per `node_sets` key. Columns: pool name and `host_type` (read-only, from template + `HostTypes.Get`); **Nodes** = `size` input (each pool `size` must be > 0). Empty template `node_sets` handling is **undecided** ([§5](#5-open-decisions)). Payload: template `host_type` + tenant/catalog `size` per pool. + +**Create payload:** Only paths in [§2.1.1](#211-static-wizard-fields) plus catalog item reference; VM hardcodes `spec.image.source_type` = `registry`; VM sends `spec.instance_type` and `spec.is_windows` explicitly, not `spec.cores` or `spec.memory_gib`. + +#### 2.1.2 Catalog overlay and defaults + +For each static **non-picker** field on **Configuration** and **Networking** steps, match `field_definitions` by `path`. **General** step fields ignore catalog `field_definitions` entirely (wizard labels, editability, and validation only). Non-matching paths are **ignored** (not on Review, not in payload). + +**Picker-backed fields (v1):** `spec.instance_type` and `spec.network_attachments` (including nested paths such as `spec.network_attachments.subnet`) load options from list APIs ([§2.1.5](#215-vm-instance-type-picker-api), [§2.1.4](#214-vm-networking-picker-apis)). Matching catalog `field_definitions` for these paths are **ignored** — wizard labels, editability, validation, and defaults come from wizard defaults and list-API behavior only. Catalog overlay on picker fields is **deferred** to a later release ([§5](#5-open-decisions)). + +| Aspect | Matching entry (non-picker fields only) | No matching entry | +| ---------- | --------------------------------------------------------------------------- | --------------------- | +| Label | `display_name` or wizard default | Wizard default | +| Editable | `editable: false` → read-only on wizard step; blank when no catalog `default` | `true` | +| Default | Catalog `default` if set; else blank | Blank | +| Validation | `validation_schema` maps to integer/enum/text widgets; inline errors on blur; full step validation on Next (see [§2.2](#22-wizard-behavior)) | API/wizard validation | + +Non-editable fields (`editable: false`) are **read-only** on the wizard step (Configuration or Networking), not hidden. With a catalog `default`, the value is included in the payload. Without a catalog `default`, the field is **blank and read-only**. Read-only fields use disabled/read-only controls (same widget type as editable fields where applicable). + +**Default rules:** Fields start **blank** unless catalog `default` is set or a **special case** applies: + + +| Case | Behavior | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | +| `spec.run_strategy` | Pre-select `Always` when no catalog `default` | +| OS family (VM) | Pre-select **Linux** (`is_windows: false`) when no catalog `default` | +| Instance type (VM) | **Auto-select** when `InstanceTypes.List` returns exactly one option | +| Networking pickers | **Auto-select** when a list returns exactly one option (VN → subnet → SGs) | + +#### 2.1.3 Open required fields + +Fields marked **?** in [§2.1.1](#211-static-wizard-fields) — resolve Required vs Optional before implementation ([§5](#5-open-decisions)). + +### 2.1.4 VM networking picker APIs + +The wizard loads picker options from the **public** fulfillment APIs (`osac.public.v1`). The UI uses the generated OpenAPI client (REST); gRPC equivalents are listed for reference. + +| Picker | gRPC | REST | Purpose | +| ------ | ---- | ---- | ------- | +| Virtual network | `VirtualNetworks.List` | `GET /api/fulfillment/v1/virtual_networks` | Tenant-visible virtual networks | +| Subnet | `Subnets.List` | `GET /api/fulfillment/v1/subnets` | Subnets in the selected virtual network | +| Security groups | `SecurityGroups.List` | `GET /api/fulfillment/v1/security_groups` | Security groups in the selected virtual network | + +**List request parameters** (all three): optional query `filter` (CEL), `limit`, `offset`, `order`. Tenant scope is implicit from the authenticated session. + +**Subnet and security group filters** (after virtual network selection): + +```text +this.spec.virtual_network == "" +``` + +**Picker display and values:** + +| Picker | Option label | Selected value | +| ------ | ------------ | -------------- | +| Virtual network | `metadata.name` (fallback `id`) | VirtualNetwork `id` — drives subnet/SG list filters only | +| Subnet | `metadata.name` (fallback `id`) | Subnet `id` | +| Security group | `metadata.name` (fallback `id`) | SecurityGroup `id` (multi-select) | + +**Create payload assembly** — one `spec.network_attachments` element: + +```json +{ + "subnet": "", + "security_groups": [""] +} +``` + +Per `NetworkAttachment` in `compute_instance_type.proto`. The wizard does not send virtual network ID in `network_attachments`; placement is implied by the subnet (security groups must belong to the same virtual network). + +**Load order:** virtual network list → on selection, load filtered subnet and security group lists → auto-select when a list returns exactly one item ([§2.1.2](#212-catalog-overlay-and-defaults)). + +### 2.1.5 VM instance type picker API + +The Configuration step loads instance type options from the **public** fulfillment API (`osac.public.v1`). The UI uses the generated OpenAPI client (REST); gRPC equivalent listed for reference. + +| Picker | gRPC | REST | Purpose | +| ------ | ---- | ---- | ------- | +| Instance type | `InstanceTypes.List` | `GET /api/fulfillment/v1/instance_types` | Tenant-visible instance types (ACTIVE and DEPRECATED by default) | + +**List request parameters:** optional query `filter` (CEL), `limit`, `offset`, `order`. Tenant scope is implicit from the authenticated session. The default list excludes **OBSOLETE** instance types (not selectable for new VMs). + +**Picker display and values:** + +| Picker | Option label | Selected value | +| ------ | ------------ | -------------- | +| Instance type | `metadata.name` plus `spec.cores` and `spec.memory_gib` (e.g. `standard-4-16 — 4 vCPU, 16 GiB`); indicate **DEPRECATED** state in the label when `spec.state` is DEPRECATED | Instance type name (`metadata.name` / `id`) → `spec.instance_type` on create | + +**Create payload:** send only the instance type **name** string: + +```json +{ + "instance_type": "standard-4-16" +} +``` + +Do **not** send `cores` or `memory_gib` — they are mutually exclusive with `instance_type` at the API ([PR #735](https://github.com/osac-project/fulfillment-service/pull/735)). + +**Deprecation handling:** if the selected type is DEPRECATED, create may succeed with **warnings** in the response; the wizard surfaces those warnings after submit (non-blocking). OBSOLETE types are not offered in the picker. + +**Load order:** load instance type list when entering Configuration → auto-select when the list returns exactly one item ([§2.1.2](#212-catalog-overlay-and-defaults)). + +### 2.2 Wizard behavior + +```mermaid +flowchart LR + A[Catalog Item] --> B[General] + B --> C[Configuration] + C --> D[Networking] + D --> E[Review and Submit] +``` + +- **Catalog Item:** Require catalog item selection. +- **Review:** Shows the same values the user sees on wizard step fields (General, Configuration, Networking) — blank, catalog- or wizard-defaulted, or user-entered — with the same labels as on each step. Submit from Review. +- **Step navigation:** Next is always enabled. On click, validate every field on the current step — including fields that have not yet blurred and therefore have no inline error shown. Surface any hidden errors inline; if validation fails, show an alert asking the user to fix the errors and do not advance. + +## 3. Acceptance Criteria + +- Wizard provisions VM or Cluster using only [§2.1.1](#211-static-wizard-fields) payload paths plus hardcoded VM `source_type` and catalog item reference. +- Five-step flow: Catalog Item → General → Configuration → Networking → Review; submit from Review. +- Review shows the same values as on wizard step fields (blank, default-driven, or user-entered). +- Catalog overlay and default rules per [§2.1.2](#212-catalog-overlay-and-defaults) on Configuration and Networking **non-picker** fields only (General and picker-backed paths ignore `field_definitions` in v1); non-editable fields without `default` appear blank and read-only on their wizard step and on Review; non-editable fields with `default` appear read-only with value on their wizard step and on Review. +- VM: single `network_attachments` entry assembled from picker APIs; instance type picker sets `spec.instance_type` (not `cores`/`memory_gib`); OS family radio sets `spec.is_windows` (default **Linux**); optional `user_data` omitted when empty; create warnings for deprecated instance types are shown to the user. +- Cluster: `node_sets` matches template pool keys with template `host_type` and tenant `size` > 0; empty template `node_sets` handling per [§5](#5-open-decisions). +- All **?** requiredness decisions resolved before release ([§5](#5-open-decisions)). +- On Next click, validate all fields on the current step (including untouched fields); surface hidden inline errors; show an alert if invalid; do not advance until the step is valid. + +## 4. Dependencies + +- `ComputeInstanceCatalogItem`, `ClusterCatalogItem` (with `field_definitions`) +- `ClusterTemplates.Get`, `HostTypes.Get` (cluster Configuration step) +- `VirtualNetworks.List`, `Subnets.List`, `SecurityGroups.List` (gRPC `osac.public.v1`) / REST `GET /api/fulfillment/v1/virtual_networks`, `.../subnets`, `.../security_groups` ([§2.1.4](#214-vm-networking-picker-apis)) +- `InstanceTypes.List` (gRPC `osac.public.v1`) / REST `GET /api/fulfillment/v1/instance_types` ([§2.1.5](#215-vm-instance-type-picker-api)) +- ComputeInstance and Cluster create APIs +- `spec.instance_type` on ComputeInstance ([OSAC-1217](https://redhat.atlassian.net/browse/OSAC-1217), [fulfillment-service PR #735](https://github.com/osac-project/fulfillment-service/pull/735)) — required for VM instance type picker +- `spec.is_windows` on ComputeInstance ([OSAC-13](https://redhat.atlassian.net/browse/OSAC-13), [fulfillment-service PR #734](https://github.com/osac-project/fulfillment-service/pull/734)) — required for VM OS family in the wizard + +## 5. Open decisions + +Resolve before implementation. + +### Required vs optional (`?`) + +| Path | Resource | +| ---- | -------- | +| `spec.ssh_key` / `spec.ssh_public_key` | Both — **one decision**: Required (block Next) vs Optional (omit when empty, like `spec.user_data`) | +| `spec.boot_disk.size_gib` | ComputeInstance | +| `spec.network.pod_cidr`, `spec.network.service_cidr` | Cluster | + +### Catalog overlay on picker-backed fields (deferred) + +**Resolved for v1:** Ignore catalog `field_definitions` for picker-backed paths (`spec.instance_type`, `spec.network_attachments`, and nested networking paths). Picker UX is API-driven only; see [§2.1.2](#212-catalog-overlay-and-defaults). + +**Deferred:** Catalog overlay on picker fields (including `display_name`, `editable`, `default`, `validation_schema`, catalog-default vs auto-select precedence, and defaults not present in list API options) is out of scope for v1 and may be addressed in a later release. + +### Cluster template `node_sets` + +| Question | Options | +| -------- | ------- | +| **Empty template `node_sets` (Cluster)?** | After catalog selection, `ClusterTemplates.Get` returns **empty** `node_sets`: **Block** — wizard fails with an error; tenant cannot proceed until the catalog item or template is corrected. **Filter catalog** — do not offer catalog items whose backing template has empty `node_sets`. **Warn and continue** — show a non-blocking warning; Configuration step shows an empty pool table. **Other** — describe behavior. Payload when pools exist: template `host_type` + tenant/catalog `size` per pool ([§2.1.1](#211-static-wizard-fields)). | + +Resolve before implementation; document the chosen rule in [§2.1.1](#211-static-wizard-fields) and update acceptance criteria. + +### Additional disks + +Not in [§2.1.1](#211-static-wizard-fields) today. **Unknown** whether v1 needs wizard UI for `spec.additional_disks[]` or boot disk + API/CLI is enough. + +| Option | Outcome | +| ------ | ------- | +| **No (default)** | Out of scope ([§1.2](#12-non-goals)); boot disk only | +| **Yes** | Add repeatable `size_gib` rows on Configuration; add to §2.1.1 | From 01e138fc822f9ec94be42c8f9226618edcf31e8d Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Tue, 23 Jun 2026 15:14:31 +0300 Subject: [PATCH 041/450] OSAC-1110: address PR #66 review feedback - Add cross-reference links to StorageBackend PRD and tenant-storage-tiers EP in the Problem Statement - Add explicit v0.1 scope assumption: automatic StorageClass refresh on QoS change is not in scope, handled by OSAC-23 Assisted-by: Claude Code Signed-off-by: Roy Golan --- enhancements/storage-tier-OSAC-1110/prd.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/enhancements/storage-tier-OSAC-1110/prd.md b/enhancements/storage-tier-OSAC-1110/prd.md index 2cf1bd0d0..733612b88 100644 --- a/enhancements/storage-tier-OSAC-1110/prd.md +++ b/enhancements/storage-tier-OSAC-1110/prd.md @@ -10,7 +10,7 @@ OSAC has no API-managed catalog of storage tier offerings. Tier configuration flows through environment variables (`STORAGE_TIERS`) and Kubernetes label conventions on StorageClasses (`osac.openshift.io/storage-tier`). Neither mechanism exposes tiers through the OSAC API, captures QoS properties (IOPS limits, encryption), or establishes referential relationships with registered storage backends (StorageBackend, OSAC-1111). Without a tier abstraction, all tenants get the same storage characteristics, and downstream workflows (Tenant Storage Onboarding, Resource Creation) have no mechanism to determine which StorageClasses to create or what QoS policies to apply. -StorageBackend (OSAC-1111) registers infrastructure — endpoints, credentials, provider type. The tenant-storage-tiers EP introduced label-based tier resolution — which StorageClass serves a given tier for a given tenant. The missing layer is a tier definition that binds a named offering to registered backends with QoS properties, enabling Cloud Provider Admins to compose differentiated storage offerings (e.g., fast, standard, archive) and making them queryable by internal services. +[StorageBackend](../storage-backend-osac-1111/prd.md) (OSAC-1111) registers infrastructure — endpoints, credentials, provider type. The [tenant-storage-tiers EP](../tenant-storage-tiers/README.md) introduced label-based tier resolution — which StorageClass serves a given tier for a given tenant. The missing layer is a tier definition that binds a named offering to registered backends with QoS properties, enabling Cloud Provider Admins to compose differentiated storage offerings (e.g., fast, standard, archive) and making them queryable by internal services. ## 2. Goals and Non-Goals @@ -75,6 +75,7 @@ StorageBackend (OSAC-1111) registers infrastructure — endpoints, credentials, - QoS property updates on a StorageTier propagate to the storage provider's policy (e.g., VAST QoS policy) and take effect for existing and new volumes. However, certain properties that are baked into Kubernetes StorageClass parameters (e.g., encryption settings, QoS policy *name*) require StorageClass recreation to take effect for new volumes — existing volumes are unaffected. A future CSI proxy layer could decouple QoS enforcement from StorageClass immutability entirely, but that is out of scope (see `.planning/storage/csi-proxy-architecture.md`). - StorageTier is a catalog entity — it declares which backends can serve a tier and with what QoS properties. Backend selection for a specific tenant (which backend within the tier to provision on) is determined at the tier-to-tenant assignment layer (OSAC-23), not by the StorageTier API. - StorageTier replaces the `STORAGE_TIERS` environment variable. Migration from env-var-based configuration to API-managed tiers is a one-time operation. +- Automatic StorageClass refresh when QoS properties change on a StorageTier is not in v0.1 scope. StorageClass lifecycle management (creation, recreation on parameter drift) is handled by the OSAC Storage Controller (OSAC-23). ## 6. Dependencies From 8608258eac0252a66d4f8f61af6ecca8cb3f85c0 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 24 Jun 2026 01:17:25 -0400 Subject: [PATCH 042/450] OSAC-1123: add PRD for CaaS cluster storage PRD for enabling persistent storage on CaaS tenant clusters. Covers the trigger logic in the OSAC Storage Controller: when a ClusterOrder reaches Ready, invoke cluster-side storage setup via AAP to install CSI driver and per-tenant StorageClasses. Tracks storage readiness on the ClusterOrder CR and handles teardown on cluster deletion. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude --- enhancements/caas-cluster-storage/prd.md | 112 +++++++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 enhancements/caas-cluster-storage/prd.md diff --git a/enhancements/caas-cluster-storage/prd.md b/enhancements/caas-cluster-storage/prd.md new file mode 100644 index 000000000..0dc66ab6c --- /dev/null +++ b/enhancements/caas-cluster-storage/prd.md @@ -0,0 +1,112 @@ +# CaaS Cluster Storage + +| Field | Value | +|-------------|---------| +| Author(s) | Akshay Nadkarni | +| Jira | https://redhat.atlassian.net/browse/OSAC-1332 | +| Date | 2026-06-24 | + +## 1. Problem Statement + +CaaS tenant clusters are provisioned without persistent storage. When a ClusterOrder reaches Ready, the cluster has compute but no StorageClasses or CSI driver, so tenant workloads cannot create PVCs. The OSAC Storage Controller already handles backend setup (Stage 1) at tenant onboarding and cluster-side setup (Stage 2) for VMaaS, but it ignores ClusterOrder events. There is no automation connecting CaaS cluster readiness to storage installation. + +## 2. Goals and Non-Goals + +### 2.1 Goals + +- Automatically install the CSI driver and per-tenant, per-tier StorageClasses on a CaaS cluster when it reaches Ready. +- Track storage readiness on the ClusterOrder CR via a `ClusterStorageReady` condition, visible to Cloud Provider Admins. +- Clean up cluster-side storage resources on CaaS cluster deletion. Leave backend resources (VAST tenant, views, quotas, hub Secret) intact for other clusters. +- Handle multiple CaaS clusters per tenant independently, each with its own storage readiness. +- Integrate with the Storage Tier API (OSAC-1110) and StorageBackend API (OSAC-1111) when available, falling back to `STORAGE_TIERS` env var and single implicit backend otherwise. + +### 2.2 Non-Goals + +- VAST provider CaaS changes (tenant-UID paths, RBAC credentials, `hcp_data_plane` target). Covered by OSAC-1122. +- UI for storage. +- Stage 1 (backend setup). Assumed complete before any CaaS cluster reaches Ready. +- VMaaS cluster-side storage. Existing flow is unchanged. + +## 3. Requirements + +### 3.1 Functional Requirements + +#### CaaS Stage 2 Trigger + +- **FR-1:** When a ClusterOrder reaches `phase=Ready` and the owning Tenant has `StorageBackendReady=True`, the storage controller invokes `osac-create-tenant-cluster-storage` with `provisioning_target=hcp_data_plane`. +- **FR-2:** The storage controller passes the CaaS cluster's connection details (kubeconfig or equivalent) to the AAP playbook as extra vars. The playbook does not discover cluster access on its own. +- **FR-3:** The storage controller resolves tiers from the Tier API (OSAC-1110) when available, falling back to `STORAGE_TIERS` env var. +- **FR-4:** The storage controller discovers backends from the StorageBackend API (OSAC-1111) when available, falling back to the single implicit backend. + +#### Storage Readiness + +- **FR-5:** The storage controller sets a `ClusterStorageReady` condition on the ClusterOrder CR: `True` when the AAP job succeeds and StorageClasses are confirmed, `False` with reason on failure. +- **FR-6:** The storage controller records per-cluster detail in the Tenant CR's `status.clusterStorage` array (ClusterOrder name, readiness, reason). +- **FR-7:** `kubectl get clusterorder -o wide` shows a `ClusterStorageReady` column. + +#### Teardown + +- **FR-8:** The storage controller places a finalizer on each ClusterOrder where storage was set up. On deletion, it triggers `osac-delete-tenant-cluster-storage` to remove StorageClasses, VolumeSnapshotClasses, and CSI Secret from the CaaS cluster. The finalizer is removed only after cleanup completes. +- **FR-9:** After cleanup, the storage controller removes the ClusterOrder's entry from `status.clusterStorage` on the Tenant CR. + +### 3.2 Non-Functional Requirements + +- **NFR-1:** The storage controller must not log or emit events containing credentials, Secret contents, or sensitive AAP parameters. + +## 4. Acceptance Criteria + +**Trigger** +- [ ] ClusterOrder reaching Ready with `StorageBackendReady=True` triggers `osac-create-tenant-cluster-storage` with `provisioning_target=hcp_data_plane` +- [ ] AAP playbook receives cluster connection details as extra vars +- [ ] Tiers resolved from Tier API when available, `STORAGE_TIERS` env var otherwise +- [ ] Backends discovered from StorageBackend API when available, single implicit backend otherwise + +**Readiness** +- [ ] `ClusterStorageReady=True` on ClusterOrder when setup succeeds +- [ ] `ClusterStorageReady=False` with reason on failure +- [ ] `kubectl get clusterorder -o wide` shows storage readiness +- [ ] Tenant CR `status.clusterStorage` has an entry per CaaS cluster + +**Teardown** +- [ ] Finalizer placed on ClusterOrders where storage was set up +- [ ] ClusterOrder deletion triggers `osac-delete-tenant-cluster-storage` +- [ ] Finalizer removed only after cleanup completes +- [ ] Backend resources unaffected by cluster deletion +- [ ] `status.clusterStorage` entry removed after cleanup + +**Multi-Cluster** +- [ ] Second ClusterOrder for the same tenant gets independent storage setup +- [ ] Deleting one ClusterOrder does not affect other clusters + +**Testing** +- [ ] Unit tests with mock AAP providers cover trigger, readiness, teardown, and multi-cluster +- [ ] (Stretch) E2E against live VAST + CaaS cluster + +## 5. Assumptions + +- Stage 1 is always complete before a CaaS cluster reaches Ready. The controller does not handle a Ready ClusterOrder with an unprovisioned backend. +- AAP playbooks accept `hcp_data_plane` and use the passed kubeconfig to target the CaaS cluster. Delivered by OSAC-1122. +- VAST is the only storage provider for v0.1. A global VIP Pool is shared by all tenants. +- The kubeconfig for a CaaS cluster is obtainable from the HostedControlPlane status via the ClusterOrder's cluster reference. Exact mechanism pending CaaS working group confirmation. + +## 6. Dependencies + +- **OSAC-23 (Storage Controller):** Implemented and merged (osac-operator PR #299). This PRD extends the controller's ClusterOrder handling. +- **osac-aap PR #338 (playbook split):** Under review. Required for CaaS setup and teardown to operate independently from backend operations. +- **OSAC-1122 (VAST for CaaS):** VAST provider must accept `hcp_data_plane` and use the passed kubeconfig. Without this, the trigger fires but the playbook cannot act. +- **OSAC-1110 (Tier API):** Not blocking. Controller integrates when available, falls back to env var. +- **OSAC-1111 (StorageBackend API):** Not blocking. In development (fulfillment-service PR #728). Controller integrates when available, falls back to single implicit backend. + +## 7. Risks + +### 7.1 CaaS cluster kubeconfig access is unconfirmed + +- **Owner:** Akshay Nadkarni / CaaS working group +- **Mitigation:** The expected path (`ClusterOrder.status.clusterReference` to `HostedControlPlane.status.kubeConfig`) is traceable through the HyperShift API. If the mechanism differs, only credential retrieval changes; trigger and condition logic are unaffected. + +## 8. Open Questions + +### 8.1 What credential should the storage controller use for CaaS cluster access? + +- **Owner:** CaaS working group +- **Impact:** Affects FR-2. The expected path is `HostedControlPlane.status.kubeConfig`, but a scoped service account token may be preferred. From 6fad36f5f59ab70f408814aaf29a9d3e1464fc97 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Wed, 24 Jun 2026 13:56:48 +0300 Subject: [PATCH 043/450] OSAC-1110: add NFR-3 clarifying delete scope for StorageTier DeleteStorageTier only removes the tier from the fulfillment-service DB. It does not cascade to Kubernetes StorageClasses. Addresses reviewer feedback from PR #66. Assisted-by: Claude Code Signed-off-by: Roy Golan --- enhancements/storage-tier-OSAC-1110/prd.md | 1 + 1 file changed, 1 insertion(+) diff --git a/enhancements/storage-tier-OSAC-1110/prd.md b/enhancements/storage-tier-OSAC-1110/prd.md index 733612b88..cfbffdc35 100644 --- a/enhancements/storage-tier-OSAC-1110/prd.md +++ b/enhancements/storage-tier-OSAC-1110/prd.md @@ -54,6 +54,7 @@ OSAC has no API-managed catalog of storage tier offerings. Tier configuration fl - **NFR-1:** The StorageTier entity must track creation and modification timestamps for auditability. - **NFR-2:** QoS properties are provider-neutral: protocol (enum: `nfs`, `block`), max read bandwidth (integer, MB/s), max write bandwidth (integer, MB/s), quota (integer, bytes), and encryption (boolean). The design document maps these to provider-specific parameters (e.g., VAST QoS policy `static_limits`, VAST view quota). Provider-specific fields are not exposed in the StorageTier API. +- **NFR-3:** Deleting a StorageTier removes the tier definition from the fulfillment-service database. It does not delete or modify Kubernetes StorageClasses that were created based on the tier. StorageClass lifecycle management is handled by the OSAC Storage Controller (OSAC-23). ## 4. Acceptance Criteria From a10e9ffa677a767d73b67ccee1c11ec6ad5b58f5 Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Thu, 7 May 2026 18:17:30 +0300 Subject: [PATCH 044/450] Add repository consolidation enhancement proposal Proposes merging fulfillment-service, osac-operator, osac-aap, and osac-installer into a single mono-repo to eliminate protobuf version skew, installer drift, and cross-repo CI gaps. Assisted-by: Claude Code Co-Authored-By: Claude Opus 4.6 (1M context) Signed-off-by: Eran Cohen --- .../repository-consolidation/README.md | 421 ++++++++++++++++++ 1 file changed, 421 insertions(+) create mode 100644 enhancements/repository-consolidation/README.md diff --git a/enhancements/repository-consolidation/README.md b/enhancements/repository-consolidation/README.md new file mode 100644 index 000000000..f414dc271 --- /dev/null +++ b/enhancements/repository-consolidation/README.md @@ -0,0 +1,421 @@ +--- +title: repository-consolidation +authors: + - ercohen@redhat.com +creation-date: 2026-05-07 +last-updated: 2026-05-07 +tracking-link: + - TBD +see-also: + - N/A +replaces: + - N/A +superseded-by: + - N/A +--- + +# Repository Consolidation: Mono-Repo for OSAC Core Components + +## Summary + +This enhancement proposes consolidating four tightly coupled OSAC repositories +(`fulfillment-service`, `osac-operator`, `osac-aap`, and `osac-installer`) into +a single mono-repo. The current multi-repo layout creates persistent friction in +cross-component changes, installer drift, protobuf version skew, and CI +coverage gaps. A single repository with per-component directories eliminates +these issues while preserving separation of concerns through directory structure +and CODEOWNERS. + +## Motivation + +OSAC's core components are tightly coupled: the `fulfillment-service` defines +protobuf types and CRDs consumed by `osac-operator`; the operator produces +Kubernetes resources that `osac-aap` provisions via Ansible; and `osac-installer` +must wire all three together with correct image references, Kustomize overlays, +and infrastructure prerequisites. Despite this coupling, the components live in +separate GitHub repositories, which creates the following pain points: + +**Installer drift.** The `osac-installer` repository frequently falls behind the +other components. Updates happen on a bi-weekly ad-hoc basis when someone +manually bumps image SHAs. In the interim, testers reproduce bugs that have +already been fixed, wasting time on stale deployments. A recent incident +required days to land an installer update PR because two weeks of accumulated +changes had broken Envoy configuration, Keycloak setup, and tenancy defaults +simultaneously. + +**Protobuf and CRD version skew.** When a data structure changes in +`fulfillment-service`, the developer must: (1) land the proto change, (2) tag a +release, (3) update the Go module reference in `osac-operator`, (4) verify the +operator still compiles, (5) open a separate PR. This multi-step dance delays +delivery and introduces windows where the two components disagree on types. +Developers routinely discover that the operator references a stale +`fulfillment-service` tag and spend time debugging why things "aren't working." + +**Cross-repo CI gaps.** Today, merging a PR in `fulfillment-service` does not +run the operator's tests or the installer's integration suite. Breaking changes +land in one repo and are only caught days later when someone attempts an +end-to-end deployment. There is no mechanism in GitHub to express "this PR in +repo A depends on that PR in repo B" and gate merging on both passing together. + +**AI tooling friction.** The team uses AI-assisted development extensively. +AI agents work best when the full context (types, controllers, installer +manifests, Ansible roles) is visible in a single repository. The meta-repo +workaround (`osac-workspace`) helps, but agents still struggle with cross-repo +references, tagging ceremonies, and multi-PR coordination. + +**PR coordination overhead.** Features that span components require two or three +synchronized PRs across repos, each needing its own review cycle, CI run, and +merge ordering. Developers must manually communicate dependency order ("merge +fulfillment-service#123 first, then osac-operator#45"). Misordering causes +broken `main` branches. + +### User Stories + +- As an OSAC developer, I want to change a protobuf message and its + corresponding controller logic in a single PR so that I do not have to + coordinate releases across repositories. +- As an OSAC developer, I want every PR to run the full installation suite so + that I catch integration regressions before merge, not days later. +- As a QA engineer, I want the installer to always reflect the latest code so + that I do not waste time reproducing already-fixed bugs on stale deployments. +- As a new contributor, I want to clone one repository and have the complete + OSAC codebase available so that I can understand how the components interact + without navigating multiple repos and module dependencies. +- As a release engineer, I want a single repository to tag and release so that + I do not have to orchestrate multi-repo release trains with precise ordering. + +### Goals + +- Eliminate protobuf/CRD version skew between `fulfillment-service` and + `osac-operator` by making them part of the same Go module (or sibling modules + in the same repo). +- Ensure every PR runs integration tests that cover the full stack (API, operator, + installer). +- Remove the manual installer-update ceremony: changes to any component's + manifests are immediately reflected in the installer overlays within the same + commit. +- Preserve clear separation of concerns through directory structure, CODEOWNERS + files, and per-component CI jobs. +- Maintain the ability to build and release independent container images for + each component. + +### Non-Goals + +- Merging `osac-test-infra` into the mono-repo. Test infrastructure has a + different lifecycle and set of maintainers. +- Merging `enhancement-proposals` or `docs` repos. These are documentation-only + and benefit from independent review processes. +- Changing the project's CI system. + +## Proposal + +Consolidate `fulfillment-service`, `osac-operator`, `osac-aap`, and +`osac-installer` into a single repository with the following top-level layout: + +``` +osac/ +├── fulfillment-service/ # gRPC server, REST gateway, proto definitions, DB +├── osac-operator/ # Kubernetes operator, CRDs, controllers +├── osac-aap/ # Ansible roles and playbooks +├── osac-installer/ # Kustomize base/overlays, prerequisites, scripts +├── proto/ # Shared proto definitions (moved from fulfillment-service/proto) +├── go.work # Go workspace file linking Go modules +├── OWNERS # Top-level ownership +├── CODEOWNERS # Per-directory GitHub review assignments +├── Makefile # Top-level build/test/lint targets +└── .github/ + └── workflows/ # CI: per-component + integration +``` + +### Workflow Description + +**Developer making a cross-component change:** + +1. The developer creates a single feature branch in the mono-repo. +2. They modify protobuf definitions under `proto/`, update the + `fulfillment-service` handler, adjust the `osac-operator` controller, and + fix the installer overlay if needed. +3. They open a single PR. CI runs unit tests for each affected component and + the full integration suite. +4. Reviewers from the relevant CODEOWNERS groups are auto-assigned. +5. The PR merges atomically. There is no version skew window. + +**Developer making a single-component change:** + +1. The developer modifies files only under `fulfillment-service/`. +2. CI detects which components are affected (via path filters) and runs only + the relevant unit tests plus the integration suite. +3. Only the `fulfillment-service` CODEOWNERS group is assigned for review. +4. The experience is equivalent to working in a single-component repo. + +**Release process:** + +1. A release tag (e.g., `v1.2.0`) is created on the mono-repo. +2. CI builds container images for each component: + `osac-fulfillment-service:v1.2.0`, `osac-operator:v1.2.0`, + `osac-aap-ee:v1.2.0`. +3. The installer overlays reference images from the same tag, guaranteeing + consistency. + +### API Extensions + +No API changes. This is a repository and build infrastructure change only. + +### Implementation Details/Notes/Constraints + +**Phase 1: Repository setup and history migration** + +- Create the new `osac` repository under `osac-project`. +- Import each component's full git history using `git subtree add` or a + history-rewriting tool so that `git log --follow` and `git blame` continue to + work for migrated files. +- Set up `go.work` to link the `fulfillment-service` and `osac-operator` Go + modules, allowing them to reference each other's packages at source (no + published tags needed). + +**Phase 2: CI setup** + +- Create GitHub Actions workflows with path-based triggers: + - `fulfillment-service/**` triggers fulfillment unit tests. + - `osac-operator/**` triggers operator unit tests. + - `osac-aap/**` triggers Ansible lint and molecule tests. + - `osac-installer/**` triggers installation dry-run. + - Any change triggers the integration test suite. +- Retain per-component Containerfile builds; add a top-level Makefile for + convenience (`make build-all`, `make test-all`, `make images`). + +**Phase 3: Proto consolidation** + +- Move `fulfillment-service/proto/` to top-level `proto/` so both + `fulfillment-service` and `osac-operator` can import generated types + directly without go module version pinning. +- Update `buf.yaml` and `buf.gen.yaml` for the new layout. +- Remove the tagged-release dependency between the two Go modules. + +**Phase 4: Installer alignment** + +- Update `osac-installer` overlays to use relative references to component + manifests within the same repo (e.g., `../../fulfillment-service/manifests`). +- CI for any component change automatically validates that the installer + overlay still applies cleanly. + +**Phase 5: Archive old repositories** + +- Archive `fulfillment-service`, `osac-operator`, `osac-aap`, and + `osac-installer` on GitHub (read-only). Add a prominent notice pointing to + the new mono-repo. +- Update all documentation, wiki links, and bookmarks. + +**Conflux / Build System Compatibility** + +Multiple container images from a single repository is a well-supported pattern. +Each component retains its own `Containerfile` and build context. GitHub Actions +matrix builds can produce all images in parallel. + +### Risks and Mitigations + +| Risk | Mitigation | +|------|------------| +| Larger PR review surface | CODEOWNERS enforces per-directory review; path-filtered CI prevents unnecessary test runs | +| Git history noise from unrelated components | `git log -- ` scopes history; CODEOWNERS ensures only relevant reviewers are notified | +| Longer CI times if all tests run on every PR | Path-based CI triggers run only affected component tests; full integration runs as a separate required check | +| Migration disrupts in-flight work | Coordinate migration during a low-activity window; freeze PRs on old repos for 1-2 days | +| Go module complexity with multiple modules in one repo | `go.work` handles this natively; Kubernetes and other large projects use this pattern | +| Permissions and access control changes | GitHub CODEOWNERS and branch protection rules provide equivalent granularity to separate repos | + +### Drawbacks + +**Larger repository size.** The combined repo will be larger to clone, though +the individual components are relatively small (Go code, YAML manifests, Ansible +roles). Shallow clones (`--depth 1`) mitigate this for CI. + +**Blast radius of broken `main`.** A bad merge in any component breaks `main` +for everyone. However, this is already the case in practice: a broken +`fulfillment-service` deploy blocks all other work today, just with a delayed +feedback loop. The mono-repo makes the breakage visible immediately rather than +hiding it behind stale installer references. + +**Cognitive load for contributors focused on one component.** A developer +working only on Ansible roles sees Go code they do not care about. Directory +structure and CODEOWNERS mitigate this: `cd osac-aap` is functionally +equivalent to working in a separate repo. IDE project filtering and sparse +checkouts are available for those who want a narrower view. + +**Loss of per-component GitHub issue/PR numbers.** Existing issue references +(e.g., `fulfillment-service#42`) will point to archived repos. New issues use +the mono-repo's tracker. Jira remains the authoritative issue tracker, so GitHub +issue numbers are less critical. + +## Alternatives (Not Implemented) + +### Alternative 1: Cross-Repo GitHub Actions Triggers + +Set up GitHub Actions in `fulfillment-service` and `osac-operator` to +automatically trigger a PR in `osac-installer` whenever a change merges to +`main`. The triggered workflow would bump image SHAs, run the installer tests, +and auto-merge if green. + +**Why this is not optimal:** + +- The auto-generated installer PR can fail for reasons unrelated to the + triggering change (e.g., accumulated drift from other repos). Debugging + requires context across repositories. +- This doubles the number of PRs: every component merge creates a follow-up + installer PR, flooding the review queue. The team already struggles with PR + review throughput. +- It doubles CI compute: the component's CI runs first, then the installer's CI + runs separately. In a mono-repo, a single CI run covers both. +- It does not solve the protobuf version-skew problem: the operator still needs + a tagged release from `fulfillment-service` to update its Go module + dependency. The cross-repo trigger only helps the installer. +- Race conditions arise when two component PRs merge in quick succession: the + second installer-update PR may conflict with or overwrite the first. +- There is no atomic merge: the component change is on `main` before the + installer is verified, creating a window where `main` across repos is + inconsistent. + +### Alternative 2: Git Submodules in the Installer Repo + +Make `osac-installer` a "super-repo" that includes the other three as git +submodules. CI in the installer repo would pull the latest submodule references +and run the full test suite. + +**Why this is not optimal:** + +- Git submodules are notoriously difficult to work with. Developers must + remember to run `git submodule update --init --recursive`, submodule pointers + drift, and PRs that update submodules are confusing to review. +- It does not solve the protobuf version-skew issue. The operator still depends + on a tagged release of `fulfillment-service`'s Go module. +- Cross-component PRs still require coordinating merges: change the submodule + repo first, then update the submodule pointer in the installer, which is + two-step just like today. +- The `osac-workspace` meta-repo is already a lighter version of this approach + (using `bootstrap.sh` instead of submodules), and the team still experiences + all the drift problems. + +### Alternative 3: Shared Go Module Published via Tags + +Publish protobuf-generated types and shared packages as a standalone Go module +(e.g., `github.com/osac-project/osac-common`) with semantic versioning. Both +`fulfillment-service` and `osac-operator` depend on this shared module. + +**Why this is not optimal:** + +- Adds a third repository to coordinate: proto changes now require a three-step + dance (land in common, tag, update both consumers). +- The release tagging ceremony remains, just shifted to a different repo. +- Does not address installer drift at all. +- Does not enable atomic cross-component changes. +- The team previously had `fulfillment-api` and `fulfillment-common` as + separate repos and already consolidated them into `fulfillment-service` + because the overhead was not worth it. This alternative repeats the same + pattern. + +### Alternative 4: Periodic Nightly Sync Job + +Run a nightly CI job that builds all components from their respective `main` +branches, deploys via the installer, and runs the integration test suite. If +the job fails, create a Jira ticket for investigation. + +**Why this is not optimal:** + +- Feedback is delayed by up to 24 hours. Developers have moved on to other work + by the time the nightly job reports failures. +- It does not prevent broken code from merging; it only detects it after the + fact. +- Debugging a nightly failure requires correlating changes across multiple repos + to identify which commit broke the integration, a time-consuming process. +- Does not solve the protobuf version-skew or the multi-PR coordination + problems. +- The team's experience shows that periodic sync attempts (bi-weekly manual + bumps) already fail to keep pace with the rate of change. + +### Alternative 5: Status Quo with Process Improvements + +Keep the multi-repo layout but add stricter process requirements: mandatory +installer-update PRs for every component change, required cross-repo CI checks, +and documented dependency ordering in PR templates. + +**Why this is not optimal:** + +- Process-based solutions degrade over time. Developers forget, skip steps under + deadline pressure, or leave the team and take the knowledge with them. +- Cross-repo CI checks in GitHub require complex webhook configurations and + still cannot express "these two PRs must merge together." +- The overhead of maintaining cross-repo automation (triggers, webhooks, status + checks, bot accounts) approaches the overhead of the migration itself, but + without the long-term simplification benefits. +- The team has informally attempted process-based solutions (asking developers + to "follow through with the installer") and the installer still drifts. + +## Open Questions + +1. Should the Go modules be fully merged into a single `go.mod` or kept as + separate modules linked by `go.work`? A single module is simpler but creates + a larger dependency tree for consumers who only need one component. Separate + modules with `go.work` provide isolation while eliminating the tag-based + release ceremony. + +2. What is the right granularity for CODEOWNERS? Per top-level directory + (e.g., `fulfillment-service/`) or per sub-directory (e.g., + `fulfillment-service/internal/networking/`)? + +3. Should the migration preserve full git history via `git subtree` (slower + migration, full blame/log) or start fresh with a squashed import (faster + migration, cleaner history, but loses per-file blame)? A middle ground is + importing history and tagging the merge point for easy reference. + +4. How does this interact with the planned Helm migration for the installer? + If the installer is moving from Kustomize to Helm, the migration should be + sequenced to avoid doing the repo consolidation on a moving target. + +## Test Plan + +- **Unit tests**: Each component retains its existing test suite, run via + path-filtered CI triggers. +- **Integration tests**: The existing `osac-test-infra` E2E suite runs on every + PR, using images built from the PR's branch. This is the key improvement over + the current setup. +- **Migration validation**: After history import, verify that `git log --follow` + works for migrated files, that all CI jobs pass, and that a full deployment + from the mono-repo produces the same result as the current multi-repo + deployment. + +## Graduation Criteria + +This is a one-time infrastructure change, not a feature with maturity stages. +Success criteria: + +- All four component repositories are archived on GitHub. +- All active development happens in the mono-repo. +- CI runs the full integration suite on every PR. +- No manual installer-update PRs are needed. +- Time from "proto change" to "operator using new types" drops from days to + minutes (same PR). + +## Upgrade / Downgrade Strategy + +Not applicable. This is a repository structure change, not a runtime feature. +Existing deployments are unaffected. The container images produced by the +mono-repo are identical to those produced by the individual repos. + +## Version Skew Strategy + +The entire point of this proposal is to eliminate version skew between +components. With a mono-repo, all components are built from the same commit, +ensuring consistency by construction. + +## Support Procedures + +Not applicable. This change affects the development workflow, not the runtime +behavior of any OSAC component. No new failure modes, metrics, or alerts are +introduced. + +## Infrastructure Needed + +- A new GitHub repository: `osac-project/osac` (or reuse an existing name). +- GitHub Actions runners with capacity for the combined CI workload (expected + to be comparable to the sum of current per-repo CI, with optimization via + path-filtered triggers). +- Updated Conflux/OSBS configurations if applicable for multi-image builds. From 7460a2778bc4634bf30a894dd12afcca312e1ccb Mon Sep 17 00:00:00 2001 From: Ilya Skornyakov Date: Wed, 24 Jun 2026 18:14:21 +0300 Subject: [PATCH 045/450] =?UTF-8?q?OSAC-1269:=20add=20PRD=20=E2=80=94=20ma?= =?UTF-8?q?naged=20version=20catalog=20for=20cluster=20provisioning?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Assisted-by: Claude Code Signed-off-by: Ilya Skornyakov --- enhancements/cluster-version-api/prd.md | 173 ++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 enhancements/cluster-version-api/prd.md diff --git a/enhancements/cluster-version-api/prd.md b/enhancements/cluster-version-api/prd.md new file mode 100644 index 000000000..c077fce23 --- /dev/null +++ b/enhancements/cluster-version-api/prd.md @@ -0,0 +1,173 @@ +# ClusterVersion — Managed Version Catalog for Cluster Provisioning + +| Field | Value | +|-------------|---------| +| Author(s) | Ilya Skornyakov | +| Jira | [OSAC-1269](https://issues.redhat.com/browse/OSAC-1269) | +| Date | 2026-06-24 | + +## 1. Problem Statement + +The Cluster API requires users to provide a full OCI release image URL (e.g., `quay.io/openshift-release-dev/ocp-release:4.17.0-multi`) when creating a cluster. This leaks infrastructure implementation details into the user-facing API: users must know the exact registry path and tag format, typos are caught only at provisioning time rather than at API validation, and there is no way to discover which versions are available or to specify an update channel. + +The cost is visible across the stack. Catalog items resort to regex `validation_schema` patterns to constrain the registry prefix — a workaround that cannot verify the version actually exists or is supported. AAP template roles independently construct release image URLs from version numbers (e.g., `{{ ocp_release_version }}-multi`), duplicating URL format knowledge across templates with no central validation. Each layer works around the same missing abstraction: a managed catalog of validated cluster versions. + +A ClusterVersion resource eliminates these workarounds. Users select from a curated, queryable list of versions — no URLs, no guessing. Catalog items reference versions by name instead of embedding URL patterns. AAP templates receive pre-validated release images from the API, removing template-level URL construction. And the version catalog becomes the foundation for future upgrade operations (OSAC-1415) and ACM version synchronization, which require a queryable set of available versions and their upgrade paths. + +## 2. Goals and Non-Goals + +### 2.1 Goals + +- Users specify an OpenShift version by number (e.g., "4.17.0") instead of a raw OCI URL when creating a cluster, with the server resolving the version to the correct release image internally. +- The server validates version and channel at API time, rejecting invalid or disabled versions with descriptive error messages before any provisioning begins. +- Cloud Provider Admins manage available versions through a ClusterVersion resource. +- Tenant Users can discover available versions through the public API, enabling self-service version selection in the CLI and wizard. + +### 2.2 Non-Goals + +- Cluster upgrade operations and channel propagation to HostedCluster — triggering, tracking, or rolling back upgrades, and propagating `channel` to `HostedCluster.spec.channel`. Both are owned by OSAC-1415. The `channel` field is validated in v0.2 but not propagated. [Clarify: R1.Q3, R2.Q2] +- ACM ClusterImageSet auto-sync — automatic discovery and population of ClusterVersion objects from ACM. Versions are admin-managed in v0.2. [Clarify: R1.Q7] +- VM image management — ComputeInstance image catalog (OSAC-979) uses a separate `ComputeImage` resource. Both features share the same architectural pattern (raw URL to managed catalog with server-side resolution) but are distinct resources. [Clarify: R1.Q4] +- In-place upgrade migration — OSAC does not support in-place upgrades. This feature assumes fresh deployment. [Clarify: R2.Q4] + +## 3. Requirements + +### 3.1 Functional Requirements + +#### ClusterVersion Resource + +- **FR-1:** OSAC must provide a ClusterVersion resource with the following domain fields: `release_image` (resolved OCI URL), `enabled` (admin availability toggle), `default` (used when version is omitted), and `available_channels` (list of update channel strings, e.g., `["stable-4.17", "fast-4.17"]`). The resource includes standard OSAC metadata (id, name, labels, annotations, timestamps). The resource `name` is the version number itself (e.g., "4.17.0") — there is no separate version string field. [Clarify: R1.Q1, R1.Q8] + +- **FR-2:** The resource must be named `ClusterVersion` and the ClusterSpec field must be named `cluster_version`. The naming is cluster-scoped, distinguishing it from other versioned resources in the platform (e.g., ComputeImage for VMaaS). [User] + +- **FR-3:** Cloud Provider Admins can create, update, and delete ClusterVersion resources. Tenant Users can list and view available versions. Tenant-visible listings are filtered to enabled versions only. Getting a specific ClusterVersion by identifier returns the resource regardless of its `enabled` status. [Clarify: R1.Q6, R3.Q1] + +- **FR-4:** At most one ClusterVersion resource may be marked as `default`. The global default ClusterVersion is the lowest-priority fallback — it is used only when the user omits `cluster_version` and neither template defaults nor catalog item field_definitions supply one. [Clarify: R1.Q1] + +- **FR-5:** Cloud Provider Admins must be able to create ClusterVersion objects via the CLI (e.g., `osac create -f cluster-versions.yaml`). [Clarify: R1.Q7] + +#### Cluster API Changes + +- **FR-6:** Users specify a `cluster_version` (e.g., "4.17.0") that references a ClusterVersion resource by name when creating a cluster. This field replaces `release_image` on the user-facing API. [Clarify: R1.Q2, R1.Q5] + +- **FR-7:** Users can specify a `channel` (e.g., "stable-4.17") when creating a cluster. When specified, the server validates the channel against the referenced ClusterVersion's `available_channels` list. The channel is not propagated to the HostedCluster in v0.2. [Clarify: R1.Q3] + +- **FR-8:** The `release_image` field must be removed from the user-facing Cluster API and ClusterTemplate defaults. Users see only `cluster_version` and `channel`; `release_image` is not visible to API consumers. [Clarify: R1.Q5, R2.Q1, R2.Q4, R3.Q3, R3.Q4] + +#### Version Resolution + +- **FR-9:** During cluster creation, the server must resolve the `cluster_version` to a `release_image` using the referenced ClusterVersion resource. Version resolution is fully contained in the fulfillment-service — the ClusterOrder CRD, osac-operator, and AAP are unchanged and continue to receive `release_image` with no downstream changes. [Clarify: R1.Q2, R1.Q5, R2.Q4, R3.Q3] + +- **FR-10:** ClusterTemplate defaults must support a `cluster_version` field replacing `release_image`. Template authors set `cluster_version: "4.17.0"` in defaults. Any ClusterVersion can be used with any template; templates provide version defaults but do not constrain version selection. The server applies template defaults before version resolution. [Clarify: R2.Q1] + +#### Validation + +- **FR-11:** The server must validate `cluster_version` and `channel` at API time, rejecting invalid requests with descriptive error messages that include the invalid value and valid alternatives. Validation applies to the effective spec after template defaults and the global default ClusterVersion have been applied. Validation covers: version not found, version not enabled, version deleted, invalid channel for the given version, channel present without a resolved cluster_version (after all defaults), and empty `available_channels` (specifying any channel is invalid). The `cluster_version` and `channel` fields are immutable after cluster creation — attempts to change either on update must be rejected. [Clarify: R2.Q3, R3.Q6] + +#### CLI and Catalog Items + +- **FR-12:** The CLI must replace the `--release-image` flag with `--cluster-version` and `--channel` flags for cluster creation. [Clarify: R1.Q5] + +- **FR-13:** Catalog item field_definitions must reference the `cluster_version` field path instead of `release_image`. [Clarify: R1.Q5] + +#### Lifecycle Protection + +- **FR-14:** Deleting a ClusterVersion must be rejected when it is referenced by active clusters or ClusterTemplate defaults. The error message must identify the referencing resource (e.g., "cannot delete cluster version '4.17.0': it is in use by at least cluster 'cluster-abc'"). [Clarify: R3.Q2] + +- **FR-15:** Creating or updating a cluster or ClusterTemplate with a `cluster_version` that references a non-existent or deleted ClusterVersion must be rejected. Concurrent create and delete operations must not produce inconsistent state. [Clarify: R3.Q2] + +- **FR-16:** Disabling a ClusterVersion is always allowed regardless of references. Disabled versions block new cluster creation (FR-11 validation) but do not affect existing clusters or templates that reference them. [Clarify: R3.Q5] + +### 3.2 Non-Functional Requirements + +- **NFR-1:** The ClusterVersion resource must be consistent with existing admin-managed reference data resources (HostType, NetworkClass) in API conventions, access patterns, and lifecycle behavior. [Clarify: R1.Q6, R3.Q1] + +- **NFR-2:** Field names and reference patterns must align with the OCM Clusters Management API where applicable: `cluster_version` as a reference by readable name (aligned with OCM's `link Version Version` pattern), `channel` as a plain string (aligned with OCM's `Channel String`). [Clarify: R1.Q8] + +- **NFR-3:** The ClusterVersion resource must be extensible. OSAC-1415 is expected to add fields such as `available_upgrades` and activate channel propagation to HostedCluster. These additions must be non-breaking proto field additions. [Clarify: R2.Q2] + +## 4. Acceptance Criteria + +- [ ] A user can create a cluster by specifying `--cluster-version 4.17.0` instead of a release image URL, and the server resolves the version to the correct release image internally. +- [ ] Specifying a non-existent or deleted `cluster_version` is rejected with a descriptive error — both when creating a cluster and when setting ClusterTemplate defaults. +- [ ] Specifying a disabled version is rejected with a message indicating the version is not enabled. +- [ ] Specifying a channel not in the ClusterVersion's `available_channels` is rejected with an error listing valid channels. A request whose effective spec contains `channel` but no `cluster_version` (after defaults) is rejected. +- [ ] Attempting to change `cluster_version` or `channel` on an existing cluster is rejected (both fields are immutable). +- [ ] Listing ClusterVersions as a Tenant User returns only enabled versions with their `available_channels`. +- [ ] A Cloud Provider Admin can create, update, and delete ClusterVersion resources. +- [ ] Deleting a ClusterVersion that is referenced by an active cluster or a ClusterTemplate's defaults is rejected with a message identifying the referencing resource. +- [ ] Disabling a ClusterVersion succeeds even when referenced by active clusters or templates; new cluster creation with that version is blocked. +- [ ] When a user omits `cluster_version` and neither template defaults nor catalog item field_definitions supply one, the server uses the ClusterVersion marked as `default`. +- [ ] At most one ClusterVersion is marked as `default` at any time — setting `default` on a second ClusterVersion either clears the previous default or is rejected. +- [ ] A cluster creation and a ClusterVersion deletion racing on the same version do not produce a cluster referencing a deleted ClusterVersion. +- [ ] Template defaults support `cluster_version` — when a template specifies `cluster_version: "4.17.0"`, the server applies it as the default and resolves it to `release_image`. +- [ ] Catalog item field_definitions can reference the `cluster_version` path. +- [ ] The `release_image` field is not visible to API consumers. Internally, the server stores and forwards `release_image` to downstream systems. +- [ ] The osac-operator and AAP continue to receive `release_image` through the ClusterOrder CRD — no downstream changes are required. + +## 5. Assumptions + +- OSAC does not support in-place upgrades. This feature assumes fresh deployment or environment re-creation. Existing clusters created with `release_image` will not automatically display a `cluster_version` value after redeployment. Administrators must re-create templates and catalog items with `cluster_version` replacing `release_image`. [Clarify: R2.Q4] +- No production catalog items exist that reference `release_image` in field_definitions, so the field path change from `release_image` to `cluster_version` requires no migration. [Clarify: R1.Q5] + +## 6. Dependencies + +- **OSAC-1531 (Default Catalog Items)** — Default ClusterVersion objects should ship alongside default catalog items. ClusterVersion creation is a prerequisite for catalog items that reference version-based clusters. +- **OSAC-1415 (Cluster Upgrade)** — The ClusterVersion resource is designed as an extension point. OSAC-1415 is expected to add fields such as `available_upgrades` and activate `channel` propagation to HostedCluster. This PRD does not constrain OSAC-1415's design. [Clarify: R2.Q2] + +## 7. Risks + +### 7.1 ClusterVersion catalog must be populated before cluster creation + +Without ClusterVersion objects, users cannot create clusters via the public API. If admins deploy OSAC but do not populate versions, the system appears broken. + +- **Owner:** CaaS team +- **Mitigation:** Ship default ClusterVersion objects alongside OSAC-1531 default catalog items. Document version population in the admin guide. + +### 7.2 Shared pattern divergence with VM image management + +OSAC-1269 (ClusterVersion) and OSAC-979 (ComputeImage) follow the same architectural pattern — raw URL to managed catalog with server-side resolution. If designed independently, their APIs may diverge in naming conventions, resolution behavior, or error handling. + +- **Owner:** CaaS / VMaaS teams +- **Mitigation:** Both resources follow the HostType/NetworkClass reference data pattern. Coordinate API conventions during design review. + +## OSAC Dimensions + +### Services + +This feature applies to **CaaS** (Cluster as a Service) only. VMaaS has an analogous pattern (OSAC-979) but uses a separate resource. + +### Personas + +| Persona | Interaction | +|---------|-------------| +| Cloud Provider Admin | Creates and manages ClusterVersion resources. Populates the version catalog. Sets default version. | +| Tenant User | Selects a version from the available list when creating a cluster. Discovers available versions via `ListClusterVersions`. | +| Cloud Infrastructure Admin | Not affected. | +| Tenant Admin | Not affected beyond Tenant User capabilities. | + +### Provisioning + +Version resolution occurs during cluster creation in the fulfillment-service. The provisioning flow downstream (ClusterOrder CRD, osac-operator, AAP) is unchanged — it continues to receive `release_image`. + +### User-Facing API + +| Surface | Impact | +|---------|--------| +| Fulfillment API (gRPC/REST) | New ClusterVersion resource (admin-managed, tenant-readable). Modified ClusterSpec (`cluster_version` and `channel` replace `release_image` for API consumers). Modified ClusterTemplate defaults (`cluster_version` replaces `release_image`). Deletion protection when referenced by clusters or templates. | +| OSAC CRDs | No change. ClusterOrder CRD retains `releaseImage`. | +| Catalog Items | Field_definitions reference `cluster_version` path instead of `release_image`. | + +### Milestone Scoping + +- **Target milestone:** v0.2 +- **Deferred:** Cluster upgrades (OSAC-1415), ACM version auto-sync, channel propagation to HostedCluster, in-place migration of existing data. +- **Upgrades:** OSAC does not support in-place upgrades. No data migration is in scope. + +### Installation + +No changes to Helm charts, kustomize manifests, or `osac-installer/setup.sh` beyond the standard fulfillment-service image version bump. + +### Tenant Onboarding, Inventory, Networking, Storage + +Not applicable. The ClusterVersion resource is global (not tenant-scoped) and does not interact with inventory, networking, or storage subsystems. From 88bafde355b4e7fe66b624c9776a9ad8efe2bafe Mon Sep 17 00:00:00 2001 From: Menny Aboush Date: Wed, 24 Jun 2026 18:50:05 +0300 Subject: [PATCH 046/450] OSAC-1550: rename ssh_key to ssh_public_key in unified-networking EP (#71) --- enhancements/unified-networking/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/unified-networking/README.md b/enhancements/unified-networking/README.md index a89d01516..5ff3a9927 100644 --- a/enhancements/unified-networking/README.md +++ b/enhancements/unified-networking/README.md @@ -663,7 +663,7 @@ message ComputeInstanceSpec { ```protobuf message BaremetalInstanceSpec { string template = 1; - optional string ssh_key = 2; + optional string ssh_public_key = 2; optional string user_data = 3; optional BaremetalInstanceRunStrategy run_strategy = 4; optional google.protobuf.Timestamp restart_requested_at = 5; From 914ca7e85033e96fa2635352f41a21cefb917b10 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 24 Jun 2026 12:59:41 -0400 Subject: [PATCH 047/450] OSAC-1123: address review feedback on CaaS cluster storage PRD Add persona coverage (Tenant Admin/User, Cloud Provider Admin, Cloud Infrastructure Admin), success metric (< 5 minutes), API surface section (ClusterOrder and Tenant CR changes), security acceptance criterion for NFR-1, network reachability assumption, and Helm chart update note. Fix osac-aap PR #338 status to merged. Tighten FR-5 StorageClass confirmation to require full expected set. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude --- enhancements/caas-cluster-storage/prd.md | 28 +++++++++++++++++++----- 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/enhancements/caas-cluster-storage/prd.md b/enhancements/caas-cluster-storage/prd.md index 0dc66ab6c..09e0159b6 100644 --- a/enhancements/caas-cluster-storage/prd.md +++ b/enhancements/caas-cluster-storage/prd.md @@ -8,7 +8,7 @@ ## 1. Problem Statement -CaaS tenant clusters are provisioned without persistent storage. When a ClusterOrder reaches Ready, the cluster has compute but no StorageClasses or CSI driver, so tenant workloads cannot create PVCs. The OSAC Storage Controller already handles backend setup (Stage 1) at tenant onboarding and cluster-side setup (Stage 2) for VMaaS, but it ignores ClusterOrder events. There is no automation connecting CaaS cluster readiness to storage installation. +CaaS tenant clusters are provisioned without persistent storage. When a ClusterOrder reaches Ready, the cluster has compute but no StorageClasses or CSI driver, so tenant workloads cannot create PVCs. The OSAC Storage Controller already handles backend setup (Stage 1) at tenant onboarding and cluster-side setup (Stage 2) for VMaaS, but it ignores ClusterOrder events. There is no automation connecting CaaS cluster readiness to storage installation. This affects Tenant Admins and Tenant Users (who provision clusters and need to know when storage is ready) and Cloud Provider Admins (who monitor tenant-level storage status across clusters). ## 2. Goals and Non-Goals @@ -20,12 +20,19 @@ CaaS tenant clusters are provisioned without persistent storage. When a ClusterO - Handle multiple CaaS clusters per tenant independently, each with its own storage readiness. - Integrate with the Storage Tier API (OSAC-1110) and StorageBackend API (OSAC-1111) when available, falling back to `STORAGE_TIERS` env var and single implicit backend otherwise. -### 2.2 Non-Goals +### 2.2 Success Metrics + +| Metric | Target | Baseline | +|--------|--------|----------| +| Time from ClusterOrder Ready to ClusterStorageReady=True | < 5 minutes | N/A (new feature) | + +### 2.3 Non-Goals - VAST provider CaaS changes (tenant-UID paths, RBAC credentials, `hcp_data_plane` target). Covered by OSAC-1122. - UI for storage. - Stage 1 (backend setup). Assumed complete before any CaaS cluster reaches Ready. - VMaaS cluster-side storage. Existing flow is unchanged. +- Cloud Infrastructure Admin responsibilities (storage backend configuration, tier definitions). Covered by OSAC-1122 and OSAC-1110. ## 3. Requirements @@ -40,8 +47,8 @@ CaaS tenant clusters are provisioned without persistent storage. When a ClusterO #### Storage Readiness -- **FR-5:** The storage controller sets a `ClusterStorageReady` condition on the ClusterOrder CR: `True` when the AAP job succeeds and StorageClasses are confirmed, `False` with reason on failure. -- **FR-6:** The storage controller records per-cluster detail in the Tenant CR's `status.clusterStorage` array (ClusterOrder name, readiness, reason). +- **FR-5:** The storage controller sets a `ClusterStorageReady` condition on the ClusterOrder CR: `True` when the AAP job succeeds and all expected StorageClasses (derived from resolved tiers) are present on the CaaS cluster, `False` with reason on failure. This is the primary readiness signal for Tenant Admins and Tenant Users. +- **FR-6:** The storage controller records per-cluster detail in the Tenant CR's `status.clusterStorage` array (ClusterOrder name, readiness, reason). This provides Cloud Provider Admins a tenant-level view of storage across all CaaS clusters. - **FR-7:** `kubectl get clusterorder -o wide` shows a `ClusterStorageReady` column. #### Teardown @@ -49,6 +56,11 @@ CaaS tenant clusters are provisioned without persistent storage. When a ClusterO - **FR-8:** The storage controller places a finalizer on each ClusterOrder where storage was set up. On deletion, it triggers `osac-delete-tenant-cluster-storage` to remove StorageClasses, VolumeSnapshotClasses, and CSI Secret from the CaaS cluster. The finalizer is removed only after cleanup completes. - **FR-9:** After cleanup, the storage controller removes the ClusterOrder's entry from `status.clusterStorage` on the Tenant CR. +#### API Surface + +- **ClusterOrder CR:** New `ClusterStorageReady` condition, `osac.openshift.io/storage` finalizer, `status.clusterStorageJobs` for AAP job tracking. +- **Tenant CR:** New entries in `status.clusterStorage[]` array (field introduced by OSAC-23, extended here for CaaS clusters). + ### 3.2 Non-Functional Requirements - **NFR-1:** The storage controller must not log or emit events containing credentials, Secret contents, or sensitive AAP parameters. @@ -78,6 +90,9 @@ CaaS tenant clusters are provisioned without persistent storage. When a ClusterO - [ ] Second ClusterOrder for the same tenant gets independent storage setup - [ ] Deleting one ClusterOrder does not affect other clusters +**Security** +- [ ] Controller logs and Kubernetes events from CaaS storage operations contain no credential values or Secret contents + **Testing** - [ ] Unit tests with mock AAP providers cover trigger, readiness, teardown, and multi-cluster - [ ] (Stretch) E2E against live VAST + CaaS cluster @@ -88,11 +103,12 @@ CaaS tenant clusters are provisioned without persistent storage. When a ClusterO - AAP playbooks accept `hcp_data_plane` and use the passed kubeconfig to target the CaaS cluster. Delivered by OSAC-1122. - VAST is the only storage provider for v0.1. A global VIP Pool is shared by all tenants. - The kubeconfig for a CaaS cluster is obtainable from the HostedControlPlane status via the ClusterOrder's cluster reference. Exact mechanism pending CaaS working group confirmation. +- CaaS cluster nodes have network reachability to the VAST VIP pools for CSI volume operations. This is an infrastructure prerequisite managed by the Cloud Infrastructure Admin. ## 6. Dependencies -- **OSAC-23 (Storage Controller):** Implemented and merged (osac-operator PR #299). This PRD extends the controller's ClusterOrder handling. -- **osac-aap PR #338 (playbook split):** Under review. Required for CaaS setup and teardown to operate independently from backend operations. +- **OSAC-23 (Storage Controller):** Implemented and merged (osac-operator PR #299). This PRD extends the controller's ClusterOrder handling. The osac-operator Helm chart needs to be updated to expose the storage controller enable flag (`controllers.storage`) and storage AAP template env vars. +- **osac-aap PR #338 (playbook split):** Merged. Registers the four split AAP templates required for CaaS setup and teardown to operate independently from backend operations. - **OSAC-1122 (VAST for CaaS):** VAST provider must accept `hcp_data_plane` and use the passed kubeconfig. Without this, the trigger fires but the playbook cannot act. - **OSAC-1110 (Tier API):** Not blocking. Controller integrates when available, falls back to env var. - **OSAC-1111 (StorageBackend API):** Not blocking. In development (fulfillment-service PR #728). Controller integrates when available, falls back to single implicit backend. From 997b38307811070eb56287edeedac979246be906 Mon Sep 17 00:00:00 2001 From: Roy Golan Date: Wed, 24 Jun 2026 21:58:53 +0300 Subject: [PATCH 048/450] OSAC-1110: add design document for StorageTier API Introduces a StorageTiers gRPC service under osac.private.v1 that enables Cloud Provider Admins to define named storage offerings backed by registered StorageBackends with typed, provider-neutral QoS properties. Assisted-by: Claude Code Signed-off-by: Roy Golan --- enhancements/storage-tier-OSAC-1110/README.md | 522 ++++++++++++++++++ 1 file changed, 522 insertions(+) create mode 100644 enhancements/storage-tier-OSAC-1110/README.md diff --git a/enhancements/storage-tier-OSAC-1110/README.md b/enhancements/storage-tier-OSAC-1110/README.md new file mode 100644 index 000000000..cf2abf0e7 --- /dev/null +++ b/enhancements/storage-tier-OSAC-1110/README.md @@ -0,0 +1,522 @@ +--- +title: storage-tier-api +authors: + - rgolan@redhat.com +creation-date: 2026-06-24 +last-updated: 2026-06-24 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-1110 +prd: + - "prd.md" +see-also: + - "/enhancements/storage-backend-osac-1111" + - "/enhancements/tenant-storage-onboarding-OSAC-23" +replaces: + - N/A +superseded-by: + - N/A +--- + +# StorageTier API + +## Summary + +This design introduces a `StorageTiers` gRPC service under `osac.private.v1` that enables Cloud Provider Admins to define named storage offerings backed by registered StorageBackends with typed, provider-neutral QoS properties. The entity is DB-backed with no CRD or controller, following the NetworkClass pattern. See [PRD](prd.md) for detailed requirements. + +## Motivation + +OSAC currently configures storage tiers through the `STORAGE_TIERS` environment variable and Kubernetes label conventions (`osac.openshift.io/storage-tier`). Neither mechanism is queryable through the OSAC API, captures QoS properties, or enforces referential integrity with registered storage backends. This prevents internal services (the OSAC Storage Controller, osac-operator) from programmatically discovering which tiers exist, what QoS properties they offer, and which backends serve them. + +StorageBackend (OSAC-1111) registers storage infrastructure — endpoints, credentials, provider type. The missing layer is a tier definition that binds a named offering (e.g., "fast", "standard", "archive") to one or more registered backends with per-backend QoS properties. StorageTier fills this gap by providing an API-managed catalog that downstream workflows — specifically Tenant Storage Onboarding (OSAC-23) — consume to determine which StorageClasses to create and what QoS policies to apply. + +The design follows the established fulfillment-service patterns: private gRPC service with REST transcoding, PostgreSQL storage via the generic DAO, referential integrity via DB triggers, and CEL-based filtering on List. + +### Goals + +- Reuse the existing generic server, DAO, and migration patterns from NetworkClass to minimize implementation risk. +- Store QoS properties as typed proto fields (not JSON) for schema evolution and compile-time safety. +- Enforce referential integrity between StorageTier and StorageBackend at the database level using triggers. +- Include Signal RPC to support future consumption by the OSAC Storage Controller. + +### Non-Goals + +- Public API for tenants — tenants discover their assigned tiers through the Tenant CR status, not by querying StorageTier directly. +- Kubernetes CRD or operator controller — StorageTier is a DB-backed entity only. +- Automatic StorageClass creation or refresh when QoS properties change — that is the OSAC Storage Controller's responsibility (OSAC-23). +- Multi-backend selection logic — v0.1 supports single backend per tier (VAST); selection logic is deferred. + +## Proposal + +StorageTier is a new private API resource in the fulfillment-service. It consists of two new proto files (`storage_tier_type.proto`, `storage_tiers_service.proto`), a private server implementation (`private_storage_tiers_server.go`), a database migration for the `storage_tiers` table with referential integrity triggers, and registration in the gRPC server startup. + +A StorageTier binds a named offering to a StorageBackend with protocol and QoS properties. The tier name is immutable after creation (like `implementation_strategy` on NetworkClass), while QoS properties are mutable to allow in-place updates that propagate to the storage provider's policies. Referential integrity ensures that backends cannot be deleted while tiers reference them, and tiers cannot be deleted while tenants reference them. + +### Workflow Description + +**Actors:** Cloud Provider Admin (manages tiers via private API). + +**Preconditions:** At least one StorageBackend (OSAC-1111) exists and is registered. + +```mermaid +sequenceDiagram + participant Admin as Cloud Provider Admin + participant API as fulfillment-service (private) + participant DB as PostgreSQL + + Note over Admin,DB: Create a storage tier + Admin->>API: POST /api/private/v1/storage_tiers
{name: "fast", backends: [{backend_id: "sb-1", protocol: BLOCK, ...}]} + API->>DB: DAO.Get("sb-1") — validate backend exists + API->>DB: INSERT into storage_tiers + DB-->>DB: Name uniqueness partial index enforced + DB-->>API: Created row + API-->>Admin: StorageTier {id: "st-1", state: ACTIVE, ...} + + Note over Admin,DB: Update QoS properties + Admin->>API: PATCH /api/private/v1/storage_tiers/st-1
{backends[0].max_read_bandwidth_mbs: 2000} + API->>DB: DAO.Get each backend_id — validate backends exist + API->>DB: UPDATE storage_tiers SET data = ... + DB-->>DB: Trigger materializes backend IDs to helper table + API-->>Admin: Updated StorageTier + + Note over Admin,DB: Delete a tier (rejected — tenant references it) + Admin->>API: DELETE /api/private/v1/storage_tiers/st-1 + API->>DB: UPDATE storage_tiers SET deletion_timestamp = now() + DB-->>DB: BEFORE UPDATE trigger: Tenant references exist → Z0003 + DB-->>API: Error Z0003 "in use" + API-->>Admin: 409 Conflict "StorageTier in use by Tenant(s)" +``` + +The diagram shows the three primary workflows: creation with backend validation, QoS update with backend re-validation, and deletion blocked by referential integrity. + +**Error paths:** + +- *Backend not found on Create/Update:* The server validates each `backend_id` via `DAO.Get` before persisting. Returns `NOT_FOUND` with the invalid backend ID. +- *Duplicate name on Create:* The partial unique index raises a constraint violation. The DAO translates it to `ALREADY_EXISTS`. +- *Concurrent update conflict:* When `lock = true` in the Update request, the DAO's optimistic concurrency check rejects stale versions with `ABORTED`. +- *Tier in use on Delete:* The `BEFORE UPDATE` trigger on `storage_tiers` checks for active Tenants referencing the tier. Returns `FAILED_PRECONDITION` with the Z0003 error. + +### API Extensions + +**New gRPC service:** `osac.private.v1.StorageTiers` — full CRUD plus Signal. No public service, no mapper, no webhooks, no finalizers, no CRDs. + +**Impact if service is unavailable:** Cloud Provider Admins cannot manage storage tier definitions. Existing tiers already persisted in the database remain readable by the OSAC Storage Controller through direct DB queries or cached gRPC responses. No impact on running tenant workloads. + +### Implementation Details/Notes/Constraints + +#### Proto Schema + +**`storage_tier_type.proto`** defines the StorageTier message and its nested types: + +```protobuf +syntax = "proto3"; +package osac.private.v1; + +import "osac/private/v1/metadata_type.proto"; + +enum StorageProtocol { + STORAGE_PROTOCOL_UNSPECIFIED = 0; + STORAGE_PROTOCOL_NFS = 1; + STORAGE_PROTOCOL_BLOCK = 2; +} + +enum StorageTierState { + STORAGE_TIER_STATE_UNSPECIFIED = 0; + STORAGE_TIER_STATE_ACTIVE = 1; +} + +message BackendAssociation { + // ID of the registered StorageBackend. + string backend_id = 1; + + // Storage protocol for this backend association. + StorageProtocol protocol = 2; + + // Maximum read bandwidth in megabytes per second. + int32 max_read_bandwidth_mbs = 3; + + // Maximum write bandwidth in megabytes per second. + int32 max_write_bandwidth_mbs = 4; + + // Storage quota in gibibytes. + int64 quota_gib = 5; + + // Whether data-at-rest encryption is enabled. + bool encryption_enabled = 6; +} + +message StorageTier { + // Unique identifier. + string id = 1; + + // Standard OSAC metadata (name, labels, annotations, version). + Metadata metadata = 2; + + // Human-readable description of the tier offering. + string description = 3; + + // Backend associations with per-backend QoS properties. + // v0.1: single backend per tier. + repeated BackendAssociation backends = 4; + + // Current state of the tier. + StorageTierState state = 5; +} +``` + +Design notes: +- `metadata.name` carries the tier name (e.g., "fast", "standard"). It is immutable after creation — enforced in the server's Update method by rejecting name changes. +- `quota_gib` uses `int64` for petabyte-scale headroom, consistent with OSAC's `_gib` suffix convention [Codebase: ComputeInstance Disk.size_gib]. +- `backends` is `repeated` to support future multi-backend tiers, but v0.1 validates that exactly one backend is provided. +- No `status` sub-message — StorageTier has no async provisioning lifecycle. The `state` field is sufficient. + +**`storage_tiers_service.proto`** follows the NetworkClasses service pattern [Codebase: network_classes_service.proto]: + +```protobuf +service StorageTiers { + rpc List(StorageTiersListRequest) returns (StorageTiersListResponse) { + option (google.api.http) = {get: "/api/private/v1/storage_tiers"}; + } + rpc Get(StorageTiersGetRequest) returns (StorageTiersGetResponse) { + option (google.api.http) = { + get: "/api/private/v1/storage_tiers/{id}" + response_body: "object" + }; + } + rpc Create(StorageTiersCreateRequest) returns (StorageTiersCreateResponse) { + option (google.api.http) = { + post: "/api/private/v1/storage_tiers" + body: "object" + response_body: "object" + }; + } + rpc Update(StorageTiersUpdateRequest) returns (StorageTiersUpdateResponse) { + option (google.api.http) = { + patch: "/api/private/v1/storage_tiers/{object.id}" + body: "object" + response_body: "object" + }; + } + rpc Delete(StorageTiersDeleteRequest) returns (StorageTiersDeleteResponse) { + option (google.api.http) = {delete: "/api/private/v1/storage_tiers/{id}"}; + } + rpc Signal(StorageTiersSignalRequest) returns (StorageTiersSignalResponse) { + } +} +``` + +Request and response messages follow the same structure as `NetworkClassesListRequest` / `NetworkClassesUpdateRequest` (offset, limit, filter, order for List; FieldMask and lock for Update). + +#### Server Implementation + +`private_storage_tiers_server.go` follows the `PrivateNetworkClassesServer` pattern [Codebase: private_network_classes_server.go]: + +- Builder pattern: `PrivateStorageTiersServerBuilder` with `SetLogger`, `SetNotifier`, `SetAttributionLogic`, `SetTenancyLogic`, `SetMetricsRegisterer`. +- Embeds `GenericServer[*privatev1.StorageTier]` for standard CRUD delegation. +- Custom validation in `Create` and `Update`: + 1. Validate exactly one backend in `backends` (v0.1 constraint). + 2. For each `backend_id`, call `storageBakendsDAO.Get(ctx, backendID)` to verify the backend exists and is active. Return `NOT_FOUND` if any backend is missing. + 3. On `Create`: set `state = STORAGE_TIER_STATE_ACTIVE`. + 4. On `Update`: reject changes to `metadata.name` (immutable field check). + +**generic_server.go change:** Add `case *privatev1.StorageTier: event.SetStorageTier(object)` to the `setPayload()` switch statement. + +**gRPC registration:** In `start_grpc_server_cmd.go`, construct `PrivateStorageTiersServer` via the builder and register with `privatev1.RegisterStorageTiersServer(grpcServer, server)`. No public server registration needed. + +#### Database Migration + +Two migrations are required. StorageBackend (OSAC-1111) takes migration 60; StorageTier takes migrations 61 and 62. + +**Migration 61: `61_create_storage_tiers_tables.up.sql`** + +Creates the `storage_tiers` and `archived_storage_tiers` tables plus the name uniqueness constraint: + +```sql +create table storage_tiers ( + id text not null primary key, + name text not null default '', + creation_timestamp timestamp with time zone not null default now(), + deletion_timestamp timestamp with time zone not null default 'epoch', + finalizers text[] not null default '{}', + creators text[] not null default '{}', + tenants text[] not null default '{}', + labels jsonb not null default '{}'::jsonb, + annotations jsonb not null default '{}'::jsonb, + data jsonb not null +); + +create table archived_storage_tiers ( + id text not null, + name text not null default '', + creation_timestamp timestamp with time zone not null, + deletion_timestamp timestamp with time zone not null, + archival_timestamp timestamp with time zone not null default now(), + creators text[] not null default '{}', + tenants text[] not null default '{}', + labels jsonb not null default '{}'::jsonb, + annotations jsonb not null default '{}'::jsonb, + data jsonb not null +); + +create index storage_tiers_by_name on storage_tiers (name); +create index storage_tiers_by_owner on storage_tiers using gin (creators); +create index storage_tiers_by_tenant on storage_tiers using gin (tenants); +create index storage_tiers_by_label on storage_tiers using gin (labels); + +-- Platform-scoped name uniqueness among active tiers: +create unique index storage_tiers_unique_name + on storage_tiers (name) + where deletion_timestamp = 'epoch' and name != ''; +``` + +**Migration 62: `62_add_storage_tier_ref_triggers.up.sql`** + +Creates the materialized helper table and referential integrity triggers. This migration depends on the `storage_backends` table from OSAC-1111 (migration 60). + +```sql +-- Helper table: extracts backend IDs from the JSONB backends array for trigger-based +-- reverse lookup. Follows the materialized helper table pattern (see AGENTS.md). +create table storage_tier_backends ( + storage_tier_id text not null references storage_tiers(id) on delete cascade, + backend_id text not null, + primary key (storage_tier_id, backend_id) +); + +create index storage_tier_backends_by_backend on storage_tier_backends (backend_id); + +-- Materialize backend IDs from JSONB on every insert/update of storage_tiers: +create function materialize_storage_tier_backends() returns trigger as $$ +declare + bid text; +begin + delete from storage_tier_backends where storage_tier_id = new.id; + + for bid in + select jsonb_array_elements(new.data->'backends')->>'backendId' + loop + insert into storage_tier_backends (storage_tier_id, backend_id) + values (new.id, bid); + end loop; + + return new; +end; +$$ language plpgsql; + +create trigger materialize_storage_tier_backends + after insert or update on storage_tiers + for each row + execute function materialize_storage_tier_backends(); + +-- Backfill existing rows (if any): +update storage_tiers set data = data; + +-- Prevent deleting a StorageBackend that is referenced by an active StorageTier: +create function check_storage_backend_not_in_use_by_tier() returns trigger as $$ +declare + tier_count bigint; +begin + select count(*) into tier_count + from storage_tier_backends stb + join storage_tiers st on st.id = stb.storage_tier_id + where stb.backend_id = old.id + and st.deletion_timestamp = 'epoch'; + + if tier_count > 0 then + raise exception using + errcode = 'Z0003', + message = format( + 'cannot delete StorageBackend ''%s'': %s StorageTier(s) still reference it', + old.id, tier_count + ); + end if; + + return new; +end; +$$ language plpgsql; + +create trigger check_storage_backend_not_in_use_by_tier + before update on storage_backends + for each row + when (old.deletion_timestamp = 'epoch' and new.deletion_timestamp != 'epoch') + execute function check_storage_backend_not_in_use_by_tier(); + +-- Prevent deleting a StorageTier that is referenced by an active Tenant: +create function check_storage_tier_not_in_use() returns trigger as $$ +declare + tenant_count bigint; +begin + select count(*) into tenant_count + from tenants + where deletion_timestamp = 'epoch' + and data->'spec'->'storageTiers' ? old.id; + + if tenant_count > 0 then + raise exception using + errcode = 'Z0003', + message = format( + 'cannot delete StorageTier ''%s'': %s Tenant(s) still reference it', + old.id, tenant_count + ); + end if; + + return new; +end; +$$ language plpgsql; + +create trigger check_storage_tier_not_in_use + before update on storage_tiers + for each row + when (old.deletion_timestamp = 'epoch' and new.deletion_timestamp != 'epoch') + execute function check_storage_tier_not_in_use(); +``` + +Design notes on the triggers: +- The `storage_tier_backends` helper table enables efficient reverse lookup from backend ID to tiers, avoiding a full-table scan of `storage_tiers` JSONB data. +- The StorageBackend deletion trigger joins `storage_tier_backends` with `storage_tiers` to check only active (non-deleted) tiers. +- The Tenant deletion trigger uses the JSONB `?` operator to check if the tier ID exists in the Tenant's `storageTiers` array. The exact JSON path (`data->'spec'->'storageTiers'`) depends on the Tenant proto schema from OSAC-23; the implementation must adjust the path when that schema is finalized. +- The `storage_tier_backends` table must be excluded from schema validation in `database_tool.go` (helper table pattern). + +#### Component Interaction + +```mermaid +graph LR + Admin["Cloud Provider Admin"] + FS["fulfillment-service
(private gRPC)"] + DB["PostgreSQL"] + SC["OSAC Storage Controller
(OSAC-23, future)"] + + Admin -->|CRUD via REST/gRPC| FS + FS -->|GenericDAO| DB + SC -->|List/Get + Signal| FS + + subgraph DB + ST["storage_tiers"] + SB["storage_backends"] + STB["storage_tier_backends
(helper)"] + T["tenants"] + end + + ST -.->|trigger: materialize| STB + SB -.->|trigger: block delete| STB + ST -.->|trigger: block delete| T +``` + +The diagram shows that the fulfillment-service is the sole writer to `storage_tiers`. The OSAC Storage Controller (future) reads tier definitions via the gRPC API. Referential integrity is enforced at the database layer through triggers on `storage_backends` and `storage_tiers`, with `storage_tier_backends` serving as the materialized lookup table. + +### Security Considerations + +StorageTier inherits the fulfillment-service's existing security model without modification: + +- **Authentication:** JWT validation via the gRPC interceptor chain. Only authenticated Cloud Provider Admin tokens can access the private API. +- **Authorization:** OPA policies enforce role-based access. The private API is restricted to admin roles; no new OPA rules are required. +- **Input validation:** Backend IDs are validated via `DAO.Get` (existence check). QoS numeric fields are typed proto fields with natural bounds (int32/int64). Protocol is a proto enum — invalid values are rejected by proto unmarshaling. +- **Data exposure:** No sensitive data in StorageTier. QoS properties and backend references are operational metadata, not credentials. StorageBackend credentials are stored in the StorageBackend entity, not in the tier. + +### Failure Handling and Recovery + +| Failure Mode | Behavior | User Observation | +|---|---|---| +| Backend validation fails on Create | Server returns `NOT_FOUND` with the invalid backend ID. No row is inserted. | Admin corrects the backend ID and retries. | +| Name uniqueness violation on Create | Partial unique index raises constraint violation. DAO translates to `ALREADY_EXISTS`. | Admin chooses a different name or deletes the existing tier first. | +| Optimistic lock conflict on Update | DAO compares metadata version; mismatch returns `ABORTED`. | Admin re-fetches the tier, reapplies changes, and retries. | +| StorageBackend deletion blocked by tier | Trigger raises Z0003. DAO translates to `FAILED_PRECONDITION`. | Admin removes the backend association from all referencing tiers first. | +| StorageTier deletion blocked by tenant | Trigger raises Z0003. DAO translates to `FAILED_PRECONDITION`. | Admin unassigns the tier from all tenants first. | +| PostgreSQL unavailable | Standard DAO error propagation; gRPC returns `UNAVAILABLE`. | Admin retries. No data corruption — transaction never committed. | +| Materialization trigger failure | Transaction that caused the trigger rolls back. No partial state in helper table. | Admin retries the Create/Update. | + +All operations are transactional. No partial state is possible — either the full Create/Update/Delete commits or it rolls back entirely. There is no async reconciliation, so controller restart mid-reconciliation is not applicable. + +### RBAC / Tenancy + +StorageTier is platform-scoped, managed exclusively by Cloud Provider Admins. It follows the same tenancy approach as NetworkClass: + +- The `tenants` column is populated by the existing `tenancyLogic` (which assigns platform-scoped entities to the system tenant or all tenants, depending on configuration). +- Tenant isolation annotations (`osac.openshift.io/tenant`) are set by the standard attribution logic. +- No new OPA policies are required. The private API's existing admin-only access control is sufficient. +- Tenants do not access StorageTier directly — they discover their assigned tiers through the Tenant resource's status (populated by OSAC-23). + +### Observability and Monitoring + +No new observability changes. Existing monitoring mechanisms apply: + +- The gRPC interceptor chain already emits Prometheus metrics for all RPC calls (request count, latency, error rate) — these automatically cover StorageTier RPCs. +- Structured logging via slog captures Create/Update/Delete operations with resource IDs. +- The notification system (event payloads via `setPayload()`) enables downstream consumers to react to StorageTier changes. + +### Risks and Mitigations + +**Trigger ordering with OSAC-1111:** StorageTier's migration 62 creates a trigger on the `storage_backends` table. If OSAC-1111 (migration 60) is not applied first, migration 62 fails. Mitigation: migrations are numbered sequentially and applied in order; OSAC-1111 is a stated dependency and must merge first. + +**Tenant reference path uncertainty:** The `check_storage_tier_not_in_use` trigger references the Tenant's JSONB path for storage tier assignments (`data->'spec'->'storageTiers'`). This path depends on the Tenant proto schema changes in OSAC-23, which is not yet implemented. Mitigation: the trigger path is updated during OSAC-23 implementation to match the actual Tenant proto field. If OSAC-23 lands after StorageTier, the trigger can be added in a separate migration at that time. + +**QoS update propagation limits:** Changes to QoS properties that map to Kubernetes StorageClass parameters (e.g., encryption settings) require StorageClass recreation to take effect for new volumes. Existing volumes are unaffected. Mitigation: the OSAC Storage Controller (OSAC-23) handles StorageClass lifecycle, including parameter drift detection. + +### Drawbacks + +The primary trade-off is adding another DB-backed entity with referential integrity triggers. This increases migration complexity and requires coordination across two features (OSAC-1110 and OSAC-1111). However, the alternative — free-form JSON for QoS or no referential integrity — creates worse problems: schema drift, no compile-time safety, and orphaned references. + +The materialized helper table (`storage_tier_backends`) adds a maintenance surface: the trigger must be updated if the JSONB schema for `backends` changes. This cost is justified by the need for efficient reverse lookups when blocking StorageBackend deletion — a full-table JSONB scan on `storage_tiers` would not scale. + +## Alternatives (Not Implemented) + +**Free-form JSON for QoS properties:** Store QoS as an untyped `google.protobuf.Struct` or `map`. Pros: maximum flexibility, no proto changes when adding properties. Cons: no compile-time type safety, no proto schema documentation, no field-level validation, CEL filtering on nested fields requires custom translator support. Rejected because typed fields provide better developer experience and catch errors at compile time. + +**Server-side-only referential integrity (no DB triggers):** Validate backend references in the Go server code without DB triggers. Pros: simpler migrations, all logic in Go. Cons: does not protect against direct DB writes or race conditions (concurrent StorageBackend delete and StorageTier create). Rejected because the trigger approach matches the established VirtualNetwork pattern and provides stronger guarantees. + +**CRD-backed StorageTier with a controller:** Define StorageTier as a Kubernetes CRD with a reconciler in osac-operator. Pros: native Kubernetes semantics, kubectl access. Cons: unnecessary complexity for a catalog entity with no async provisioning lifecycle; would require syncing state between the CRD and the DB. Rejected because StorageTier has no reconciliation logic — it is a static catalog entry, not a managed resource. + +**Do nothing (keep `STORAGE_TIERS` env var):** Continue using environment variables for tier configuration. Pros: zero implementation effort. Cons: tiers are not queryable via API, no referential integrity with backends, no QoS metadata, no audit trail, blocks OSAC-23 (Tenant Storage Onboarding). Rejected because OSAC-23 requires an API-managed tier catalog. + +## Test Plan + +Test plan will be developed during implementation. Expected coverage: + +**Unit tests** (`ginkgo run -r internal`): +- `PrivateStorageTiersServer` CRUD lifecycle: create with valid backend, get by ID, list with pagination and filtering, update QoS properties, delete. +- Backend validation: reject create/update with non-existent backend ID. +- Name immutability: reject update that changes `metadata.name`. +- Optimistic concurrency: reject update with stale version. +- `setPayload()` switch coverage for the new StorageTier case. + +**Integration tests** (`ginkgo run it`): +- Full CRUD via gRPC and REST endpoints against a kind cluster with PostgreSQL. +- Referential integrity: create StorageTier referencing a StorageBackend, then attempt to delete the backend — verify Z0003 rejection. +- Name uniqueness: create two tiers with the same name — verify constraint violation. Delete first, recreate — verify name reuse. +- Trigger behavior: verify `storage_tier_backends` helper table is populated correctly after create and update. + +## Graduation Criteria + +Graduation criteria will be defined when targeting a release. Expected stages: Dev Preview -> Tech Preview -> GA based on production deployment feedback. + +## Upgrade / Downgrade Strategy + +This is a new API with no upgrade impact. The database migration (61, 62) adds new tables and triggers without modifying existing tables. + +Downgrade requires: +1. Deleting all StorageTier instances via the API. +2. Reverting migration 62 (drop triggers, drop helper table). +3. Reverting migration 61 (drop `storage_tiers` and `archived_storage_tiers` tables). + +No other components depend on StorageTier at initial deployment. When OSAC-23 (Tenant Storage Onboarding) is deployed, downgrade must also consider removing tier references from Tenant resources. + +## Version Skew Strategy + +StorageTier is entirely within the fulfillment-service — no cross-component version skew applies. The gRPC service and database migration are deployed together as part of the fulfillment-service image. The OSAC Storage Controller (OSAC-23, future) consumes StorageTier via the gRPC API; standard proto backward compatibility (additive-only field changes) ensures version skew tolerance. + +## Support Procedures + +**Failure detection:** +- gRPC error rate increase on `osac.private.v1.StorageTiers/*` RPCs (visible in existing Prometheus dashboards). +- PostgreSQL migration failure logs during service startup (migration 61/62 apply errors). +- Trigger errors (Z0003) logged at WARN level with the resource ID and referencing entity count. + +**Disabling the feature:** +- Remove the `PrivateStorageTiersServer` registration from the gRPC server startup. StorageTier RPCs return `UNIMPLEMENTED`. +- Existing data in `storage_tiers` is inert — no controllers read it without the OSAC Storage Controller. +- No impact on existing workloads. Tenant onboarding (OSAC-23) would fail to look up tiers, but that feature depends on StorageTier being available. + +**Recovery:** +- Re-register the server. The database tables and triggers remain intact. No data loss or consistency issues. + +## Infrastructure Needed + +None. From 464649e35b25ff1ba733f402a96f88e6b1e25963 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 24 Jun 2026 17:04:32 -0400 Subject: [PATCH 049/450] OSAC-1123: rewrite PRD as user-focused capabilities Rewrite the PRD to focus on user-observable outcomes instead of implementation details. Capabilities are grouped by persona and workflow (automatic setup, readiness visibility, cleanup, multi-cluster independence). Acceptance criteria are PM-verifiable. Removes API field names, controller references, env vars, and playbook names. Resolves the kubeconfig open question (confirmed by CaaS team via HostedControlPlane kubeconfig Secret). Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude --- enhancements/caas-cluster-storage/prd.md | 125 ++++++++--------------- 1 file changed, 44 insertions(+), 81 deletions(-) diff --git a/enhancements/caas-cluster-storage/prd.md b/enhancements/caas-cluster-storage/prd.md index 09e0159b6..80ae83a82 100644 --- a/enhancements/caas-cluster-storage/prd.md +++ b/enhancements/caas-cluster-storage/prd.md @@ -8,121 +8,84 @@ ## 1. Problem Statement -CaaS tenant clusters are provisioned without persistent storage. When a ClusterOrder reaches Ready, the cluster has compute but no StorageClasses or CSI driver, so tenant workloads cannot create PVCs. The OSAC Storage Controller already handles backend setup (Stage 1) at tenant onboarding and cluster-side setup (Stage 2) for VMaaS, but it ignores ClusterOrder events. There is no automation connecting CaaS cluster readiness to storage installation. This affects Tenant Admins and Tenant Users (who provision clusters and need to know when storage is ready) and Cloud Provider Admins (who monitor tenant-level storage status across clusters). +CaaS tenant clusters are provisioned without persistent storage. When a cluster is ready, it has compute but no way for tenant workloads to create persistent volumes. Storage backend resources (credentials, views, quotas) are already provisioned during tenant onboarding, but nothing connects cluster readiness to storage installation on that cluster. Cloud Provider Admins have no visibility into whether a cluster's storage is ready, and Tenant Admins and Users cannot use persistent storage until someone manually configures it. ## 2. Goals and Non-Goals ### 2.1 Goals -- Automatically install the CSI driver and per-tenant, per-tier StorageClasses on a CaaS cluster when it reaches Ready. -- Track storage readiness on the ClusterOrder CR via a `ClusterStorageReady` condition, visible to Cloud Provider Admins. -- Clean up cluster-side storage resources on CaaS cluster deletion. Leave backend resources (VAST tenant, views, quotas, hub Secret) intact for other clusters. -- Handle multiple CaaS clusters per tenant independently, each with its own storage readiness. -- Integrate with the Storage Tier API (OSAC-1110) and StorageBackend API (OSAC-1111) when available, falling back to `STORAGE_TIERS` env var and single implicit backend otherwise. +- When a CaaS cluster is provisioned and ready, persistent storage (CSI driver and per-tenant, per-tier StorageClasses) is automatically available without manual configuration. +- Cloud Provider Admins can see tenant-level storage status across all CaaS clusters. +- Tenant Admins and Users can see whether their cluster's storage is ready. +- When a CaaS cluster is deleted, storage resources on that cluster are cleaned up. Backend resources are left intact for other clusters. +- Multiple CaaS clusters per tenant are handled independently, each with its own storage readiness. +- When the Storage Tier API (OSAC-1110) and StorageBackend API (OSAC-1111) are available, the system uses them. Otherwise it falls back to the existing configuration. ### 2.2 Success Metrics | Metric | Target | Baseline | |--------|--------|----------| -| Time from ClusterOrder Ready to ClusterStorageReady=True | < 5 minutes | N/A (new feature) | +| Time from cluster ready to storage available | < 5 minutes | N/A (new feature) | ### 2.3 Non-Goals -- VAST provider CaaS changes (tenant-UID paths, RBAC credentials, `hcp_data_plane` target). Covered by OSAC-1122. -- UI for storage. -- Stage 1 (backend setup). Assumed complete before any CaaS cluster reaches Ready. -- VMaaS cluster-side storage. Existing flow is unchanged. +- VAST provider CaaS-specific changes (storage path conventions, credential scoping, CaaS provisioning targets). Covered by OSAC-1122. +- Storage UI. +- Storage backend provisioning (runs during tenant onboarding, assumed complete before any cluster is ready). +- VMaaS cluster-side storage changes. Existing flow is unchanged. - Cloud Infrastructure Admin responsibilities (storage backend configuration, tier definitions). Covered by OSAC-1122 and OSAC-1110. -## 3. Requirements +## 3. Capabilities -### 3.1 Functional Requirements +### Automatic Storage Setup -#### CaaS Stage 2 Trigger +When a CaaS cluster is provisioned and ready, the system automatically installs the CSI driver and creates per-tenant, per-tier StorageClasses on that cluster. The tenant's storage backend must be fully provisioned before this happens. No manual intervention is required from any persona. -- **FR-1:** When a ClusterOrder reaches `phase=Ready` and the owning Tenant has `StorageBackendReady=True`, the storage controller invokes `osac-create-tenant-cluster-storage` with `provisioning_target=hcp_data_plane`. -- **FR-2:** The storage controller passes the CaaS cluster's connection details (kubeconfig or equivalent) to the AAP playbook as extra vars. The playbook does not discover cluster access on its own. -- **FR-3:** The storage controller resolves tiers from the Tier API (OSAC-1110) when available, falling back to `STORAGE_TIERS` env var. -- **FR-4:** The storage controller discovers backends from the StorageBackend API (OSAC-1111) when available, falling back to the single implicit backend. +The system determines which tiers and backends are available for the tenant. If the Tier API or StorageBackend API are deployed, they are used. Otherwise the system falls back to the existing tier and backend configuration. -#### Storage Readiness +### Storage Readiness Visibility -- **FR-5:** The storage controller sets a `ClusterStorageReady` condition on the ClusterOrder CR: `True` when the AAP job succeeds and all expected StorageClasses (derived from resolved tiers) are present on the CaaS cluster, `False` with reason on failure. This is the primary readiness signal for Tenant Admins and Tenant Users. -- **FR-6:** The storage controller records per-cluster detail in the Tenant CR's `status.clusterStorage` array (ClusterOrder name, readiness, reason). This provides Cloud Provider Admins a tenant-level view of storage across all CaaS clusters. -- **FR-7:** `kubectl get clusterorder -o wide` shows a `ClusterStorageReady` column. +As a Cloud Provider Admin, I can see the storage readiness of each tenant's CaaS clusters at the tenant level, so I can monitor onboarding progress and troubleshoot failures across tenants. -#### Teardown +As a Tenant Admin or Tenant User, I can see whether my specific CaaS cluster's storage is ready, so I know when I can start creating persistent volumes. If storage setup failed, I can see the reason. -- **FR-8:** The storage controller places a finalizer on each ClusterOrder where storage was set up. On deletion, it triggers `osac-delete-tenant-cluster-storage` to remove StorageClasses, VolumeSnapshotClasses, and CSI Secret from the CaaS cluster. The finalizer is removed only after cleanup completes. -- **FR-9:** After cleanup, the storage controller removes the ClusterOrder's entry from `status.clusterStorage` on the Tenant CR. +### Cluster Deletion and Cleanup -#### API Surface +When a CaaS cluster is deleted, storage resources installed on that cluster (StorageClasses, CSI credentials, snapshot classes) are cleaned up. Backend resources (storage provider tenant, views, quotas, credentials on the hub) are not affected, because other clusters belonging to the same tenant may still use them. -- **ClusterOrder CR:** New `ClusterStorageReady` condition, `osac.openshift.io/storage` finalizer, `status.clusterStorageJobs` for AAP job tracking. -- **Tenant CR:** New entries in `status.clusterStorage[]` array (field introduced by OSAC-23, extended here for CaaS clusters). +### Multi-Cluster Independence -### 3.2 Non-Functional Requirements +A tenant can have multiple CaaS clusters. Each cluster's storage is set up and torn down independently. Adding or removing a CaaS cluster does not affect storage on other clusters belonging to the same tenant. -- **NFR-1:** The storage controller must not log or emit events containing credentials, Secret contents, or sensitive AAP parameters. +### 3.1 Operational Expectations -## 4. Acceptance Criteria - -**Trigger** -- [ ] ClusterOrder reaching Ready with `StorageBackendReady=True` triggers `osac-create-tenant-cluster-storage` with `provisioning_target=hcp_data_plane` -- [ ] AAP playbook receives cluster connection details as extra vars -- [ ] Tiers resolved from Tier API when available, `STORAGE_TIERS` env var otherwise -- [ ] Backends discovered from StorageBackend API when available, single implicit backend otherwise - -**Readiness** -- [ ] `ClusterStorageReady=True` on ClusterOrder when setup succeeds -- [ ] `ClusterStorageReady=False` with reason on failure -- [ ] `kubectl get clusterorder -o wide` shows storage readiness -- [ ] Tenant CR `status.clusterStorage` has an entry per CaaS cluster +- Storage setup completes within 5 minutes of a CaaS cluster becoming ready. +- Credentials and secrets are never exposed in user-visible error messages, status fields, or events. -**Teardown** -- [ ] Finalizer placed on ClusterOrders where storage was set up -- [ ] ClusterOrder deletion triggers `osac-delete-tenant-cluster-storage` -- [ ] Finalizer removed only after cleanup completes -- [ ] Backend resources unaffected by cluster deletion -- [ ] `status.clusterStorage` entry removed after cleanup - -**Multi-Cluster** -- [ ] Second ClusterOrder for the same tenant gets independent storage setup -- [ ] Deleting one ClusterOrder does not affect other clusters - -**Security** -- [ ] Controller logs and Kubernetes events from CaaS storage operations contain no credential values or Secret contents +## 4. Acceptance Criteria -**Testing** -- [ ] Unit tests with mock AAP providers cover trigger, readiness, teardown, and multi-cluster -- [ ] (Stretch) E2E against live VAST + CaaS cluster +- [ ] A tenant user can create a PVC on a newly provisioned CaaS cluster without manual storage configuration +- [ ] A Cloud Provider Admin can see storage readiness status for each tenant's CaaS clusters +- [ ] A Tenant Admin can see whether their specific CaaS cluster's storage is ready and the reason if it failed +- [ ] When a CaaS cluster is deleted, storage resources on that cluster are cleaned up +- [ ] Deleting one CaaS cluster does not affect storage on other clusters belonging to the same tenant +- [ ] Backend resources are unaffected by CaaS cluster deletion +- [ ] A second CaaS cluster for the same tenant gets independent storage setup +- [ ] Unit tests cover setup, readiness, teardown, and multi-cluster scenarios ## 5. Assumptions -- Stage 1 is always complete before a CaaS cluster reaches Ready. The controller does not handle a Ready ClusterOrder with an unprovisioned backend. -- AAP playbooks accept `hcp_data_plane` and use the passed kubeconfig to target the CaaS cluster. Delivered by OSAC-1122. +- Storage backend provisioning is always complete before a CaaS cluster reaches ready. The system does not handle a ready cluster with an unprovisioned backend. +- The storage provider's automation accepts CaaS clusters as a target and uses the provided cluster credentials. This is delivered by OSAC-1122. - VAST is the only storage provider for v0.1. A global VIP Pool is shared by all tenants. -- The kubeconfig for a CaaS cluster is obtainable from the HostedControlPlane status via the ClusterOrder's cluster reference. Exact mechanism pending CaaS working group confirmation. -- CaaS cluster nodes have network reachability to the VAST VIP pools for CSI volume operations. This is an infrastructure prerequisite managed by the Cloud Infrastructure Admin. +- Cluster credentials are obtainable from the cluster provisioning infrastructure via the HostedControlPlane kubeconfig Secret. Confirmed by the CaaS team. +- CaaS cluster nodes have network reachability to storage VIP pools. This is an infrastructure prerequisite managed by the Cloud Infrastructure Admin. ## 6. Dependencies -- **OSAC-23 (Storage Controller):** Implemented and merged (osac-operator PR #299). This PRD extends the controller's ClusterOrder handling. The osac-operator Helm chart needs to be updated to expose the storage controller enable flag (`controllers.storage`) and storage AAP template env vars. -- **osac-aap PR #338 (playbook split):** Merged. Registers the four split AAP templates required for CaaS setup and teardown to operate independently from backend operations. -- **OSAC-1122 (VAST for CaaS):** VAST provider must accept `hcp_data_plane` and use the passed kubeconfig. Without this, the trigger fires but the playbook cannot act. -- **OSAC-1110 (Tier API):** Not blocking. Controller integrates when available, falls back to env var. -- **OSAC-1111 (StorageBackend API):** Not blocking. In development (fulfillment-service PR #728). Controller integrates when available, falls back to single implicit backend. - -## 7. Risks - -### 7.1 CaaS cluster kubeconfig access is unconfirmed - -- **Owner:** Akshay Nadkarni / CaaS working group -- **Mitigation:** The expected path (`ClusterOrder.status.clusterReference` to `HostedControlPlane.status.kubeConfig`) is traceable through the HyperShift API. If the mechanism differs, only credential retrieval changes; trigger and condition logic are unaffected. - -## 8. Open Questions - -### 8.1 What credential should the storage controller use for CaaS cluster access? +- **Tenant Storage Onboarding (OSAC-23):** The storage automation framework, two-stage model, and cluster watch are implemented and merged (osac-operator PR #299). This PRD extends the framework to act on CaaS cluster events. The Helm chart needs to be updated to expose the storage controller configuration. +- **Playbook split (osac-aap PR #338):** Merged. Provides the independent setup and teardown actions required for CaaS cluster-scoped operations. +- **VAST for CaaS (OSAC-1122):** The storage provider automation must support CaaS clusters as a provisioning target. Without this, the trigger fires but setup cannot complete. +- **Tier API (OSAC-1110):** Not blocking. The system integrates when available, falls back to existing configuration otherwise. +- **StorageBackend API (OSAC-1111):** Not blocking. In development (fulfillment-service PR #728). The system integrates when available, falls back to single implicit backend otherwise. -- **Owner:** CaaS working group -- **Impact:** Affects FR-2. The expected path is `HostedControlPlane.status.kubeConfig`, but a scoped service account token may be preferred. From c8c0570b3636d60a85122d0b6e1962122c9facbd Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 24 Jun 2026 17:29:00 -0400 Subject: [PATCH 050/450] OSAC-1123: address review feedback, fix pre-commit Drop arbitrary 5-minute success metric (no defensible basis). Remove unit test AC (implementation concern, not product requirement). Remove Helm chart detail from dependencies (design concern). Resolve kubeconfig open question (confirmed by CaaS team). Fix trailing blank line for pre-commit. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude --- enhancements/caas-cluster-storage/prd.md | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/enhancements/caas-cluster-storage/prd.md b/enhancements/caas-cluster-storage/prd.md index 80ae83a82..a6796bc04 100644 --- a/enhancements/caas-cluster-storage/prd.md +++ b/enhancements/caas-cluster-storage/prd.md @@ -21,13 +21,7 @@ CaaS tenant clusters are provisioned without persistent storage. When a cluster - Multiple CaaS clusters per tenant are handled independently, each with its own storage readiness. - When the Storage Tier API (OSAC-1110) and StorageBackend API (OSAC-1111) are available, the system uses them. Otherwise it falls back to the existing configuration. -### 2.2 Success Metrics - -| Metric | Target | Baseline | -|--------|--------|----------| -| Time from cluster ready to storage available | < 5 minutes | N/A (new feature) | - -### 2.3 Non-Goals +### 2.2 Non-Goals - VAST provider CaaS-specific changes (storage path conventions, credential scoping, CaaS provisioning targets). Covered by OSAC-1122. - Storage UI. @@ -59,19 +53,18 @@ A tenant can have multiple CaaS clusters. Each cluster's storage is set up and t ### 3.1 Operational Expectations -- Storage setup completes within 5 minutes of a CaaS cluster becoming ready. +- Storage should be available shortly after the cluster is ready. - Credentials and secrets are never exposed in user-visible error messages, status fields, or events. ## 4. Acceptance Criteria - [ ] A tenant user can create a PVC on a newly provisioned CaaS cluster without manual storage configuration - [ ] A Cloud Provider Admin can see storage readiness status for each tenant's CaaS clusters -- [ ] A Tenant Admin can see whether their specific CaaS cluster's storage is ready and the reason if it failed +- [ ] A Tenant Admin or User can see whether their specific CaaS cluster's storage is ready and the reason if it failed - [ ] When a CaaS cluster is deleted, storage resources on that cluster are cleaned up - [ ] Deleting one CaaS cluster does not affect storage on other clusters belonging to the same tenant - [ ] Backend resources are unaffected by CaaS cluster deletion - [ ] A second CaaS cluster for the same tenant gets independent storage setup -- [ ] Unit tests cover setup, readiness, teardown, and multi-cluster scenarios ## 5. Assumptions @@ -83,9 +76,8 @@ A tenant can have multiple CaaS clusters. Each cluster's storage is set up and t ## 6. Dependencies -- **Tenant Storage Onboarding (OSAC-23):** The storage automation framework, two-stage model, and cluster watch are implemented and merged (osac-operator PR #299). This PRD extends the framework to act on CaaS cluster events. The Helm chart needs to be updated to expose the storage controller configuration. +- **Tenant Storage Onboarding (OSAC-23):** The storage automation framework, two-stage model, and cluster watch are implemented and merged (osac-operator PR #299). This PRD extends the framework to act on CaaS cluster events. - **Playbook split (osac-aap PR #338):** Merged. Provides the independent setup and teardown actions required for CaaS cluster-scoped operations. - **VAST for CaaS (OSAC-1122):** The storage provider automation must support CaaS clusters as a provisioning target. Without this, the trigger fires but setup cannot complete. - **Tier API (OSAC-1110):** Not blocking. The system integrates when available, falls back to existing configuration otherwise. - **StorageBackend API (OSAC-1111):** Not blocking. In development (fulfillment-service PR #728). The system integrates when available, falls back to single implicit backend otherwise. - From 7ea6b4dea50a84aae8a086dac700513fcd8d3278 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 24 Jun 2026 18:35:46 -0400 Subject: [PATCH 051/450] OSAC-1123: remove remaining design leakage, consistent dependencies Remove implementation details from non-goals, cleanup section, and assumptions (StorageClasses, CSI credentials, VIP Pool, HostedControlPlane). Standardize dependency format: status first, then what it provides. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude --- enhancements/caas-cluster-storage/prd.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/enhancements/caas-cluster-storage/prd.md b/enhancements/caas-cluster-storage/prd.md index a6796bc04..db4dad6f2 100644 --- a/enhancements/caas-cluster-storage/prd.md +++ b/enhancements/caas-cluster-storage/prd.md @@ -23,7 +23,7 @@ CaaS tenant clusters are provisioned without persistent storage. When a cluster ### 2.2 Non-Goals -- VAST provider CaaS-specific changes (storage path conventions, credential scoping, CaaS provisioning targets). Covered by OSAC-1122. +- Making the storage provider CaaS-ready. Covered by OSAC-1122. - Storage UI. - Storage backend provisioning (runs during tenant onboarding, assumed complete before any cluster is ready). - VMaaS cluster-side storage changes. Existing flow is unchanged. @@ -45,7 +45,7 @@ As a Tenant Admin or Tenant User, I can see whether my specific CaaS cluster's s ### Cluster Deletion and Cleanup -When a CaaS cluster is deleted, storage resources installed on that cluster (StorageClasses, CSI credentials, snapshot classes) are cleaned up. Backend resources (storage provider tenant, views, quotas, credentials on the hub) are not affected, because other clusters belonging to the same tenant may still use them. +When a CaaS cluster is deleted, storage resources installed on that cluster are cleaned up. Backend resources shared across the tenant's clusters are not affected, because other clusters may still use them. ### Multi-Cluster Independence @@ -70,14 +70,14 @@ A tenant can have multiple CaaS clusters. Each cluster's storage is set up and t - Storage backend provisioning is always complete before a CaaS cluster reaches ready. The system does not handle a ready cluster with an unprovisioned backend. - The storage provider's automation accepts CaaS clusters as a target and uses the provided cluster credentials. This is delivered by OSAC-1122. -- VAST is the only storage provider for v0.1. A global VIP Pool is shared by all tenants. -- Cluster credentials are obtainable from the cluster provisioning infrastructure via the HostedControlPlane kubeconfig Secret. Confirmed by the CaaS team. -- CaaS cluster nodes have network reachability to storage VIP pools. This is an infrastructure prerequisite managed by the Cloud Infrastructure Admin. +- VAST is the only storage provider for v0.1. +- Cluster credentials are obtainable from the cluster provisioning infrastructure. Confirmed by the CaaS team. +- CaaS cluster nodes have network reachability to the storage backend. This is an infrastructure prerequisite managed by the Cloud Infrastructure Admin. ## 6. Dependencies -- **Tenant Storage Onboarding (OSAC-23):** The storage automation framework, two-stage model, and cluster watch are implemented and merged (osac-operator PR #299). This PRD extends the framework to act on CaaS cluster events. -- **Playbook split (osac-aap PR #338):** Merged. Provides the independent setup and teardown actions required for CaaS cluster-scoped operations. -- **VAST for CaaS (OSAC-1122):** The storage provider automation must support CaaS clusters as a provisioning target. Without this, the trigger fires but setup cannot complete. -- **Tier API (OSAC-1110):** Not blocking. The system integrates when available, falls back to existing configuration otherwise. -- **StorageBackend API (OSAC-1111):** Not blocking. In development (fulfillment-service PR #728). The system integrates when available, falls back to single implicit backend otherwise. +- **Tenant Storage Onboarding (OSAC-23):** Merged. Provides the storage automation framework that this PRD extends for CaaS clusters. +- **Playbook split (osac-aap PR #338):** Merged. Provides independent setup and teardown actions required for per-cluster operations. +- **VAST for CaaS (OSAC-1122):** In progress. The storage provider must support CaaS clusters as a target. Required for end-to-end functionality. +- **Tier API (OSAC-1110):** In progress. Not blocking. The system integrates when available, falls back to existing configuration. +- **StorageBackend API (OSAC-1111):** In progress. Not blocking. The system integrates when available, falls back to single implicit backend. From b6b564f9c6804d50cc7d8bc27ac54b2642aa9bf1 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Wed, 24 Jun 2026 18:41:58 -0400 Subject: [PATCH 052/450] OSAC-1123: consolidate OSAC-23 dependencies into single entry Merge playbook split (osac-aap PR #338) into the OSAC-23 dependency since both are part of the same epic and both are merged. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude --- enhancements/caas-cluster-storage/prd.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/enhancements/caas-cluster-storage/prd.md b/enhancements/caas-cluster-storage/prd.md index db4dad6f2..d61804834 100644 --- a/enhancements/caas-cluster-storage/prd.md +++ b/enhancements/caas-cluster-storage/prd.md @@ -76,8 +76,7 @@ A tenant can have multiple CaaS clusters. Each cluster's storage is set up and t ## 6. Dependencies -- **Tenant Storage Onboarding (OSAC-23):** Merged. Provides the storage automation framework that this PRD extends for CaaS clusters. -- **Playbook split (osac-aap PR #338):** Merged. Provides independent setup and teardown actions required for per-cluster operations. +- **Tenant Storage Onboarding (OSAC-23):** Merged. Provides the storage automation framework and independent per-cluster setup and teardown actions that this PRD extends for CaaS clusters. - **VAST for CaaS (OSAC-1122):** In progress. The storage provider must support CaaS clusters as a target. Required for end-to-end functionality. - **Tier API (OSAC-1110):** In progress. Not blocking. The system integrates when available, falls back to existing configuration. - **StorageBackend API (OSAC-1111):** In progress. Not blocking. The system integrates when available, falls back to single implicit backend. From c2c59cdd2410897035ca8e91a4fcc8f50c4a4fb5 Mon Sep 17 00:00:00 2001 From: Eran Cohen Date: Thu, 25 Jun 2026 10:37:01 +0300 Subject: [PATCH 053/450] OSAC-1771: add OWNERS file for Prow integration MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add OWNERS file with approvers set to OSAC architects, leads, and managers — the people who review and approve enhancement proposals. Reviewers include all approvers plus active EP contributors. Assisted-by: Claude Code Signed-off-by: Eran Cohen --- OWNERS | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 OWNERS diff --git a/OWNERS b/OWNERS new file mode 100644 index 000000000..a0df32aa2 --- /dev/null +++ b/OWNERS @@ -0,0 +1,57 @@ +approvers: +# Architects +- avishayt +- carbonin +- eliorerz +- jhernand +- larsks +- maorfr +- masayag +- mhrivnak +- rgolangh +- vladikr +- ygalblum +# Leads +- adriengentil +- akshaynadkarni +- CrystalChun +- danmanor +- rawagner +- rccrdpccl +# Managers +- AlonaKaplan +- chenyosef +- eranco74 +- gamli75 +- liatb-rh +- oourfali +- ronniel1 +reviewers: +- adriengentil +- akshaynadkarni +- AlonaKaplan +- avishayt +- carbonin +- chenyosef +- CrystalChun +- danmanor +- eliorerz +- eranco74 +- gamli75 +- jhernand +- larsks +- liatb-rh +- mhrivnak +- maorfr +- masayag +- mennyaboush +- omer-vishlitzky +- oourfali +- rawagner +- rccrdpccl +- rgolangh +- ronniel1 +- trewest +- tzvatot +- vladikr +- ygalblum From 6a18736a7836faffe878198b142f096ebc4a1baa Mon Sep 17 00:00:00 2001 From: Ilya Skornyakov Date: Thu, 25 Jun 2026 15:35:59 +0300 Subject: [PATCH 054/450] =?UTF-8?q?OSAC-1269:=20revise=20PRD=20=E2=80=94?= =?UTF-8?q?=20lifecycle=20states,=20consolidated=20ACs,=20goal/persona=20r?= =?UTF-8?q?efinements?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace enabled/disabled toggle with three-state lifecycle (active/deprecated/ obsolete) following InstanceType pattern. Consolidate 16 acceptance criteria into 6 grouped by theme. Rewrite goals as user-facing outcomes. Scope FR-4 visibility to cluster responses. Add replacement-version qualifier throughout. Update Tenant Admin persona for catalog item authoring. Drop channel from ClusterSpec (deferred to OSAC-1415). Clean problem statement and non-goals. Assisted-by: Claude Code Signed-off-by: Ilya Skornyakov --- enhancements/cluster-version-api/prd.md | 127 +++++++++++------------- 1 file changed, 56 insertions(+), 71 deletions(-) diff --git a/enhancements/cluster-version-api/prd.md b/enhancements/cluster-version-api/prd.md index c077fce23..a9f961750 100644 --- a/enhancements/cluster-version-api/prd.md +++ b/enhancements/cluster-version-api/prd.md @@ -8,128 +8,112 @@ ## 1. Problem Statement -The Cluster API requires users to provide a full OCI release image URL (e.g., `quay.io/openshift-release-dev/ocp-release:4.17.0-multi`) when creating a cluster. This leaks infrastructure implementation details into the user-facing API: users must know the exact registry path and tag format, typos are caught only at provisioning time rather than at API validation, and there is no way to discover which versions are available or to specify an update channel. +The Cluster API requires users to provide a full OCI release image URL (e.g., `quay.io/openshift-release-dev/ocp-release:4.17.0-multi`) when creating a cluster. This leaks infrastructure implementation details into the user-facing API: users must know the exact registry path and tag format, typos are caught only at provisioning time rather than at API validation, and there is no way to discover which versions are available. -The cost is visible across the stack. Catalog items resort to regex `validation_schema` patterns to constrain the registry prefix — a workaround that cannot verify the version actually exists or is supported. AAP template roles independently construct release image URLs from version numbers (e.g., `{{ ocp_release_version }}-multi`), duplicating URL format knowledge across templates with no central validation. Each layer works around the same missing abstraction: a managed catalog of validated cluster versions. +The cost compounds across layers. Catalog items resort to regex patterns to constrain the image URL prefix — a workaround that cannot verify the version actually exists or is supported. Multiple components independently construct release image URLs from version numbers, duplicating format knowledge with no central validation. Each layer works around the same missing abstraction: a managed catalog of validated cluster versions. -A ClusterVersion resource eliminates these workarounds. Users select from a curated, queryable list of versions — no URLs, no guessing. Catalog items reference versions by name instead of embedding URL patterns. AAP templates receive pre-validated release images from the API, removing template-level URL construction. And the version catalog becomes the foundation for future upgrade operations (OSAC-1415) and ACM version synchronization, which require a queryable set of available versions and their upgrade paths. +A ClusterVersion resource eliminates these workarounds. Users select from a curated, queryable list of versions — no URLs, no guessing. Catalog items reference versions by name instead of embedding URL patterns. Automation receives pre-validated release images from the API, removing URL construction. And the version catalog becomes the foundation for future upgrade operations (OSAC-1415) and ACM version synchronization, which require a queryable set of available versions and their upgrade paths. ## 2. Goals and Non-Goals ### 2.1 Goals - Users specify an OpenShift version by number (e.g., "4.17.0") instead of a raw OCI URL when creating a cluster, with the server resolving the version to the correct release image internally. -- The server validates version and channel at API time, rejecting invalid or disabled versions with descriptive error messages before any provisioning begins. -- Cloud Provider Admins manage available versions through a ClusterVersion resource. -- Tenant Users can discover available versions through the public API, enabling self-service version selection in the CLI and wizard. +- Users receive immediate, descriptive feedback when specifying an invalid, obsolete, or deprecated version — before any provisioning begins. +- Cloud Provider Admins manage available cluster versions through the CLI and UI console. +- Tenant Users can discover and select from available versions when creating a cluster — in the CLI, UI, and API. ### 2.2 Non-Goals -- Cluster upgrade operations and channel propagation to HostedCluster — triggering, tracking, or rolling back upgrades, and propagating `channel` to `HostedCluster.spec.channel`. Both are owned by OSAC-1415. The `channel` field is validated in v0.2 but not propagated. [Clarify: R1.Q3, R2.Q2] -- ACM ClusterImageSet auto-sync — automatic discovery and population of ClusterVersion objects from ACM. Versions are admin-managed in v0.2. [Clarify: R1.Q7] -- VM image management — ComputeInstance image catalog (OSAC-979) uses a separate `ComputeImage` resource. Both features share the same architectural pattern (raw URL to managed catalog with server-side resolution) but are distinct resources. [Clarify: R1.Q4] +- Cluster upgrade operations and channel propagation to the hosted cluster — triggering, tracking, or rolling back upgrades, and propagating the version's channel to the hosted cluster. Owned by OSAC-1415. Channels are stored as version catalog metadata but not propagated to the cluster in v0.2. [Clarify: R1.Q3, R2.Q2] +- ACM ClusterImageSet auto-sync — automatic discovery and population of version catalog entries from ACM. Versions are admin-managed in v0.2. [Clarify: R1.Q7] +- VM image management — the VM image catalog (OSAC-979) uses a separate resource. Both features share the same architectural pattern (raw URL to managed catalog with server-side resolution) but are distinct resources. [Clarify: R1.Q4] - In-place upgrade migration — OSAC does not support in-place upgrades. This feature assumes fresh deployment. [Clarify: R2.Q4] ## 3. Requirements ### 3.1 Functional Requirements -#### ClusterVersion Resource +#### Version Catalog Management -- **FR-1:** OSAC must provide a ClusterVersion resource with the following domain fields: `release_image` (resolved OCI URL), `enabled` (admin availability toggle), `default` (used when version is omitted), and `available_channels` (list of update channel strings, e.g., `["stable-4.17", "fast-4.17"]`). The resource includes standard OSAC metadata (id, name, labels, annotations, timestamps). The resource `name` is the version number itself (e.g., "4.17.0") — there is no separate version string field. [Clarify: R1.Q1, R1.Q8] +- **FR-1:** Each version in the catalog is identified by its version number (e.g., "4.17.0"), maps to a release image, has a lifecycle state (active, deprecated, or obsolete), can optionally be marked as the default, and is associated with update channels (e.g., "stable-4.17", "fast-4.17"). Channels are informational metadata in v0.2; channel-based selection and propagation are deferred to OSAC-1415. [Clarify: R1.Q1, R1.Q8] -- **FR-2:** The resource must be named `ClusterVersion` and the ClusterSpec field must be named `cluster_version`. The naming is cluster-scoped, distinguishing it from other versioned resources in the platform (e.g., ComputeImage for VMaaS). [User] +- **FR-2:** Admins can create, update, and delete catalog entries. Users can list versions filtered to active and deprecated entries by default; obsolete versions are hidden from the default list but visible via explicit filter. Viewing a specific version by identifier returns it regardless of its lifecycle state, so users can inspect versions referenced by their existing clusters. [Clarify: R1.Q6, R3.Q1] -- **FR-3:** Cloud Provider Admins can create, update, and delete ClusterVersion resources. Tenant Users can list and view available versions. Tenant-visible listings are filtered to enabled versions only. Getting a specific ClusterVersion by identifier returns the resource regardless of its `enabled` status. [Clarify: R1.Q6, R3.Q1] +- **FR-3:** At most one version can be marked as default. The default version is used when no version is specified by the user or template. [Clarify: R1.Q1] -- **FR-4:** At most one ClusterVersion resource may be marked as `default`. The global default ClusterVersion is the lowest-priority fallback — it is used only when the user omits `cluster_version` and neither template defaults nor catalog item field_definitions supply one. [Clarify: R1.Q1] +#### Cluster Creation -- **FR-5:** Cloud Provider Admins must be able to create ClusterVersion objects via the CLI (e.g., `osac create -f cluster-versions.yaml`). [Clarify: R1.Q7] +- **FR-4:** Users specify a version number (e.g., "4.17.0") when creating a cluster. Release image URLs are not visible in cluster creation or listing responses — only the version is exposed. [Clarify: R1.Q2, R1.Q5, R2.Q1, R2.Q4, R3.Q3, R3.Q4] -#### Cluster API Changes +- **FR-5:** Templates can specify a default version (e.g., "4.17.0"). Any version can be used with any template; templates provide defaults but do not constrain version selection. [Clarify: R2.Q1] -- **FR-6:** Users specify a `cluster_version` (e.g., "4.17.0") that references a ClusterVersion resource by name when creating a cluster. This field replaces `release_image` on the user-facing API. [Clarify: R1.Q2, R1.Q5] +- **FR-6:** A cluster's version is visible in responses when viewing or listing clusters. -- **FR-7:** Users can specify a `channel` (e.g., "stable-4.17") when creating a cluster. When specified, the server validates the channel against the referenced ClusterVersion's `available_channels` list. The channel is not propagated to the HostedCluster in v0.2. [Clarify: R1.Q3] - -- **FR-8:** The `release_image` field must be removed from the user-facing Cluster API and ClusterTemplate defaults. Users see only `cluster_version` and `channel`; `release_image` is not visible to API consumers. [Clarify: R1.Q5, R2.Q1, R2.Q4, R3.Q3, R3.Q4] - -#### Version Resolution +#### Validation -- **FR-9:** During cluster creation, the server must resolve the `cluster_version` to a `release_image` using the referenced ClusterVersion resource. Version resolution is fully contained in the fulfillment-service — the ClusterOrder CRD, osac-operator, and AAP are unchanged and continue to receive `release_image` with no downstream changes. [Clarify: R1.Q2, R1.Q5, R2.Q4, R3.Q3] +- **FR-7:** Version is validated at creation time with descriptive error messages that include the invalid value and valid alternatives. Validation covers: version not found and version obsolete. Creating a cluster with a deprecated version succeeds but includes a warning identifying the replacement version, if one is set. [Clarify: R2.Q3, R3.Q6] -- **FR-10:** ClusterTemplate defaults must support a `cluster_version` field replacing `release_image`. Template authors set `cluster_version: "4.17.0"` in defaults. Any ClusterVersion can be used with any template; templates provide version defaults but do not constrain version selection. The server applies template defaults before version resolution. [Clarify: R2.Q1] +#### User Interfaces -#### Validation +- **FR-8:** The CLI supports catalog management (create, update, delete) and cluster creation with a version option replacing the release image option. [Clarify: R1.Q5, R1.Q7] -- **FR-11:** The server must validate `cluster_version` and `channel` at API time, rejecting invalid requests with descriptive error messages that include the invalid value and valid alternatives. Validation applies to the effective spec after template defaults and the global default ClusterVersion have been applied. Validation covers: version not found, version not enabled, version deleted, invalid channel for the given version, channel present without a resolved cluster_version (after all defaults), and empty `available_channels` (specifying any channel is invalid). The `cluster_version` and `channel` fields are immutable after cluster creation — attempts to change either on update must be rejected. [Clarify: R2.Q3, R3.Q6] +- **FR-9:** The UI console supports catalog management for admins and version selection in the cluster creation wizard for users. -#### CLI and Catalog Items +- **FR-10:** Catalog items reference version instead of release image in their field definitions, following the existing catalog item pattern. [Clarify: R1.Q5] -- **FR-12:** The CLI must replace the `--release-image` flag with `--cluster-version` and `--channel` flags for cluster creation. [Clarify: R1.Q5] +#### Lifecycle Protection -- **FR-13:** Catalog item field_definitions must reference the `cluster_version` field path instead of `release_image`. [Clarify: R1.Q5] +- **FR-11:** Deleting a version is rejected when it is referenced by active clusters or template defaults. The error identifies the referencing resource (e.g., "cannot delete version '4.17.0': in use by cluster 'cluster-abc'"). [Clarify: R3.Q2] -#### Lifecycle Protection +- **FR-12:** Creating or updating a cluster or template with a version that does not exist or has been deleted is rejected. A cluster cannot reference a deleted version. [Clarify: R3.Q2] -- **FR-14:** Deleting a ClusterVersion must be rejected when it is referenced by active clusters or ClusterTemplate defaults. The error message must identify the referencing resource (e.g., "cannot delete cluster version '4.17.0': it is in use by at least cluster 'cluster-abc'"). [Clarify: R3.Q2] +- **FR-13:** Lifecycle state transitions (active, deprecated, obsolete) are always allowed regardless of references. Existing clusters and templates referencing a deprecated or obsolete version are not affected. Admins can transition a version between any states in any direction. [Clarify: R3.Q5] -- **FR-15:** Creating or updating a cluster or ClusterTemplate with a `cluster_version` that references a non-existent or deleted ClusterVersion must be rejected. Concurrent create and delete operations must not produce inconsistent state. [Clarify: R3.Q2] +- **FR-14:** Version is immutable after cluster creation. [Clarify: R2.Q3] -- **FR-16:** Disabling a ClusterVersion is always allowed regardless of references. Disabled versions block new cluster creation (FR-11 validation) but do not affect existing clusters or templates that reference them. [Clarify: R3.Q5] +- **FR-15:** A version can be marked as deprecated with an optional replacement version. Deprecated versions remain available for new cluster creation but include a warning identifying the replacement, if one is set. Obsolete versions are blocked for new cluster creation. Deprecation and obsolescence timestamps are recorded automatically when the version enters each state. ### 3.2 Non-Functional Requirements -- **NFR-1:** The ClusterVersion resource must be consistent with existing admin-managed reference data resources (HostType, NetworkClass) in API conventions, access patterns, and lifecycle behavior. [Clarify: R1.Q6, R3.Q1] - -- **NFR-2:** Field names and reference patterns must align with the OCM Clusters Management API where applicable: `cluster_version` as a reference by readable name (aligned with OCM's `link Version Version` pattern), `channel` as a plain string (aligned with OCM's `Channel String`). [Clarify: R1.Q8] +- **NFR-1:** Admins manage the version catalog using the same patterns they already know from other admin-managed resources — consistent commands, access model, lifecycle states (active/deprecated/obsolete), and visibility rules with no new interaction model to learn. [Clarify: R1.Q6, R3.Q1] -- **NFR-3:** The ClusterVersion resource must be extensible. OSAC-1415 is expected to add fields such as `available_upgrades` and activate channel propagation to HostedCluster. These additions must be non-breaking proto field additions. [Clarify: R2.Q2] +- **NFR-2:** Future additions to the version catalog (e.g., upgrade paths, channel-based selection) must not break existing cluster creation workflows. [Clarify: R2.Q2] ## 4. Acceptance Criteria -- [ ] A user can create a cluster by specifying `--cluster-version 4.17.0` instead of a release image URL, and the server resolves the version to the correct release image internally. -- [ ] Specifying a non-existent or deleted `cluster_version` is rejected with a descriptive error — both when creating a cluster and when setting ClusterTemplate defaults. -- [ ] Specifying a disabled version is rejected with a message indicating the version is not enabled. -- [ ] Specifying a channel not in the ClusterVersion's `available_channels` is rejected with an error listing valid channels. A request whose effective spec contains `channel` but no `cluster_version` (after defaults) is rejected. -- [ ] Attempting to change `cluster_version` or `channel` on an existing cluster is rejected (both fields are immutable). -- [ ] Listing ClusterVersions as a Tenant User returns only enabled versions with their `available_channels`. -- [ ] A Cloud Provider Admin can create, update, and delete ClusterVersion resources. -- [ ] Deleting a ClusterVersion that is referenced by an active cluster or a ClusterTemplate's defaults is rejected with a message identifying the referencing resource. -- [ ] Disabling a ClusterVersion succeeds even when referenced by active clusters or templates; new cluster creation with that version is blocked. -- [ ] When a user omits `cluster_version` and neither template defaults nor catalog item field_definitions supply one, the server uses the ClusterVersion marked as `default`. -- [ ] At most one ClusterVersion is marked as `default` at any time — setting `default` on a second ClusterVersion either clears the previous default or is rejected. -- [ ] A cluster creation and a ClusterVersion deletion racing on the same version do not produce a cluster referencing a deleted ClusterVersion. -- [ ] Template defaults support `cluster_version` — when a template specifies `cluster_version: "4.17.0"`, the server applies it as the default and resolves it to `release_image`. -- [ ] Catalog item field_definitions can reference the `cluster_version` path. -- [ ] The `release_image` field is not visible to API consumers. Internally, the server stores and forwards `release_image` to downstream systems. -- [ ] The osac-operator and AAP continue to receive `release_image` through the ClusterOrder CRD — no downstream changes are required. +- [ ] A user can create a cluster by specifying a version number (e.g., 4.17.0) instead of a release image URL. The server resolves the version to the correct release image internally. Release image URLs are not visible in cluster creation or listing responses. Version is immutable after creation. +- [ ] Specifying a non-existent, deleted, or obsolete version is rejected with a descriptive error indicating the reason and identifying the replacement if one is set. Deprecated versions allow creation with a warning. Validation applies to both cluster creation and template defaults. +- [ ] Admins can create, update, and delete version catalog entries. Deleting a version referenced by an active cluster or template defaults is rejected with a message identifying the referencing resource. +- [ ] Admins can transition a version between active, deprecated, and obsolete in any direction, even when referenced by active clusters or templates. Listing versions returns active and deprecated entries by default; obsolete versions are hidden unless explicitly filtered. Deprecation and obsolescence timestamps are recorded automatically on each transition. +- [ ] At most one version is marked as default at any time — setting a new default clears the previous one. When a user omits the version and template defaults do not supply one, the server uses the default version. Templates can specify a default version, and the server resolves it to a release image. +- [ ] The CLI supports version catalog management and cluster creation with a version option. Catalog items can reference version in their field definitions. The UI console supports catalog management for admins and version selection in the cluster creation wizard. ## 5. Assumptions -- OSAC does not support in-place upgrades. This feature assumes fresh deployment or environment re-creation. Existing clusters created with `release_image` will not automatically display a `cluster_version` value after redeployment. Administrators must re-create templates and catalog items with `cluster_version` replacing `release_image`. [Clarify: R2.Q4] -- No production catalog items exist that reference `release_image` in field_definitions, so the field path change from `release_image` to `cluster_version` requires no migration. [Clarify: R1.Q5] +- OSAC does not support in-place upgrades. This feature assumes fresh deployment or environment re-creation. Existing clusters created with a release image URL will not automatically display a version value after redeployment. Administrators must re-create templates and catalog items with version references replacing release image URLs. [Clarify: R2.Q4] +- No production catalog items exist that reference the release image, so the change to version-based selection requires no migration. [Clarify: R1.Q5] ## 6. Dependencies -- **OSAC-1531 (Default Catalog Items)** — Default ClusterVersion objects should ship alongside default catalog items. ClusterVersion creation is a prerequisite for catalog items that reference version-based clusters. -- **OSAC-1415 (Cluster Upgrade)** — The ClusterVersion resource is designed as an extension point. OSAC-1415 is expected to add fields such as `available_upgrades` and activate `channel` propagation to HostedCluster. This PRD does not constrain OSAC-1415's design. [Clarify: R2.Q2] +- **OSAC-1531 (Default Catalog Items)** — Default version catalog entries should ship alongside default catalog items. Version catalog population is a prerequisite for catalog items that reference version-based clusters. +- **OSAC-1415 (Cluster Upgrade)** — The version catalog is designed as an extension point. OSAC-1415 is expected to add upgrade capabilities, channel-based version selection, and channel propagation to the hosted cluster. This PRD does not constrain OSAC-1415's design. [Clarify: R2.Q2] ## 7. Risks -### 7.1 ClusterVersion catalog must be populated before cluster creation +### 7.1 Version catalog must be populated before cluster creation -Without ClusterVersion objects, users cannot create clusters via the public API. If admins deploy OSAC but do not populate versions, the system appears broken. +Without version catalog entries, users cannot create clusters via the public API. If admins deploy OSAC but do not populate versions, the system appears broken. - **Owner:** CaaS team -- **Mitigation:** Ship default ClusterVersion objects alongside OSAC-1531 default catalog items. Document version population in the admin guide. +- **Mitigation:** Ship default version catalog entries alongside OSAC-1531 default catalog items. Document version population in the admin guide. ### 7.2 Shared pattern divergence with VM image management -OSAC-1269 (ClusterVersion) and OSAC-979 (ComputeImage) follow the same architectural pattern — raw URL to managed catalog with server-side resolution. If designed independently, their APIs may diverge in naming conventions, resolution behavior, or error handling. +OSAC-1269 (cluster version catalog) and OSAC-979 (VM image catalog) follow the same architectural pattern — raw URL to managed catalog with server-side resolution. If designed independently, their APIs may diverge in naming conventions, resolution behavior, or error handling. - **Owner:** CaaS / VMaaS teams -- **Mitigation:** Both resources follow the HostType/NetworkClass reference data pattern. Coordinate API conventions during design review. +- **Mitigation:** Both catalogs follow the existing admin-managed reference data pattern. Coordinate API conventions during design review. ## OSAC Dimensions @@ -141,27 +125,28 @@ This feature applies to **CaaS** (Cluster as a Service) only. VMaaS has an analo | Persona | Interaction | |---------|-------------| -| Cloud Provider Admin | Creates and manages ClusterVersion resources. Populates the version catalog. Sets default version. | -| Tenant User | Selects a version from the available list when creating a cluster. Discovers available versions via `ListClusterVersions`. | +| Cloud Provider Admin | Creates and manages the version catalog via CLI, API, and UI. Populates available versions. Sets the default version. | +| Tenant User | Selects a version when creating a cluster via CLI, UI wizard, or catalog items. Discovers available versions via the API and UI. | | Cloud Infrastructure Admin | Not affected. | -| Tenant Admin | Not affected beyond Tenant User capabilities. | +| Tenant Admin | Same as Tenant User. May also reference versions when authoring org-specific catalog items. | ### Provisioning -Version resolution occurs during cluster creation in the fulfillment-service. The provisioning flow downstream (ClusterOrder CRD, osac-operator, AAP) is unchanged — it continues to receive `release_image`. +Version resolution occurs during cluster creation. The provisioning flow for the cluster itself is unchanged — this feature adds version selection and validation, not new provisioning steps. ### User-Facing API | Surface | Impact | |---------|--------| -| Fulfillment API (gRPC/REST) | New ClusterVersion resource (admin-managed, tenant-readable). Modified ClusterSpec (`cluster_version` and `channel` replace `release_image` for API consumers). Modified ClusterTemplate defaults (`cluster_version` replaces `release_image`). Deletion protection when referenced by clusters or templates. | -| OSAC CRDs | No change. ClusterOrder CRD retains `releaseImage`. | -| Catalog Items | Field_definitions reference `cluster_version` path instead of `release_image`. | +| Fulfillment API (gRPC/REST) | New version catalog resource. Cluster creation uses version instead of release image URL. Template defaults updated. Deletion protection when referenced. | +| OSAC CRDs | No change. | +| UI Console | Version catalog management for admins. Version selection in the cluster creation wizard. | +| Catalog Items | Support version selection. | ### Milestone Scoping - **Target milestone:** v0.2 -- **Deferred:** Cluster upgrades (OSAC-1415), ACM version auto-sync, channel propagation to HostedCluster, in-place migration of existing data. +- **Deferred:** Cluster upgrades (OSAC-1415), ACM version auto-sync, channel-based version selection, channel propagation to hosted cluster, in-place migration of existing data. - **Upgrades:** OSAC does not support in-place upgrades. No data migration is in scope. ### Installation @@ -170,4 +155,4 @@ No changes to Helm charts, kustomize manifests, or `osac-installer/setup.sh` bey ### Tenant Onboarding, Inventory, Networking, Storage -Not applicable. The ClusterVersion resource is global (not tenant-scoped) and does not interact with inventory, networking, or storage subsystems. +Not applicable. The version catalog is global (not tenant-scoped) and does not interact with inventory, networking, or storage subsystems. From 5365aa4c4195944d819ff80a78934941fed4d1df Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Thu, 25 Jun 2026 11:06:58 -0400 Subject: [PATCH 055/450] OSAC-1123: simplify PRD to user stories and scope Rewrite the CaaS cluster storage PRD based on reviewer feedback from Michael Hrivnak and Avishay Traeger. The PRD now focuses on user stories grouped by persona instead of design-level capabilities and repeated acceptance criteria. Key changes: - Replace Goals/Non-Goals with In Scope/Out of Scope to avoid repeating what User Stories already cover - Replace Capabilities and Acceptance Criteria with User Stories grouped by persona (Tenant Admin/User, Cloud Provider Admin) - Remove design leakage: CSI driver references, per-tenant/per-tier StorageClasses, Tier API/StorageBackend API fallback, VAST-specific assumptions, cluster credential details - Remove ticket numbers from non-goals and dependencies - Trim assumptions to product-level constraints only Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- enhancements/caas-cluster-storage/prd.md | 87 +++++++----------------- 1 file changed, 26 insertions(+), 61 deletions(-) diff --git a/enhancements/caas-cluster-storage/prd.md b/enhancements/caas-cluster-storage/prd.md index d61804834..c9bd38b2c 100644 --- a/enhancements/caas-cluster-storage/prd.md +++ b/enhancements/caas-cluster-storage/prd.md @@ -4,79 +4,44 @@ |-------------|---------| | Author(s) | Akshay Nadkarni | | Jira | https://redhat.atlassian.net/browse/OSAC-1332 | -| Date | 2026-06-24 | +| Date | 2026-06-25 | -## 1. Problem Statement +## Problem Statement -CaaS tenant clusters are provisioned without persistent storage. When a cluster is ready, it has compute but no way for tenant workloads to create persistent volumes. Storage backend resources (credentials, views, quotas) are already provisioned during tenant onboarding, but nothing connects cluster readiness to storage installation on that cluster. Cloud Provider Admins have no visibility into whether a cluster's storage is ready, and Tenant Admins and Users cannot use persistent storage until someone manually configures it. +CaaS tenant clusters are provisioned without persistent storage. When a tenant's cluster becomes ready, it has compute but no way to create persistent volumes. Tenants cannot run stateful workloads until someone manually configures storage, and there is no visibility into whether storage is available on a given cluster. -## 2. Goals and Non-Goals +## In Scope -### 2.1 Goals +- Automatic storage provisioning on CaaS clusters +- Storage readiness visibility for tenants and cloud admins +- Storage cleanup on CaaS cluster deletion -- When a CaaS cluster is provisioned and ready, persistent storage (CSI driver and per-tenant, per-tier StorageClasses) is automatically available without manual configuration. -- Cloud Provider Admins can see tenant-level storage status across all CaaS clusters. -- Tenant Admins and Users can see whether their cluster's storage is ready. -- When a CaaS cluster is deleted, storage resources on that cluster are cleaned up. Backend resources are left intact for other clusters. -- Multiple CaaS clusters per tenant are handled independently, each with its own storage readiness. -- When the Storage Tier API (OSAC-1110) and StorageBackend API (OSAC-1111) are available, the system uses them. Otherwise it falls back to the existing configuration. +## Out of Scope -### 2.2 Non-Goals +- Storage provider changes for CaaS support +- Storage UI +- Storage backend provisioning for a tenant (runs during tenant onboarding, before any cluster is ready) +- VMaaS storage changes -- Making the storage provider CaaS-ready. Covered by OSAC-1122. -- Storage UI. -- Storage backend provisioning (runs during tenant onboarding, assumed complete before any cluster is ready). -- VMaaS cluster-side storage changes. Existing flow is unchanged. -- Cloud Infrastructure Admin responsibilities (storage backend configuration, tier definitions). Covered by OSAC-1122 and OSAC-1110. +## User Stories -## 3. Capabilities +### Tenant Admin / Tenant User -### Automatic Storage Setup +- As a Tenant Admin or Tenant User, I want persistent storage to be available on my CaaS cluster when it is ready, so that I can run stateful workloads without waiting for manual configuration. +- As a Tenant Admin or Tenant User, I want to select a storage tier via StorageClasses when creating persistent volumes, so that I can choose the right performance level for my workload. +- As a Tenant Admin or Tenant User, I want to see whether my CaaS cluster's storage is ready and the reason if it is not, so that I know when I can deploy stateful workloads. +- As a Tenant Admin or Tenant User, I want storage resources on my CaaS cluster to be cleaned up when the cluster is deleted, without affecting my other clusters or backend resources. -When a CaaS cluster is provisioned and ready, the system automatically installs the CSI driver and creates per-tenant, per-tier StorageClasses on that cluster. The tenant's storage backend must be fully provisioned before this happens. No manual intervention is required from any persona. +### Cloud Provider Admin -The system determines which tiers and backends are available for the tenant. If the Tier API or StorageBackend API are deployed, they are used. Otherwise the system falls back to the existing tier and backend configuration. +- As a Cloud Provider Admin, I want to see storage readiness across all tenant clusters, so that I can identify and troubleshoot failures. -### Storage Readiness Visibility +## Assumptions -As a Cloud Provider Admin, I can see the storage readiness of each tenant's CaaS clusters at the tenant level, so I can monitor onboarding progress and troubleshoot failures across tenants. +- Storage backend provisioning is complete before a CaaS cluster becomes ready. +- CaaS cluster nodes have network reachability to the storage backend. -As a Tenant Admin or Tenant User, I can see whether my specific CaaS cluster's storage is ready, so I know when I can start creating persistent volumes. If storage setup failed, I can see the reason. +## Dependencies -### Cluster Deletion and Cleanup - -When a CaaS cluster is deleted, storage resources installed on that cluster are cleaned up. Backend resources shared across the tenant's clusters are not affected, because other clusters may still use them. - -### Multi-Cluster Independence - -A tenant can have multiple CaaS clusters. Each cluster's storage is set up and torn down independently. Adding or removing a CaaS cluster does not affect storage on other clusters belonging to the same tenant. - -### 3.1 Operational Expectations - -- Storage should be available shortly after the cluster is ready. -- Credentials and secrets are never exposed in user-visible error messages, status fields, or events. - -## 4. Acceptance Criteria - -- [ ] A tenant user can create a PVC on a newly provisioned CaaS cluster without manual storage configuration -- [ ] A Cloud Provider Admin can see storage readiness status for each tenant's CaaS clusters -- [ ] A Tenant Admin or User can see whether their specific CaaS cluster's storage is ready and the reason if it failed -- [ ] When a CaaS cluster is deleted, storage resources on that cluster are cleaned up -- [ ] Deleting one CaaS cluster does not affect storage on other clusters belonging to the same tenant -- [ ] Backend resources are unaffected by CaaS cluster deletion -- [ ] A second CaaS cluster for the same tenant gets independent storage setup - -## 5. Assumptions - -- Storage backend provisioning is always complete before a CaaS cluster reaches ready. The system does not handle a ready cluster with an unprovisioned backend. -- The storage provider's automation accepts CaaS clusters as a target and uses the provided cluster credentials. This is delivered by OSAC-1122. -- VAST is the only storage provider for v0.1. -- Cluster credentials are obtainable from the cluster provisioning infrastructure. Confirmed by the CaaS team. -- CaaS cluster nodes have network reachability to the storage backend. This is an infrastructure prerequisite managed by the Cloud Infrastructure Admin. - -## 6. Dependencies - -- **Tenant Storage Onboarding (OSAC-23):** Merged. Provides the storage automation framework and independent per-cluster setup and teardown actions that this PRD extends for CaaS clusters. -- **VAST for CaaS (OSAC-1122):** In progress. The storage provider must support CaaS clusters as a target. Required for end-to-end functionality. -- **Tier API (OSAC-1110):** In progress. Not blocking. The system integrates when available, falls back to existing configuration. -- **StorageBackend API (OSAC-1111):** In progress. Not blocking. The system integrates when available, falls back to single implicit backend. +- **Tenant Storage Onboarding:** Provides the storage automation framework that CaaS cluster storage builds on. +- **Storage provider CaaS support:** The storage provider must accept CaaS clusters as a target. Required for end-to-end functionality. From 5e43d86c64b5b3adfabe98670a4632a34f29bee3 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Thu, 25 Jun 2026 23:45:53 -0400 Subject: [PATCH 056/450] OSAC-1123: design document for CaaS cluster storage Extend the OSAC Storage Controller to provision persistent storage on CaaS tenant clusters. When a ClusterOrder reaches Phase=Ready, the controller retrieves the cluster's kubeconfig via the HyperShift HostedControlPlane API, triggers AAP to install the CSI driver and per-tenant StorageClasses, and tracks readiness as a ClusterStorageReady condition on the ClusterOrder CR. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- enhancements/caas-cluster-storage/design.md | 360 ++++++++++++++++++++ 1 file changed, 360 insertions(+) create mode 100644 enhancements/caas-cluster-storage/design.md diff --git a/enhancements/caas-cluster-storage/design.md b/enhancements/caas-cluster-storage/design.md new file mode 100644 index 000000000..1ed6ccb5f --- /dev/null +++ b/enhancements/caas-cluster-storage/design.md @@ -0,0 +1,360 @@ +--- +title: caas-cluster-storage +authors: + - anadkarn@redhat.com +creation-date: 2026-06-25 +last-updated: 2026-06-25 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-1123 +prd: + - "prd.md" +see-also: + - "/enhancements/tenant-storage-onboarding-OSAC-23" +replaces: + - N/A +superseded-by: + - N/A +--- + +# CaaS Cluster Storage + +## Summary + +Extend the OSAC Storage Controller to provision persistent storage on CaaS tenant clusters. When a ClusterOrder reaches `Phase=Ready`, the storage controller retrieves the cluster's kubeconfig via the HyperShift HostedControlPlane API, triggers AAP to install the CSI driver and per-tenant StorageClasses, and tracks readiness as a `ClusterStorageReady` condition on the ClusterOrder CR. See [PRD](prd.md) for detailed requirements. + +## Motivation + +The OSAC Storage Controller ([OSAC-1001](https://redhat.atlassian.net/browse/OSAC-1001)) provisions storage for tenants on a single preconfigured VMaaS target cluster in two stages: backend setup (Stage 1: VAST tenant, views, quotas, credentials) and cluster-side setup (Stage 2: CSI driver, per-tenant StorageClasses). Both stages are triggered when a Tenant CR reaches `Phase=Ready`. + +CaaS clusters require three things the current architecture does not provide: + +1. **Multiple target clusters.** Each ClusterOrder has its own hosted control plane. The controller must obtain a kubeconfig dynamically rather than using a preconfigured target. + +2. **Different trigger.** CaaS Stage 2 is triggered by `ClusterOrder.Phase=Ready`, not `Tenant.Phase=Ready`. Stage 1 has already completed during tenant onboarding. + +3. **Per-cluster visibility.** Storage readiness must be visible on each ClusterOrder, not only aggregated on the Tenant CR. + +### Goals + +- Extend the existing storage controller rather than introducing a new controller or duplicating provisioning logic. +- Retrieve the CaaS cluster kubeconfig from `ClusterOrder.status.clusterReference` via the HyperShift HostedControlPlane API. +- Track per-ClusterOrder storage readiness as a separate condition from the cluster's compute readiness (`Phase`). +- Keep VMaaS storage provisioning unchanged. + +### Non-Goals + +- StorageTier API integration ([OSAC-1110](https://redhat.atlassian.net/browse/OSAC-1110), separate epic). +- VAST provider CaaS changes ([OSAC-1122](https://redhat.atlassian.net/browse/OSAC-1122), separate epic). +- Dynamic tier addition to running CaaS clusters (beyond v0.1). +- Storage UI ([OSAC-1252](https://redhat.atlassian.net/browse/OSAC-1252), backlog). +- Dedicated AAP ServiceAccount with scoped RBAC for storage operations ([OSAC-499](https://redhat.atlassian.net/browse/OSAC-499), stretch goal). The AAP `storage-operations-ig` currently uses the shared `osac-sa`. + +## Proposal + +1. **Extend the Storage Controller** to provision and tear down cluster-side storage on CaaS clusters, triggered by ClusterOrder lifecycle events. The controller retrieves the CaaS cluster kubeconfig and passes it to AAP as an extra variable. + +2. **Update AAP storage roles** to pass the resolved kubeconfig to all `kubernetes.core.k8s` calls, so StorageClasses and CSI resources are created on the CaaS cluster instead of the hub. Kubeconfig acceptance and resolution in the playbooks is handled by [OSAC-1327](https://redhat.atlassian.net/browse/OSAC-1327). + +No new controllers, CRDs, or fulfillment-service changes are required. + +### Changes Per Repository + +| Repository | Changes | +|---|---| +| **osac-operator** | Extend storage controller (kubeconfig retrieval, `mapClusterOrderToTenant`, CaaS reconciliation loop, teardown). Add `ClusterStorageReady` condition and `osac.openshift.io/cluster-storage` finalizer to ClusterOrder CRD. Expand RBAC for ClusterOrder status/finalizers and HostedControlPlane read access. | +| **osac-aap** | Add `kubeconfig` parameter to all `kubernetes.core.k8s` calls in `ensure_storage_class.yaml` and `teardown_cluster_storage.yaml`. Kubeconfig resolution and `vmaas`-only guard removal are handled by [OSAC-1327](https://redhat.atlassian.net/browse/OSAC-1327) (PR [#377](https://github.com/osac-project/osac-aap/pull/377)). | +| **fulfillment-service** | None. The `osac.openshift.io/tenant` annotation is already set on ClusterOrder CRs by the cluster reconciler. | +| **osac-installer** | Update RBAC manifests to reflect the storage controller's expanded permissions. | + +### Workflow Description + +#### Personas + +- **Cloud Provider Admin:** Registers storage backends, creates Tenant CRs, monitors storage readiness. +- **Tenant Admin / Tenant User:** Orders CaaS clusters, consumes persistent storage via PVCs. + +#### Components + +- **Storage Controller (osac-operator):** Reconciles storage state on Tenant and ClusterOrder CRs. +- **AAP (osac-aap):** Executes cluster-side storage provisioning and teardown playbooks. + +#### Prerequisites + +1. Tenant CR has `StorageBackendReady=True` (Stage 1 completed during tenant onboarding). +2. ClusterOrder has `osac.openshift.io/tenant=` annotation and `Phase=Ready`. +3. ClusterOrder's `status.clusterReference` points to a HostedCluster with a populated `status.kubeConfig`. + +#### CaaS Storage Provisioning + +```mermaid +sequenceDiagram + participant FS as Fulfillment Service + participant SC as Storage Controller + participant CO as ClusterOrder CR + participant HC as HostedControlPlane + participant AAP as AAP + participant CaaS as CaaS Cluster + + FS->>CO: Create ClusterOrder with osac.openshift.io/tenant annotation + Note over CO: ClusterOrder progresses to Phase=Ready + + SC->>SC: ClusterOrder watch fires, maps to Tenant via annotation + SC->>CO: Read ClusterOrder, check Phase=Ready + SC->>SC: Verify Tenant has StorageBackendReady=True + + SC->>HC: Read HostedControlPlane via clusterReference + SC->>HC: Read kubeconfig Secret from status.kubeConfig + + SC->>AAP: Trigger osac-create-tenant-cluster-storage (with kubeconfig) + AAP->>CaaS: Install CSI operator (if needed) + AAP->>CaaS: Create per-tenant StorageClasses and CSI Secret + SC->>SC: Poll job until Succeeded + + SC->>CaaS: Discover StorageClasses by osac.openshift.io/tenant label + SC->>CO: Set ClusterStorageReady=True condition + SC->>SC: Update Tenant status.clusterStorage with CaaS cluster entry +``` + +When a ClusterOrder changes, the storage controller looks up the owning Tenant via the `osac.openshift.io/tenant` annotation. It then lists all Ready ClusterOrders for that tenant that don't yet have `ClusterStorageReady=True`, retrieves each cluster's kubeconfig, triggers the AAP provisioning job, and sets the condition on success. + +#### CaaS Storage Teardown + +```mermaid +sequenceDiagram + participant SC as Storage Controller + participant CO as ClusterOrder CR + participant HC as HostedControlPlane + participant AAP as AAP + participant CaaS as CaaS Cluster + + Note over CO: ClusterOrder DeletionTimestamp set + + SC->>SC: ClusterOrder watch fires, maps to Tenant + SC->>CO: Read ClusterOrder, check DeletionTimestamp + SC->>HC: Read kubeconfig from HostedControlPlane + + SC->>AAP: Trigger osac-delete-tenant-cluster-storage (with kubeconfig) + AAP->>CaaS: Delete StorageClasses, VolumeSnapshotClasses, CSI Secrets + SC->>SC: Poll job until Succeeded + + SC->>SC: Remove CaaS cluster entry from Tenant status.clusterStorage + SC->>CO: Remove storage finalizer from ClusterOrder +``` + +The storage finalizer (`osac.openshift.io/cluster-storage`) blocks ClusterOrder deletion until cleanup completes. The controller triggers the teardown AAP job, polls until completion, removes the Tenant's `status.clusterStorage` entry, then removes the finalizer. + +If the HostedControlPlane or kubeconfig Secret is already gone (the HostedCluster was deleted outside of OSAC's control), the CaaS cluster no longer exists and there are no resources to clean up. The controller logs a warning, removes the `status.clusterStorage` entry, and removes the finalizer. + +Backend teardown (Stage 1 reverse) is not triggered by ClusterOrder deletion. Backend resources are shared across a tenant's clusters and only torn down when the Tenant itself is deleted. + +Error handling follows the existing OSAC controller pattern: set the condition to `False` with a descriptive reason, and wait for a change to the ClusterOrder or related resources before retrying. See [Failure Handling and Recovery](#failure-handling-and-recovery) for the full failure mode table. + +### API Extensions + +#### ClusterOrder CRD + +New condition: + +```go +ClusterOrderConditionClusterStorageReady ClusterOrderConditionType = "ClusterStorageReady" +``` + +Owned by the storage controller. A ClusterOrder can reach `Phase=Ready` without `ClusterStorageReady=True`; compute and storage readiness are independent. + +New finalizer: `osac.openshift.io/cluster-storage`. Added when provisioning begins, removed after teardown. + +`ClusterStorageJobs []JobStatus` already exists in `ClusterOrderStatus`. + +#### Tenant CRD + +No schema changes. The existing `status.clusterStorage []ClusterStorageStatus` list (keyed by `clusterName`) accommodates CaaS entries alongside VMaaS. For CaaS, `clusterName` is the ClusterOrder name. + +#### RBAC + +New permissions for the storage controller's ClusterRole: + +```go +// +kubebuilder:rbac:groups=osac.openshift.io,resources=clusterorders/status,verbs=get;update;patch +// +kubebuilder:rbac:groups=osac.openshift.io,resources=clusterorders/finalizers,verbs=update +// +kubebuilder:rbac:groups=hypershift.openshift.io,resources=hostedcontrolplanes,verbs=get;list;watch +``` + +These are added to the existing `osac-operator-controller-manager` ClusterRole. + +If the storage controller is down, ClusterOrders still reach `Phase=Ready`. The `ClusterStorageReady` condition remains stale until the controller recovers. Deletion is blocked by the finalizer. + +### Implementation Details/Notes/Constraints + +#### Kubeconfig Retrieval + +Three-step lookup: + +1. `ClusterOrder.status.clusterReference` provides the HostedCluster namespace and name. +2. Read the `HostedControlPlane` resource in that namespace (`hypershift.openshift.io/v1beta1`). +3. Read the Secret from `HostedControlPlane.status.kubeConfig` (name and key). + +The kubeconfig is used to construct a `client.Client` for StorageClass discovery and passed to AAP for CSI driver and StorageClass installation. + +#### Reconciliation Flow + +The storage controller already watches ClusterOrders via `mapClusterOrderToTenant`, but today that function is a stub returning nil. It will read the `osac.openshift.io/tenant` annotation from the ClusterOrder and look up the owning Tenant (this annotation is already set by the fulfillment-service cluster reconciler). + +`handleUpdate` extends with a CaaS loop after the existing VMaaS logic. For each Ready ClusterOrder without `ClusterStorageReady=True`: + +1. Add the `osac.openshift.io/cluster-storage` finalizer. +2. Retrieve the kubeconfig. +3. Trigger or poll the AAP job (using `ClusterOrder.Status.ClusterStorageJobs`). +4. On success, discover StorageClasses on the CaaS cluster and set the condition. +5. Update the Tenant's `status.clusterStorage`. + +Any ClusterOrder with a `DeletionTimestamp` and the storage finalizer triggers teardown instead. + +#### AAP Changes: Target Cluster Kubeconfig + +For VMaaS, storage roles run K8s calls against the AAP pod's default cluster context. For CaaS, each cluster has a different kubeconfig that must be passed per-job. + +[OSAC-1327](https://redhat.atlassian.net/browse/OSAC-1327) (PR [#377](https://github.com/osac-project/osac-aap/pull/377), WIP) handles kubeconfig resolution: accepts `ansible_eda.event.admin_kubeconfig` as an extra variable, writes it to a temp file, removes the `vmaas`-only guards, and adds cluster name discovery. However, the `kubernetes.core.k8s` calls in `ensure_storage_class.yaml` and `teardown_cluster_storage.yaml` still don't pass the resolved kubeconfig, so StorageClasses and CSI resources would be created on the hub cluster instead of the CaaS cluster. This design adds `kubeconfig: "{{ _remote_kubeconfig | default(omit) }}"` to those K8s calls. + +On the operator side, the storage controller retrieves the kubeconfig from the HostedControlPlane Secret and injects it as `admin_kubeconfig` in the `ansible_eda.event` payload (matching PR #377's convention). This uses the existing `extra_vars_context.go` pattern with a new `WithAdminKubeconfig` function. + +#### StorageClass Properties + +CaaS clusters use the same StorageClass properties as VMaaS (same AAP role, same values): + +| Property | NFS | Block | +|---|---|---| +| `reclaimPolicy` | `Delete` | `Delete` | +| `volumeBindingMode` | `Immediate` | `WaitForFirstConsumer` | +| Provisioner | `csi.vastdata.com` | `blockcsi.vastdata.com` | + +For NFS, `Delete` means the backing VAST view/export is removed when the PVC is deleted. For block, the backing volume is removed. This is consistent across VMaaS and CaaS. + +Labels on all StorageClasses: +- `osac.openshift.io/tenant=` +- `osac.openshift.io/storage-tier=` +- `osac.openshift.io/storage-protocol=` +- `app.kubernetes.io/managed-by=osac-aap` + +The tier resolution algorithm (`storage_tier_resolution.go`) is reused without modification. The only difference is which cluster it runs against. + +#### Extension Point: Tier API + +When the Tier API ([OSAC-1110](https://redhat.atlassian.net/browse/OSAC-1110)) is available, the storage controller will query it for available tiers and pass them to AAP as extra variables. The tier resolution algorithm on the target cluster remains unchanged. + +### Security Considerations + +CaaS cluster storage inherits the existing security model: + +- **Kubeconfig handling:** The kubeconfig Secret is read from the HostedCluster namespace on the hub cluster and passed to AAP without being persisted. The storage controller's RBAC permissions ([API Extensions](#rbac)) grant it read access to HostedControlPlane resources and Secrets in those namespaces. +- **Tenant isolation:** StorageClasses are scoped to tenants via the `osac.openshift.io/tenant` label, and the tier resolution algorithm only returns StorageClasses matching the tenant. CSI credentials are scoped per-tenant via VAST RBAC Realms ([OSAC-1326](https://redhat.atlassian.net/browse/OSAC-1326)). +- **API-level authorization:** No new Open Policy Agent (OPA) authorization policies are needed in fulfillment-service, because storage provisioning is triggered by the platform (storage controller), not by tenant API calls. + +### Failure Handling and Recovery + +| Failure Mode | Behavior | Recovery | User Observes | +|---|---|---|---| +| Kubeconfig not available | `ClusterStorageReady=False`, reason `KubeConfigNotAvailable` | Automatic when HostedControlPlane populates `status.kubeConfig`. | ClusterOrder Ready but `ClusterStorageReady=False`. | +| AAP provisioning fails | `ClusterStorageReady=False`, reason `ProvisionFailed` | Retries when the ClusterOrder or a related resource changes. | `kubectl describe` shows job failure message. | +| AAP teardown fails | Finalizer remains. | Retries on next reconciliation trigger. | ClusterOrder stuck in Deleting. | +| HostedControlPlane gone during teardown | No resources to clean up. | Controller removes `status.clusterStorage` entry and finalizer. | ClusterOrder deletion proceeds. Warning event logged. | +| Storage controller down | Conditions stale. Phase=Ready unaffected. | Automatic on restart; the controller re-evaluates all Tenants and ClusterOrders. | New clusters show Ready without storage condition until recovery. | +| Duplicate StorageClasses per tier | Warning event, `ClusterStorageReady=False`, reason `MultipleFound` | Admin resolves duplicates on the CaaS cluster. | Warning event on Tenant CR. | + +### RBAC / Tenancy + +No tenant-facing RBAC changes. Storage provisioning is platform-managed; tenants consume storage through standard PVC APIs. Tenant isolation is enforced by label-based filtering in the tier resolution algorithm. Controller permissions are described under [RBAC in API Extensions](#rbac). + +### Observability and Monitoring + +New Kubernetes events on ClusterOrder: + +| Event Type | Reason | When | +|---|---|---| +| Normal | `ClusterStorageProvisioned` | Provisioning succeeded and StorageClasses discovered. | +| Warning | `ClusterStorageProvisionFailed` | AAP provisioning job failed. | +| Warning | `KubeConfigNotAvailable` | Cannot retrieve kubeconfig for the CaaS cluster. | +| Warning | `DuplicateStorageClass` | Multiple StorageClasses for the same tier. | + +Structured log entries include `clusterOrder`, `tenant`, and `clusterName` fields. + +### Risks and Mitigations + +| Risk | Mitigation | +|---|---| +| Kubeconfig Secret rotation between provisioning and discovery | Controller reads kubeconfig fresh on each reconciliation. | +| ClusterOrder count per tenant grows large | Acceptable for v0.1 (single-digit expected). Can split to a separate controller later. | +| ClusterOrder deleted while provisioning is in progress | Finalizer prevents premature deletion. Controller transitions to teardown on next reconciliation. | +| HyperShift API changes break kubeconfig path | Import HyperShift API types as a Go module for compile-time checking. | + +### Drawbacks + +The storage controller now writes conditions and finalizers on both Tenant and ClusterOrder CRs, increasing its surface area. A separate CaaS storage controller would be simpler per-controller but would duplicate provisioning logic and introduce dual-writer conflicts on Tenant status. + +## Alternatives (Not Implemented) + +### 1. Separate CaaSStorageReconciler + +A standalone controller reconciling ClusterOrders directly. + +**Pros:** Clean separation of VMaaS and CaaS concerns. +**Cons:** Duplicates provisioning lifecycle logic. Two controllers writing to Tenant `status.clusterStorage` creates ordering issues. +**Rejected:** The provisioning logic (AAP job triggering, polling, status updates, finalizer management) is identical for VMaaS and CaaS. Keeping it in one controller avoids maintaining two copies and coordinating two writers on Tenant status. + +### 2. ClusterOrder controller owns storage provisioning + +The ClusterOrder controller triggers storage directly when a cluster reaches Ready. + +**Pros:** No cross-resource condition writes. +**Cons:** Mixes storage into the cluster controller. Reverses the separation established by [OSAC-1001](https://redhat.atlassian.net/browse/OSAC-1001). +**Rejected:** Storage logic belongs in the storage controller. + +### 3. Storage readiness gates ClusterOrder Phase=Ready + +Require storage before a ClusterOrder can reach Ready. + +**Pros:** Tenants never see a Ready cluster without storage. +**Cons:** Blocks compute availability on storage provisioning. Unnecessarily restrictive for stateless workloads. +**Rejected:** Compute and storage readiness are independent concerns. The `ClusterStorageReady` condition provides visibility without coupling. + +## Test Plan + +### Unit Tests + +Controller tests using envtest for the Kubernetes API and mock provisioning providers for AAP, following the existing pattern in `storage_controller_test.go`: + +- `mapClusterOrderToTenant`: Tenant lookup from annotation. Cover missing, empty, and non-existent Tenant cases. +- Kubeconfig retrieval: Cover missing `clusterReference`, missing HostedControlPlane, missing Secret, and missing key. +- CaaS provisioning: Create Tenant + ClusterOrder, verify provisioning triggers when both are ready. Verify condition setting on ClusterOrder, Tenant `status.clusterStorage` updates, and finalizer management. +- CaaS teardown: Delete a ClusterOrder, verify teardown completes before finalizer is removed. Cover the case where HostedControlPlane is already gone. +- Duplicate StorageClass detection emits warning events. +- VMaaS regression: Verify VMaaS flow is unchanged when CaaS clusters exist. + +### E2E Tests + +Via osac-test-infra ([OSAC-1329](https://redhat.atlassian.net/browse/OSAC-1329)): +- Provision a CaaS cluster, verify StorageClasses are installed and PVCs work. +- Delete a CaaS cluster, verify storage resources are cleaned up. +- Tenant isolation: StorageClasses from one tenant are not visible on another tenant's cluster. + +## Graduation Criteria + +N/A. OSAC is in active development and has not been released to customers. + +## Upgrade / Downgrade Strategy + +Pre-GA change. This enhancement adds a new condition (`ClusterStorageReady`) and finalizer to ClusterOrder. No existing fields are modified, no data migration needed. + +## Version Skew Strategy + +No fulfillment-service changes are needed. The storage controller and AAP roles can be deployed independently. If the operator is deployed before the AAP changes, the controller triggers templates that don't exist yet; AAP returns "template not found", the controller records a failed job, and retries once AAP is updated. + +## Support Procedures + +- `kubectl get clusterorder -o wide` shows `ClusterStorageReady` (priority=1 column). +- `kubectl describe clusterorder ` shows conditions, events, and `ClusterStorageJobs` history. +- `kubectl get tenant -o wide` shows aggregate storage status. +- Controller logs: filter by `clusterOrder=`. + +## Infrastructure Needed + +None. From 125e1a9600864851b9a1d0cd91cce71736f6ac30 Mon Sep 17 00:00:00 2001 From: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Date: Fri, 26 Jun 2026 00:52:59 -0400 Subject: [PATCH 057/450] OSAC-1123: address CodeRabbit review feedback Add Secrets RBAC for kubeconfig retrieval via hub-access-hosted-clusters ClusterRole. Clarify kubeconfig security handling with no_log. Update kubeconfig rotation risk to cover mid-job rotation. Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> Assisted-by: Cursor/Claude Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com> --- enhancements/caas-cluster-storage/design.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/enhancements/caas-cluster-storage/design.md b/enhancements/caas-cluster-storage/design.md index 1ed6ccb5f..12f8f739b 100644 --- a/enhancements/caas-cluster-storage/design.md +++ b/enhancements/caas-cluster-storage/design.md @@ -176,11 +176,15 @@ New permissions for the storage controller's ClusterRole: ```go // +kubebuilder:rbac:groups=osac.openshift.io,resources=clusterorders/status,verbs=get;update;patch // +kubebuilder:rbac:groups=osac.openshift.io,resources=clusterorders/finalizers,verbs=update -// +kubebuilder:rbac:groups=hypershift.openshift.io,resources=hostedcontrolplanes,verbs=get;list;watch ``` These are added to the existing `osac-operator-controller-manager` ClusterRole. +To read the kubeconfig, the controller needs access to `hostedcontrolplanes` and `secrets` in HostedCluster namespaces on the hub cluster. Today, the `hub-access-hosted-clusters` ClusterRole (in osac-installer) grants `get` on `hostedclusters` and `secrets`, and the ClusterOrder controller binds it per namespace. Two changes needed: + +1. Add `hostedcontrolplanes: get` to the `hub-access-hosted-clusters` ClusterRole. +2. Add the storage controller's ServiceAccount to the per-namespace RoleBinding. + If the storage controller is down, ClusterOrders still reach `Phase=Ready`. The `ClusterStorageReady` condition remains stale until the controller recovers. Deletion is blocked by the finalizer. ### Implementation Details/Notes/Constraints @@ -245,7 +249,7 @@ When the Tier API ([OSAC-1110](https://redhat.atlassian.net/browse/OSAC-1110)) i CaaS cluster storage inherits the existing security model: -- **Kubeconfig handling:** The kubeconfig Secret is read from the HostedCluster namespace on the hub cluster and passed to AAP without being persisted. The storage controller's RBAC permissions ([API Extensions](#rbac)) grant it read access to HostedControlPlane resources and Secrets in those namespaces. +- **Kubeconfig handling:** The storage controller reads the kubeconfig Secret from the hub cluster and passes it to AAP as an extra variable. It is not stored anywhere else. On the AAP side, playbook tasks that handle the kubeconfig use `no_log: true` so it does not appear in job output or logs. - **Tenant isolation:** StorageClasses are scoped to tenants via the `osac.openshift.io/tenant` label, and the tier resolution algorithm only returns StorageClasses matching the tenant. CSI credentials are scoped per-tenant via VAST RBAC Realms ([OSAC-1326](https://redhat.atlassian.net/browse/OSAC-1326)). - **API-level authorization:** No new Open Policy Agent (OPA) authorization policies are needed in fulfillment-service, because storage provisioning is triggered by the platform (storage controller), not by tenant API calls. @@ -281,7 +285,7 @@ Structured log entries include `clusterOrder`, `tenant`, and `clusterName` field | Risk | Mitigation | |---|---| -| Kubeconfig Secret rotation between provisioning and discovery | Controller reads kubeconfig fresh on each reconciliation. | +| Kubeconfig Secret rotates while an AAP job is running | The job fails with an authentication error (`ProvisionFailed`). The controller reads the new kubeconfig on the next reconciliation and retries. | | ClusterOrder count per tenant grows large | Acceptable for v0.1 (single-digit expected). Can split to a separate controller later. | | ClusterOrder deleted while provisioning is in progress | Finalizer prevents premature deletion. Controller transitions to teardown on next reconciliation. | | HyperShift API changes break kubeconfig path | Import HyperShift API types as a Go module for compile-time checking. | From e3a7994faa739f490374d759cb4ff724b1ed1353 Mon Sep 17 00:00:00 2001 From: Ilya Skornyakov Date: Sun, 28 Jun 2026 16:16:56 +0300 Subject: [PATCH 058/450] OSAC-1269: clarify admin/user visibility, lifecycle surfacing, and validation gaps Assisted-by: Claude Code Signed-off-by: Ilya Skornyakov --- enhancements/cluster-version-api/prd.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/enhancements/cluster-version-api/prd.md b/enhancements/cluster-version-api/prd.md index a9f961750..a4ae6c6b7 100644 --- a/enhancements/cluster-version-api/prd.md +++ b/enhancements/cluster-version-api/prd.md @@ -36,23 +36,23 @@ A ClusterVersion resource eliminates these workarounds. Users select from a cura #### Version Catalog Management -- **FR-1:** Each version in the catalog is identified by its version number (e.g., "4.17.0"), maps to a release image, has a lifecycle state (active, deprecated, or obsolete), can optionally be marked as the default, and is associated with update channels (e.g., "stable-4.17", "fast-4.17"). Channels are informational metadata in v0.2; channel-based selection and propagation are deferred to OSAC-1415. [Clarify: R1.Q1, R1.Q8] +- **FR-1:** Admins can create, update, and delete version catalog entries. Each entry requires a version number (e.g., "4.17.0") and release image URL; lifecycle state, channels (e.g., "stable-4.17", "fast-4.17"), and default flag are optional. Channels are informational metadata in v0.2; channel-based selection and propagation are deferred to OSAC-1415. [Clarify: R1.Q1, R1.Q6, R1.Q8, R3.Q1] -- **FR-2:** Admins can create, update, and delete catalog entries. Users can list versions filtered to active and deprecated entries by default; obsolete versions are hidden from the default list but visible via explicit filter. Viewing a specific version by identifier returns it regardless of its lifecycle state, so users can inspect versions referenced by their existing clusters. [Clarify: R1.Q6, R3.Q1] +- **FR-2:** Users can browse and view available versions. Release image URLs are not exposed to users — they see version number, lifecycle state, and channels. Listings return active and deprecated versions by default; obsolete versions require an explicit filter. A specific version can be viewed regardless of its state, so users can inspect versions referenced by their existing clusters. [Clarify: R1.Q6, R3.Q1] -- **FR-3:** At most one version can be marked as default. The default version is used when no version is specified by the user or template. [Clarify: R1.Q1] +- **FR-3:** At most one version can be marked as default. An obsolete version cannot be marked as default. The default is used when no version is specified by the user or template. [Clarify: R1.Q1] #### Cluster Creation -- **FR-4:** Users specify a version number (e.g., "4.17.0") when creating a cluster. Release image URLs are not visible in cluster creation or listing responses — only the version is exposed. [Clarify: R1.Q2, R1.Q5, R2.Q1, R2.Q4, R3.Q3, R3.Q4] +- **FR-4:** Users specify a version number (e.g., "4.17.0") when creating a cluster. [Clarify: R1.Q2, R1.Q5, R2.Q1, R2.Q4, R3.Q3, R3.Q4] - **FR-5:** Templates can specify a default version (e.g., "4.17.0"). Any version can be used with any template; templates provide defaults but do not constrain version selection. [Clarify: R2.Q1] -- **FR-6:** A cluster's version is visible in responses when viewing or listing clusters. +- **FR-6:** A cluster's version and its current lifecycle state are visible when viewing or listing clusters. If the version is deprecated or obsolete, the state is surfaced so users can identify clusters that need attention. #### Validation -- **FR-7:** Version is validated at creation time with descriptive error messages that include the invalid value and valid alternatives. Validation covers: version not found and version obsolete. Creating a cluster with a deprecated version succeeds but includes a warning identifying the replacement version, if one is set. [Clarify: R2.Q3, R3.Q6] +- **FR-7:** Version is validated at creation time with descriptive error messages. Validation covers: version not found, version obsolete, and no version resolvable (no explicit version, no template default, and no system default). Creating a cluster with a deprecated version succeeds but includes a warning identifying the replacement version, if one is set. [Clarify: R2.Q3, R3.Q6] #### User Interfaces @@ -82,11 +82,11 @@ A ClusterVersion resource eliminates these workarounds. Users select from a cura ## 4. Acceptance Criteria -- [ ] A user can create a cluster by specifying a version number (e.g., 4.17.0) instead of a release image URL. The server resolves the version to the correct release image internally. Release image URLs are not visible in cluster creation or listing responses. Version is immutable after creation. +- [ ] A user can create a cluster by specifying a version number (e.g., 4.17.0) instead of a release image URL. The server resolves the version to the correct release image internally. Release image URLs are not exposed to users — neither in the version catalog nor in cluster responses. Version is immutable after creation. The version's current lifecycle state is visible when viewing or listing clusters. - [ ] Specifying a non-existent, deleted, or obsolete version is rejected with a descriptive error indicating the reason and identifying the replacement if one is set. Deprecated versions allow creation with a warning. Validation applies to both cluster creation and template defaults. - [ ] Admins can create, update, and delete version catalog entries. Deleting a version referenced by an active cluster or template defaults is rejected with a message identifying the referencing resource. - [ ] Admins can transition a version between active, deprecated, and obsolete in any direction, even when referenced by active clusters or templates. Listing versions returns active and deprecated entries by default; obsolete versions are hidden unless explicitly filtered. Deprecation and obsolescence timestamps are recorded automatically on each transition. -- [ ] At most one version is marked as default at any time — setting a new default clears the previous one. When a user omits the version and template defaults do not supply one, the server uses the default version. Templates can specify a default version, and the server resolves it to a release image. +- [ ] At most one version is marked as default at any time — setting a new default clears the previous one. An obsolete version cannot be marked as default. When a user omits the version and template defaults do not supply one, the server uses the default version. Templates can specify a default version, and the server resolves it to a release image. - [ ] The CLI supports version catalog management and cluster creation with a version option. Catalog items can reference version in their field definitions. The UI console supports catalog management for admins and version selection in the cluster creation wizard. ## 5. Assumptions @@ -125,8 +125,8 @@ This feature applies to **CaaS** (Cluster as a Service) only. VMaaS has an analo | Persona | Interaction | |---------|-------------| -| Cloud Provider Admin | Creates and manages the version catalog via CLI, API, and UI. Populates available versions. Sets the default version. | -| Tenant User | Selects a version when creating a cluster via CLI, UI wizard, or catalog items. Discovers available versions via the API and UI. | +| Cloud Provider Admin | Creates version entries by providing version number, release image URL, channels, and lifecycle state. Manages the catalog via CLI, API, and UI. Sets the default version. | +| Tenant User | Browses available versions (version number, state, channels — no release image URLs). Selects a version when creating a cluster via CLI, UI wizard, or catalog items. | | Cloud Infrastructure Admin | Not affected. | | Tenant Admin | Same as Tenant User. May also reference versions when authoring org-specific catalog items. | From 748b8b9439a040b82a3fbeea22acf40cc31b143e Mon Sep 17 00:00:00 2001 From: Ilya Skornyakov Date: Sun, 28 Jun 2026 16:20:05 +0300 Subject: [PATCH 059/450] =?UTF-8?q?OSAC-1269:=20tighten=20problem=20statem?= =?UTF-8?q?ent=20and=20goals=20=E2=80=94=20remove=20implementation=20detai?= =?UTF-8?q?ls?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Assisted-by: Claude Code Signed-off-by: Ilya Skornyakov --- enhancements/cluster-version-api/prd.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/enhancements/cluster-version-api/prd.md b/enhancements/cluster-version-api/prd.md index a4ae6c6b7..91ceef970 100644 --- a/enhancements/cluster-version-api/prd.md +++ b/enhancements/cluster-version-api/prd.md @@ -10,15 +10,15 @@ The Cluster API requires users to provide a full OCI release image URL (e.g., `quay.io/openshift-release-dev/ocp-release:4.17.0-multi`) when creating a cluster. This leaks infrastructure implementation details into the user-facing API: users must know the exact registry path and tag format, typos are caught only at provisioning time rather than at API validation, and there is no way to discover which versions are available. -The cost compounds across layers. Catalog items resort to regex patterns to constrain the image URL prefix — a workaround that cannot verify the version actually exists or is supported. Multiple components independently construct release image URLs from version numbers, duplicating format knowledge with no central validation. Each layer works around the same missing abstraction: a managed catalog of validated cluster versions. +Without a managed version catalog, every layer that needs to validate or constrain versions must do so ad hoc — there is no central source of which versions exist, which are supported, and which are end-of-life. -A ClusterVersion resource eliminates these workarounds. Users select from a curated, queryable list of versions — no URLs, no guessing. Catalog items reference versions by name instead of embedding URL patterns. Automation receives pre-validated release images from the API, removing URL construction. And the version catalog becomes the foundation for future upgrade operations (OSAC-1415) and ACM version synchronization, which require a queryable set of available versions and their upgrade paths. +A managed version catalog provides that missing abstraction. Users select from a curated, queryable list of versions — no URLs, no guessing. It also becomes the foundation for future upgrade operations (OSAC-1415) and ACM version synchronization, which require a queryable set of available versions and their lifecycle states. ## 2. Goals and Non-Goals ### 2.1 Goals -- Users specify an OpenShift version by number (e.g., "4.17.0") instead of a raw OCI URL when creating a cluster, with the server resolving the version to the correct release image internally. +- Users specify an OpenShift version by number (e.g., "4.17.0") instead of a raw OCI release image URL when creating a cluster. - Users receive immediate, descriptive feedback when specifying an invalid, obsolete, or deprecated version — before any provisioning begins. - Cloud Provider Admins manage available cluster versions through the CLI and UI console. - Tenant Users can discover and select from available versions when creating a cluster — in the CLI, UI, and API. From aec49c35616892bc3b0a0571540aef199d18a086 Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Sun, 28 Jun 2026 18:34:31 +0300 Subject: [PATCH 060/450] Add PRD for OSAC-1425: Tenant UI Networking Section Assisted-by: Claude Code Signed-off-by: Elay Aharoni --- enhancements/networking-ui-vmaas-scope/prd.md | 201 ++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 enhancements/networking-ui-vmaas-scope/prd.md diff --git a/enhancements/networking-ui-vmaas-scope/prd.md b/enhancements/networking-ui-vmaas-scope/prd.md new file mode 100644 index 000000000..cfc731802 --- /dev/null +++ b/enhancements/networking-ui-vmaas-scope/prd.md @@ -0,0 +1,201 @@ +# OSAC Tenant UI: Networking Section + +| Field | Value | +|-------------|---------| +| Author(s) | Elay Aharoni, Dan Manor | +| Jira | [OSAC-1425](https://redhat.atlassian.net/browse/OSAC-1425) | +| Date | 2026-06-28 | + +## 1. Problem Statement + +Tenant users and tenant admins currently lack a dedicated UI for managing networking resources (VirtualNetworks, Subnets, SecurityGroups, PublicIPs) in the OSAC platform. While the fulfillment API provides full CRUD capabilities for these resources, users must either use the CLI or construct API calls directly. Additionally, when creating VMs through the VMaaS wizard, users have no integrated way to create or select networking resources inline—they must context-switch to external tools or pre-create resources before provisioning compute instances. This creates friction in the user experience, increases time-to-first-VM for new tenants, and reduces discoverability of networking capabilities. + +## 2. Goals and Non-Goals + +### 2.1 Goals + +- Tenant users can create, list, view, and delete VirtualNetworks through a dedicated Networking section in the OSAC UI sidebar +- Tenant users can manage Subnets and SecurityGroups scoped to a VirtualNetwork from the VirtualNetwork detail page +- Tenant users can allocate and release PublicIPs from provider-managed pools through a dedicated PublicIPs page +- New tenants creating their first VM can provision all required networking resources (VirtualNetwork, Subnet, SecurityGroup) inline within the VM creation wizard without leaving the wizard flow +- Returning tenants can select from existing networking resources in the VM creation wizard with smart defaults (auto-select when only one option exists) + +### 2.2 Non-Goals + +- NATGateway, ExternalIPAttachment, NetworkClass management (these are provider-only or future scope) +- PublicIPPool CRUD operations (pools are provider-managed, read-only for tenants) +- BaremetalInstance networking or Cluster networking (out of scope for VMaaS phase) +- Attaching/detaching PublicIPs to compute instances (covered in a separate initiative) +- Migration of the existing AdminNetworksPage topology view (preserved as-is, enhancements are future work) + +## 3. Requirements + +### 3.1 Functional Requirements + +#### Virtual Networks + +- **FR-1:** The UI must provide a VirtualNetworks list page (`/networking/virtual-networks`) displaying all VirtualNetworks owned by the tenant with columns for Name, Region, IPv4 CIDR, Subnets count, and Status +- **FR-2:** The VirtualNetworks list page must support filtering by Region and Status, sorting by Name/Region/Status, and searching by Name +- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for Name (required, DNS-valid, unique within tenant), Region (required, dropdown), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) +- **FR-4:** The Create VirtualNetwork form must validate inputs inline (show validation errors below each field) and disable the Create button until all required fields are valid +- **FR-5:** After successful VirtualNetwork creation, the UI must close the form, refresh the list, show a success toast notification, and display the new VN with "Provisioning" status +- **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, region, CIDR as key properties, and a Delete action in the header +- **FR-7:** The VirtualNetwork detail page must display three tabs: Subnets (default), Security Groups, and Details +- **FR-8:** The UI must block deletion of a VirtualNetwork if it has subnets or security groups, showing an error message directing the user to delete child resources first + +#### Subnets + +- **FR-9:** The Subnets tab on the VirtualNetwork detail page must display all subnets belonging to that VN in a table with columns for Name, CIDR, Available IPs (used/total), and Status +- **FR-10:** The Subnets tab must provide a "Create subnet" action that opens a side panel form with the parent VN pre-selected and fields for Name (required, DNS-valid) and CIDR (required, must be within parent VN CIDR, must not overlap existing subnets) +- **FR-11:** The Create Subnet form must show the parent VN CIDR as context and display existing subnet CIDRs to help users select a non-overlapping range +- **FR-12:** Clicking on a subnet name must show a detail panel (side drawer or inline expansion) displaying subnet metadata and a list of attached resources (compute instances) +- **FR-13:** Subnets must not have their own top-level sidebar entry—they are managed exclusively from the VirtualNetwork detail page +- **FR-14:** The UI must block deletion of a Subnet if it has attached compute instances, showing an error message directing the user to remove or migrate instances first + +#### Security Groups + +- **FR-15:** The UI must provide a SecurityGroups list page (`/networking/security-groups`) displaying all SecurityGroups across all VirtualNetworks with columns for Name, Virtual Network (link to VN detail), Inbound Rules count, Outbound Rules count, and Status +- **FR-16:** The SecurityGroups list page must support filtering by Virtual Network and Status, and searching by Name +- **FR-17:** The UI must provide a "Create security group" action (available from both the SG list page and the VN detail Security Groups tab) that opens a side panel form with fields for Virtual Network (required dropdown, pre-selected if triggered from VN detail), Name (required, DNS-valid), and expandable Inbound/Outbound Rules sections +- **FR-18:** The Create SecurityGroup form must allow users to add multiple inbound and outbound rules inline, with each rule having fields for Protocol (dropdown: TCP/UDP/ICMP/All), Port Range (text input, disabled for ICMP), Source/Destination CIDR (text with CIDR validation), and Description (optional) +- **FR-19:** The UI must provide a SecurityGroup detail page (`/networking/security-groups/:id`) with tabs for Inbound Rules, Outbound Rules, and Details +- **FR-20:** The Inbound Rules and Outbound Rules tabs must display rules in a table (columns: Priority, Protocol, Port Range, Source/Destination CIDR, Description) with "Add Rule" action and row-level Edit/Delete actions, ordered by priority +- **FR-21:** SecurityGroups must be accessible from both the sidebar (top-level) and the VirtualNetwork detail page (Pattern 3: top-level but VN-scoped) +- **FR-22:** The UI must block deletion of a SecurityGroup if it is attached to compute instances, showing an error message directing the user to remove it from all instances first + +#### Public IPs + +- **FR-23:** The UI must provide a PublicIPs list page (`/networking/public-ips`) displaying all PublicIPs allocated to the tenant with columns for Address, Pool, Attached To (resource name + type, or dash if unattached), and Status +- **FR-24:** The PublicIPs list page must support filtering by Pool, Status, and Attached (yes/no) +- **FR-25:** The UI must provide an "Allocate IP" action that opens a modal dialog with fields for Pool (required dropdown showing available pools with remaining IP count, e.g., "external-pool-1 (Available: 245 IPs)") and Name (required, label for this allocation) +- **FR-26:** After successful IP allocation, the UI must close the modal and show the new IP as "Available" (unattached) in the list +- **FR-27:** The UI must provide Attach and Detach row actions for PublicIPs—Attach (if Available) opens a side panel listing eligible resources (ComputeInstances/VMs) in a searchable table; Detach (if Attached) opens a confirmation modal warning the user that the VM will lose external connectivity on this IP +- **FR-28:** The UI must provide a Release (delete) action for PublicIPs with confirmation, blocked if the IP is currently attached + +#### Sidebar Navigation + +- **FR-29:** The tenant user sidebar must add a new "Networking" section with sub-items: Virtual Networks, Security Groups, Public IPs (Subnets are not a sidebar item) +- **FR-30:** The tenant admin sidebar must add the same "Networking" section under the existing Management section, preserving the existing Infrastructure (Networks topology view) section + +#### VMaaS Wizard Integration + +- **FR-31:** The VM creation wizard (`/vms/create/:catalogItemId`) must include a Network Configuration step after basic VM settings (name, SSH key, etc.) with sections for Network Attachment and Public IP (optional) +- **FR-32:** The Network Attachment section must provide fields for Virtual Network (required dropdown showing all tenant VNs with name and CIDR, with "Create new VN" link), Subnet (required dropdown filtered to selected VN, showing CIDR and available IP count, with "Create new Subnet" link), and Security Groups (optional multi-select checkboxes showing SGs scoped to selected VN with rule count summary, with "Create new Security Group" link) +- **FR-33:** When a user clicks "Create new VN" from the wizard, the UI must open the Create VN side panel that overlays the wizard—after VN creation, the new VN must be auto-selected in the wizard +- **FR-34:** The wizard must support multi-NIC configuration via an "[+ Add another network attachment]" button—each attachment has its own VN/Subnet/SG selection, all subnets must belong to the same VN (enforced with validation error), and exactly one attachment must be marked as Primary (radio button) which determines the default gateway +- **FR-35:** The wizard's Public IP section must provide radio button options: No public IP, Use existing (dropdown of Available PublicIPs), or Allocate new from pool (dropdown of pools with IP count)—if "Allocate new" is selected, a new PublicIP must be allocated and attached during VM creation +- **FR-36:** The wizard must enforce that VirtualNetwork is required (block VM creation without network attachment), Subnet is required and must belong to the selected VN, and show a warning if no Security Groups are selected ("No security groups selected. Your VM will have no firewall rules.") +- **FR-37:** The wizard must implement smart defaults: if the tenant has exactly one VN auto-select it, if the selected VN has exactly one subnet auto-select it, if the selected VN has exactly one SG pre-check it, if only one PublicIP pool exists pre-select it for "Allocate new" +- **FR-38:** If no VNs exist when the wizard reaches the Network Configuration step, the UI must show a prominent message "You need to create a virtual network before provisioning a VM" with a "Create Virtual Network" button that navigates to `/networking/virtual-networks` (or opens inline creation with return path to wizard) + +#### Error States and Edge Cases + +- **FR-39:** When no resources exist, list pages must show empty states with illustrations, helpful headings (e.g., "No virtual networks yet"), descriptions, and primary action buttons (e.g., "Create virtual network") +- **FR-40:** When PublicIP pools are unavailable, the PublicIPs list page must show an informational banner: "No IP pools available. Contact your provider to provision IP address pools." +- **FR-41:** While a resource is in Provisioning or Deleting state, the UI must disable Delete actions, show a spinner next to the status badge, and auto-refresh the list every 5 seconds until the resource reaches a terminal state (Ready or Failed) +- **FR-42:** For failed resources, the UI must show the error message from the API in a collapsible alert on the detail page and provide Retry and Delete actions +- **FR-43:** The UI must use TanStack Query's stale-while-revalidate pattern for concurrent modification handling, show a Refresh button in the toolbar, auto-refetch on window focus, and show optimistic updates for delete operations (gray out the row immediately) + +#### API Integration + +- **FR-44:** The UI must use protobuf-generated types from `libs/types/src/osac/public/v1/` and create TanStack Query hooks for each resource (useVirtualNetworks, useVirtualNetwork, useCreateVirtualNetwork, useDeleteVirtualNetwork, etc.) +- **FR-45:** The UI must call the following REST gateway endpoints: VirtualNetworks (GET /v1/virtualnetworks, GET /v1/virtualnetworks/{id}, POST /v1/virtualnetworks, PATCH /v1/virtualnetworks/{id}, DELETE /v1/virtualnetworks/{id}), Subnets (GET /v1/subnets with filter by virtual_network, GET /v1/subnets/{id}, POST /v1/subnets, DELETE /v1/subnets/{id}), SecurityGroups (GET /v1/securitygroups with filter by virtual_network, GET /v1/securitygroups/{id}, POST /v1/securitygroups, PATCH /v1/securitygroups/{id}, DELETE /v1/securitygroups/{id}), PublicIPs (GET /v1/publicips, GET /v1/publicips/{id}, POST /v1/publicips, DELETE /v1/publicips/{id}), PublicIPPools (GET /v1/publicippools, read-only) + +### 3.2 Non-Functional Requirements + +- **NFR-1:** The UI must be built with React 19 and TypeScript using the PatternFly 6 design system (Red Hat design system) +- **NFR-2:** The UI must use react-router-dom v7 for routing, TanStack Query (React Query) for data fetching, and Connect-ES for gRPC-Web client integration +- **NFR-3:** The UI must follow the file structure pattern: list pages in `libs/ui-components/src/pages/networking/`, detail pages in the same directory, table/form components in `libs/ui-components/src/components/networking/`, and TanStack Query hooks in `libs/ui-components/src/api/v1/` +- **NFR-4:** The UI must use PatternFly components: PageSection/Title/Text for page layout, Table/Thead/Tr/Th/Tbody/Td for tables, Toolbar/ToolbarContent/ToolbarItem for search/filter toolbars, Form/FormGroup/TextInput/FormSelect for forms, DrawerPanelContent for side panels, Modal for confirmations, Label for status badges, EmptyState for empty states, Tabs/Tab for detail pages, Breadcrumb/BreadcrumbItem for navigation, Wizard/WizardStep for the VM wizard +- **NFR-5:** Status badges must use consistent colors: Ready (green pf-m-green), Provisioning (blue pf-m-blue with spinner), Failed (red pf-m-red), Deleting (orange pf-m-orange), Available (green for PublicIPs), Attached (blue for PublicIPs) +- **NFR-6:** Tables must be responsive using PatternFly's responsive table breakpoints—on small screens, use compound expansion to show row details instead of navigating to a detail page +- **NFR-7:** Side panels (create forms) must become full-width drawers on mobile, and the sidebar must remain collapsible as in the current OSAC UI +- **NFR-8:** All form inputs must have associated labels (PatternFly FormGroup handles this), status badges must have aria-labels (include text like "Status: Ready," not just color), table row actions must be keyboard-navigable, modal focus must be trapped (PatternFly Modal handles this), status changes must be announced via aria-live regions, and CIDR inputs must have helper text explaining the expected format +- **NFR-9:** The UI must use PatternFly 6 components as specified in NFR-1 and NFR-4, consistent with the current OSAC UI codebase +- **NFR-10:** The UI must handle at least 100 VirtualNetworks, 500 Subnets, 200 SecurityGroups, and 1000 PublicIPs per tenant with acceptable performance—pagination should be enforced when counts exceed these thresholds + +## 4. Acceptance Criteria + +- [ ] A tenant user can navigate to Networking > Virtual Networks from the sidebar and see a list of all their VirtualNetworks with Name, Region, IPv4 CIDR, Subnets count, and Status columns +- [ ] A tenant user can click "Create virtual network" and successfully create a VN by filling in Name, Region, and IPv4 CIDR fields with inline validation +- [ ] A tenant user can click on a VirtualNetwork name to see the detail page with Subnets, Security Groups, and Details tabs +- [ ] A tenant user can create a Subnet from the Subnets tab with the parent VN pre-selected, and the form validates that the subnet CIDR is within the parent VN CIDR and does not overlap existing subnets +- [ ] A tenant user can view all SecurityGroups across all VNs from the Networking > Security Groups page, filtered by VirtualNetwork +- [ ] A tenant user can create a SecurityGroup from either the SG list page or the VN detail page, selecting a VN and adding inbound/outbound rules inline with Protocol, Port Range, Source/Destination CIDR, and Description fields +- [ ] A tenant user can view SecurityGroup inbound and outbound rules on the SG detail page in separate tabs, with Add Rule, Edit, and Delete actions +- [ ] A tenant user can allocate a PublicIP from the Networking > Public IPs page by selecting a pool and providing a name +- [ ] A tenant user can see Attach and Detach actions for PublicIPs—Attach shows a searchable list of VMs, Detach shows a confirmation modal +- [ ] A new tenant creating their first VM sees "You need to create a virtual network before provisioning a VM" when reaching the Network Configuration step in the VM wizard +- [ ] A new tenant can click "Create Virtual Network" from the wizard, create a VN inline (side panel overlay), and the new VN is auto-selected in the wizard after creation +- [ ] A new tenant can create Subnets and SecurityGroups inline from the wizard using "Create new Subnet" and "Create new Security Group" links, with the current VN pre-selected +- [ ] A returning tenant with one VN, one Subnet, and one SG sees all three auto-selected in the VM wizard Network Configuration step +- [ ] A tenant user can add multiple network attachments in the VM wizard via "[+ Add another network attachment]" and mark one as Primary +- [ ] The wizard blocks VM creation if no VirtualNetwork is selected and shows a warning if no SecurityGroups are selected +- [ ] Empty states show helpful illustrations and "Create" buttons on all list pages when no resources exist +- [ ] Resources in Provisioning or Deleting state show a spinner, disable Delete actions, and auto-refresh every 5 seconds +- [ ] Deletion of a VirtualNetwork is blocked if it has subnets or security groups, with an error message directing the user to delete children first +- [ ] Deletion of a Subnet is blocked if it has attached compute instances, with an error message +- [ ] Deletion of a SecurityGroup is blocked if it is attached to compute instances, with an error message +- [ ] Deletion of a PublicIP is blocked if it is currently attached to a resource +- [ ] All forms validate inputs inline and disable the Create/Submit button until all required fields are valid +- [ ] Status badges use correct colors (Ready=green, Provisioning=blue with spinner, Failed=red, Deleting=orange, Available=green for PublicIPs, Attached=blue for PublicIPs) and include aria-labels for accessibility + +## 5. Assumptions + +- The OSAC UI codebase uses a pnpm monorepo structure with the UI components in `libs/ui-components/src/` +- The fulfillment API endpoints listed in Section 3.1 (FR-45) are already implemented and stable—no API changes are required for this UI work +- The existing VM creation wizard at `/vms/create/:catalogItemId` has a NetworkAttachmentFields component that can be extended or replaced with the new Network Configuration step +- The existing AdminNetworksPage (topology view) does not require changes as part of this feature—it will be preserved as-is under Infrastructure > Networks for tenant admins +- PatternFly components (DrawerPanelContent, Modal, Wizard, etc.) provide sufficient accessibility support (keyboard navigation, focus management, aria-labels) without additional custom implementation +- The protobuf-generated types in `libs/types/src/osac/public/v1/` include all necessary types for VirtualNetwork, Subnet, SecurityGroup, PublicIP, and PublicIPPool + +## 6. Dependencies + +- fulfillment-service REST gateway endpoints must be available and stable (GET/POST/PATCH/DELETE for /v1/virtualnetworks, /v1/subnets, /v1/securitygroups, /v1/publicips) +- protobuf-generated TypeScript types must be available in `libs/types/src/osac/public/v1/` for VirtualNetwork, Subnet, SecurityGroup, PublicIP, PublicIPPool +- The existing OSAC UI navigation/sidebar framework must support adding a new "Networking" section with sub-items +- The existing VM creation wizard framework must support adding a new Network Configuration step with inline resource creation + +## 7. Risks + +### 7.1 PatternFly version mismatch + +- **Owner:** To be determined +- **Mitigation:** Confirm the current PatternFly version used by OSAC UI before implementation. If OSAC UI is on PatternFly 5, implement components using PF5 and plan a migration path to PF6 in a future phase. If OSAC UI is already on PF6, proceed as specified. + +### 7.2 Inline resource creation from wizard complicates state management + +- **Owner:** UI team +- **Mitigation:** Use TanStack Query's cache invalidation and refetch mechanisms to ensure the wizard's VN/Subnet/SG dropdowns refresh after inline creation. Test the round-trip flow (create VN → auto-select → create Subnet → auto-select → continue wizard) thoroughly to ensure state consistency. + +### 7.3 Performance degradation with large resource counts + +- **Owner:** UI team +- **Mitigation:** Implement pagination on list pages when resource counts exceed 100 VNs, 500 Subnets, 200 SGs, or 1000 PublicIPs. Use TanStack Query's pagination support and the API's `page` and `size` parameters. Monitor performance in testing with realistic data volumes. + +## 8. Open Questions + +### 8.1 Should the "Create new VN" link in the wizard navigate away or use an inline overlay? + +- **Owner:** UX team / Product Owner +- **Impact:** Affects the wizard integration implementation (FR-33). Option (A) is an inline side panel overlay that preserves wizard state. Option (B) navigates to `/networking/virtual-networks` and loses wizard state. Option (C) is an inline drawer within the wizard step itself. The specification leans toward Option (A) but doesn't definitively resolve this. + +### 8.2 What specific enhancements to the AdminNetworksPage topology view are in scope? + +- **Owner:** Product Owner +- **Impact:** Non-Goals section states topology view enhancements are future work, but the specification says the topology view "should be preserved and enhanced." Clarify whether any enhancements (e.g., integration with the new Networking section, updated styling) are expected in this phase or if it remains fully as-is. + +### 8.3 What testing strategy is required for this feature? + +- **Owner:** QE team / Product Owner +- **Impact:** Determines deliverables and acceptance criteria. Should the feature include: (a) unit tests for all components, (b) E2E tests for critical flows (VN creation, wizard integration), (c) accessibility testing with specific tools (e.g., axe, NVDA), (d) manual testing only, or (e) a combination? The specification does not specify testing requirements. + +### 8.4 What are the exact error message strings for validation failures? + +- **Owner:** UX team / Technical Writer +- **Impact:** Affects form validation implementation (FR-4, FR-11, FR-18). The specification describes validation rules (e.g., "CIDR must be within parent VN CIDR") but does not provide exact error message strings. Should these follow a standard template or be defined per-field? + +### 8.5 What performance SLOs should the UI meet for list page load times? + +- **Owner:** Product Owner +- **Impact:** NFR-10 assumes pagination thresholds (100 VNs, 500 Subnets, etc.) but does not specify target load times. Should list pages load within 2 seconds at p95? What is the acceptable performance envelope for filter/search operations? From 29591fac9fa0a5ba64dbbd8ad469f3de8d7baba7 Mon Sep 17 00:00:00 2001 From: Siddarth Royapally <60885840+SiddarthR56@users.noreply.github.com> Date: Mon, 29 Jun 2026 01:32:22 -0400 Subject: [PATCH 061/450] Document CIDR immutability for VirtualNetwork and Subnet resources (#63) Assisted-by: Cursor/Claude --- enhancements/networking/README.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/enhancements/networking/README.md b/enhancements/networking/README.md index 788272afa..c28ed5828 100644 --- a/enhancements/networking/README.md +++ b/enhancements/networking/README.md @@ -45,22 +45,27 @@ This section defines the key networking terms used throughout this enhancement: to an AWS VPC or Azure VNet. VirtualNetwork is scoped to a Region. It provides logical isolation and defines the overall address space via optional `ipv4` and `ipv6` sections (each with a `cidr`). Single-stack = one section; - dual-stack = both. + dual-stack = both. The `region`, `networkClass`, `implementationStrategy`, and + CIDRs are immutable after creation. - **Subnet**: A subdivision of a VirtualNetwork's IP address space, scoped to a Region (the same Region as the VirtualNetwork). Subnets use optional `ipv4` and `ipv6` sections (each with a `cidr`), consistent with the VirtualNetwork. - Resources are attached to Subnets to receive IP addresses and network - connectivity. + The `virtualNetwork` (parent reference) and CIDRs are immutable after + creation. Resources are attached to Subnets to receive IP addresses and + network connectivity. - **SecurityGroup**: A stateful firewall that controls inbound and outbound traffic for resources. Rules specify allowed protocols, ports, and source/ destination addresses (IPv4 or IPv6 CIDRs). SecurityGroups are applied to - resources within a VirtualNetwork. + resources within a VirtualNetwork. The `virtualNetwork` and + `implementationStrategy` are immutable after creation. - **PublicIPPool**: A provider-defined pool of public IP addresses. Each pool is either IPv4 or IPv6 (not both), defined via `ipv4.cidrs` or `ipv6.cidrs`. - Pools are scoped to a region. Tenants allocate PublicIPs from a PublicIPPool. + Pools are scoped to a region. The `cidrs`, IP family (IPv4 vs IPv6), and + `implementationStrategy` are immutable after creation. Tenants allocate + PublicIPs from a PublicIPPool. - **PublicIP** (also known as **Floating IP**): A public IP address (IPv4 or IPv6) allocated from a PublicIPPool. PublicIPs can be dynamically attached to @@ -687,6 +692,16 @@ apply SecurityGroups for traffic control. pool's family. SecurityGroup rules use CIDR notation (e.g. `0.0.0.0/0`, `::/0`). Implementation support for IPv6 depends on the NetworkClass and underlying platform (e.g. OVN-Kubernetes / UDN). +- **Field Immutability**: Several spec fields are immutable after creation + across networking resources. The Fulfillment Service and O-SAC Operator both + reject update requests that modify immutable fields; to change them, the + resource must be deleted and re-created. + - **VirtualNetwork**: `region`, `networkClass`, `implementationStrategy`, + `ipv4.cidr`, `ipv6.cidr` + - **Subnet**: `virtualNetwork`, `ipv4.cidr`, `ipv6.cidr` + - **SecurityGroup**: `virtualNetwork`, `implementationStrategy` + - **PublicIPPool**: `cidrs`, IP family (IPv4 vs IPv6), + `implementationStrategy` - **Namespace per Subnet** (`udn-net`): Each Subnet has its own namespace and UserDefinedNetwork; VirtualNetwork is a logical grouping. UDN and namespace are created with the Subnet and removed with it; VirtualNetwork can be deleted From 1ff1c51c6810b9cacc1404b5b07eb8c3f2231b0e Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Mon, 29 Jun 2026 12:52:24 +0300 Subject: [PATCH 062/450] PRD OSAC-1425: address review feedback - Remove PublicIP attach/detach from Non-Goals (FR-27/28/35 cover this) - Remove all Region references (Region lives on NetworkClass, not VirtualNetwork) --- enhancements/networking-ui-vmaas-scope/prd.md | 65 ++----------------- 1 file changed, 6 insertions(+), 59 deletions(-) diff --git a/enhancements/networking-ui-vmaas-scope/prd.md b/enhancements/networking-ui-vmaas-scope/prd.md index cfc731802..b405f3170 100644 --- a/enhancements/networking-ui-vmaas-scope/prd.md +++ b/enhancements/networking-ui-vmaas-scope/prd.md @@ -1,59 +1,42 @@ # OSAC Tenant UI: Networking Section - | Field | Value | |-------------|---------| | Author(s) | Elay Aharoni, Dan Manor | | Jira | [OSAC-1425](https://redhat.atlassian.net/browse/OSAC-1425) | | Date | 2026-06-28 | - ## 1. Problem Statement - Tenant users and tenant admins currently lack a dedicated UI for managing networking resources (VirtualNetworks, Subnets, SecurityGroups, PublicIPs) in the OSAC platform. While the fulfillment API provides full CRUD capabilities for these resources, users must either use the CLI or construct API calls directly. Additionally, when creating VMs through the VMaaS wizard, users have no integrated way to create or select networking resources inline—they must context-switch to external tools or pre-create resources before provisioning compute instances. This creates friction in the user experience, increases time-to-first-VM for new tenants, and reduces discoverability of networking capabilities. - ## 2. Goals and Non-Goals - ### 2.1 Goals - - Tenant users can create, list, view, and delete VirtualNetworks through a dedicated Networking section in the OSAC UI sidebar - Tenant users can manage Subnets and SecurityGroups scoped to a VirtualNetwork from the VirtualNetwork detail page - Tenant users can allocate and release PublicIPs from provider-managed pools through a dedicated PublicIPs page - New tenants creating their first VM can provision all required networking resources (VirtualNetwork, Subnet, SecurityGroup) inline within the VM creation wizard without leaving the wizard flow - Returning tenants can select from existing networking resources in the VM creation wizard with smart defaults (auto-select when only one option exists) - ### 2.2 Non-Goals - - NATGateway, ExternalIPAttachment, NetworkClass management (these are provider-only or future scope) - PublicIPPool CRUD operations (pools are provider-managed, read-only for tenants) - BaremetalInstance networking or Cluster networking (out of scope for VMaaS phase) -- Attaching/detaching PublicIPs to compute instances (covered in a separate initiative) - Migration of the existing AdminNetworksPage topology view (preserved as-is, enhancements are future work) - ## 3. Requirements - ### 3.1 Functional Requirements - #### Virtual Networks - -- **FR-1:** The UI must provide a VirtualNetworks list page (`/networking/virtual-networks`) displaying all VirtualNetworks owned by the tenant with columns for Name, Region, IPv4 CIDR, Subnets count, and Status -- **FR-2:** The VirtualNetworks list page must support filtering by Region and Status, sorting by Name/Region/Status, and searching by Name -- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for Name (required, DNS-valid, unique within tenant), Region (required, dropdown), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) +- **FR-1:** The UI must provide a VirtualNetworks list page (`/networking/virtual-networks`) displaying all VirtualNetworks owned by the tenant with columns for Name, IPv4 CIDR, Subnets count, and Status +- **FR-2:** The VirtualNetworks list page must support filtering by Status, sorting by Name/Status, and searching by Name +- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) - **FR-4:** The Create VirtualNetwork form must validate inputs inline (show validation errors below each field) and disable the Create button until all required fields are valid - **FR-5:** After successful VirtualNetwork creation, the UI must close the form, refresh the list, show a success toast notification, and display the new VN with "Provisioning" status -- **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, region, CIDR as key properties, and a Delete action in the header +- **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, CIDR as key properties, and a Delete action in the header - **FR-7:** The VirtualNetwork detail page must display three tabs: Subnets (default), Security Groups, and Details - **FR-8:** The UI must block deletion of a VirtualNetwork if it has subnets or security groups, showing an error message directing the user to delete child resources first - #### Subnets - - **FR-9:** The Subnets tab on the VirtualNetwork detail page must display all subnets belonging to that VN in a table with columns for Name, CIDR, Available IPs (used/total), and Status - **FR-10:** The Subnets tab must provide a "Create subnet" action that opens a side panel form with the parent VN pre-selected and fields for Name (required, DNS-valid) and CIDR (required, must be within parent VN CIDR, must not overlap existing subnets) - **FR-11:** The Create Subnet form must show the parent VN CIDR as context and display existing subnet CIDRs to help users select a non-overlapping range - **FR-12:** Clicking on a subnet name must show a detail panel (side drawer or inline expansion) displaying subnet metadata and a list of attached resources (compute instances) - **FR-13:** Subnets must not have their own top-level sidebar entry—they are managed exclusively from the VirtualNetwork detail page - **FR-14:** The UI must block deletion of a Subnet if it has attached compute instances, showing an error message directing the user to remove or migrate instances first - #### Security Groups - - **FR-15:** The UI must provide a SecurityGroups list page (`/networking/security-groups`) displaying all SecurityGroups across all VirtualNetworks with columns for Name, Virtual Network (link to VN detail), Inbound Rules count, Outbound Rules count, and Status - **FR-16:** The SecurityGroups list page must support filtering by Virtual Network and Status, and searching by Name - **FR-17:** The UI must provide a "Create security group" action (available from both the SG list page and the VN detail Security Groups tab) that opens a side panel form with fields for Virtual Network (required dropdown, pre-selected if triggered from VN detail), Name (required, DNS-valid), and expandable Inbound/Outbound Rules sections @@ -62,23 +45,17 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **FR-20:** The Inbound Rules and Outbound Rules tabs must display rules in a table (columns: Priority, Protocol, Port Range, Source/Destination CIDR, Description) with "Add Rule" action and row-level Edit/Delete actions, ordered by priority - **FR-21:** SecurityGroups must be accessible from both the sidebar (top-level) and the VirtualNetwork detail page (Pattern 3: top-level but VN-scoped) - **FR-22:** The UI must block deletion of a SecurityGroup if it is attached to compute instances, showing an error message directing the user to remove it from all instances first - #### Public IPs - - **FR-23:** The UI must provide a PublicIPs list page (`/networking/public-ips`) displaying all PublicIPs allocated to the tenant with columns for Address, Pool, Attached To (resource name + type, or dash if unattached), and Status - **FR-24:** The PublicIPs list page must support filtering by Pool, Status, and Attached (yes/no) - **FR-25:** The UI must provide an "Allocate IP" action that opens a modal dialog with fields for Pool (required dropdown showing available pools with remaining IP count, e.g., "external-pool-1 (Available: 245 IPs)") and Name (required, label for this allocation) - **FR-26:** After successful IP allocation, the UI must close the modal and show the new IP as "Available" (unattached) in the list - **FR-27:** The UI must provide Attach and Detach row actions for PublicIPs—Attach (if Available) opens a side panel listing eligible resources (ComputeInstances/VMs) in a searchable table; Detach (if Attached) opens a confirmation modal warning the user that the VM will lose external connectivity on this IP - **FR-28:** The UI must provide a Release (delete) action for PublicIPs with confirmation, blocked if the IP is currently attached - #### Sidebar Navigation - - **FR-29:** The tenant user sidebar must add a new "Networking" section with sub-items: Virtual Networks, Security Groups, Public IPs (Subnets are not a sidebar item) - **FR-30:** The tenant admin sidebar must add the same "Networking" section under the existing Management section, preserving the existing Infrastructure (Networks topology view) section - #### VMaaS Wizard Integration - - **FR-31:** The VM creation wizard (`/vms/create/:catalogItemId`) must include a Network Configuration step after basic VM settings (name, SSH key, etc.) with sections for Network Attachment and Public IP (optional) - **FR-32:** The Network Attachment section must provide fields for Virtual Network (required dropdown showing all tenant VNs with name and CIDR, with "Create new VN" link), Subnet (required dropdown filtered to selected VN, showing CIDR and available IP count, with "Create new Subnet" link), and Security Groups (optional multi-select checkboxes showing SGs scoped to selected VN with rule count summary, with "Create new Security Group" link) - **FR-33:** When a user clicks "Create new VN" from the wizard, the UI must open the Create VN side panel that overlays the wizard—after VN creation, the new VN must be auto-selected in the wizard @@ -87,22 +64,16 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **FR-36:** The wizard must enforce that VirtualNetwork is required (block VM creation without network attachment), Subnet is required and must belong to the selected VN, and show a warning if no Security Groups are selected ("No security groups selected. Your VM will have no firewall rules.") - **FR-37:** The wizard must implement smart defaults: if the tenant has exactly one VN auto-select it, if the selected VN has exactly one subnet auto-select it, if the selected VN has exactly one SG pre-check it, if only one PublicIP pool exists pre-select it for "Allocate new" - **FR-38:** If no VNs exist when the wizard reaches the Network Configuration step, the UI must show a prominent message "You need to create a virtual network before provisioning a VM" with a "Create Virtual Network" button that navigates to `/networking/virtual-networks` (or opens inline creation with return path to wizard) - #### Error States and Edge Cases - - **FR-39:** When no resources exist, list pages must show empty states with illustrations, helpful headings (e.g., "No virtual networks yet"), descriptions, and primary action buttons (e.g., "Create virtual network") - **FR-40:** When PublicIP pools are unavailable, the PublicIPs list page must show an informational banner: "No IP pools available. Contact your provider to provision IP address pools." - **FR-41:** While a resource is in Provisioning or Deleting state, the UI must disable Delete actions, show a spinner next to the status badge, and auto-refresh the list every 5 seconds until the resource reaches a terminal state (Ready or Failed) - **FR-42:** For failed resources, the UI must show the error message from the API in a collapsible alert on the detail page and provide Retry and Delete actions - **FR-43:** The UI must use TanStack Query's stale-while-revalidate pattern for concurrent modification handling, show a Refresh button in the toolbar, auto-refetch on window focus, and show optimistic updates for delete operations (gray out the row immediately) - #### API Integration - - **FR-44:** The UI must use protobuf-generated types from `libs/types/src/osac/public/v1/` and create TanStack Query hooks for each resource (useVirtualNetworks, useVirtualNetwork, useCreateVirtualNetwork, useDeleteVirtualNetwork, etc.) - **FR-45:** The UI must call the following REST gateway endpoints: VirtualNetworks (GET /v1/virtualnetworks, GET /v1/virtualnetworks/{id}, POST /v1/virtualnetworks, PATCH /v1/virtualnetworks/{id}, DELETE /v1/virtualnetworks/{id}), Subnets (GET /v1/subnets with filter by virtual_network, GET /v1/subnets/{id}, POST /v1/subnets, DELETE /v1/subnets/{id}), SecurityGroups (GET /v1/securitygroups with filter by virtual_network, GET /v1/securitygroups/{id}, POST /v1/securitygroups, PATCH /v1/securitygroups/{id}, DELETE /v1/securitygroups/{id}), PublicIPs (GET /v1/publicips, GET /v1/publicips/{id}, POST /v1/publicips, DELETE /v1/publicips/{id}), PublicIPPools (GET /v1/publicippools, read-only) - ### 3.2 Non-Functional Requirements - - **NFR-1:** The UI must be built with React 19 and TypeScript using the PatternFly 6 design system (Red Hat design system) - **NFR-2:** The UI must use react-router-dom v7 for routing, TanStack Query (React Query) for data fetching, and Connect-ES for gRPC-Web client integration - **NFR-3:** The UI must follow the file structure pattern: list pages in `libs/ui-components/src/pages/networking/`, detail pages in the same directory, table/form components in `libs/ui-components/src/components/networking/`, and TanStack Query hooks in `libs/ui-components/src/api/v1/` @@ -113,11 +84,9 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **NFR-8:** All form inputs must have associated labels (PatternFly FormGroup handles this), status badges must have aria-labels (include text like "Status: Ready," not just color), table row actions must be keyboard-navigable, modal focus must be trapped (PatternFly Modal handles this), status changes must be announced via aria-live regions, and CIDR inputs must have helper text explaining the expected format - **NFR-9:** The UI must use PatternFly 6 components as specified in NFR-1 and NFR-4, consistent with the current OSAC UI codebase - **NFR-10:** The UI must handle at least 100 VirtualNetworks, 500 Subnets, 200 SecurityGroups, and 1000 PublicIPs per tenant with acceptable performance—pagination should be enforced when counts exceed these thresholds - ## 4. Acceptance Criteria - -- [ ] A tenant user can navigate to Networking > Virtual Networks from the sidebar and see a list of all their VirtualNetworks with Name, Region, IPv4 CIDR, Subnets count, and Status columns -- [ ] A tenant user can click "Create virtual network" and successfully create a VN by filling in Name, Region, and IPv4 CIDR fields with inline validation +- [ ] A tenant user can navigate to Networking > Virtual Networks from the sidebar and see a list of all their VirtualNetworks with Name, IPv4 CIDR, Subnets count, and Status columns +- [ ] A tenant user can click "Create virtual network" and successfully create a VN by filling in Name and IPv4 CIDR fields with inline validation - [ ] A tenant user can click on a VirtualNetwork name to see the detail page with Subnets, Security Groups, and Details tabs - [ ] A tenant user can create a Subnet from the Subnets tab with the parent VN pre-selected, and the form validates that the subnet CIDR is within the parent VN CIDR and does not overlap existing subnets - [ ] A tenant user can view all SecurityGroups across all VNs from the Networking > Security Groups page, filtered by VirtualNetwork @@ -139,63 +108,41 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - [ ] Deletion of a PublicIP is blocked if it is currently attached to a resource - [ ] All forms validate inputs inline and disable the Create/Submit button until all required fields are valid - [ ] Status badges use correct colors (Ready=green, Provisioning=blue with spinner, Failed=red, Deleting=orange, Available=green for PublicIPs, Attached=blue for PublicIPs) and include aria-labels for accessibility - ## 5. Assumptions - - The OSAC UI codebase uses a pnpm monorepo structure with the UI components in `libs/ui-components/src/` - The fulfillment API endpoints listed in Section 3.1 (FR-45) are already implemented and stable—no API changes are required for this UI work - The existing VM creation wizard at `/vms/create/:catalogItemId` has a NetworkAttachmentFields component that can be extended or replaced with the new Network Configuration step - The existing AdminNetworksPage (topology view) does not require changes as part of this feature—it will be preserved as-is under Infrastructure > Networks for tenant admins - PatternFly components (DrawerPanelContent, Modal, Wizard, etc.) provide sufficient accessibility support (keyboard navigation, focus management, aria-labels) without additional custom implementation - The protobuf-generated types in `libs/types/src/osac/public/v1/` include all necessary types for VirtualNetwork, Subnet, SecurityGroup, PublicIP, and PublicIPPool - ## 6. Dependencies - - fulfillment-service REST gateway endpoints must be available and stable (GET/POST/PATCH/DELETE for /v1/virtualnetworks, /v1/subnets, /v1/securitygroups, /v1/publicips) - protobuf-generated TypeScript types must be available in `libs/types/src/osac/public/v1/` for VirtualNetwork, Subnet, SecurityGroup, PublicIP, PublicIPPool - The existing OSAC UI navigation/sidebar framework must support adding a new "Networking" section with sub-items - The existing VM creation wizard framework must support adding a new Network Configuration step with inline resource creation - ## 7. Risks - ### 7.1 PatternFly version mismatch - - **Owner:** To be determined - **Mitigation:** Confirm the current PatternFly version used by OSAC UI before implementation. If OSAC UI is on PatternFly 5, implement components using PF5 and plan a migration path to PF6 in a future phase. If OSAC UI is already on PF6, proceed as specified. - ### 7.2 Inline resource creation from wizard complicates state management - - **Owner:** UI team - **Mitigation:** Use TanStack Query's cache invalidation and refetch mechanisms to ensure the wizard's VN/Subnet/SG dropdowns refresh after inline creation. Test the round-trip flow (create VN → auto-select → create Subnet → auto-select → continue wizard) thoroughly to ensure state consistency. - ### 7.3 Performance degradation with large resource counts - - **Owner:** UI team - **Mitigation:** Implement pagination on list pages when resource counts exceed 100 VNs, 500 Subnets, 200 SGs, or 1000 PublicIPs. Use TanStack Query's pagination support and the API's `page` and `size` parameters. Monitor performance in testing with realistic data volumes. - ## 8. Open Questions - ### 8.1 Should the "Create new VN" link in the wizard navigate away or use an inline overlay? - - **Owner:** UX team / Product Owner - **Impact:** Affects the wizard integration implementation (FR-33). Option (A) is an inline side panel overlay that preserves wizard state. Option (B) navigates to `/networking/virtual-networks` and loses wizard state. Option (C) is an inline drawer within the wizard step itself. The specification leans toward Option (A) but doesn't definitively resolve this. - ### 8.2 What specific enhancements to the AdminNetworksPage topology view are in scope? - - **Owner:** Product Owner - **Impact:** Non-Goals section states topology view enhancements are future work, but the specification says the topology view "should be preserved and enhanced." Clarify whether any enhancements (e.g., integration with the new Networking section, updated styling) are expected in this phase or if it remains fully as-is. - ### 8.3 What testing strategy is required for this feature? - - **Owner:** QE team / Product Owner - **Impact:** Determines deliverables and acceptance criteria. Should the feature include: (a) unit tests for all components, (b) E2E tests for critical flows (VN creation, wizard integration), (c) accessibility testing with specific tools (e.g., axe, NVDA), (d) manual testing only, or (e) a combination? The specification does not specify testing requirements. - ### 8.4 What are the exact error message strings for validation failures? - - **Owner:** UX team / Technical Writer - **Impact:** Affects form validation implementation (FR-4, FR-11, FR-18). The specification describes validation rules (e.g., "CIDR must be within parent VN CIDR") but does not provide exact error message strings. Should these follow a standard template or be defined per-field? - ### 8.5 What performance SLOs should the UI meet for list page load times? - - **Owner:** Product Owner - **Impact:** NFR-10 assumes pagination thresholds (100 VNs, 500 Subnets, etc.) but does not specify target load times. Should list pages load within 2 seconds at p95? What is the acceptable performance envelope for filter/search operations? From 69a3acd37bdcacd7d7b02c870aed4a9001407d7e Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Mon, 29 Jun 2026 13:55:31 +0300 Subject: [PATCH 063/450] PRD OSAC-1425: address CodeRabbit review feedback - Resolve AdminNetworksPage topology view wording contradiction (issue #3) - Specify IPv4/IPv6 CIDRs explicitly in FR-6 (issue #4) - Pick side drawer pattern for subnet detail display in FR-12 (issue #5) - Add Priority field to SecurityGroup rule specification in FR-18 (issue #6) - Standardize PublicIP action terminology to 'Release' in FR-28 (issue #7) - Document multi-NIC same-VN constraint rationale in FR-34 (issue #8) - Align wizard empty-state flow with inline overlay pattern in FR-38 (issue #9) - Define Retry action API contract in FR-42 (issue #10) - Clarify Subnet endpoints are create/delete only in FR-45 (issue #11) - Remove redundant NFR-9 (issue #12) - Update Open Question 8.2 wording to match Non-Goals Assisted-by: Claude Code Signed-off-by: Elay Aharoni --- enhancements/networking-ui-vmaas-scope/prd.md | 23 +++++++++---------- 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/enhancements/networking-ui-vmaas-scope/prd.md b/enhancements/networking-ui-vmaas-scope/prd.md index b405f3170..adc3a145b 100644 --- a/enhancements/networking-ui-vmaas-scope/prd.md +++ b/enhancements/networking-ui-vmaas-scope/prd.md @@ -17,7 +17,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - NATGateway, ExternalIPAttachment, NetworkClass management (these are provider-only or future scope) - PublicIPPool CRUD operations (pools are provider-managed, read-only for tenants) - BaremetalInstance networking or Cluster networking (out of scope for VMaaS phase) -- Migration of the existing AdminNetworksPage topology view (preserved as-is, enhancements are future work) +- Migration or enhancement of the existing AdminNetworksPage topology view (future scope) ## 3. Requirements ### 3.1 Functional Requirements #### Virtual Networks @@ -26,21 +26,21 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) - **FR-4:** The Create VirtualNetwork form must validate inputs inline (show validation errors below each field) and disable the Create button until all required fields are valid - **FR-5:** After successful VirtualNetwork creation, the UI must close the form, refresh the list, show a success toast notification, and display the new VN with "Provisioning" status -- **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, CIDR as key properties, and a Delete action in the header +- **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, IPv4 CIDR (and IPv6 CIDR if configured) as key properties, and a Delete action in the header - **FR-7:** The VirtualNetwork detail page must display three tabs: Subnets (default), Security Groups, and Details - **FR-8:** The UI must block deletion of a VirtualNetwork if it has subnets or security groups, showing an error message directing the user to delete child resources first #### Subnets - **FR-9:** The Subnets tab on the VirtualNetwork detail page must display all subnets belonging to that VN in a table with columns for Name, CIDR, Available IPs (used/total), and Status - **FR-10:** The Subnets tab must provide a "Create subnet" action that opens a side panel form with the parent VN pre-selected and fields for Name (required, DNS-valid) and CIDR (required, must be within parent VN CIDR, must not overlap existing subnets) - **FR-11:** The Create Subnet form must show the parent VN CIDR as context and display existing subnet CIDRs to help users select a non-overlapping range -- **FR-12:** Clicking on a subnet name must show a detail panel (side drawer or inline expansion) displaying subnet metadata and a list of attached resources (compute instances) +- **FR-12:** Clicking on a subnet name must show subnet metadata and a list of attached resources (compute instances) in a side drawer - **FR-13:** Subnets must not have their own top-level sidebar entry—they are managed exclusively from the VirtualNetwork detail page - **FR-14:** The UI must block deletion of a Subnet if it has attached compute instances, showing an error message directing the user to remove or migrate instances first #### Security Groups - **FR-15:** The UI must provide a SecurityGroups list page (`/networking/security-groups`) displaying all SecurityGroups across all VirtualNetworks with columns for Name, Virtual Network (link to VN detail), Inbound Rules count, Outbound Rules count, and Status - **FR-16:** The SecurityGroups list page must support filtering by Virtual Network and Status, and searching by Name - **FR-17:** The UI must provide a "Create security group" action (available from both the SG list page and the VN detail Security Groups tab) that opens a side panel form with fields for Virtual Network (required dropdown, pre-selected if triggered from VN detail), Name (required, DNS-valid), and expandable Inbound/Outbound Rules sections -- **FR-18:** The Create SecurityGroup form must allow users to add multiple inbound and outbound rules inline, with each rule having fields for Protocol (dropdown: TCP/UDP/ICMP/All), Port Range (text input, disabled for ICMP), Source/Destination CIDR (text with CIDR validation), and Description (optional) +- **FR-18:** The Create SecurityGroup form must allow users to add multiple inbound and outbound rules inline, with each rule having fields for Protocol (dropdown: TCP/UDP/ICMP/All), Port Range (text input, disabled for ICMP), Source/Destination CIDR (text with CIDR validation), Description (optional), and Priority (integer, determines rule evaluation order) - **FR-19:** The UI must provide a SecurityGroup detail page (`/networking/security-groups/:id`) with tabs for Inbound Rules, Outbound Rules, and Details - **FR-20:** The Inbound Rules and Outbound Rules tabs must display rules in a table (columns: Priority, Protocol, Port Range, Source/Destination CIDR, Description) with "Add Rule" action and row-level Edit/Delete actions, ordered by priority - **FR-21:** SecurityGroups must be accessible from both the sidebar (top-level) and the VirtualNetwork detail page (Pattern 3: top-level but VN-scoped) @@ -51,7 +51,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **FR-25:** The UI must provide an "Allocate IP" action that opens a modal dialog with fields for Pool (required dropdown showing available pools with remaining IP count, e.g., "external-pool-1 (Available: 245 IPs)") and Name (required, label for this allocation) - **FR-26:** After successful IP allocation, the UI must close the modal and show the new IP as "Available" (unattached) in the list - **FR-27:** The UI must provide Attach and Detach row actions for PublicIPs—Attach (if Available) opens a side panel listing eligible resources (ComputeInstances/VMs) in a searchable table; Detach (if Attached) opens a confirmation modal warning the user that the VM will lose external connectivity on this IP -- **FR-28:** The UI must provide a Release (delete) action for PublicIPs with confirmation, blocked if the IP is currently attached +- **FR-28:** The UI must provide a Release action for PublicIPs with confirmation, blocked if the IP is currently attached #### Sidebar Navigation - **FR-29:** The tenant user sidebar must add a new "Networking" section with sub-items: Virtual Networks, Security Groups, Public IPs (Subnets are not a sidebar item) - **FR-30:** The tenant admin sidebar must add the same "Networking" section under the existing Management section, preserving the existing Infrastructure (Networks topology view) section @@ -59,20 +59,20 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **FR-31:** The VM creation wizard (`/vms/create/:catalogItemId`) must include a Network Configuration step after basic VM settings (name, SSH key, etc.) with sections for Network Attachment and Public IP (optional) - **FR-32:** The Network Attachment section must provide fields for Virtual Network (required dropdown showing all tenant VNs with name and CIDR, with "Create new VN" link), Subnet (required dropdown filtered to selected VN, showing CIDR and available IP count, with "Create new Subnet" link), and Security Groups (optional multi-select checkboxes showing SGs scoped to selected VN with rule count summary, with "Create new Security Group" link) - **FR-33:** When a user clicks "Create new VN" from the wizard, the UI must open the Create VN side panel that overlays the wizard—after VN creation, the new VN must be auto-selected in the wizard -- **FR-34:** The wizard must support multi-NIC configuration via an "[+ Add another network attachment]" button—each attachment has its own VN/Subnet/SG selection, all subnets must belong to the same VN (enforced with validation error), and exactly one attachment must be marked as Primary (radio button) which determines the default gateway +- **FR-34:** The wizard must support multi-NIC configuration via an "[+ Add another network attachment]" button—each attachment has its own VN/Subnet/SG selection, and exactly one attachment must be marked as Primary (radio button) which determines the default gateway. Note: All subnets must belong to the same VirtualNetwork (OSAC platform constraint to simplify initial networking model; future phases may support cross-VN attachments) - **FR-35:** The wizard's Public IP section must provide radio button options: No public IP, Use existing (dropdown of Available PublicIPs), or Allocate new from pool (dropdown of pools with IP count)—if "Allocate new" is selected, a new PublicIP must be allocated and attached during VM creation - **FR-36:** The wizard must enforce that VirtualNetwork is required (block VM creation without network attachment), Subnet is required and must belong to the selected VN, and show a warning if no Security Groups are selected ("No security groups selected. Your VM will have no firewall rules.") - **FR-37:** The wizard must implement smart defaults: if the tenant has exactly one VN auto-select it, if the selected VN has exactly one subnet auto-select it, if the selected VN has exactly one SG pre-check it, if only one PublicIP pool exists pre-select it for "Allocate new" -- **FR-38:** If no VNs exist when the wizard reaches the Network Configuration step, the UI must show a prominent message "You need to create a virtual network before provisioning a VM" with a "Create Virtual Network" button that navigates to `/networking/virtual-networks` (or opens inline creation with return path to wizard) +- **FR-38:** If no VNs exist when the wizard reaches the Network Configuration step, the UI must show a prominent message "You need to create a virtual network before provisioning a VM" with a "Create Virtual Network" button that opens the inline Create VN side panel (see FR-33) with automatic return to the wizard after creation #### Error States and Edge Cases - **FR-39:** When no resources exist, list pages must show empty states with illustrations, helpful headings (e.g., "No virtual networks yet"), descriptions, and primary action buttons (e.g., "Create virtual network") - **FR-40:** When PublicIP pools are unavailable, the PublicIPs list page must show an informational banner: "No IP pools available. Contact your provider to provision IP address pools." - **FR-41:** While a resource is in Provisioning or Deleting state, the UI must disable Delete actions, show a spinner next to the status badge, and auto-refresh the list every 5 seconds until the resource reaches a terminal state (Ready or Failed) -- **FR-42:** For failed resources, the UI must show the error message from the API in a collapsible alert on the detail page and provide Retry and Delete actions +- **FR-42:** For failed resources, the UI must show the error message from the API in a collapsible alert on the detail page and provide Retry and Delete actions. Retry re-submits the original creation request (POST) to the same endpoint; the UI does not persist intermediate state—users must re-enter form data if the retry also fails - **FR-43:** The UI must use TanStack Query's stale-while-revalidate pattern for concurrent modification handling, show a Refresh button in the toolbar, auto-refetch on window focus, and show optimistic updates for delete operations (gray out the row immediately) #### API Integration - **FR-44:** The UI must use protobuf-generated types from `libs/types/src/osac/public/v1/` and create TanStack Query hooks for each resource (useVirtualNetworks, useVirtualNetwork, useCreateVirtualNetwork, useDeleteVirtualNetwork, etc.) -- **FR-45:** The UI must call the following REST gateway endpoints: VirtualNetworks (GET /v1/virtualnetworks, GET /v1/virtualnetworks/{id}, POST /v1/virtualnetworks, PATCH /v1/virtualnetworks/{id}, DELETE /v1/virtualnetworks/{id}), Subnets (GET /v1/subnets with filter by virtual_network, GET /v1/subnets/{id}, POST /v1/subnets, DELETE /v1/subnets/{id}), SecurityGroups (GET /v1/securitygroups with filter by virtual_network, GET /v1/securitygroups/{id}, POST /v1/securitygroups, PATCH /v1/securitygroups/{id}, DELETE /v1/securitygroups/{id}), PublicIPs (GET /v1/publicips, GET /v1/publicips/{id}, POST /v1/publicips, DELETE /v1/publicips/{id}), PublicIPPools (GET /v1/publicippools, read-only) +- **FR-45:** The UI must call the following REST gateway endpoints: VirtualNetworks (GET /v1/virtualnetworks, GET /v1/virtualnetworks/{id}, POST /v1/virtualnetworks, PATCH /v1/virtualnetworks/{id}, DELETE /v1/virtualnetworks/{id}), Subnets (GET /v1/subnets with filter by virtual_network, GET /v1/subnets/{id}, POST /v1/subnets, DELETE /v1/subnets/{id}—note: Subnets are create/delete only, no PATCH endpoint), SecurityGroups (GET /v1/securitygroups with filter by virtual_network, GET /v1/securitygroups/{id}, POST /v1/securitygroups, PATCH /v1/securitygroups/{id}, DELETE /v1/securitygroups/{id}), PublicIPs (GET /v1/publicips, GET /v1/publicips/{id}, POST /v1/publicips, DELETE /v1/publicips/{id}), PublicIPPools (GET /v1/publicippools, read-only) ### 3.2 Non-Functional Requirements - **NFR-1:** The UI must be built with React 19 and TypeScript using the PatternFly 6 design system (Red Hat design system) - **NFR-2:** The UI must use react-router-dom v7 for routing, TanStack Query (React Query) for data fetching, and Connect-ES for gRPC-Web client integration @@ -82,8 +82,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **NFR-6:** Tables must be responsive using PatternFly's responsive table breakpoints—on small screens, use compound expansion to show row details instead of navigating to a detail page - **NFR-7:** Side panels (create forms) must become full-width drawers on mobile, and the sidebar must remain collapsible as in the current OSAC UI - **NFR-8:** All form inputs must have associated labels (PatternFly FormGroup handles this), status badges must have aria-labels (include text like "Status: Ready," not just color), table row actions must be keyboard-navigable, modal focus must be trapped (PatternFly Modal handles this), status changes must be announced via aria-live regions, and CIDR inputs must have helper text explaining the expected format -- **NFR-9:** The UI must use PatternFly 6 components as specified in NFR-1 and NFR-4, consistent with the current OSAC UI codebase -- **NFR-10:** The UI must handle at least 100 VirtualNetworks, 500 Subnets, 200 SecurityGroups, and 1000 PublicIPs per tenant with acceptable performance—pagination should be enforced when counts exceed these thresholds +- **NFR-9:** The UI must handle at least 100 VirtualNetworks, 500 Subnets, 200 SecurityGroups, and 1000 PublicIPs per tenant with acceptable performance—pagination should be enforced when counts exceed these thresholds ## 4. Acceptance Criteria - [ ] A tenant user can navigate to Networking > Virtual Networks from the sidebar and see a list of all their VirtualNetworks with Name, IPv4 CIDR, Subnets count, and Status columns - [ ] A tenant user can click "Create virtual network" and successfully create a VN by filling in Name and IPv4 CIDR fields with inline validation @@ -136,7 +135,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **Impact:** Affects the wizard integration implementation (FR-33). Option (A) is an inline side panel overlay that preserves wizard state. Option (B) navigates to `/networking/virtual-networks` and loses wizard state. Option (C) is an inline drawer within the wizard step itself. The specification leans toward Option (A) but doesn't definitively resolve this. ### 8.2 What specific enhancements to the AdminNetworksPage topology view are in scope? - **Owner:** Product Owner -- **Impact:** Non-Goals section states topology view enhancements are future work, but the specification says the topology view "should be preserved and enhanced." Clarify whether any enhancements (e.g., integration with the new Networking section, updated styling) are expected in this phase or if it remains fully as-is. +- **Impact:** Non-Goals section states topology view enhancements are future scope. Clarify whether any enhancements (e.g., integration with the new Networking section, updated styling) are expected in a future phase. ### 8.3 What testing strategy is required for this feature? - **Owner:** QE team / Product Owner - **Impact:** Determines deliverables and acceptance criteria. Should the feature include: (a) unit tests for all components, (b) E2E tests for critical flows (VN creation, wizard integration), (c) accessibility testing with specific tools (e.g., axe, NVDA), (d) manual testing only, or (e) a combination? The specification does not specify testing requirements. From 4b7477db4b312f1caf8ba1e91a32e7c89c5565c4 Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Mon, 29 Jun 2026 14:38:29 +0300 Subject: [PATCH 064/450] PRD OSAC-1425: address danmanor and CodeRabbit review feedback (round 2) Critical fixes from danmanor: - Add NetworkClass dropdown to VN create form (FR-3) - required field - Fix all REST endpoint paths to use /api/fulfillment prefix and snake_case (FR-45) - Clarify PublicIP attach/detach requires private API endpoints (Assumption 5, Section 6) Major fixes from danmanor: - Remove Priority and Description from SecurityRule fields (FR-18, FR-20) - not in proto - Remove 'Available IPs (used/total)' column from Subnets table (FR-9) - no API backing - Mark PatternFly version mismatch risk as RESOLVED (Risk 7.1) CodeRabbit feedback: - Add blank lines around all section headings (MD022 linter) Assisted-by: Claude Code Signed-off-by: Elay Aharoni --- enhancements/networking-ui-vmaas-scope/prd.md | 34 +++++++++++++------ 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/enhancements/networking-ui-vmaas-scope/prd.md b/enhancements/networking-ui-vmaas-scope/prd.md index adc3a145b..7bac93184 100644 --- a/enhancements/networking-ui-vmaas-scope/prd.md +++ b/enhancements/networking-ui-vmaas-scope/prd.md @@ -23,39 +23,49 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ #### Virtual Networks - **FR-1:** The UI must provide a VirtualNetworks list page (`/networking/virtual-networks`) displaying all VirtualNetworks owned by the tenant with columns for Name, IPv4 CIDR, Subnets count, and Status - **FR-2:** The VirtualNetworks list page must support filtering by Status, sorting by Name/Status, and searching by Name -- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) +- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for NetworkClass (required dropdown showing available NetworkClasses with name and region), Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) - **FR-4:** The Create VirtualNetwork form must validate inputs inline (show validation errors below each field) and disable the Create button until all required fields are valid - **FR-5:** After successful VirtualNetwork creation, the UI must close the form, refresh the list, show a success toast notification, and display the new VN with "Provisioning" status - **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, IPv4 CIDR (and IPv6 CIDR if configured) as key properties, and a Delete action in the header - **FR-7:** The VirtualNetwork detail page must display three tabs: Subnets (default), Security Groups, and Details - **FR-8:** The UI must block deletion of a VirtualNetwork if it has subnets or security groups, showing an error message directing the user to delete child resources first + #### Subnets -- **FR-9:** The Subnets tab on the VirtualNetwork detail page must display all subnets belonging to that VN in a table with columns for Name, CIDR, Available IPs (used/total), and Status + +- **FR-9:** The Subnets tab on the VirtualNetwork detail page must display all subnets belonging to that VN in a table with columns for Name, CIDR, and Status - **FR-10:** The Subnets tab must provide a "Create subnet" action that opens a side panel form with the parent VN pre-selected and fields for Name (required, DNS-valid) and CIDR (required, must be within parent VN CIDR, must not overlap existing subnets) - **FR-11:** The Create Subnet form must show the parent VN CIDR as context and display existing subnet CIDRs to help users select a non-overlapping range - **FR-12:** Clicking on a subnet name must show subnet metadata and a list of attached resources (compute instances) in a side drawer - **FR-13:** Subnets must not have their own top-level sidebar entry—they are managed exclusively from the VirtualNetwork detail page - **FR-14:** The UI must block deletion of a Subnet if it has attached compute instances, showing an error message directing the user to remove or migrate instances first + #### Security Groups + - **FR-15:** The UI must provide a SecurityGroups list page (`/networking/security-groups`) displaying all SecurityGroups across all VirtualNetworks with columns for Name, Virtual Network (link to VN detail), Inbound Rules count, Outbound Rules count, and Status - **FR-16:** The SecurityGroups list page must support filtering by Virtual Network and Status, and searching by Name - **FR-17:** The UI must provide a "Create security group" action (available from both the SG list page and the VN detail Security Groups tab) that opens a side panel form with fields for Virtual Network (required dropdown, pre-selected if triggered from VN detail), Name (required, DNS-valid), and expandable Inbound/Outbound Rules sections -- **FR-18:** The Create SecurityGroup form must allow users to add multiple inbound and outbound rules inline, with each rule having fields for Protocol (dropdown: TCP/UDP/ICMP/All), Port Range (text input, disabled for ICMP), Source/Destination CIDR (text with CIDR validation), Description (optional), and Priority (integer, determines rule evaluation order) +- **FR-18:** The Create SecurityGroup form must allow users to add multiple inbound and outbound rules inline, with each rule having fields for Protocol (dropdown: TCP/UDP/ICMP/All), Port Range (text input, disabled for ICMP), and Source/Destination CIDR (text with CIDR validation) - **FR-19:** The UI must provide a SecurityGroup detail page (`/networking/security-groups/:id`) with tabs for Inbound Rules, Outbound Rules, and Details -- **FR-20:** The Inbound Rules and Outbound Rules tabs must display rules in a table (columns: Priority, Protocol, Port Range, Source/Destination CIDR, Description) with "Add Rule" action and row-level Edit/Delete actions, ordered by priority +- **FR-20:** The Inbound Rules and Outbound Rules tabs must display rules in a table (columns: Protocol, Port Range, Source/Destination CIDR) with "Add Rule" action and row-level Edit/Delete actions. Rule edits use `PATCH /v1/securitygroups/{id}` to replace the full rule set (the API does not support individual rule-level endpoints) - **FR-21:** SecurityGroups must be accessible from both the sidebar (top-level) and the VirtualNetwork detail page (Pattern 3: top-level but VN-scoped) - **FR-22:** The UI must block deletion of a SecurityGroup if it is attached to compute instances, showing an error message directing the user to remove it from all instances first + #### Public IPs + - **FR-23:** The UI must provide a PublicIPs list page (`/networking/public-ips`) displaying all PublicIPs allocated to the tenant with columns for Address, Pool, Attached To (resource name + type, or dash if unattached), and Status - **FR-24:** The PublicIPs list page must support filtering by Pool, Status, and Attached (yes/no) - **FR-25:** The UI must provide an "Allocate IP" action that opens a modal dialog with fields for Pool (required dropdown showing available pools with remaining IP count, e.g., "external-pool-1 (Available: 245 IPs)") and Name (required, label for this allocation) - **FR-26:** After successful IP allocation, the UI must close the modal and show the new IP as "Available" (unattached) in the list - **FR-27:** The UI must provide Attach and Detach row actions for PublicIPs—Attach (if Available) opens a side panel listing eligible resources (ComputeInstances/VMs) in a searchable table; Detach (if Attached) opens a confirmation modal warning the user that the VM will lose external connectivity on this IP - **FR-28:** The UI must provide a Release action for PublicIPs with confirmation, blocked if the IP is currently attached + #### Sidebar Navigation + - **FR-29:** The tenant user sidebar must add a new "Networking" section with sub-items: Virtual Networks, Security Groups, Public IPs (Subnets are not a sidebar item) - **FR-30:** The tenant admin sidebar must add the same "Networking" section under the existing Management section, preserving the existing Infrastructure (Networks topology view) section + #### VMaaS Wizard Integration + - **FR-31:** The VM creation wizard (`/vms/create/:catalogItemId`) must include a Network Configuration step after basic VM settings (name, SSH key, etc.) with sections for Network Attachment and Public IP (optional) - **FR-32:** The Network Attachment section must provide fields for Virtual Network (required dropdown showing all tenant VNs with name and CIDR, with "Create new VN" link), Subnet (required dropdown filtered to selected VN, showing CIDR and available IP count, with "Create new Subnet" link), and Security Groups (optional multi-select checkboxes showing SGs scoped to selected VN with rule count summary, with "Create new Security Group" link) - **FR-33:** When a user clicks "Create new VN" from the wizard, the UI must open the Create VN side panel that overlays the wizard—after VN creation, the new VN must be auto-selected in the wizard @@ -64,15 +74,19 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **FR-36:** The wizard must enforce that VirtualNetwork is required (block VM creation without network attachment), Subnet is required and must belong to the selected VN, and show a warning if no Security Groups are selected ("No security groups selected. Your VM will have no firewall rules.") - **FR-37:** The wizard must implement smart defaults: if the tenant has exactly one VN auto-select it, if the selected VN has exactly one subnet auto-select it, if the selected VN has exactly one SG pre-check it, if only one PublicIP pool exists pre-select it for "Allocate new" - **FR-38:** If no VNs exist when the wizard reaches the Network Configuration step, the UI must show a prominent message "You need to create a virtual network before provisioning a VM" with a "Create Virtual Network" button that opens the inline Create VN side panel (see FR-33) with automatic return to the wizard after creation + #### Error States and Edge Cases + - **FR-39:** When no resources exist, list pages must show empty states with illustrations, helpful headings (e.g., "No virtual networks yet"), descriptions, and primary action buttons (e.g., "Create virtual network") - **FR-40:** When PublicIP pools are unavailable, the PublicIPs list page must show an informational banner: "No IP pools available. Contact your provider to provision IP address pools." - **FR-41:** While a resource is in Provisioning or Deleting state, the UI must disable Delete actions, show a spinner next to the status badge, and auto-refresh the list every 5 seconds until the resource reaches a terminal state (Ready or Failed) - **FR-42:** For failed resources, the UI must show the error message from the API in a collapsible alert on the detail page and provide Retry and Delete actions. Retry re-submits the original creation request (POST) to the same endpoint; the UI does not persist intermediate state—users must re-enter form data if the retry also fails - **FR-43:** The UI must use TanStack Query's stale-while-revalidate pattern for concurrent modification handling, show a Refresh button in the toolbar, auto-refetch on window focus, and show optimistic updates for delete operations (gray out the row immediately) + #### API Integration + - **FR-44:** The UI must use protobuf-generated types from `libs/types/src/osac/public/v1/` and create TanStack Query hooks for each resource (useVirtualNetworks, useVirtualNetwork, useCreateVirtualNetwork, useDeleteVirtualNetwork, etc.) -- **FR-45:** The UI must call the following REST gateway endpoints: VirtualNetworks (GET /v1/virtualnetworks, GET /v1/virtualnetworks/{id}, POST /v1/virtualnetworks, PATCH /v1/virtualnetworks/{id}, DELETE /v1/virtualnetworks/{id}), Subnets (GET /v1/subnets with filter by virtual_network, GET /v1/subnets/{id}, POST /v1/subnets, DELETE /v1/subnets/{id}—note: Subnets are create/delete only, no PATCH endpoint), SecurityGroups (GET /v1/securitygroups with filter by virtual_network, GET /v1/securitygroups/{id}, POST /v1/securitygroups, PATCH /v1/securitygroups/{id}, DELETE /v1/securitygroups/{id}), PublicIPs (GET /v1/publicips, GET /v1/publicips/{id}, POST /v1/publicips, DELETE /v1/publicips/{id}), PublicIPPools (GET /v1/publicippools, read-only) +- **FR-45:** The UI must call the following REST gateway endpoints: VirtualNetworks (GET /api/fulfillment/v1/virtual_networks, GET /api/fulfillment/v1/virtual_networks/{id}, POST /api/fulfillment/v1/virtual_networks, PATCH /api/fulfillment/v1/virtual_networks/{id}, DELETE /api/fulfillment/v1/virtual_networks/{id}), Subnets (GET /api/fulfillment/v1/subnets with filter by virtual_network, GET /api/fulfillment/v1/subnets/{id}, POST /api/fulfillment/v1/subnets, DELETE /api/fulfillment/v1/subnets/{id}—note: Subnets are create/delete only, no PATCH endpoint), SecurityGroups (GET /api/fulfillment/v1/security_groups with filter by virtual_network, GET /api/fulfillment/v1/security_groups/{id}, POST /api/fulfillment/v1/security_groups, PATCH /api/fulfillment/v1/security_groups/{id}, DELETE /api/fulfillment/v1/security_groups/{id}), PublicIPs (GET /api/fulfillment/v1/public_ips, GET /api/fulfillment/v1/public_ips/{id}, POST /api/fulfillment/v1/public_ips, DELETE /api/fulfillment/v1/public_ips/{id}), PublicIPPools (GET /api/fulfillment/v1/public_ip_pools, read-only), NetworkClasses (GET /api/fulfillment/v1/network_classes, read-only) ### 3.2 Non-Functional Requirements - **NFR-1:** The UI must be built with React 19 and TypeScript using the PatternFly 6 design system (Red Hat design system) - **NFR-2:** The UI must use react-router-dom v7 for routing, TanStack Query (React Query) for data fetching, and Connect-ES for gRPC-Web client integration @@ -109,20 +123,20 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - [ ] Status badges use correct colors (Ready=green, Provisioning=blue with spinner, Failed=red, Deleting=orange, Available=green for PublicIPs, Attached=blue for PublicIPs) and include aria-labels for accessibility ## 5. Assumptions - The OSAC UI codebase uses a pnpm monorepo structure with the UI components in `libs/ui-components/src/` -- The fulfillment API endpoints listed in Section 3.1 (FR-45) are already implemented and stable—no API changes are required for this UI work +- The fulfillment API endpoints listed in Section 3.1 (FR-45) are already implemented and stable. Note: PublicIP attach/detach (FR-27, FR-28, FR-35) requires the private API endpoint `/api/private/v1/public_ip_attachments`, which is not exposed in the public API—the UI will need appropriate permissions to call the private API for these operations - The existing VM creation wizard at `/vms/create/:catalogItemId` has a NetworkAttachmentFields component that can be extended or replaced with the new Network Configuration step - The existing AdminNetworksPage (topology view) does not require changes as part of this feature—it will be preserved as-is under Infrastructure > Networks for tenant admins - PatternFly components (DrawerPanelContent, Modal, Wizard, etc.) provide sufficient accessibility support (keyboard navigation, focus management, aria-labels) without additional custom implementation - The protobuf-generated types in `libs/types/src/osac/public/v1/` include all necessary types for VirtualNetwork, Subnet, SecurityGroup, PublicIP, and PublicIPPool ## 6. Dependencies -- fulfillment-service REST gateway endpoints must be available and stable (GET/POST/PATCH/DELETE for /v1/virtualnetworks, /v1/subnets, /v1/securitygroups, /v1/publicips) -- protobuf-generated TypeScript types must be available in `libs/types/src/osac/public/v1/` for VirtualNetwork, Subnet, SecurityGroup, PublicIP, PublicIPPool +- fulfillment-service REST gateway endpoints must be available and stable (GET/POST/PATCH/DELETE for /api/fulfillment/v1/virtual_networks, /api/fulfillment/v1/subnets, /api/fulfillment/v1/security_groups, /api/fulfillment/v1/public_ips) +- fulfillment-service private API endpoint `/api/private/v1/public_ip_attachments` (POST/DELETE) for PublicIP attach/detach operations (FR-27, FR-28, FR-35) +- protobuf-generated TypeScript types must be available in `libs/types/src/osac/public/v1/` for VirtualNetwork, Subnet, SecurityGroup, PublicIP, PublicIPPool, NetworkClass - The existing OSAC UI navigation/sidebar framework must support adding a new "Networking" section with sub-items - The existing VM creation wizard framework must support adding a new Network Configuration step with inline resource creation ## 7. Risks ### 7.1 PatternFly version mismatch -- **Owner:** To be determined -- **Mitigation:** Confirm the current PatternFly version used by OSAC UI before implementation. If OSAC UI is on PatternFly 5, implement components using PF5 and plan a migration path to PF6 in a future phase. If OSAC UI is already on PF6, proceed as specified. +- **Status:** RESOLVED — OSAC UI is confirmed to be on PatternFly 6.4.x, so the PRD's PF6 requirement is satisfied ### 7.2 Inline resource creation from wizard complicates state management - **Owner:** UI team - **Mitigation:** Use TanStack Query's cache invalidation and refetch mechanisms to ensure the wizard's VN/Subnet/SG dropdowns refresh after inline creation. Test the round-trip flow (create VN → auto-select → create Subnet → auto-select → continue wizard) thoroughly to ensure state consistency. From 801b82ae8cebc3d4f50af3932d080ea5627bc8d0 Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Mon, 29 Jun 2026 18:05:45 +0300 Subject: [PATCH 065/450] PRD OSAC-1425: fix NetworkClass dropdown description - remove region, add description NetworkClass proto has no region field. The dropdown should show name and description instead. Assisted-by: Claude Code Signed-off-by: Elay Aharoni --- enhancements/networking-ui-vmaas-scope/prd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/enhancements/networking-ui-vmaas-scope/prd.md b/enhancements/networking-ui-vmaas-scope/prd.md index 7bac93184..5134cabd0 100644 --- a/enhancements/networking-ui-vmaas-scope/prd.md +++ b/enhancements/networking-ui-vmaas-scope/prd.md @@ -23,7 +23,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ #### Virtual Networks - **FR-1:** The UI must provide a VirtualNetworks list page (`/networking/virtual-networks`) displaying all VirtualNetworks owned by the tenant with columns for Name, IPv4 CIDR, Subnets count, and Status - **FR-2:** The VirtualNetworks list page must support filtering by Status, sorting by Name/Status, and searching by Name -- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for NetworkClass (required dropdown showing available NetworkClasses with name and region), Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) +- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for NetworkClass (required dropdown showing available NetworkClasses with name and description), Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) - **FR-4:** The Create VirtualNetwork form must validate inputs inline (show validation errors below each field) and disable the Create button until all required fields are valid - **FR-5:** After successful VirtualNetwork creation, the UI must close the form, refresh the list, show a success toast notification, and display the new VN with "Provisioning" status - **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, IPv4 CIDR (and IPv6 CIDR if configured) as key properties, and a Delete action in the header From 0054f974f9ef8252931ce94d34a40039e9eb2d2f Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Mon, 29 Jun 2026 18:07:17 +0300 Subject: [PATCH 066/450] PRD OSAC-1425: address danmanor final review comments 1. Fix FR-20: correct SecurityGroup PATCH endpoint path format - Changed from /v1/securitygroups/{id} to /api/fulfillment/v1/security_groups/{id} 2. Fix FR-32: remove 'available IP count' from subnet dropdown - SubnetStatus doesn't include IP count data 3. Fix Assumption 5 & Section 6: PublicIPAttachments is in public API - Remove references to private API endpoint - Update to use /api/fulfillment/v1/public_ip_attachments (public API) - Add PublicIPAttachment to required types list Assisted-by: Claude Code Signed-off-by: Elay Aharoni --- enhancements/networking-ui-vmaas-scope/prd.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/enhancements/networking-ui-vmaas-scope/prd.md b/enhancements/networking-ui-vmaas-scope/prd.md index 5134cabd0..f37ab2077 100644 --- a/enhancements/networking-ui-vmaas-scope/prd.md +++ b/enhancements/networking-ui-vmaas-scope/prd.md @@ -46,7 +46,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **FR-17:** The UI must provide a "Create security group" action (available from both the SG list page and the VN detail Security Groups tab) that opens a side panel form with fields for Virtual Network (required dropdown, pre-selected if triggered from VN detail), Name (required, DNS-valid), and expandable Inbound/Outbound Rules sections - **FR-18:** The Create SecurityGroup form must allow users to add multiple inbound and outbound rules inline, with each rule having fields for Protocol (dropdown: TCP/UDP/ICMP/All), Port Range (text input, disabled for ICMP), and Source/Destination CIDR (text with CIDR validation) - **FR-19:** The UI must provide a SecurityGroup detail page (`/networking/security-groups/:id`) with tabs for Inbound Rules, Outbound Rules, and Details -- **FR-20:** The Inbound Rules and Outbound Rules tabs must display rules in a table (columns: Protocol, Port Range, Source/Destination CIDR) with "Add Rule" action and row-level Edit/Delete actions. Rule edits use `PATCH /v1/securitygroups/{id}` to replace the full rule set (the API does not support individual rule-level endpoints) +- **FR-20:** The Inbound Rules and Outbound Rules tabs must display rules in a table (columns: Protocol, Port Range, Source/Destination CIDR) with "Add Rule" action and row-level Edit/Delete actions. Rule edits use `PATCH /api/fulfillment/v1/security_groups/{id}` to replace the full rule set (the API does not support individual rule-level endpoints) - **FR-21:** SecurityGroups must be accessible from both the sidebar (top-level) and the VirtualNetwork detail page (Pattern 3: top-level but VN-scoped) - **FR-22:** The UI must block deletion of a SecurityGroup if it is attached to compute instances, showing an error message directing the user to remove it from all instances first @@ -67,7 +67,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ #### VMaaS Wizard Integration - **FR-31:** The VM creation wizard (`/vms/create/:catalogItemId`) must include a Network Configuration step after basic VM settings (name, SSH key, etc.) with sections for Network Attachment and Public IP (optional) -- **FR-32:** The Network Attachment section must provide fields for Virtual Network (required dropdown showing all tenant VNs with name and CIDR, with "Create new VN" link), Subnet (required dropdown filtered to selected VN, showing CIDR and available IP count, with "Create new Subnet" link), and Security Groups (optional multi-select checkboxes showing SGs scoped to selected VN with rule count summary, with "Create new Security Group" link) +- **FR-32:** The Network Attachment section must provide fields for Virtual Network (required dropdown showing all tenant VNs with name and CIDR, with "Create new VN" link), Subnet (required dropdown filtered to selected VN, showing CIDR, with "Create new Subnet" link), and Security Groups (optional multi-select checkboxes showing SGs scoped to selected VN with rule count summary, with "Create new Security Group" link) - **FR-33:** When a user clicks "Create new VN" from the wizard, the UI must open the Create VN side panel that overlays the wizard—after VN creation, the new VN must be auto-selected in the wizard - **FR-34:** The wizard must support multi-NIC configuration via an "[+ Add another network attachment]" button—each attachment has its own VN/Subnet/SG selection, and exactly one attachment must be marked as Primary (radio button) which determines the default gateway. Note: All subnets must belong to the same VirtualNetwork (OSAC platform constraint to simplify initial networking model; future phases may support cross-VN attachments) - **FR-35:** The wizard's Public IP section must provide radio button options: No public IP, Use existing (dropdown of Available PublicIPs), or Allocate new from pool (dropdown of pools with IP count)—if "Allocate new" is selected, a new PublicIP must be allocated and attached during VM creation @@ -123,15 +123,14 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - [ ] Status badges use correct colors (Ready=green, Provisioning=blue with spinner, Failed=red, Deleting=orange, Available=green for PublicIPs, Attached=blue for PublicIPs) and include aria-labels for accessibility ## 5. Assumptions - The OSAC UI codebase uses a pnpm monorepo structure with the UI components in `libs/ui-components/src/` -- The fulfillment API endpoints listed in Section 3.1 (FR-45) are already implemented and stable. Note: PublicIP attach/detach (FR-27, FR-28, FR-35) requires the private API endpoint `/api/private/v1/public_ip_attachments`, which is not exposed in the public API—the UI will need appropriate permissions to call the private API for these operations +- The fulfillment API endpoints listed in Section 3.1 (FR-45) are already implemented and stable, including PublicIP attach/detach operations (FR-27, FR-28, FR-35) which are available in the public API at `/api/fulfillment/v1/public_ip_attachments` - The existing VM creation wizard at `/vms/create/:catalogItemId` has a NetworkAttachmentFields component that can be extended or replaced with the new Network Configuration step - The existing AdminNetworksPage (topology view) does not require changes as part of this feature—it will be preserved as-is under Infrastructure > Networks for tenant admins - PatternFly components (DrawerPanelContent, Modal, Wizard, etc.) provide sufficient accessibility support (keyboard navigation, focus management, aria-labels) without additional custom implementation - The protobuf-generated types in `libs/types/src/osac/public/v1/` include all necessary types for VirtualNetwork, Subnet, SecurityGroup, PublicIP, and PublicIPPool ## 6. Dependencies -- fulfillment-service REST gateway endpoints must be available and stable (GET/POST/PATCH/DELETE for /api/fulfillment/v1/virtual_networks, /api/fulfillment/v1/subnets, /api/fulfillment/v1/security_groups, /api/fulfillment/v1/public_ips) -- fulfillment-service private API endpoint `/api/private/v1/public_ip_attachments` (POST/DELETE) for PublicIP attach/detach operations (FR-27, FR-28, FR-35) -- protobuf-generated TypeScript types must be available in `libs/types/src/osac/public/v1/` for VirtualNetwork, Subnet, SecurityGroup, PublicIP, PublicIPPool, NetworkClass +- fulfillment-service REST gateway endpoints must be available and stable (GET/POST/PATCH/DELETE for /api/fulfillment/v1/virtual_networks, /api/fulfillment/v1/subnets, /api/fulfillment/v1/security_groups, /api/fulfillment/v1/public_ips, /api/fulfillment/v1/public_ip_attachments) +- protobuf-generated TypeScript types must be available in `libs/types/src/osac/public/v1/` for VirtualNetwork, Subnet, SecurityGroup, PublicIP, PublicIPPool, PublicIPAttachment, NetworkClass - The existing OSAC UI navigation/sidebar framework must support adding a new "Networking" section with sub-items - The existing VM creation wizard framework must support adding a new Network Configuration step with inline resource creation ## 7. Risks From 46ae7efdd3f986b64094f7bc257aa183d86655bb Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Tue, 30 Jun 2026 15:06:48 +0300 Subject: [PATCH 067/450] OSAC-1425: address PR#82 feedback - simplify NetworkClass and PublicIP UX MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Changes based on @AlonaKaplan's feedback: 1. NetworkClass: Hidden from tenant users (platform-assigned) - Removed NetworkClass dropdown from VirtualNetwork creation form (FR-3) - Added note that NetworkClass is assigned automatically - Updated assumptions to clarify platform assignment 2. PublicIP allocation: Simplified to automatic IP family-based allocation - Changed from 3-option radio (No IP / Use existing / Allocate from pool) to checkbox + IP Family dropdown (FR-34, was FR-35) - PublicIP now allocated automatically from appropriate pool based on IP family - Updated smart defaults to pre-check 'Allocate Public IP' for new tenants (FR-36) 3. Multi-NIC support: Removed (deferred to future phase) - Deleted FR-34 (multi-NIC configuration requirement) - Added multi-NIC to Non-Goals section - Removed multi-NIC acceptance criteria - Renumbered subsequent FRs (FR-35 → FR-34, etc.) All FRs renumbered consistently. Total requirements: 44 FRs (was 45). Assisted-by: Claude Code Signed-off-by: Elay Aharoni --- enhancements/networking-ui-vmaas-scope/prd.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/enhancements/networking-ui-vmaas-scope/prd.md b/enhancements/networking-ui-vmaas-scope/prd.md index f37ab2077..274034802 100644 --- a/enhancements/networking-ui-vmaas-scope/prd.md +++ b/enhancements/networking-ui-vmaas-scope/prd.md @@ -17,13 +17,15 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - NATGateway, ExternalIPAttachment, NetworkClass management (these are provider-only or future scope) - PublicIPPool CRUD operations (pools are provider-managed, read-only for tenants) - BaremetalInstance networking or Cluster networking (out of scope for VMaaS phase) + - Multi-NIC (multiple network attachments per VM) support (future phase) - Migration or enhancement of the existing AdminNetworksPage topology view (future scope) ## 3. Requirements ### 3.1 Functional Requirements #### Virtual Networks - **FR-1:** The UI must provide a VirtualNetworks list page (`/networking/virtual-networks`) displaying all VirtualNetworks owned by the tenant with columns for Name, IPv4 CIDR, Subnets count, and Status - **FR-2:** The VirtualNetworks list page must support filtering by Status, sorting by Name/Status, and searching by Name -- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for NetworkClass (required dropdown showing available NetworkClasses with name and description), Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) +- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) + Note: NetworkClass is assigned automatically by the platform and hidden from tenant users. - **FR-4:** The Create VirtualNetwork form must validate inputs inline (show validation errors below each field) and disable the Create button until all required fields are valid - **FR-5:** After successful VirtualNetwork creation, the UI must close the form, refresh the list, show a success toast notification, and display the new VN with "Provisioning" status - **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, IPv4 CIDR (and IPv6 CIDR if configured) as key properties, and a Delete action in the header @@ -69,24 +71,23 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - **FR-31:** The VM creation wizard (`/vms/create/:catalogItemId`) must include a Network Configuration step after basic VM settings (name, SSH key, etc.) with sections for Network Attachment and Public IP (optional) - **FR-32:** The Network Attachment section must provide fields for Virtual Network (required dropdown showing all tenant VNs with name and CIDR, with "Create new VN" link), Subnet (required dropdown filtered to selected VN, showing CIDR, with "Create new Subnet" link), and Security Groups (optional multi-select checkboxes showing SGs scoped to selected VN with rule count summary, with "Create new Security Group" link) - **FR-33:** When a user clicks "Create new VN" from the wizard, the UI must open the Create VN side panel that overlays the wizard—after VN creation, the new VN must be auto-selected in the wizard -- **FR-34:** The wizard must support multi-NIC configuration via an "[+ Add another network attachment]" button—each attachment has its own VN/Subnet/SG selection, and exactly one attachment must be marked as Primary (radio button) which determines the default gateway. Note: All subnets must belong to the same VirtualNetwork (OSAC platform constraint to simplify initial networking model; future phases may support cross-VN attachments) -- **FR-35:** The wizard's Public IP section must provide radio button options: No public IP, Use existing (dropdown of Available PublicIPs), or Allocate new from pool (dropdown of pools with IP count)—if "Allocate new" is selected, a new PublicIP must be allocated and attached during VM creation -- **FR-36:** The wizard must enforce that VirtualNetwork is required (block VM creation without network attachment), Subnet is required and must belong to the selected VN, and show a warning if no Security Groups are selected ("No security groups selected. Your VM will have no firewall rules.") -- **FR-37:** The wizard must implement smart defaults: if the tenant has exactly one VN auto-select it, if the selected VN has exactly one subnet auto-select it, if the selected VN has exactly one SG pre-check it, if only one PublicIP pool exists pre-select it for "Allocate new" -- **FR-38:** If no VNs exist when the wizard reaches the Network Configuration step, the UI must show a prominent message "You need to create a virtual network before provisioning a VM" with a "Create Virtual Network" button that opens the inline Create VN side panel (see FR-33) with automatic return to the wizard after creation +- **FR-34:** The wizard's Public IP section must provide a checkbox "Allocate Public IP" and an IP Family dropdown (IPv4/IPv6). When checked, a PublicIP is automatically allocated from an available pool based on the selected IP family and attached during VM creation +- **FR-35:** The wizard must enforce that VirtualNetwork is required (block VM creation without network attachment), Subnet is required, and show a warning if no Security Groups are selected ("No security groups selected. Your VM will have no firewall rules.") +- **FR-36:** The wizard must implement smart defaults: if the tenant has exactly one VN auto-select it, if the selected VN has exactly one subnet auto-select it, if the selected VN has exactly one SG pre-check it, if the tenant has no existing PublicIPs, pre-check "Allocate Public IP" checkbox and default IP Family to IPv4 +- **FR-37:** If no VNs exist when the wizard reaches the Network Configuration step, the UI must show a prominent message "You need to create a virtual network before provisioning a VM" with a "Create Virtual Network" button that opens the inline Create VN side panel (see FR-33) with automatic return to the wizard after creation #### Error States and Edge Cases -- **FR-39:** When no resources exist, list pages must show empty states with illustrations, helpful headings (e.g., "No virtual networks yet"), descriptions, and primary action buttons (e.g., "Create virtual network") -- **FR-40:** When PublicIP pools are unavailable, the PublicIPs list page must show an informational banner: "No IP pools available. Contact your provider to provision IP address pools." -- **FR-41:** While a resource is in Provisioning or Deleting state, the UI must disable Delete actions, show a spinner next to the status badge, and auto-refresh the list every 5 seconds until the resource reaches a terminal state (Ready or Failed) -- **FR-42:** For failed resources, the UI must show the error message from the API in a collapsible alert on the detail page and provide Retry and Delete actions. Retry re-submits the original creation request (POST) to the same endpoint; the UI does not persist intermediate state—users must re-enter form data if the retry also fails -- **FR-43:** The UI must use TanStack Query's stale-while-revalidate pattern for concurrent modification handling, show a Refresh button in the toolbar, auto-refetch on window focus, and show optimistic updates for delete operations (gray out the row immediately) +- **FR-38:** When no resources exist, list pages must show empty states with illustrations, helpful headings (e.g., "No virtual networks yet"), descriptions, and primary action buttons (e.g., "Create virtual network") +- **FR-39:** When PublicIP pools are unavailable, the PublicIPs list page must show an informational banner: "No IP pools available. Contact your provider to provision IP address pools." +- **FR-40:** While a resource is in Provisioning or Deleting state, the UI must disable Delete actions, show a spinner next to the status badge, and auto-refresh the list every 5 seconds until the resource reaches a terminal state (Ready or Failed) +- **FR-41:** For failed resources, the UI must show the error message from the API in a collapsible alert on the detail page and provide Retry and Delete actions. Retry re-submits the original creation request (POST) to the same endpoint; the UI does not persist intermediate state—users must re-enter form data if the retry also fails +- **FR-42:** The UI must use TanStack Query's stale-while-revalidate pattern for concurrent modification handling, show a Refresh button in the toolbar, auto-refetch on window focus, and show optimistic updates for delete operations (gray out the row immediately) #### API Integration -- **FR-44:** The UI must use protobuf-generated types from `libs/types/src/osac/public/v1/` and create TanStack Query hooks for each resource (useVirtualNetworks, useVirtualNetwork, useCreateVirtualNetwork, useDeleteVirtualNetwork, etc.) -- **FR-45:** The UI must call the following REST gateway endpoints: VirtualNetworks (GET /api/fulfillment/v1/virtual_networks, GET /api/fulfillment/v1/virtual_networks/{id}, POST /api/fulfillment/v1/virtual_networks, PATCH /api/fulfillment/v1/virtual_networks/{id}, DELETE /api/fulfillment/v1/virtual_networks/{id}), Subnets (GET /api/fulfillment/v1/subnets with filter by virtual_network, GET /api/fulfillment/v1/subnets/{id}, POST /api/fulfillment/v1/subnets, DELETE /api/fulfillment/v1/subnets/{id}—note: Subnets are create/delete only, no PATCH endpoint), SecurityGroups (GET /api/fulfillment/v1/security_groups with filter by virtual_network, GET /api/fulfillment/v1/security_groups/{id}, POST /api/fulfillment/v1/security_groups, PATCH /api/fulfillment/v1/security_groups/{id}, DELETE /api/fulfillment/v1/security_groups/{id}), PublicIPs (GET /api/fulfillment/v1/public_ips, GET /api/fulfillment/v1/public_ips/{id}, POST /api/fulfillment/v1/public_ips, DELETE /api/fulfillment/v1/public_ips/{id}), PublicIPPools (GET /api/fulfillment/v1/public_ip_pools, read-only), NetworkClasses (GET /api/fulfillment/v1/network_classes, read-only) +- **FR-43:** The UI must use protobuf-generated types from `libs/types/src/osac/public/v1/` and create TanStack Query hooks for each resource (useVirtualNetworks, useVirtualNetwork, useCreateVirtualNetwork, useDeleteVirtualNetwork, etc.) +- **FR-44:** The UI must call the following REST gateway endpoints: VirtualNetworks (GET /api/fulfillment/v1/virtual_networks, GET /api/fulfillment/v1/virtual_networks/{id}, POST /api/fulfillment/v1/virtual_networks, PATCH /api/fulfillment/v1/virtual_networks/{id}, DELETE /api/fulfillment/v1/virtual_networks/{id}), Subnets (GET /api/fulfillment/v1/subnets with filter by virtual_network, GET /api/fulfillment/v1/subnets/{id}, POST /api/fulfillment/v1/subnets, DELETE /api/fulfillment/v1/subnets/{id}—note: Subnets are create/delete only, no PATCH endpoint), SecurityGroups (GET /api/fulfillment/v1/security_groups with filter by virtual_network, GET /api/fulfillment/v1/security_groups/{id}, POST /api/fulfillment/v1/security_groups, PATCH /api/fulfillment/v1/security_groups/{id}, DELETE /api/fulfillment/v1/security_groups/{id}), PublicIPs (GET /api/fulfillment/v1/public_ips, GET /api/fulfillment/v1/public_ips/{id}, POST /api/fulfillment/v1/public_ips, DELETE /api/fulfillment/v1/public_ips/{id}), PublicIPPools (GET /api/fulfillment/v1/public_ip_pools, read-only), NetworkClasses (GET /api/fulfillment/v1/network_classes, read-only) ### 3.2 Non-Functional Requirements - **NFR-1:** The UI must be built with React 19 and TypeScript using the PatternFly 6 design system (Red Hat design system) - **NFR-2:** The UI must use react-router-dom v7 for routing, TanStack Query (React Query) for data fetching, and Connect-ES for gRPC-Web client integration @@ -111,7 +112,6 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - [ ] A new tenant can click "Create Virtual Network" from the wizard, create a VN inline (side panel overlay), and the new VN is auto-selected in the wizard after creation - [ ] A new tenant can create Subnets and SecurityGroups inline from the wizard using "Create new Subnet" and "Create new Security Group" links, with the current VN pre-selected - [ ] A returning tenant with one VN, one Subnet, and one SG sees all three auto-selected in the VM wizard Network Configuration step -- [ ] A tenant user can add multiple network attachments in the VM wizard via "[+ Add another network attachment]" and mark one as Primary - [ ] The wizard blocks VM creation if no VirtualNetwork is selected and shows a warning if no SecurityGroups are selected - [ ] Empty states show helpful illustrations and "Create" buttons on all list pages when no resources exist - [ ] Resources in Provisioning or Deleting state show a spinner, disable Delete actions, and auto-refresh every 5 seconds @@ -127,7 +127,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - The existing VM creation wizard at `/vms/create/:catalogItemId` has a NetworkAttachmentFields component that can be extended or replaced with the new Network Configuration step - The existing AdminNetworksPage (topology view) does not require changes as part of this feature—it will be preserved as-is under Infrastructure > Networks for tenant admins - PatternFly components (DrawerPanelContent, Modal, Wizard, etc.) provide sufficient accessibility support (keyboard navigation, focus management, aria-labels) without additional custom implementation -- The protobuf-generated types in `libs/types/src/osac/public/v1/` include all necessary types for VirtualNetwork, Subnet, SecurityGroup, PublicIP, and PublicIPPool +- The protobuf-generated types in `libs/types/src/osac/public/v1/` include all necessary types for VirtualNetwork, Subnet, SecurityGroup, PublicIP, and PublicIPPool. NetworkClass is assigned automatically by the platform and is not selectable by tenant users. ## 6. Dependencies - fulfillment-service REST gateway endpoints must be available and stable (GET/POST/PATCH/DELETE for /api/fulfillment/v1/virtual_networks, /api/fulfillment/v1/subnets, /api/fulfillment/v1/security_groups, /api/fulfillment/v1/public_ips, /api/fulfillment/v1/public_ip_attachments) - protobuf-generated TypeScript types must be available in `libs/types/src/osac/public/v1/` for VirtualNetwork, Subnet, SecurityGroup, PublicIP, PublicIPPool, PublicIPAttachment, NetworkClass From 7dddc5e070295b07d3c0db51bdf0fd06fea54fad Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Tue, 30 Jun 2026 15:24:19 +0300 Subject: [PATCH 068/450] Fix Code Rabbit review feedback Applied all Code Rabbit markdown and content fixes: 1. Markdown formatting (MD022): - Added blank lines around section headings (lines 22-24) - Added blank lines before/after "3. Requirements", "3.1 Functional Requirements", and "Virtual Networks" headings 2. NetworkClass note clarity: - Moved NetworkClass note inline with FR-3 instead of detached - Changed phrasing from "hidden from" to "not exposed to" for consistency 3. Dangling reference fix: - Updated FR-45 reference to "FR-43 to FR-44" in Assumptions section (line 129) 4. Missing acceptance criterion: - Added criterion for PublicIP smart default: "A new tenant creating their first VM has 'Allocate Public IP' checkbox pre-checked and IP Family defaulted to IPv4" 5. Split combined assumption: - Separated protobuf types assumption from NetworkClass behavior - NetworkClass now has its own bullet with aligned phrasing All markdownlint warnings resolved. Assisted-by: Claude Code Signed-off-by: Elay Aharoni --- enhancements/networking-ui-vmaas-scope/prd.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/enhancements/networking-ui-vmaas-scope/prd.md b/enhancements/networking-ui-vmaas-scope/prd.md index 274034802..8feac71e4 100644 --- a/enhancements/networking-ui-vmaas-scope/prd.md +++ b/enhancements/networking-ui-vmaas-scope/prd.md @@ -19,13 +19,16 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - BaremetalInstance networking or Cluster networking (out of scope for VMaaS phase) - Multi-NIC (multiple network attachments per VM) support (future phase) - Migration or enhancement of the existing AdminNetworksPage topology view (future scope) + ## 3. Requirements + ### 3.1 Functional Requirements + #### Virtual Networks + - **FR-1:** The UI must provide a VirtualNetworks list page (`/networking/virtual-networks`) displaying all VirtualNetworks owned by the tenant with columns for Name, IPv4 CIDR, Subnets count, and Status - **FR-2:** The VirtualNetworks list page must support filtering by Status, sorting by Name/Status, and searching by Name -- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional) - Note: NetworkClass is assigned automatically by the platform and hidden from tenant users. +- **FR-3:** The UI must provide a "Create virtual network" action that opens a side panel form with fields for Name (required, DNS-valid, unique within tenant), IPv4 CIDR (required, /16 to /24 range), and IPv6 CIDR (optional). NetworkClass is assigned automatically by the platform and is not exposed to tenant users. - **FR-4:** The Create VirtualNetwork form must validate inputs inline (show validation errors below each field) and disable the Create button until all required fields are valid - **FR-5:** After successful VirtualNetwork creation, the UI must close the form, refresh the list, show a success toast notification, and display the new VN with "Provisioning" status - **FR-6:** The UI must provide a VirtualNetwork detail page (`/networking/virtual-networks/:id`) showing the VN name as page title, breadcrumb navigation, status badge, IPv4 CIDR (and IPv6 CIDR if configured) as key properties, and a Delete action in the header @@ -112,6 +115,7 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - [ ] A new tenant can click "Create Virtual Network" from the wizard, create a VN inline (side panel overlay), and the new VN is auto-selected in the wizard after creation - [ ] A new tenant can create Subnets and SecurityGroups inline from the wizard using "Create new Subnet" and "Create new Security Group" links, with the current VN pre-selected - [ ] A returning tenant with one VN, one Subnet, and one SG sees all three auto-selected in the VM wizard Network Configuration step +- [ ] A new tenant creating their first VM has "Allocate Public IP" checkbox pre-checked and IP Family defaulted to IPv4 in the Network Configuration step - [ ] The wizard blocks VM creation if no VirtualNetwork is selected and shows a warning if no SecurityGroups are selected - [ ] Empty states show helpful illustrations and "Create" buttons on all list pages when no resources exist - [ ] Resources in Provisioning or Deleting state show a spinner, disable Delete actions, and auto-refresh every 5 seconds @@ -123,11 +127,12 @@ Tenant users and tenant admins currently lack a dedicated UI for managing networ - [ ] Status badges use correct colors (Ready=green, Provisioning=blue with spinner, Failed=red, Deleting=orange, Available=green for PublicIPs, Attached=blue for PublicIPs) and include aria-labels for accessibility ## 5. Assumptions - The OSAC UI codebase uses a pnpm monorepo structure with the UI components in `libs/ui-components/src/` -- The fulfillment API endpoints listed in Section 3.1 (FR-45) are already implemented and stable, including PublicIP attach/detach operations (FR-27, FR-28, FR-35) which are available in the public API at `/api/fulfillment/v1/public_ip_attachments` +- The fulfillment API endpoints listed in Section 3.1 (FR-43 to FR-44) are already implemented and stable, including PublicIP attach/detach operations (FR-27, FR-28, FR-35) which are available in the public API at `/api/fulfillment/v1/public_ip_attachments` - The existing VM creation wizard at `/vms/create/:catalogItemId` has a NetworkAttachmentFields component that can be extended or replaced with the new Network Configuration step - The existing AdminNetworksPage (topology view) does not require changes as part of this feature—it will be preserved as-is under Infrastructure > Networks for tenant admins - PatternFly components (DrawerPanelContent, Modal, Wizard, etc.) provide sufficient accessibility support (keyboard navigation, focus management, aria-labels) without additional custom implementation -- The protobuf-generated types in `libs/types/src/osac/public/v1/` include all necessary types for VirtualNetwork, Subnet, SecurityGroup, PublicIP, and PublicIPPool. NetworkClass is assigned automatically by the platform and is not selectable by tenant users. +- The protobuf-generated types in `libs/types/src/osac/public/v1/` include all necessary types for VirtualNetwork, Subnet, SecurityGroup, PublicIP, and PublicIPPool +- NetworkClass is assigned automatically by the platform and hidden from tenant users (not exposed in the tenant API) ## 6. Dependencies - fulfillment-service REST gateway endpoints must be available and stable (GET/POST/PATCH/DELETE for /api/fulfillment/v1/virtual_networks, /api/fulfillment/v1/subnets, /api/fulfillment/v1/security_groups, /api/fulfillment/v1/public_ips, /api/fulfillment/v1/public_ip_attachments) - protobuf-generated TypeScript types must be available in `libs/types/src/osac/public/v1/` for VirtualNetwork, Subnet, SecurityGroup, PublicIP, PublicIPPool, PublicIPAttachment, NetworkClass From 6b8f0d062ebc1bfb0543fffb31b24f0d888d31ad Mon Sep 17 00:00:00 2001 From: Elay Aharoni Date: Tue, 30 Jun 2026 17:47:52 +0300 Subject: [PATCH 069/450] Add design document for OSAC-1425: OSAC Tenant UI Networking Section Design document for the tenant-facing networking UI feature covering: - VirtualNetworks, Subnets, SecurityGroups, and PublicIPs management - VMaaS wizard integration with inline resource creation - React 19 + PatternFly 6 implementation patterns This design aligns with the updated PRD (PR #83) which simplified: - NetworkClass: Platform-assigned, hidden from users - PublicIP allocation: Checkbox + IP family (automatic pool selection) - Multi-NIC: Removed from current scope (deferred to future phase) Jira: https://redhat.atlassian.net/browse/OSAC-1425 PRD: PR #83 Assisted-by: Claude Code Signed-off-by: Elay Aharoni --- .../networking-ui-vmaas-scope/README.md | 879 ++++++++++++++++++ 1 file changed, 879 insertions(+) create mode 100644 enhancements/networking-ui-vmaas-scope/README.md diff --git a/enhancements/networking-ui-vmaas-scope/README.md b/enhancements/networking-ui-vmaas-scope/README.md new file mode 100644 index 000000000..a19d4c103 --- /dev/null +++ b/enhancements/networking-ui-vmaas-scope/README.md @@ -0,0 +1,879 @@ +--- +title: networking-ui-vmaas-scope +authors: + - eaharoni@redhat.com + - dmanor@redhat.com +creation-date: 2026-06-29 +last-updated: 2026-06-29 +tracking-link: + - https://redhat.atlassian.net/browse/OSAC-1425 +prd: + - prd.md +see-also: + - /enhancements/networking + - /enhancements/vmaas +replaces: + - N/A +superseded-by: + - N/A +--- + +# OSAC Tenant UI: Networking Section + +This enhancement adds a dedicated Networking section to the OSAC tenant UI for managing VirtualNetworks, Subnets, SecurityGroups, and PublicIPs, with integrated inline resource creation in the VMaaS wizard. See [PRD](prd.md) for detailed requirements. + +## Summary + +This design implements a tenant-facing UI for networking resource management using React 19, PatternFly 6, and TanStack Query. The implementation adds list/detail pages for VirtualNetworks, SecurityGroups, and PublicIPs, extends the existing VMaaS wizard with inline networking resource creation, and provides mutation hooks for create/update/delete operations against the fulfillment API. The design follows existing osac-ui patterns for page layout, query hooks, form validation, and wizard integration. + +## Motivation + +Tenant users and tenant admins currently manage networking resources (VirtualNetworks, Subnets, SecurityGroups, PublicIPs) via CLI or direct API calls. When provisioning VMs through the VMaaS wizard, users must pre-create networking resources in separate tools, then context-switch back to the wizard to select them. This creates friction for new tenants attempting their first VM provisioning and reduces discoverability of networking capabilities. + +The fulfillment API already provides full CRUD operations for these resources. The osac-ui codebase includes read-only query hooks (`useVirtualNetworks`, `useSubnets`, `useSecurityGroups`) and a basic wizard networking step that lists resources for selection. This design extends the existing foundation with create/update/delete mutation hooks, dedicated pages for resource management, and inline creation workflows in the wizard. + +The proposed UI leverages PatternFly 6 components (Table, Drawer, Modal, Wizard), TanStack Query for data fetching and cache management, and react-router-dom v7 for navigation. The design reuses existing form components (`SelectField`, `MultiSelectField`, `InputField`) and follows the established pattern for query hooks, page layout, and wizard adapters. + +### Goals + +- Reuse existing osac-ui patterns for query hooks, page layout, form validation, and wizard integration +- Support inline networking resource creation from the VMaaS wizard without leaving the wizard flow +- Provide accessible, responsive UI following PatternFly 6 design system and WCAG standards +- Handle resource lifecycle states (Provisioning, Ready, Failed, Deleting) with appropriate UI feedback and auto-refresh +- Enforce platform constraints (e.g., all NIC attachments must use the same VirtualNetwork) in the wizard UX + +### Non-Goals + +- Provider-only resource management (NetworkClass CRUD, PublicIPPool CRUD, NATGateway, ExternalIPAttachment) +- BaremetalInstance or Cluster networking UI (out of scope for VMaaas phase) +- Migration or enhancement of the existing AdminNetworksPage topology view +- Multi-region VirtualNetwork support, cross-VN NIC attachments, and multi-NIC (multiple network attachments per VM) support (deferred to future phase) + +## Proposal + +This design adds three categories of UI components to osac-ui: + +1. **Pages and components** in `libs/ui-components/src`: list pages for VirtualNetworks, SecurityGroups, and PublicIPs; detail pages with tabbed views for Subnets (VN detail) and Rules (SG detail); create/edit forms in side drawers; delete confirmation modals. + +2. **API hooks** in `libs/ui-components/src/api/v1/networking.ts`: mutation hooks (`useCreateVirtualNetwork`, `useDeleteVirtualNetwork`, `usePatchSecurityGroup`, etc.) following the established pattern in `compute-instance.ts`; single-resource query hooks (`useVirtualNetwork(id)`, `useSecurityGroup(id)`, `usePublicIP(id)`); and invalidation helpers for cache management. + +3. **Wizard extensions** in `libs/ui-components/src/components/catalogProvision/wizard/adapters/computeInstance/VmNetworkingStep.tsx`: inline create drawers for VirtualNetwork, Subnet, and SecurityGroup; single network attachment UI; PublicIP allocation with IP family selection. + +Routing changes in `apps/app-frontend/src/shell` add a "Networking" section to the tenant user and tenant admin sidebars with navigation to `/networking/virtual-networks`, `/networking/security-groups`, and `/networking/public-ips`. + +The design leverages the existing fulfillment API endpoints (`/api/fulfillment/v1/virtual_networks`, `/api/fulfillment/v1/subnets`, `/api/fulfillment/v1/security_groups`, `/api/fulfillment/v1/public_ips`, `/api/fulfillment/v1/public_ip_attachments`). No backend changes are required—the API surface is already stable. + +### Workflow Description + +#### Persona: Tenant User + +**Workflow 1: Create a VirtualNetwork** + +1. User navigates to Networking > Virtual Networks from the sidebar +2. User clicks "Create virtual network" button +3. Side panel opens with form fields: Name (text input), IPv4 CIDR (text input with /16-/24 validation), IPv6 CIDR (optional text input). Note: NetworkClass is assigned automatically by the platform. +4. User fills required fields, sees inline validation errors below invalid fields +5. User clicks "Create" (disabled until all validations pass) +6. POST `/api/fulfillment/v1/virtual_networks` with request body `{ object: { metadata: { name }, spec: { ipv4_cidr, ipv6_cidr } } }` (network_class assigned by backend) +7. Side panel closes, TanStack Query invalidates the VirtualNetworks list query, success toast appears +8. New VirtualNetwork appears in list with "Provisioning" status badge (blue, spinner) +9. Auto-refresh polls every 5 seconds until status becomes "Ready" (green) or "Failed" (red) + +**Workflow 2: Create a Subnet from VirtualNetwork detail page** + +1. User navigates to VirtualNetwork detail page (`/networking/virtual-networks/{id}`) +2. User is on the Subnets tab (default) +3. User clicks "Create subnet" button +4. Side panel opens with parent VN pre-selected (read-only), helper text shows parent VN CIDR and existing subnet CIDRs +5. User enters Name and CIDR (validated: within parent VN CIDR, no overlap with existing subnets) +6. User clicks "Create" +7. POST `/api/fulfillment/v1/subnets` with request body `{ object: { metadata: { name }, spec: { virtual_network, ipv4_cidr } } }` +8. Side panel closes, Subnets table refreshes, new Subnet appears with "Provisioning" status + +**Workflow 3: Create a SecurityGroup with inbound/outbound rules** + +1. User navigates to Networking > Security Groups +2. User clicks "Create security group" +3. Side panel opens with fields: Virtual Network (dropdown), Name (text input), Inbound Rules (expandable section with "Add rule" button), Outbound Rules (expandable section) +4. User selects VirtualNetwork, enters Name +5. User expands Inbound Rules, clicks "Add rule" +6. Inline rule form appears: Protocol (dropdown: TCP/UDP/ICMP/All), Port Range (text input, disabled if ICMP selected), Source CIDR (text input with CIDR validation) +7. User adds multiple rules, clicks "Create" +8. POST `/api/fulfillment/v1/security_groups` with request body `{ object: { metadata: { name }, spec: { virtual_network, inbound_rules: [...], outbound_rules: [...] } } }` +9. Side panel closes, SecurityGroups list refreshes + +**Workflow 4: Allocate and Attach a PublicIP to a VM** + +1. User navigates to Networking > Public IPs +2. User clicks "Allocate IP" +3. Modal dialog opens with fields: Pool (dropdown showing "pool-name (Available: N IPs)"), Name (text input) +4. User selects pool, enters name, clicks "Allocate" +5. POST `/api/fulfillment/v1/public_ips` with request body `{ object: { metadata: { name }, spec: { pool } } }` +6. Modal closes, new PublicIP appears in list with "Available" status +7. User clicks "Attach" action on the row +8. Side panel opens with searchable table of ComputeInstances (VMs) +9. User selects a VM, clicks "Attach" +10. POST `/api/fulfillment/v1/public_ip_attachments` with request body `{ public_ip_id, resource_id, resource_type: "ComputeInstance" }` +11. Side panel closes, PublicIP row updates to show "Attached" status and VM name in "Attached To" column + +**Workflow 5: Provision a VM with inline networking resource creation** + +1. User navigates to VMaaS catalog, selects a template, clicks "Create VM" +2. Wizard opens, user fills basic fields (name, SSH key) +3. User reaches Network Configuration step +4. If no VirtualNetworks exist: prominent message "You need to create a virtual network before provisioning a VM" with "Create Virtual Network" button + - User clicks button, Create VN side drawer overlays wizard + - User creates VN, drawer closes, wizard auto-selects the new VN +5. Network Attachment section shows: Virtual Network (dropdown, auto-selected if only 1), "Create new VN" link +6. Subnet dropdown filters to selected VN (auto-selected if only 1), "Create new Subnet" link +7. Security Groups multi-select checkboxes filter to selected VN (pre-checked if only 1), "Create new Security Group" link +8. Public IP section: checkbox "Allocate Public IP" (default checked for new tenants with no existing IPs), IP Family dropdown (IPv4/IPv6, default IPv4) +9. If checkbox is checked, PublicIP is automatically allocated from available pool based on IP family +10. User clicks "Create VM" +11. POST `/api/fulfillment/v1/public_ips` (if checkbox checked, with IP family), then POST `/api/fulfillment/v1/compute_instances` with networking spec, then POST `/api/fulfillment/v1/public_ip_attachments` (if PublicIP allocated) +12. Wizard closes, user redirected to VM detail page + +**Error handling variations:** + +- **Validation failure:** Form submission blocked, inline error messages appear below invalid fields with specific guidance (e.g., "CIDR must be within parent VN range 10.0.0.0/16") +- **API error (4xx/5xx):** Toast notification with error message, form remains open with user's input preserved, "Retry" button available +- **Provisioning failure:** Resource transitions to "Failed" status, detail page shows collapsible alert with error message from `status.message`, "Retry" and "Delete" actions available. Retry re-submits the original POST request. +- **Delete blocked:** If VirtualNetwork has subnets or security groups, DELETE request returns 400, UI shows error modal: "Cannot delete VirtualNetwork. Delete all subnets and security groups first." + +```mermaid +sequenceDiagram + participant User + participant UI as osac-ui (React) + participant Proxy as Go Proxy + participant API as fulfillment-service + + User->>UI: Navigate to /networking/virtual-networks + UI->>Proxy: GET /api/fulfillment/v1/virtual_networks + Proxy->>API: GET /api/fulfillment/v1/virtual_networks (with tenant context) + API-->>Proxy: VirtualNetworksListResponse + Proxy-->>UI: VirtualNetworksListResponse + UI-->>User: Render list page + + User->>UI: Click "Create virtual network" + UI-->>User: Open side panel form + + User->>UI: Fill form, click "Create" + UI->>Proxy: POST /api/fulfillment/v1/virtual_networks + Proxy->>API: POST (with tenant annotation injection) + API-->>Proxy: VirtualNetwork (status: PENDING) + Proxy-->>UI: VirtualNetwork + UI->>UI: Invalidate list query, refetch + UI-->>User: Close panel, show toast, display new VN with "Provisioning" badge + + loop Every 5 seconds while status != READY|FAILED + UI->>Proxy: GET /api/fulfillment/v1/virtual_networks + Proxy->>API: GET + API-->>Proxy: VirtualNetworksListResponse (status updated) + Proxy-->>UI: VirtualNetworksListResponse + UI-->>User: Update status badge + end +``` + +This sequence diagram illustrates the list-fetch, create, and polling workflow for VirtualNetwork creation. The same pattern applies to Subnets, SecurityGroups, and PublicIPs. + +### API Extensions + +This design does not introduce new API extensions. The fulfillment API already provides the required gRPC services and REST gateway endpoints: + +- `VirtualNetworks` service: List, Get, Create, Patch, Delete +- `Subnets` service: List, Get, Create, Delete (no Patch—subnets are immutable after creation) +- `SecurityGroups` service: List, Get, Create, Patch, Delete +- `PublicIPs` service: List, Get, Create, Delete +- `PublicIPAttachments` service: Create (attach), Delete (detach) +- `PublicIPPools` service: List (read-only for tenants) +- `NetworkClasses` service: List (read-only for tenants) + +The UI consumes these services via the REST gateway (`/api/fulfillment/v1/*`). No CRD changes, webhooks, or finalizers are required—this is a pure frontend implementation. + +### Implementation Details/Notes/Constraints + +#### File Structure + +Following NFR-3, the implementation adds these files to `libs/ui-components/src`: + +**Pages** (new directory: `pages/networking/`): +- `pages/networking/VirtualNetworksPage.tsx` — list page with table, toolbar (search/filter/sort), empty state, "Create" button +- `pages/networking/VirtualNetworkDetailPage.tsx` — detail page with tabs (Subnets, Security Groups, Details), breadcrumb, status badge, Delete action +- `pages/networking/SecurityGroupsPage.tsx` — list page +- `pages/networking/SecurityGroupDetailPage.tsx` — detail page with tabs (Inbound Rules, Outbound Rules, Details) +- `pages/networking/PublicIPsPage.tsx` — list page with Allocate/Attach/Detach/Release actions + +**Components** (new directory: `components/networking/`): +- `components/networking/VirtualNetworkForm.tsx` — create/edit form for VirtualNetwork (used in side drawer and wizard inline creation) +- `components/networking/SubnetForm.tsx` — create form with parent VN CIDR helper text +- `components/networking/SecurityGroupForm.tsx` — create/edit form with inline rule management +- `components/networking/SecurityGroupRuleRow.tsx` — reusable rule input row (Protocol dropdown, Port Range input, CIDR input) +- `components/networking/PublicIPAllocateModal.tsx` — modal dialog for IP allocation +- `components/networking/PublicIPAttachDrawer.tsx` — side drawer with VM selection table +- `components/networking/SubnetsTable.tsx` — table component for Subnets tab +- `components/networking/SecurityGroupsTable.tsx` — table component for SG list and VN detail SG tab +- `components/networking/SecurityGroupRulesTable.tsx` — table component for Inbound/Outbound rules tabs with Add/Edit/Delete actions + +**API hooks** (extend `api/v1/networking.ts`): + +Current state: +```typescript +// Read-only hooks (already exist) +export const useVirtualNetworks = (params?, options?) => useApiQuery(...) +export const useSubnets = (params?, options?) => useApiQuery(...) +export const useSecurityGroups = (params?, options?) => useApiQuery(...) +``` + +New additions: +```typescript +// Single-resource getters +export const useVirtualNetwork = (id: string) => + useApiQuery({ + queryKey: ['v1/virtual_networks', [id]], + meta: { decode: VirtualNetworkSchema }, + enabled: Boolean(id?.trim()), + }); + +export const useSubnet = (id: string) => useApiQuery({ queryKey: ['v1/subnets', [id]], ... }); +export const useSecurityGroup = (id: string) => useApiQuery({ queryKey: ['v1/security_groups', [id]], ... }); +export const usePublicIP = (id: string) => useApiQuery({ queryKey: ['v1/public_ips', [id]], ... }); + +// Mutation hooks +export const useCreateVirtualNetwork = () => { + const apiFetch = useApiFetch(); + const qc = useApiQueryClient(); + return useMutation({ + mutationFn: async (vn: VirtualNetworkInput) => + apiFetch('v1/virtual_networks', { + method: 'POST', + body: { object: vn }, + decode: VirtualNetworkSchema, + }), + onSuccess: () => invalidateVirtualNetworksQueries(qc), + }); +}; + +export const useDeleteVirtualNetwork = () => { + const apiFetch = useApiFetch(); + const qc = useApiQueryClient(); + return useMutation({ + mutationFn: (id: string) => + apiFetch('v1/virtual_networks', { pathParams: [id], method: 'DELETE' }), + onSuccess: () => invalidateVirtualNetworksQueries(qc), + }); +}; + +// Patch hook for SecurityGroups (rule updates) +export const usePatchSecurityGroup = () => { + const apiFetch = useApiFetch(); + const qc = useApiQueryClient(); + return useMutation({ + mutationFn: ({ id, patch }: { id: string; patch: Partial }) => + apiFetch('v1/security_groups', { + pathParams: [id], + method: 'PATCH', + body: { object: patch, field_mask: { paths: Object.keys(patch) } }, + decode: SecurityGroupSchema, + }), + onSuccess: () => invalidateSecurityGroupsQueries(qc), + }); +}; + +// PublicIP attach/detach hooks +export const useAttachPublicIP = () => { + const apiFetch = useApiFetch(); + const qc = useApiQueryClient(); + return useMutation({ + mutationFn: ({ publicIpId, resourceId, resourceType }: AttachPublicIPInput) => + apiFetch('v1/public_ip_attachments', { + method: 'POST', + body: { public_ip_id: publicIpId, resource_id: resourceId, resource_type: resourceType }, + decode: PublicIPAttachmentSchema, + }), + onSuccess: () => { + invalidatePublicIPsQueries(qc); + invalidateComputeInstancesQueries(qc); // Refresh VM list to show attached IP + }, + }); +}; + +export const useDetachPublicIP = () => { + const apiFetch = useApiFetch(); + const qc = useApiQueryClient(); + return useMutation({ + mutationFn: (attachmentId: string) => + apiFetch('v1/public_ip_attachments', { pathParams: [attachmentId], method: 'DELETE' }), + onSuccess: () => { + invalidatePublicIPsQueries(qc); + invalidateComputeInstancesQueries(qc); + }, + }); +}; + +// Cache invalidation helpers +const invalidateVirtualNetworksQueries = async (qc: ReturnType) => { + await qc.invalidateQueries({ queryKey: apiQueryKey('v1/virtual_networks', null) }); +}; +// ... similar invalidation helpers for subnets, security_groups, public_ips +``` + +Pattern follows `api/v1/compute-instance.ts`: mutation hooks use `useMutation` from TanStack Query, call `apiFetch` with method/body/decode, and invalidate relevant queries in `onSuccess`. + +**Wizard integration** (extend existing file): + +`libs/ui-components/src/components/catalogProvision/wizard/adapters/computeInstance/VmNetworkingStep.tsx` currently fetches VirtualNetworks, Subnets, SecurityGroups and renders SelectField/MultiSelectField. The design extends it with: + +1. **Inline create drawers:** "Create new VN" link opens a Drawer overlay with ``. After successful creation (onSuccess callback), the drawer closes and the wizard's VirtualNetwork dropdown refetches and auto-selects the new VN. Same pattern for Subnet and SecurityGroup inline creation. + +2. **Single network attachment:** State variable `attachment: { virtualNetworkId, subnetId, securityGroupIds }`. Platform constraint enforced: exactly one network attachment per VM in this phase (multi-NIC deferred to future phase). + +3. **PublicIP allocation:** Checkbox "Allocate Public IP" (default checked for new tenants with no existing IPs) and IP Family dropdown (IPv4/IPv6, default IPv4). When checked, wizard includes a pre-flight POST to allocate an IP from an appropriate pool based on IP family, then attaches it via POST to public_ip_attachments. + +**Navigation changes:** + +`apps/app-frontend/src/shell/shellNav.ts`: + +```typescript +// In getTenantUserNav(): +{ + kind: 'section', + sectionId: 'nav-tenant-networking', + label: t('Networking'), + children: [ + { id: 'virtual-networks', label: t('Virtual Networks'), path: '/networking/virtual-networks' }, + { id: 'security-groups', label: t('Security Groups'), path: '/networking/security-groups' }, + { id: 'public-ips', label: t('Public IPs'), path: '/networking/public-ips' }, + ], +}, + +// In getTenantAdminNav(), under 'nav-admin-mgmt' section: +{ + kind: 'section', + sectionId: 'nav-admin-networking', + label: t('Networking'), + children: [ + { id: 'virtual-networks', label: t('Virtual Networks'), path: '/networking/virtual-networks' }, + { id: 'security-groups', label: t('Security Groups'), path: '/networking/security-groups' }, + { id: 'public-ips', label: t('Public IPs'), path: '/networking/public-ips' }, + ], +}, +// Preserve existing 'nav-admin-infra' section with admin-networks (topology view) +``` + +`apps/app-frontend/src/shell/AppShell.tsx` adds routes: + +```typescript +} /> +} /> +} /> +} /> +} /> +``` + +#### Data Models + +The UI consumes protobuf-generated TypeScript types from `libs/types/src/osac/public/v1/`. Key interfaces: + +```typescript +interface VirtualNetwork { + id: string; + metadata?: { + name?: string; + labels?: Record; + annotations?: Record; + created_at?: Timestamp; + }; + spec?: { + network_class?: string; + ipv4_cidr?: string; // Required, /16 to /24 + ipv6_cidr?: string; // Optional + }; + status?: { + state?: VirtualNetworkState; // PENDING, READY, FAILED, DELETING + message?: string; + }; +} + +interface Subnet { + id: string; + metadata?: { name?: string; ... }; + spec?: { + virtual_network?: string; // Parent VN ID + ipv4_cidr?: string; // Required, within parent VN CIDR + }; + status?: { + state?: SubnetState; + message?: string; + }; +} + +interface SecurityGroup { + id: string; + metadata?: { name?: string; ... }; + spec?: { + virtual_network?: string; + inbound_rules?: SecurityGroupRule[]; + outbound_rules?: SecurityGroupRule[]; + }; + status?: { state?: SecurityGroupState; message?: string; }; +} + +interface SecurityGroupRule { + protocol?: string; // "TCP" | "UDP" | "ICMP" | "All" + port_range?: string; // e.g., "22", "80-443", empty for ICMP + cidr?: string; // Source (inbound) or Destination (outbound) CIDR + description?: string; +} + +interface PublicIP { + id: string; + metadata?: { name?: string; ... }; + spec?: { + pool?: string; // PublicIPPool ID + address?: string; // IP address (assigned by provider) + }; + status?: { + state?: PublicIPState; // PENDING, AVAILABLE, ATTACHED, FAILED, DELETING + message?: string; + }; +} + +interface PublicIPPool { + id: string; + metadata?: { name?: string; ... }; + spec?: { + cidr?: string; + available_count?: number; + }; +} + +interface NetworkClass { + id: string; + metadata?: { name?: string; ... }; + spec?: { description?: string; }; +} +``` + +#### Form Validation + +All forms use Formik for state management and Yup for schema validation. Validation rules follow PRD requirements: + +**VirtualNetwork:** +- `metadata.name`: required, DNS-valid (RFC 1123 subdomain: lowercase alphanumeric, hyphens, max 63 chars), unique within tenant (uniqueness checked server-side, client shows conflict error from 409 response) +- `spec.network_class`: assigned automatically by the platform (hidden from tenant users, omitted from create request) +- `spec.ipv4_cidr`: required, valid CIDR notation, prefix length between /16 and /24 (Yup regex: `/^(\d{1,3}\.){3}\d{1,3}\/(1[6-9]|2[0-4])$/`) +- `spec.ipv6_cidr`: optional, valid IPv6 CIDR if provided + +**Subnet:** +- `metadata.name`: required, DNS-valid +- `spec.virtual_network`: required (pre-filled from parent VN, read-only in create form) +- `spec.ipv4_cidr`: required, valid CIDR, must be within parent VN CIDR range (client-side check via ip-address library), must not overlap existing subnets (checked by fetching existing subnets for the VN and validating ranges) + +**SecurityGroup:** +- `metadata.name`: required, DNS-valid +- `spec.virtual_network`: required +- `spec.inbound_rules` / `spec.outbound_rules`: + - `protocol`: required, one of ["TCP", "UDP", "ICMP", "All"] + - `port_range`: required if protocol is TCP or UDP, disabled if ICMP, format: single port ("22") or range ("80-443"), validated via regex `/^\d+(-\d+)?$/` + - `cidr`: required, valid CIDR notation + +**PublicIP:** +- `metadata.name`: required (user-provided label) +- `spec.pool`: required, must match an existing PublicIPPool ID + +Error messages follow a consistent template: "{Field name} {validation rule}". Examples: +- "Name is required" +- "IPv4 CIDR must be in /16 to /24 range" +- "Subnet CIDR must be within parent VirtualNetwork CIDR 10.0.0.0/16" +- "Port range is invalid. Use a single port (22) or range (80-443)" + +#### Status Handling and Auto-Refresh + +Resources transition through states: PENDING → READY, PENDING → FAILED, READY → DELETING → (deleted), AVAILABLE → ATTACHED (PublicIPs). + +**Status badge rendering:** +- PENDING (Provisioning): blue (`pf-m-blue`), spinner icon +- READY: green (`pf-m-green`) +- FAILED: red (`pf-m-red`) +- DELETING: orange (`pf-m-orange`), spinner icon +- AVAILABLE (PublicIPs): green +- ATTACHED (PublicIPs): blue + +**Auto-refresh logic:** + +When a resource is in a non-terminal state (PENDING or DELETING), the list page enables auto-refresh via TanStack Query's `refetchInterval`: + +```typescript +const { data: virtualNetworks = [], refetch } = useVirtualNetworks( + {}, + { + refetchInterval: (data) => { + const hasNonTerminalState = data?.some( + (vn) => vn.status?.state === 'PENDING' || vn.status?.state === 'DELETING' + ); + return hasNonTerminalState ? 5000 : false; // 5 seconds if any resource is provisioning/deleting + }, + } +); +``` + +Detail pages use the same pattern. On window focus, TanStack Query auto-refetches (default behavior, no config needed). + +**Delete action behavior:** + +- If resource has children (VN with subnets/SGs, Subnet with attached VMs, SG attached to VMs, PublicIP in ATTACHED state), DELETE request returns 400 with error message. UI shows error modal with message from API response and does not retry. +- If resource is in PENDING or DELETING state, Delete action is disabled (button grayed out). +- Optimistic update: when user clicks Delete and confirms, the row is immediately grayed out and a spinner appears. If the DELETE request fails, the row returns to normal state and an error toast is shown. + +#### Accessibility + +Following NFR-8: + +- **Form labels:** All inputs use PatternFly `FormGroup` with `label` prop, which generates associated `