diff --git a/.ai-bot/instructions.md b/.ai-bot/instructions.md deleted file mode 100644 index 75d32bbc..00000000 --- a/.ai-bot/instructions.md +++ /dev/null @@ -1,4 +0,0 @@ -# Bot Instructions - -Before diagnosing the bug, read `reference/CONVENTIONS.md` -for codebase patterns and naming conventions. diff --git a/AGENTS.md b/AGENTS.md index d0401f70..c40203ae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -135,7 +135,7 @@ workflow) at release time — not by a feature PR. What still needs an explicit - **New CRD types** in `osac-operator` → register in the `fulfillment-service` reconciler (an in-repo change, same PR) - **`osac-ui`** → a real external dependency (OCI chart + image, version-tagged), bumped deliberately when a new release is needed -See `reference/CONVENTIONS.md` for the full dependency table (regenerated via the repo-intel tooling, not hand-edited). +See `reference/CONVENTIONS.md` for the full dependency table (a hand-trimmed excerpt symlinked to `osac/docs/CONVENTIONS.md` — see that file's `README.md` for provenance). ## Enhancement Proposals diff --git a/reference/ARCHITECTURE.md b/reference/ARCHITECTURE.md deleted file mode 100644 index b73eccf0..00000000 --- a/reference/ARCHITECTURE.md +++ /dev/null @@ -1,250 +0,0 @@ -# Architecture - -**Analysis Date:** 2026-07-27 - -## Pattern Overview - -**Overall:** Multi-tier distributed system using Protocol Buffers for API definition, gRPC for service communication, and Kubernetes operators for resource orchestration. - -**Key Characteristics:** -- Public and private API split at the Protocol Buffer definition level -- Generic server implementation for CRUD operations across all resources -- Database-driven (PostgreSQL) with DAO abstraction layer -- Kubernetes operators for orchestration and provisioning -- Provider-based architecture for pluggable provisioning backends (AAP, EDA webhooks) -- Multi-tenancy with tenant isolation at database and authorization layers -- Feedback loop integration between operator and fulfillment service - -## Layers - -**API Layer (Proto-based):** -- Purpose: Defines resource schemas and service contracts -- Location: `fulfillment-service/proto/public/osac/public/v1/` and `fulfillment-service/proto/private/osac/private/v1/` -- Contains: Type definitions (messages), service definitions (RPCs), enums for status -- Depends on: Protocol Buffers standard library, Google API annotations -- Used by: Generated Go code, OpenAPI specifications - -**Server Layer (gRPC):** -- Purpose: Implements gRPC service definitions and REST gateway -- Location: `fulfillment-service/internal/servers/` -- Contains: Resource-specific servers (e.g., `clusters_server.go`), generic server base class, mappers for public/private API translation -- Depends on: Generated API code, database layer, authentication/authorization, notifications -- Used by: gRPC and REST clients via gateway -- Pattern: Builder pattern for server configuration; public servers wrap private servers and add tenant/auth logic - -**Private Servers (Internal RPC):** -- Purpose: Full CRUD operations without user-facing restrictions -- Location: `fulfillment-service/internal/servers/private_*_server.go` -- Contains: All CRUD operations plus Signal RPC for controller feedback -- Depends on: Database DAO, generic server infrastructure -- Used by: Public servers, controllers via gRPC client - -**Public Servers (User-facing):** -- Purpose: Tenant-aware API with read-heavy, limited write operations -- Location: `fulfillment-service/internal/servers/*_server.go` -- Contains: Get/List operations, restricted Create/Update/Delete (system-only) -- Depends on: Private servers, attribution/tenancy logic, mappers -- Used by: External clients (gRPC and REST) - -**Database Layer:** -- Purpose: Persistent storage with multi-tenancy support -- Location: `fulfillment-service/internal/database/`, `fulfillment-service/internal/database/dao/` -- Contains: Generic DAO for type-safe CRUD, filter/sort translation, migrations -- Depends on: PostgreSQL, Protocol Buffers serialization -- Used by: Server layer, controller reconcilers - -**Authentication & Authorization:** -- Purpose: Identity verification, tenancy enforcement, permission checks -- Location: `fulfillment-service/internal/auth/`, `fulfillment-service/internal/idp/`, `fulfillment-service/internal/oauth/` -- Contains: Attribution logic (creator tracking), tenancy logic (tenant identification), OPA integration, identity provider management, OAuth token handling -- Depends on: JWT tokens, OpenPolicy Agent -- Used by: Server layer, generic server infrastructure - -**Supporting Packages (fulfillment-service):** -- `internal/console/` — VM console access support -- `internal/events/` — Event tracking and publishing -- `internal/rendering/` — Table rendering for CLI output (YAML-driven table definitions) -- `internal/network/` — Network-related utilities -- `internal/provisioners/` — Provisioner abstractions -- `internal/computeinstancespec/` — ComputeInstance spec building -- `internal/cache/`, `internal/collections/`, `internal/masks/`, `internal/templating/`, `internal/validation/` — Shared utilities - -**Controller Layer (Kubernetes Operators):** -- Purpose: Reconciliation logic for resource provisioning and lifecycle management -- Location: `osac-operator/internal/controller/`, `fulfillment-service/internal/controllers/` -- Contains: Resource-specific controllers and feedback reconcilers for 30+ resource types including Cluster, ComputeInstance, VirtualNetwork, Subnet, SecurityGroup, PublicIP/ExternalIP, PublicIPPool/ExternalIPPool, PublicIPAttachment/ExternalIPAttachment, NatGateway, Tenant, BaremetalInstance, Project, ProjectMembership, IdentityProvider, Role, RoleBinding, User, and more -- Depends on: controller-runtime, provisioning providers, gRPC client to fulfillment service -- Used by: Kubernetes operator manager - -**Provisioning Providers:** -- Purpose: Pluggable backends for resource provisioning -- Location: `osac-operator/pkg/provisioning/`, `osac-operator/pkg/aap/` -- Contains: Provider interface and factory, AAP provider, provisioning lifecycle helpers, extra vars context -- Depends on: HTTP clients, Ansible API -- Used by: Controllers for provision/deprovision operations - -**Networking/Ansible:** -- Purpose: Low-level provisioning of VMs and network infrastructure -- Location: `osac-aap/collections/ansible_collections/massopencloud/esi/` -- Contains: Ansible roles and playbooks for compute and network provisioning -- Depends on: Ansible, cloud provider SDKs (e.g., OpenStack) -- Used by: AAP job execution triggered by provisioning providers - -## Data Flow - -**Create Resource (Client to Fulfillment):** - -1. Client calls gRPC/REST endpoint on public server (e.g., CreateCluster) -2. Public server maps request to private API representation -3. Private server applies tenancy/attribution metadata -4. Generic server validates and creates database record -5. Success response returned to client -6. Notifier broadcasts change event if configured - -**Resource Provisioning (Fulfillment to Operator):** - -1. Controller polls or watches fulfillment service via Reconciler -2. Reconciler calls GetCluster with filter/watch expressions -3. Fulfillment service returns resource with current status -4. Controller reconciles spec vs status (checks conditions) -5. If action needed, controller triggers provisioning provider -6. Provider (AAP) executes Ansible jobs or webhooks - -**Provisioning Feedback (Operator to Fulfillment):** - -1. Controller receives event from provisioning provider (job completion, status) -2. Feedback controller sends Signal RPC to fulfillment service private API -3. Private server updates resource status, conditions, and finalizers -4. Status change persists to database -5. Public server reflects updated status when queried - -**State Management:** - -- Desired state: Stored in resource Spec (creation_timestamp, deletion_timestamp, labels, annotations) -- Observed state: Stored in resource Status (conditions, phase, last-observed-generation) -- Controller duty: Reconcile observed state toward desired state -- Database as source of truth: Controllers read from DB via gRPC, write back via Signal RPC -- Tenancy enforcement: All queries filtered by tenant ID in annotations - -## Key Abstractions - -**Resource Object (Proto Message):** -- Purpose: Represents infrastructure entities across 35+ resource types spanning compute (Cluster, ComputeInstance, BaremetalInstance), networking (VirtualNetwork, Subnet, SecurityGroup, NatGateway, NetworkClass), IP management (PublicIP, PublicIPPool, PublicIPAttachment, ExternalIP, ExternalIPPool, ExternalIPAttachment), identity (Tenant, Project, User, Role, RoleBinding, ProjectMembership, IdentityProvider), catalog (ClusterCatalogItem, ComputeInstanceCatalogItem, BaremetalInstanceCatalogItem), templates (ClusterTemplate, ComputeInstanceTemplate, BaremetalInstanceTemplate), and platform (HostType, InstanceType, ClusterVersion, Hub, StorageBackend, StorageTier, Event, BreakGlassCredentials) -- Examples: `fulfillment-service/proto/public/osac/public/v1/cluster_type.proto`, `virtual_network_type.proto` -- Pattern: All have metadata (name, labels, annotations, timestamps), spec (user configuration), status (observed state) - -**Generic DAO[O Object]:** -- Purpose: Type-safe database abstraction for any protobuf message -- Examples: `fulfillment-service/internal/database/dao/generic_dao.go` -- Pattern: Uses Go generics; implements Create, Get, List, Update, Delete, Exists; handles JSON serialization - -**Generic Server[O Object]:** -- Purpose: Implements CRUD gRPC operations for any resource -- Examples: `fulfillment-service/internal/servers/generic_server.go` -- Pattern: Parameterized by object type; delegates to DAO; handles field masking, filtering, sorting; enforces tenancy - -**Server Builder:** -- Purpose: Configures server with logger, DAO, notifier, auth logic before instantiation -- Examples: `ClustersServerBuilder`, `NewGenericServer[O]().SetLogger(...).SetService(...).Build()` -- Pattern: Builder pattern eliminates constructor parameter overload; mandatory vs optional configuration explicit - -**Reconciler[O Object]:** -- Purpose: Polls/watches resource changes and executes custom reconciliation logic -- Examples: `fulfillment-service/internal/controllers/reconciler.go`, `osac-operator/internal/controller/` -- Pattern: Takes a ReconcilerFunction that receives current resource state and performs work - -**Provisioning Provider:** -- Purpose: Abstraction for different provisioning backends -- Examples: `osac-operator/pkg/provisioning/aap_provider.go`, `provider.go`, `factory.go` -- Pattern: Interface with Provision/Deprovision/GetStatus methods; factory-based instantiation; provisioning lifecycle helpers for shared provision/deprovision flow - -## Entry Points - -**fulfillment-service binary:** -- Location: `fulfillment-service/cmd/fulfillment-service/main.go` → `internal/cmd/service/Root()` -- Triggers: Service start commands (grpc-server, rest-gateway, controller), dev/probe commands -- Responsibilities: Service initialization, subcommand routing, logging setup - -**gRPC Server:** -- Location: `fulfillment-service/internal/cmd/service/start/grpcserver/` -- Triggers: `fulfillment-service start grpc-server` CLI command -- Responsibilities: Listen on gRPC port, register all service implementations, set up interceptors (panic recovery, metrics, logging, auth, transactions) - -**REST Gateway:** -- Location: `fulfillment-service/internal/cmd/service/start/restgateway/` -- Triggers: `fulfillment-service start rest-gateway` CLI command -- Responsibilities: Translate HTTP/JSON to gRPC; proxy requests to gRPC server; serve OpenAPI specs - -**Controller (Fulfillment):** -- Location: `fulfillment-service/internal/cmd/service/start/controller/` -- Triggers: `fulfillment-service start controller` CLI command -- Responsibilities: Run reconcilers for in-process resource monitoring and feedback -- Controllers: 22+ resource-specific controllers in `internal/controllers/` covering baremetalinstance, cluster, computeinstance, externalip, externalipattachment, externalippool, identityprovider, natgateway, onboarding, project, projectmembership, publicip, publicipattachment, publicippool, role, rolebinding, securitygroup, subnet, tenant, user, virtualnetwork, plus finalizers management - -**osac-operator binary:** -- Location: `osac-operator/cmd/main.go` -- Triggers: Kubernetes operator deployment (helm/kustomize) -- Responsibilities: Initialize multicluster manager, register controllers, set up gRPC client to fulfillment service, start reconciliation loops - -**Operator Controllers (osac-operator):** -- Location: `osac-operator/internal/controller/` -- Pattern: Most resource types have a primary controller and a feedback controller (e.g., `computeinstance_controller.go` + `computeinstance_feedback_controller.go`). Exceptions: BaremetalInstance has a feedback controller only, and Storage has a standalone controller with no feedback pair. A generic `feedback_controller.go` provides shared feedback logic. -- **ClusterOrder**: Provisions OpenShift clusters via Hosted Control Planes -- **ComputeInstance**: Creates KubeVirt VMs, attaches to networks, applies security groups -- **Networking** (VirtualNetwork, Subnet, SecurityGroup, NatGateway): Translates logical network specs to AAP provisioning -- **IP Management** (PublicIP, PublicIPPool, PublicIPAttachment, ExternalIP, ExternalIPPool, ExternalIPAttachment): Manages IP allocation and attachment lifecycle -- **Tenant**: Reconciles tenant namespace isolation -- **BaremetalInstance** (feedback only): Processes bare metal provisioning feedback -- **Storage**: Resolves storage tier configuration -- Triggers: Resource created/updated in Kubernetes or fulfillment service -- Responsibilities: Reconcile spec vs status, trigger provisioning providers, send feedback signals - -**CLI binary:** -- Location: `fulfillment-service/cmd/osac/main.go` → `internal/cmd/cli/Root()` -- Triggers: Manual CLI invocation for cluster/host/compute instance management -- Responsibilities: Provide kubectl-like CLI interface, call fulfillment service gRPC APIs - -**Console Proxy (Operator):** -- Location: `osac-operator/cmd/console-proxy/main.go` → `internal/consoleproxy/` -- Triggers: Deployed as a Kubernetes aggregated API server alongside the operator -- Responsibilities: Provides WebSocket proxy for KubeVirt VM console and VNC access; handles authentication, discovery, health checks, and subresource routing - -**bare-metal-fulfillment-operator:** -- Location: `bare-metal-fulfillment-operator/` -- CRD types: BareMetalInstance, BareMetalPool (`api/v1alpha1/`) -- Controllers: `baremetalinstance_controller.go`, `baremetalpool_controller.go` (`internal/controller/`) -- Triggers: BareMetalInstance/BareMetalPool resources created/updated in Kubernetes -- Responsibilities: Orchestrates bare-metal host provisioning via Metal3 integration; manages pool-based allocation of bare-metal hosts - -## Error Handling - -**Strategy:** Hierarchical error wrapping with context preservation; gRPC status codes mapped to domain errors. - -**Patterns:** - -- DAO errors: Wrapped with table/operation context (e.g., "failed to create cluster in table 'clusters'") -- Server errors: Translated to gRPC status codes (NotFound → Code.NOT_FOUND, AlreadyExists → Code.ALREADY_EXISTS) -- Controller errors: Logged with full context; reconciliation retried with exponential backoff -- Auth errors: Return Unauthenticated or PermissionDenied gRPC codes; tenancy violations logged -- Database transaction errors: Automatically rolled back; client receives clear error message - -## Cross-Cutting Concerns - -**Logging:** slog (Go's structured logging); each layer logs with context (request ID, resource ID, tenant ID); configuration via CLI flags (`--log-level`, `--log-format`) - -**Validation:** CEL expressions in database layer for filtering; Protocol Buffer field presence/constraints at message definition; server-side validation before storage - -**Authentication:** JWT token extraction from gRPC metadata; OAuth2 token file reading for service-to-service auth; token verification delegated to Keycloak - -**Multi-tenancy:** Tenant ID stored in resource annotations (`osac.openshift.io/tenant`); all database queries filtered by tenant automatically; OPA policies enforce isolation at authorization layer - -**Observability:** -- Metrics: Prometheus instrumentation at gRPC interceptor level; controller reconciliation metrics tracked -- Health checks: gRPC health check protocol; Kubernetes liveness/readiness probes -- Events: Database change events published via Notifier; watch streams propagate changes to controllers - -**Transactions:** gRPC interceptor wraps each RPC in database transaction; automatic rollback on error; ensures consistency across CRUD operations - ---- - -*Architecture analysis: 2026-07-27* diff --git a/reference/ARCHITECTURE.md b/reference/ARCHITECTURE.md new file mode 120000 index 00000000..404c7840 --- /dev/null +++ b/reference/ARCHITECTURE.md @@ -0,0 +1 @@ +../osac/docs/ARCHITECTURE.md \ No newline at end of file diff --git a/reference/CONCERNS.md b/reference/CONCERNS.md deleted file mode 100644 index 91245043..00000000 --- a/reference/CONCERNS.md +++ /dev/null @@ -1,477 +0,0 @@ -# Codebase Concerns - -**Analysis Date:** 2026-07-27 - -## Tech Debt - -### Generic Server Type Coupling - -**Area:** Event Notification System - -**Issue:** The `GenericServer[O]` in `fulfillment-service/internal/servers/generic_server.go` contains hard-coded type switches for event payload handling. These type switches couple the generic server to specific message types (ClusterTemplate, Cluster, HostClass, Host, Hub, ComputeInstanceTemplate, ComputeInstance, NetworkClass, VirtualNetwork, Subnet, SecurityGroup), breaking the abstraction. - -**Files:** -- `fulfillment-service/internal/servers/generic_server.go` (lines 669-691, 693-730) - -**Impact:** -- Adding new resource types requires modifying the generic server (violates Open/Closed principle) -- Makes the generic server harder to test — test suite must cover all resource types -- Code duplication exists in both `notifyEvent()` and `setPayload()` methods -- Maintenance burden increases with each new resource type - -**Fix Approach:** -- Extract event payload mapping to a plugin interface or registrable type handler -- Use reflection or a registry pattern to map protobuf message types to event setters -- Consider moving to a `proto.Message` -> oneof handler pattern with dynamic dispatch - ---- - -### PostgreSQL NOTIFY Payload Size Limitation - -**Area:** Event Notification - -**Issue:** Hub objects often exceed PostgreSQL's default 8000 byte notification limit. Currently, the code works around this by cloning the Hub message and stripping the Kubeconfig field (line 705-709), but this is a temporary workaround. - -**Files:** -- `fulfillment-service/internal/servers/generic_server.go` (lines 705-709) - -**Impact:** -- Very large Hub kubeconfigs (>8KB) will silently fail to deliver the Hub object in event notifications -- Event consumers won't have the full object state and must fetch separately -- Workaround is fragile — if kubeconfig can't be stripped (e.g., future message types), notifications break -- Database performance degrades as table of event payloads grows unbounded - -**Fix Approach:** -- Implement a separate event payload table with foreign key to events -- Store full payloads in table, reference by ID in NOTIFY message -- Implement garbage collection for old payloads -- Alternative: switch to LISTEN/NOTIFY for state changes only, clients fetch state via API - ---- - -### Incomplete Phase Mapping in Operator Feedback - -**Area:** Kubernetes Operator Feedback - -**Issue:** The `feedback_controller.go` in osac-operator has unimplemented phase mappings. Specifically, `ClusterOrderPhaseDeleting` has no equivalent phase in the fulfillment service, and the code just returns without updating status. - -**Files:** -- `osac-operator/internal/controller/feedback_controller.go` (line 269) - -**Impact:** -- Clusters in "Deleting" phase will not have their status synced to the fulfillment service -- Operator won't reflect actual deletion state, causing stale status in the API -- Users won't know if a cluster deletion is in progress or stuck -- No monitoring/alerting possible for stuck deletion operations - -**Fix Approach:** -- Map `ClusterOrderPhaseDeleting` to `CLUSTER_STATE_DELETING` (or add this state if missing) -- Implement `syncPhaseDeleting()` to properly signal cluster state during deletion lifecycle -- Add integration test verifying deletion status propagation - ---- - -### Unimplemented Object Filtering - -**Area:** Event Reconciliation - -**Issue:** The `Reconciler` builder in `fulfillment-service/internal/controllers/reconciler.go` (line 194) initializes `objectFilter` to empty string with a TODO comment indicating object filtering support hasn't been implemented yet. - -**Files:** -- `fulfillment-service/internal/controllers/reconciler.go` (lines 193-195) - -**Impact:** -- Reconcilers cannot filter objects before processing, wasting CPU on irrelevant events -- At scale, all reconcilers process all object events regardless of relevance -- No way to optimize reconciliation — e.g., "only reconcile Clusters in 'Progressing' state" -- Performance degrades linearly with total number of resources in the system - -**Fix Approach:** -- Implement CEL filter expression parsing for objects (same pattern as event filters) -- Translate object filters to database WHERE clauses in `ListEvents` calls -- Add filter examples to reconciler builder documentation -- Test with large datasets to validate performance improvement - ---- - -## Known Bugs - -### KubeVirt Console Authentication Limitation - -**Area:** Console Access - -**Issue:** The `kubevirt_backend.go` in `fulfillment-service/internal/console/kubevirt_backend.go` only supports BearerToken authentication. If a hub kubeconfig uses BearerTokenFile, ExecProvider, or client certificates, the WebSocket connection will be unauthenticated. - -**Files:** -- `fulfillment-service/internal/console/kubevirt_backend.go` (lines 134-145) - -**Symptoms:** -- Console connections fail with "unauthenticated" error when hub uses certificate or exec-based auth -- Logs show warning: "Hub REST config has no BearerToken; WebSocket connection may be unauthenticated" -- Users cannot access console for hubs using Kubernetes token files or certificate auth - -**Trigger:** -- Hub kubeconfig generated with auth method other than inline BearerToken -- Any hub using token files, certificate authentication, or authentication plugins - -**Workaround:** -- Export hub kubeconfig token explicitly and patch it into the hub object -- Limitation is documented in the warning message - -**Fix Approach:** -- Use `k8s.io/client-go/transport.New()` to build a round-tripper from the REST config -- Extract all supported auth headers (Authorization, X-Custom-Headers, certificates) from round-tripper -- Pass headers to WebSocket dial configuration -- Test with kubeconfigs using token files, exec providers, and certificates - ---- - -## Security Considerations - -### ALPN Disabled for OpenShift Router Compatibility - -**Area:** gRPC Client TLS Configuration - -**Risk:** The gRPC client in `fulfillment-service/internal/network/grpc_client.go` uses experimental TLS credentials with ALPN disabled to work around OpenShift router limitations. This bypasses ALPN protocol negotiation, which is a security defense-in-depth mechanism. - -**Files:** -- `fulfillment-service/internal/network/grpc_client.go` (lines 307-315) - -**Current Mitigation:** -- Code comment documents the reason and references upstream issues -- Only affects controller-to-service communication, not client-facing API -- OpenShift router is the bottleneck, not our code - -**Recommendations:** -- Track upstream issue: https://github.com/grpc/grpc-go/pull/7980 -- Upgrade gRPC when ALPN support becomes standard in grpc-go (1.67+) -- Consider feature request to OpenShift to enable ALPN in the default router -- Monitor for gRPC 2.0+ which may have different ALPN handling -- Document this as a deployment constraint in operator README - ---- - -### Custom Kubernetes Token Audience Not Configurable - -**Area:** Authentication Configuration - -**Risk:** The Authorino AuthConfig in `fulfillment-service/charts/service/templates/grpc-server/authconfig.yaml` hardcodes Kubernetes API server audiences instead of using a custom audience for the fulfillment service (line 26). - -**Files:** -- `fulfillment-service/charts/service/templates/grpc-server/authconfig.yaml` (lines 26-56) - -**Current Mitigation:** -- Falls back to cluster default audiences (`kubernetes.default.svc`, `kubernetes.default.svc.cluster.local`) -- Works for service accounts with auto-generated tokens -- Different Kubernetes flavors have different audience conventions (Kind vs. OpenShift) - -**Recommendations:** -- Create a custom audience (e.g., `fulfillment-api`) and update AuthConfig -- Users would need to create tokens with custom audience: `kubectl create token -n osac --audience=fulfillment-api client` -- Investigate projected service account token support for controller pods -- Document token creation in the installation guide -- Test with different Kubernetes distributions before rolling out - ---- - -## Performance Bottlenecks - -### Large Integration Test Tool (1.2k+ lines) - -**Area:** Integration Test Infrastructure - -**Problem:** The `it_tool.go` file is 1,263 lines, handling cluster setup, deployment, health checks, and cleanup. This makes it hard to understand test flow and difficult to extend. - -**Files:** -- `fulfillment-service/it/it_tool.go` (1,263 lines) -- `osac-installer/base/osac-fulfillment-service/it/it_tool.go` (1,295 lines — duplicated) - -**Cause:** Combines multiple concerns (Kind management, Helm deployment, health checking, token generation, gRPC client setup) - -**Improvement Path:** -- Split into smaller components: `kind_manager.go`, `helm_deployer.go`, `health_checker.go`, `client_factory.go` -- Extract common test utilities into `internal/testing/` reusable packages -- Reduce code duplication between main and installer versions -- Add unit tests for the tool's individual components - ---- - -### Complex Filter Translator (905 lines) - -**Area:** Database Query Translation - -**Problem:** The `filter_translator.go` in `fulfillment-service/internal/database/dao/` is 905 lines of SQL generation logic for CEL filter expressions. - -**Files:** -- `fulfillment-service/internal/database/dao/filter_translator.go` (905 lines) - -**Impact:** -- Difficult to maintain and test — requires deep knowledge of CEL, SQL, and protobuf reflection -- Query generation logic is hard to verify for correctness and SQL injection safety -- Adding support for new operators requires editing a monolithic file - -**Improvement Path:** -- Break into smaller visitor pattern components for each expression type -- Extract CEL evaluation logic into separate concern from SQL translation -- Add comprehensive integration tests with SQL injection payloads -- Consider using a proven library like `github.com/google/cel-go` + `sqlc` for safer translation - ---- - -### Missing Tests in Core Public Servers - -**Area:** Test Coverage - -**Problem:** Several critical public API servers lack unit tests or have minimal coverage: -- `security_groups_server.go` -- `virtual_networks_server.go` -- `subnets_server.go` -- `clusters_server.go` -- Mapper functions in `generic_mapper.go` - -**Files:** -- `fulfillment-service/internal/servers/*_server.go` (see above) - -**Impact:** -- Regressions in public APIs may not be caught until integration tests -- Tenant isolation bugs could go undetected in unit test phase -- Permission check logic is only tested end-to-end, not in isolation - -**Priority:** High - -**Approach:** -- Add unit test files following existing test pattern (Ginkgo + Gomega) -- Mock the database layer with `gomock` -- Test both positive cases (success) and negative cases (permission denied, not found) -- Test multi-tenant isolation with different tenant IDs - ---- - -## Fragile Areas - -### Boilerplate Test Code in Operator Controllers - -**Files:** -- `osac-operator/internal/controller/*_controller_test.go` - -**Why Fragile:** -- Generated template code contains multiple `TODO(user):` comments -- Tests are incomplete and never fully implemented — assertions are empty or placeholder -- Each controller test file copies the same pattern, creating maintenance burden -- If the template changes, multiple test files get out of sync - -**Safe Modification:** -- Create a test helper package to share common setup logic (context creation, resource creation, reconciliation) -- Use Ginkgo shared behaviors to avoid duplication -- Fill in actual assertions instead of TODO comments -- Add pre-commit hook to catch remaining `TODO(user):` markers - ---- - -### Context.TODO() Usages in Tests - -**Files:** Multiple test files use `context.TODO()` instead of proper context management: -- `osac-operator/internal/controller/webhook_common_test.go:18` -- `osac-operator/internal/controller/suite_test.go:69` -- `osac-operator/internal/controller/*_controller_test.go` (multiple files) - -**Risk:** -- Tests don't respect context cancellation or timeouts -- Goroutine leaks possible if tests hang -- Resource cleanup may not happen properly if context dies - -**Safe Modification:** -- Use `context.WithCancel()` in BeforeSuite with proper cleanup via DeferCleanup -- Apply timeout contexts to individual test cases -- Verify no goroutines are left running after tests - ---- - -## Scaling Limits - -### Hard-Coded Event Payload Type Switch in Generic Server - -**Area:** Resource Type Scalability - -**Limit:** The `GenericServer.setPayload()` method has explicit case statements for each resource type. Supporting more than ~15-20 resource types becomes unwieldy. - -**Current Count:** 30+ resource types (ClusterTemplate, Cluster, HostClass, Host, Hub, ComputeInstanceTemplate, ComputeInstance, NetworkClass, VirtualNetwork, Subnet, SecurityGroup, and others) - -**Scaling Path:** -- Implement type registry pattern: `EventPayloadHandler` interface with `Handle(msg proto.Message, event *Event) error` -- Register handlers at startup: `registry.Register(&ClusterPayloadHandler{})` -- Call `registry.Handle(object, event)` instead of switch statement -- New resource types only need to implement the interface, no server changes needed - ---- - -### gRPC Method Allowlist in OPA Policy - -**Area:** Authorization Policy Maintenance - -**Problem:** The Authorino AuthConfig contains a hard-coded list of allowed gRPC methods for client access (lines 132-164 in `authconfig.yaml`). Adding new public endpoints requires manual list maintenance and redeploy. - -**Files:** -- `fulfillment-service/charts/service/templates/grpc-server/authconfig.yaml` - -**Current Methods:** ~40 client-accessible methods listed - -**Scaling Path:** -- Move method allowlist to ConfigMap for runtime updates without redeploy -- Implement service discovery of public methods via reflection API -- Use annotation-based method classification (`osac.openshift.io/public="true"`) -- Auto-generate allowlist from proto service definitions - ---- - -## Dependencies at Risk - -### Experimental gRPC ALPN Workaround Dependency - -**Risk:** The codebase depends on `google.golang.org/grpc/security/advancedtls/experimentalcredentials.NewTLSWithALPNDisabled()` which is marked experimental. This function may be removed or changed in future gRPC versions. - -**Impact:** If experimental function is removed, gRPC clients to OpenShift will fail - -**Migration Plan:** -1. Monitor gRPC releases for ALPN support in OpenShift -2. Upgrade to new gRPC version when ALPN works with OpenShift router -3. Remove experimental credentials usage and switch back to standard `grpc.WithTransportCredentials(creds.NewTLS(tlsConfig))` -4. Have fallback: use standard credentials, only use experimental if gRPC version < 1.67 - ---- - -### OPA Version Constraint - -**Risk:** Uses `open-policy-agent/opa v1.18.2`. OPA updates may change policy syntax or require policy rewrites. - -**Files:** -- `fulfillment-service/go.mod` lists `github.com/open-policy-agent/opa v1.18.2` -- `fulfillment-service/charts/service/templates/grpc-server/authconfig.yaml` contains Rego policies - -**Current Mitigation:** -- OPA 1.18.x is stable and widely used -- Policies use standard Rego (not bleeding-edge syntax) - -**Recommendations:** -- Pin OPA version in Helm values and document upgrade procedure -- Test policy changes against new OPA versions before upgrading -- Keep Rego policies simple to avoid syntax incompatibilities - ---- - -## Test Coverage Gaps - -### Missing Unit Tests for Server Implementations - -**What's Not Tested:** Public API server implementations for networking resources - -**Files:** -- `fulfillment-service/internal/servers/virtual_networks_server.go` -- `fulfillment-service/internal/servers/subnets_server.go` -- `fulfillment-service/internal/servers/security_groups_server.go` -- `fulfillment-service/internal/servers/network_classes_server.go` - -**Risk:** -- Tenant isolation bugs in CIDR validation or SecurityGroup rules won't be caught until integration tests -- Authorization logic may have edge cases (e.g., cross-tenant subnet creation) -- Permission checks for public vs. admin operations untested - -**Priority:** High - -**Fix:** Add unit test suite following pattern in `fulfillment-service/internal/servers/*_test.go`: -- Mock database layer with gomock -- Test Create with valid/invalid CIDR blocks -- Test List filtering by tenant ID -- Test permission checks (client vs. admin) -- Test multi-tenant isolation - ---- - -### Operator Controller Reconciliation Logic Not Fully Tested - -**What's Not Tested:** Feedback synchronization and phase mapping in operator controllers - -**Files:** -- `osac-operator/internal/controller/feedback_controller.go` — `syncPhase()` logic incomplete -- `osac-operator/internal/controller/clusterorder_controller.go` — complex reconciliation loop - -**Risk:** -- Stale status if phase mapping is incomplete -- Reconciliation loops may miss objects or requeue unnecessarily -- No detection of race conditions in controller logic - -**Priority:** High - -**Fix:** Expand controller tests to cover: -- All phases and transitions (Progressing → Ready, Failed, Deleting) -- Multiple reconciliation cycles -- Concurrent updates from multiple controllers - ---- - -## Missing Critical Features - -### No Garbage Collection for Event Notifications - -**Problem:** Event notifications are stored in PostgreSQL NOTIFY queue, but there's no cleanup mechanism for old or failed notifications. - -**Impact:** -- Database grows unbounded with abandoned events -- Notification queue may get congested -- No way to replay missed events - -**Blocks:** Historical event auditing, event replaying for failed reconcilers - -**Priority:** Medium - -**Fix Approach:** -- Implement event retention policy (e.g., keep events 30 days) -- Add background job to purge old events -- Consider event replication to S3 or external event store for audit trail - ---- - -### No Default Object Filter Implementation in Reconcilers - -**Problem:** Reconcilers cannot filter objects, so all reconcilers process all object change events regardless of relevance. - -**Impact:** -- Scalability limited — CPU usage scales with total resource count, not relevant resources -- No way to optimize reconciliation — e.g., "only process Clusters with state=Progressing" - -**Blocks:** Efficient reconciliation at scale (1000+ resources) - ---- - -## Summary - -| Category | Count | Severity | Effort | -|----------|-------|----------|--------| -| Tech Debt | 3 | Medium | Medium | -| Known Bugs | 1 | Low | Medium | -| Security | 2 | Low | Medium | -| Performance | 3 | Low | Low | -| Fragile Areas | 2 | Medium | Medium | -| Scaling Limits | 2 | Medium | High | -| Dependencies at Risk | 2 | Low | Medium | -| Test Gaps | 3 | High | High | -| Missing Features | 2 | Medium | High | - -**High Priority (Address First):** -1. Add unit tests for public API servers (test coverage) -2. Implement object filtering in reconcilers (blocking scalability) -3. Fix phase mapping in operator feedback (correctness) - -**Medium Priority (Address in Next Cycle):** -1. Refactor generic server type coupling (maintainability) -2. Implement event payload table (robustness) -3. Extract integration test tool components (code quality) - -**Low Priority (Address when time permits):** -1. Update ALPN handling when gRPC/OpenShift alignment improves -2. Fix KubeVirt console authentication for all auth types -3. Implement garbage collection for events - ---- - -*Concerns audit: 2026-07-27* diff --git a/reference/CONVENTIONS.md b/reference/CONVENTIONS.md deleted file mode 100644 index 0116dc87..00000000 --- a/reference/CONVENTIONS.md +++ /dev/null @@ -1,203 +0,0 @@ -# Coding Conventions - -**Analysis Date:** 2026-07-27 - -## Naming Patterns - -**Files:** -- Source files: `snake_case.go` (e.g., `auth_context.go`, `clusters_server.go`) -- Test files: `*_test.go` suffix in same package (e.g., `auth_context_test.go`) -- Mock files: `*_mock.go` generated via mockgen (e.g., `attribution_logic_mock.go`) -- Suite test files: `*_suite_test.go` for package-level setup (e.g., `auth_suite_test.go`) - -**Functions:** -- Public: `PascalCase` (e.g., `NewClustersServer()`, `SubjectFromContext()`, `DetermineAssignedCreators()`) -- Private: `camelCase` (e.g., `getDeletionTimestamp()`, `buildFilter()`) -- Constructor builders: `NewTypeName()` returning `*TypeBuilder` (e.g., `NewClustersServer()`, `NewLogger()`) -- Builder methods: `SetFieldName()` returning `*Builder` for method chaining (e.g., `SetLogger()`, `SetAttributionLogic()`) -- Getter/Setter pairs for single responsibility: `GetName()` / `SetName()` (e.g., `GetId()` / `SetId()`) - -**Variables:** -- Constants: `camelCase` or `SCREAMING_SNAKE_CASE` depending on scope (e.g., `subjectContextKey`, `defaultLimit`) -- Type assertions: `value.(TypeName)` with switch case pattern (e.g., `subject := subject.(type)`) -- Interface types: `camelCase` private (e.g., `metadataIface`) or singular `InterfaceName` public - -**Types:** -- Struct names: `PascalCase` (e.g., `ClustersServer`, `GenericDAO`, `LoggerBuilder`) -- Interface names: `PascalCase` (e.g., `AttributionLogic`, `Object`, `EventCallback`) -- Generic type parameters: Single capital letters (e.g., `[O Object]` for object type, `[T]` for table type) -- Error types: `Err` prefix for error types (e.g., `ErrNotFound`, `ErrAlreadyExists`, `ErrDenied`) - -## Code Style - -**Formatting:** -- Standard Go formatting via `gofmt` (enforced implicitly, no explicit .gofmtrc config) -- Indentation: tabs (Go default) -- Line length: no hard limit, but keep readable - -**Linting:** -- Proto files: `buf lint` (configured in `buf.yaml`) -- YAML files: `yamllint` with strict mode (configured in `.pre-commit-config.yaml`) -- Pre-commit hooks: trailing whitespace, merge conflict markers, large files, JSON validation - -**License Headers:** -- All `.go` and `.proto` files start with Apache 2.0 copyright header -- Format: Multi-line comment block with Red Hat Inc. copyright and Apache 2.0 license - -## Import Organization - -**Order:** -1. Standard library imports (e.g., `context`, `fmt`, `log/slog`) -2. Third-party imports (e.g., `google.golang.org`, `github.com/onsi/ginkgo`) -3. Project internal imports (e.g., `github.com/osac-project/fulfillment-service/internal/...`) - -**Path Aliases:** -- No custom import aliases in standard code -- Generated proto packages under `internal/api/osac/{public,private}/v1/` -- Private internal packages under `internal/` use absolute paths - -**Blank lines:** -- One blank line between import groups -- No custom aliases, use full paths for clarity - -## Error Handling - -**Patterns:** -- Custom error types as structs with fields (e.g., `ErrNotFound{IDs []string}`) -- Error interface implementation via `Error() string` method on error types -- Human-friendly error messages (no technical details in user-facing errors) -- Type switch for error inspection in handlers: - ```go - switch err := err.(type) { - case *ErrNotFound: - // handle not found - case *ErrAlreadyExists: - // handle exists - case *ErrDenied: - // handle denied - default: - // handle generic error - } - ``` -- Panics used for invariant violations (e.g., `SubjectFromContext` panics if subject missing) -- Builder validation errors returned as error strings (e.g., "logger is mandatory") - -## Logging - -**Framework:** `log/slog` (Go 1.21+ standard library) - -**Patterns:** -- Logger passed as `*slog.Logger` in builder/constructor patterns -- Structured logging with context methods: `logger.ErrorContext()`, `logger.InfoContext()` -- Contextual logger creation: `logger.With(...)` adds fields to new logger -- Logger output configured via builder: `SetWriter()`, `SetOut()`, `SetErr()` -- Log level set via builder: `SetLevel()` with string values (e.g., `slog.LevelDebug.String()`) -- Special field handling in logger: `%p` placeholder replaced with process ID -- Test integration: logs written to `GinkgoWriter` in tests for suite output - -## Comments - -**When to Comment:** -- Package-level: Describe the purpose of the file/package (e.g., `// This file contains functions that extract information from the context.`) -- Public types and functions: Always comment exported symbols (required by Go conventions) -- Interface methods: Document the contract and behavior expectations -- Complex logic: Explain why something is done, not just what -- Context keys: Comment purpose (e.g., `// contextKey is the type used to store...`) - -**JSDoc/TSDoc:** -- Go uses standard comment format above exported items -- No special tags (not a requirement of Go style) -- Documentation comments are prose, not structured tags -- Example: - ```go - // SubjectFromContext extracts the subject from the context. Panics if there is no subject in the - // context. - func SubjectFromContext(ctx context.Context) *Subject { - ``` - -## Function Design - -**Size:** -- Functions kept to single responsibility -- Builder methods intentionally short (single setter + return) -- Server methods grouped by operation (Create, Get, List, Update, Delete) - -**Parameters:** -- Context always first parameter in functions that need it -- Builders use typed receiver: `(b *Builder) Method()` -- Interfaces prefer simple parameter types (avoid varargs for clarity) - -**Return Values:** -- Builders return `*Builder` for chaining (e.g., `SetLogger(...).SetNotifier(...).Build()`) -- `Build()` methods return `(Type, error)` with validation errors -- Server methods return `(*ResponseType, error)` matching gRPC pattern -- Error as last return value (Go convention) - -## Module Design - -**Exports:** -- Public types: `PascalCase` (visible outside package) -- Public functions: `PascalCase` (visible outside package) -- Private types: `camelCase` or prefix-only (e.g., `contextKey`) -- Private functions: `camelCase` (not exported) -- Interfaces marked with `var _ InterfaceName = (*ConcreteType)(nil)` assertion - -**Barrel Files:** -- No explicit barrel/index files -- Each package is a unit with its own exports -- Proto-generated code in `internal/api/` organized by version (public/v1, private/v1) - -**Package Organization:** -- `internal/auth/` - Authentication logic, context, token handling -- `internal/servers/` - gRPC service implementations -- `internal/database/` - Database abstraction, migrations -- `internal/database/dao/` - Generic DAO and filter translation -- `internal/logging/` - Logger and interceptor builders -- `internal/testing/` - Test utilities and fixtures -- `proto/` - Protocol Buffer definitions split by API level - -## Builder Pattern - -**Usage:** -- All major components use builder pattern for complex initialization -- Mandatory fields checked in `Build()` method -- Optional fields have sensible defaults -- Method chaining enabled via returning `*Builder` -- Example: - ```go - server, err := NewClustersServer(). - SetLogger(logger). - SetAttributionLogic(attribution). - SetTenancyLogic(tenancy). - Build() - ``` - -## Proto Naming Conventions - -**Files:** `snake_case.proto` (e.g., `virtual_network_type.proto`) - -**Messages:** `PascalCase` (e.g., `VirtualNetwork`, `Cluster`) - -**Fields:** `snake_case` (e.g., `creation_timestamp`, `ipv4_cidr`) - -**Enums:** `SCREAMING_SNAKE_CASE` (e.g., `STATE_PENDING`, `STATE_READY`) - -**Services:** `PascalCase` (e.g., `Clusters`, `VirtualNetworks`) - -**RPC Methods:** `PascalCase` (e.g., `CreateCluster`, `GetVirtualNetwork`) - -## Cross-Repo Dependencies - -When changing one repo, check all dependent repos in this table before submitting: - -| Change in | Also check | Why | -|-----------|-----------|-----| -| `osac-operator` CRD types | `fulfillment-service` reconciler registration | New CRD types must be registered in the fulfillment-service reconciler (in-repo change, same PR) | -| `osac-operator` CRD spec changes | `osac-aap` roles that read CRD fields | Adding a field to `ClusterOrderSpec` requires the AAP playbook to extract and use it | -| `fulfillment-service` CLI flag changes | `osac-test-infra` test helpers | Adding `--pull-secret-file` required updating `OsacCLI.create_cluster` in the test infra (separate repo, separate PR) | - -Evidence: MGMT-24226 eval scored 3/5 because the agent fixed `fulfillment-service` and `osac-aap` but missed updating `osac-installer`'s pinned image version — this was pre-mono-repo-merge (OSAC-1739), when these were still separate repos and image tags were pinned per-commit. That specific mechanism (a separate pinned tag needing a manual re-sync via `sync-image-tags.sh`) no longer exists post-mono-repo — `osac/osac-installer/scripts/sync-image-tags.sh` was removed upstream (`OSAC-3367`) and `osac/osac-installer/values/*/values.yaml` now leaves mono-repo-resident components' image tags unpinned. See the CRD-registration and CLI-flag rows above for what still needs a cross-file check today. - ---- - -*Convention analysis: 2026-07-27* diff --git a/reference/CONVENTIONS.md b/reference/CONVENTIONS.md new file mode 120000 index 00000000..4245b103 --- /dev/null +++ b/reference/CONVENTIONS.md @@ -0,0 +1 @@ +../osac/docs/CONVENTIONS.md \ No newline at end of file diff --git a/reference/INTEGRATIONS.md b/reference/INTEGRATIONS.md deleted file mode 100644 index 012a2ba6..00000000 --- a/reference/INTEGRATIONS.md +++ /dev/null @@ -1,208 +0,0 @@ -# External Integrations - -**Analysis Date:** 2026-07-27 - -## APIs & External Services - -**Kubernetes/OpenShift:** -- OpenShift Container Platform - Deployment target for production - - SDK: github.com/openshift/hypershift/api - HyperShift for hosted control planes (`osac-operator/go.mod`) - - SDK: github.com/openshift/api - OpenShift API extensions - - Purpose: Cluster provisioning and management via HyperShift - - SDK: kubevirt.io/api v1.8.4 - KubeVirt VM management (`osac-operator/go.mod`) - - Purpose: ComputeInstance VM lifecycle via KubeVirt virtualization - - SDK: sigs.k8s.io/multicluster-runtime v0.24.1 - Hub/remote cluster operation (`osac-operator/go.mod`) - - Purpose: Management and workload cluster separation for multi-cluster controllers - -**Infrastructure Provisioning:** -- AWS (Amazon Web Services) - - SDK: boto3, aiobotocore (Python) - Infrastructure provisioning - - Auth: AWS credentials (via AWS_* environment variables or IAM roles) - - Purpose: Compute, networking, and storage provisioning - -- OpenStack - - SDK: openstacksdk, python-openstackclient - Infrastructure management - - Auth: OpenStack credentials (Keystone authentication) - - Purpose: Virtual machine and networking provisioning - -- Ironic (Bare Metal Service) - - SDK: python-ironicclient - Bare metal node management - - Auth: OpenStack/Keystone credentials - - Purpose: Physical server provisioning and control - -- ESI (Emergence System Initiative) - - SDK: esisdk, python-esiclient - Specialized infrastructure - - Auth: Keystone/OpenStack credentials - - Purpose: Advanced infrastructure provisioning workflows - -- ESI LEAP - - SDK: python-esileapclient - Location-based resource provisioning - - Auth: OpenStack credentials - - Purpose: Distributed edge infrastructure management - -**Networking:** -- OVN (Open Virtual Network) - - SDK: github.com/ovn-org/ovn-kubernetes/go-controller (`osac-operator/go.mod`) - - Purpose: Advanced SDN networking for Kubernetes - -- Envoy Gateway - - Configuration: Via Authorino AuthConfig manifests (`fulfillment-service/charts/service/templates/grpc-server/authconfig.yaml`) - - Purpose: TLS/SNI routing, advanced traffic management - -## Data Storage - -**Databases:** -- PostgreSQL 15+ - - Connection: `--db-url` flag (format: `postgres://user:pass@host:5432/db`) - - Client: github.com/jackc/pgx/v5 - Native PostgreSQL driver - - ORM/Query: Direct SQL via pgx (type-safe generic DAO pattern) - - Location: `fulfillment-service/internal/database/` - - Purpose: Persistent storage for all infrastructure resources (clusters, hosts, networks) - -**File Storage:** -- Local filesystem only - No external file storage service -- Kubernetes ConfigMaps/Secrets for configuration storage - -**Caching:** -- None detected - Services use direct database queries with CEL filtering - -## Authentication & Identity - -**Auth Provider:** -- Keycloak (Optional, configurable) - - Implementation: JWT token validation - - Auth types: guest (no auth) or external (JWT-based) - - Configuration: `--grpc-authn-type` flag (guest or external) and `--grpc-authn-external-address` flag - - Location: `fulfillment-service/internal/auth/` - - Purpose: OIDC/OpenID Connect identity provider for users - -**Custom Implementation:** -- Guest authentication (no verification) -- Service account authentication (Kubernetes ServiceAccount tokens) -- Multi-tenancy enforcement via annotations: `osac.openshift.io/tenant` and `osac.openshift.io/owner-reference` -- Location: `fulfillment-service/internal/auth/auth_*.go` - -**Token Management:** -- JWT validation: github.com/golang-jwt/jwt/v5 -- Token sources: File-based (`auth_file_token_source.go`) or memory-based (`auth_memory_token_store.go`) -- External Auth Service: gRPC Envoy ext_authz protocol support for external authorization - -## Authorization - -**Policy Engine:** -- Open Policy Agent (OPA) v1.14.1 - - Purpose: Fine-grained authorization policies - - Integration: Runtime policy enforcement - - Location: `fulfillment-service/internal/` (authorization integration) - - Configuration: OPA policies (location not visible in Go code) - -**Tenancy Logic:** -- Three modes: guest, default, serviceaccount -- Tenant isolation via resource annotations and OPA policies -- Configuration: `--tenancy-logic` flag - -## Monitoring & Observability - -**Error Tracking:** -- None detected - No external error tracking service (Sentry, DataDog, etc.) -- Internal error handling via context-based error propagation - -**Logs:** -- slog (structured logging library) - - Framework: standard Go log/slog - - Output: stdout/stderr (typical Kubernetes container logs) - - Format: Structured JSON-compatible - - Configuration: `--log-level` flag (debug, info, warn, error) - - Location: `fulfillment-service/internal/logging/` - -**Metrics:** -- Prometheus - - Client: github.com/prometheus/client_golang v1.23.2 - - Metrics endpoint: `/metrics` on metrics listener (default `localhost:8080`) - - gRPC metrics: Automatic collection via interceptor chain (`fulfillment-service/internal/metrics/grpc_metrics_interceptor.go`) - - Purpose: Performance monitoring, request latency, error rates - -**Tracing:** -- OpenTelemetry support detected in osac-operator (`go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp`) -- No active span/trace collector configured in fulfillment-service - -## CI/CD & Deployment - -**Hosting:** -- Kubernetes-native deployment (Kind for testing, OpenShift for production) -- Helm-based deployment (`fulfillment-service/charts/service/`) -- Kustomize overlays for alternative deployment (`fulfillment-service/manifests/`) - -**CI Pipeline:** -- GitHub Actions (implied by gh CLI references in CLAUDE.md) -- No explicit CI pipeline files visible in codebase -- Operator SDK v1.39.1 for operator bundle generation and OLM publication - -**Container Registry:** -- Default: ghcr.io (GitHub Container Registry) for osac-operator -- Build targets: `image-build`, `image-push` via Makefile -- Container tool: Podman (default) or Docker - -## Environment Configuration - -**Required Environment Variables:** -- None required at runtime - All configuration via command-line flags -- PostgreSQL connection details passed via `--db-url` flag - -**Optional Environment Variables:** -- AWS credentials: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` (for osac-aap) -- OpenStack credentials: `OS_*` environment variables (for osac-aap) -- Kubernetes config: `KUBECONFIG` for cluster access - -**Secrets Location:** -- `.env` files (not visible in codebase - location TBD) -- Kubernetes Secrets mounted as volumes -- Service Account tokens at `/var/run/secrets/kubernetes.io/serviceaccount/` -- JWT token files passed via `--auth-token-file` flag (if configured) - -## Webhooks & Callbacks - -**Incoming:** -- gRPC streaming endpoints for watch operations (Events Watch) -- Location: `fulfillment-service/internal/servers/events_server.go` -- No HTTP webhooks detected - -**Outgoing:** -- None detected - Services are passive (respond to requests) -- Integration tests use port mappings for TLS/SNI testing (`fulfillment-service/internal/testing/kind.go`) - -## Service-to-Service Communication - -**Within fulfillment-service:** -- gRPC between internal services -- Public API (read operations) wraps Private API (full CRUD) -- Architecture: Public servers delegate to Private servers - -**Between Components:** -- fulfillment-service to osac-operator: Kubernetes custom resources and controllers -- fulfillment-service to osac-aap: Likely via Ansible API or HTTP callbacks (details in osac-aap) -- osac-operator to HyperShift: Via OpenShift API - -## Integration Testing Infrastructure - -**Test Environment:** -- Kind (Kubernetes-in-Docker) cluster named `fulfillment-service-it` -- Requires `/etc/hosts` entries: - - `127.0.0.1 keycloak.keycloak.svc.cluster.local` - - `127.0.0.1 fulfillment-api.osac.svc.cluster.local` -- Deployment modes: Helm (default) or Kustomize (`IT_DEPLOY_MODE` env var) -- Cluster preservation: `IT_KEEP_KIND=true` for debugging - -**Local Development Database:** -```bash -podman run -d --name postgresql_database \ - -e POSTGRESQL_USER=user \ - -e POSTGRESQL_PASSWORD=pass \ - -e POSTGRESQL_DATABASE=db \ - -p 127.0.0.1:5432:5432 \ - quay.io/sclorg/postgresql-15-c9s:latest -``` - ---- - -*Integration audit: 2026-07-27* diff --git a/reference/STACK.md b/reference/STACK.md deleted file mode 100644 index ed1e5f09..00000000 --- a/reference/STACK.md +++ /dev/null @@ -1,135 +0,0 @@ -# Technology Stack - -**Analysis Date:** 2026-07-27 - -## Languages - -**Primary:** -- Go 1.26.3 - Used in `fulfillment-service`, `osac-operator`, and `bare-metal-fulfillment-operator` -- Python 3.13+ - Used in `osac-aap` for Ansible Automation Platform roles and playbooks (`osac-aap/pyproject.toml`) -- Python 3.14+ - Used in `fulfillment-service` for dev tooling (`fulfillment-service/pyproject.toml`) -- Python 3.11+ - Used in `osac-test-infra` for E2E tests (`osac-test-infra/pyproject.toml`) - -**Secondary:** -- YAML - Kubernetes manifests, Helm charts, Ansible playbooks -- Protocol Buffers (proto3) - API definitions for gRPC services - -## Runtime - -**Environment:** -- Go 1.26.3 runtime for compiled services -- Python 3.14+ runtime for fulfillment-service dev tooling -- Python 3.13+ runtime for Ansible automation -- Python 3.11+ runtime for E2E tests (osac-test-infra) -- Kubernetes 1.31+ as deployment target (via Kind for testing, OpenShift for production) - -**Package Manager:** -- Go modules for Go projects -- UV/pip for Python dependencies (`osac-aap/pyproject.toml`) - -## Frameworks - -**Core Services:** -- gRPC v1.82.1 - RPC framework for inter-service communication (`fulfillment-service/go.mod`) -- grpc-gateway v2.29.0 - HTTP/JSON to gRPC transcoding (`fulfillment-service/go.mod`) -- Buf v2 - Protocol Buffer management and code generation (`fulfillment-service/buf.yaml`) - -**Web/API:** -- HTTP/2 support via `golang.org/x/net/http2` and `golang.org/x/net/http2/h2c` -- Prometheus HTTP handlers for metrics exposure - -**Kubernetes:** -- controller-runtime v0.24.1 - Kubernetes operator development framework (`fulfillment-service/go.mod`, `osac-operator/go.mod`, `bare-metal-fulfillment-operator/go.mod`) -- Kubernetes client-go v0.36.2 - Official Kubernetes Go client (`fulfillment-service/go.mod`) -- kubevirt.io/api v1.8.4 - KubeVirt VM management (`osac-operator/go.mod`) -- multicluster-runtime v0.24.1 - Multi-cluster hub/remote operation (`osac-operator/go.mod`) -- OpenShift HyperShift API v0.0.0-20250331235933 - For cluster provisioning (`osac-operator/go.mod`) -- OVN-Kubernetes v0.0.0-20251211123925 - Advanced networking (`osac-operator/go.mod`) - -**Testing:** -- Ginkgo v2.32.0 - BDD testing framework (same version across all repos) -- Gomega v1.42.1 - Assertion library -- Kind (Kubernetes-in-Docker) - Container-based Kubernetes clusters for integration tests (`fulfillment-service/internal/testing/kind.go`) - -**Build/Dev:** -- Cobra v1.10.2 - CLI framework (`fulfillment-service/go.mod`) -- Make - Build orchestration -- Operator SDK v1.39.1 - Kubernetes operator tooling (`osac-operator/Makefile`) - -**Ansible:** -- Ansible 11.4+ - Infrastructure automation platform (`osac-aap/pyproject.toml`) -- Ansible collections: amazon/aws, kubernetes/core, openstack/cloud - Infrastructure providers - -## Key Dependencies - -**Critical:** -- google.golang.org/protobuf v1.36.12 - Protocol Buffer runtime (`fulfillment-service/go.mod`) -- jackc/pgx/v5 v5.10.0 - PostgreSQL driver (`fulfillment-service/go.mod`) -- open-policy-agent/opa v1.18.2 - Authorization and policy evaluation (`fulfillment-service/go.mod`) -- golang-jwt/jwt v5.3.1 - JWT token handling (`fulfillment-service/go.mod`) - -**Infrastructure:** -- prometheus/client_golang v1.23.2 - Prometheus metrics client (`fulfillment-service/go.mod`) -- jackc/pgerrcode - PostgreSQL error code handling (`fulfillment-service/go.mod`) -- golang-migrate/migrate v4.19.1 - Database migration tool (`fulfillment-service/go.mod`) - -**Cloud/Infrastructure:** -- boto3/botocore - AWS SDK for Python (`osac-aap/pyproject.toml`) -- aiobotocore - Async AWS SDK (`osac-aap/pyproject.toml`) -- kubernetes - Python Kubernetes client (`osac-aap/pyproject.toml`) -- openstacksdk - OpenStack SDK (`osac-aap/pyproject.toml`) -- python-ironicclient - Ironic (bare metal service) client (`osac-aap/pyproject.toml`) -- python-esiclient - ESI (Emergence System Initiative) client (`osac-aap/pyproject.toml`) -- python-esileapclient - ESI LEAP (Location Elasticity and Provisioning) client (`osac-aap/pyproject.toml`) -- pydantic v2.11.3+ - Data validation framework (`osac-aap/pyproject.toml`) - -**Networking:** -- envoyproxy/protoc-gen-validate - Protocol Buffer validation (`fulfillment-service/go.mod`) -- dnspython - DNS manipulation library (`osac-aap/pyproject.toml`) - -**Development Utilities:** -- google/cel-go v0.29.2 - Common Expression Language for filtering and authorization (`fulfillment-service/go.mod`) -- go.uber.org/mock v0.6.0 - Mock code generation (`fulfillment-service/go.mod`) - -## Configuration - -**Environment:** -Services accept command-line flags rather than environment variables directly: -- Database URL: `--db-url` flag for PostgreSQL connection (`fulfillment-service/internal/database/database_flags.go`) -- gRPC Listener: `--grpc-listener-address` (default `localhost:8000`) -- HTTP Listener: `--http-listener-address` (default `localhost:8001`) -- Metrics Listener: `--metrics-listener-address` (default `localhost:8080`) -- Authentication type: `--grpc-authn-type` (guest or external) -- Tenancy logic: `--tenancy-logic` (guest, default, or serviceaccount) - -**Build:** -- `buf.yaml` - Protocol Buffer linting and code generation configuration (`fulfillment-service/buf.yaml`) -- Makefile in each component - Build targets and container image management -- Containerfile (Podman/Docker) - Container image definitions - -**Kubernetes Deployment:** -- Helm chart in `fulfillment-service/charts/service/` - Service deployment manifests -- Kustomize overlays in `fulfillment-service/manifests/` - Alternative deployment approach - - Kind overlay for local testing - -## Platform Requirements - -**Development:** -- Go 1.26.3 -- Python 3.14+ (fulfillment-service tooling), 3.13+ (osac-aap), 3.11+ (osac-test-infra) -- Kubernetes cluster (Kind for local testing) or access to OpenShift cluster -- PostgreSQL 15+ (container-based for local development) -- Protocol Buffer compiler (protoc) with Buf -- Docker or Podman for container operations -- Make and standard Unix tools - -**Production:** -- OpenShift 4.x cluster (required for HyperShift and OVN-Kubernetes integration) -- PostgreSQL 15+ database -- Helm 3.x for deployment -- Keycloak for identity provider integration -- Prometheus for metrics collection (optional but recommended) - ---- - -*Stack analysis: 2026-07-27* diff --git a/reference/STRUCTURE.md b/reference/STRUCTURE.md deleted file mode 100644 index dfd296d9..00000000 --- a/reference/STRUCTURE.md +++ /dev/null @@ -1,387 +0,0 @@ -# Codebase Structure - -**Analysis Date:** 2026-07-27 - -## Directory Layout - -``` -osac-workspace/ # Meta-workspace root -├── osac/ # Mono-repo: fulfillment-service + osac-operator + osac-aap + osac-installer + bare-metal-fulfillment-operator + osac-csi-driver -│ ├── fulfillment-service/ # gRPC API server with REST gateway -│ │ ├── cmd/ # Binary entry points -│ │ ├── internal/ # Private implementation -│ │ ├── proto/ # Protocol Buffer definitions -│ │ ├── it/ # Integration tests -│ │ ├── charts/ # Helm deployment manifests -│ │ ├── manifests/ # Kustomize deployment manifests -│ │ ├── docs/ # API documentation -│ │ └── go.mod # Go module definition -│ ├── osac-operator/ # Kubernetes operator -│ │ ├── cmd/ # Operator binary + console-proxy -│ │ ├── api/ # Kubernetes CRD definitions -│ │ ├── internal/ # Private implementation -│ │ ├── pkg/ # Shared packages (provisioning, aap) -│ │ ├── config/ # Kustomize configuration -│ │ ├── test/ # E2E and unit tests -│ │ └── go.mod # Go module definition -│ ├── bare-metal-fulfillment-operator/ # Bare metal host provisioning operator -│ │ ├── cmd/ # Operator binary entry point -│ │ ├── api/ # CRD definitions (BareMetalInstance, BareMetalPool) -│ │ ├── internal/ # Controllers, inventory, profile management -│ │ ├── config/ # Kustomize configuration -│ │ └── charts/ # Helm deployment manifests -│ ├── osac-aap/ # Ansible provisioning -│ │ └── collections/ # Ansible collection with roles/playbooks -│ ├── osac-installer/ # Installation manifests -│ └── osac-csi-driver/ # CSI storage driver -├── osac-test-infra/ # Integration test utilities -├── osac-ui/ # Web console (React, PatternFly 6) -├── osac-ux/ # Read-only UI reference (React 19, PatternFly 6) -├── osac-docs/ # Architecture docs and guides -├── docs/ # Additional documentation -├── enhancement-proposals/ # Design documents -├── CLAUDE.md # Project development guide -└── reference/ # Codebase reference docs (architecture, conventions, stack) -``` - -## Directory Purposes - -All component-relative paths below (e.g., `fulfillment-service/cmd`, -`osac-operator/api/v1alpha1`) are relative to `osac/` — i.e., -`fulfillment-service/cmd` means `osac/fulfillment-service/cmd`. -`osac-test-infra` and `enhancement-proposals` are genuinely separate -top-level repos, not `osac/` subdirectories. - -**fulfillment-service/cmd:** -- Purpose: Binary entry points for service, CLI, and dev tools -- Contains: main.go files for `fulfillment-service`, `osac` (CLI), `osac-dev`, `buf-plugin-osac-lint`, and `test-server` -- Key files: `cmd/fulfillment-service/main.go`, `cmd/osac/main.go`, `cmd/osac-dev/main.go` - -**fulfillment-service/internal/cmd:** -- Purpose: Command hierarchy and CLI structure -- Contains: Root commands, subcommand implementations, service startup logic -- Key files: `service/root_cmd.go`, `service/start/grpcserver/`, `cli/root_cmd.go` - -**fulfillment-service/internal/cmd/service/start:** -- Purpose: Server startup implementations -- Contains: gRPC server, REST gateway, controller initialization -- Key files: `grpcserver/cmd.go`, `restgateway/cmd.go`, `controller/cmd.go` - -**fulfillment-service/internal/servers:** -- Purpose: gRPC service implementations -- Contains: Resource-specific servers (one per resource type) plus generic server base -- Key files: `clusters_server.go`, `private_clusters_server.go`, `generic_server.go`, `generic_mapper.go` -- Pattern: Public server files like `clusters_server.go` contain public API + builder; private implementations in `private_clusters_server.go` - -**fulfillment-service/internal/api/osac:** -- Purpose: Generated code from Protocol Buffers (never edit manually) -- Contains: Auto-generated Go protobuf message types and service stubs -- Subdirectories: `public/v1/` (user-facing), `private/v1/` (admin), `tests/v1/` (test utilities) - -**fulfillment-service/internal/database:** -- Purpose: Database access and persistence -- Contains: Generic DAO implementation, schema management, migration files -- Key files: `dao/generic_dao.go`, `dao/filter_translator.go`, `migrations/*.up.sql` - -**fulfillment-service/internal/database/dao:** -- Purpose: Data access object implementations -- Contains: CRUD operations, event callbacks, table management -- Key files: `generic_dao.go`, `generic_dao_create.go`, `generic_dao_get.go`, `generic_dao_list.go`, `generic_dao_update.go`, `generic_dao_delete.go` - -**fulfillment-service/internal/database/migrations:** -- Purpose: SQL schema migrations -- Contains: Numbered migration files (.up.sql) -- Pattern: Each resource gets a migration creating its table with standard columns (id, name, creation_timestamp, deletion_timestamp, finalizers, creators, tenants, labels, annotations, data) - -**fulfillment-service/internal/auth:** -- Purpose: Authentication, authorization, tenancy -- Contains: Attribution logic (creator tracking), tenancy logic (tenant identification), OPA integration -- Key files: `attribution_logic.go`, `tenancy_logic.go` - -**fulfillment-service/internal/controllers:** -- Purpose: Resource reconciliation logic -- Contains: Reconciler base class, resource-specific controller implementations -- Subdirectories: `cluster/`, `computeinstance/`, `virtualnetwork/`, `subnet/`, `securitygroup/`, `publicip/`, `publicippool/`, `publicipattachment/`, `externalip/`, `externalippool/`, `externalipattachment/`, `natgateway/`, `baremetalinstance/`, `tenant/`, `onboarding/`, `project/`, `projectmembership/`, `user/`, `role/`, `rolebinding/`, `identityprovider/`, `finalizers/` -- Key file: `reconciler.go` (generic reconciliation framework) - -**fulfillment-service/internal/kubernetes:** -- Purpose: Kubernetes integration utilities -- Contains: Label/annotation helpers, CRD helpers, GroupVersionKind definitions -- Subdirectories: `labels/`, `annotations/`, `gvks/` - -**fulfillment-service/proto/public:** -- Purpose: User-facing API definitions -- Contains: Type definitions and service definitions split by resource -- Structure: `osac/public/v1/_type.proto` (messages) and `s_service.proto` (RPC operations) - -**fulfillment-service/proto/private:** -- Purpose: Admin/controller API definitions -- Contains: Full CRUD plus Signal RPC for feedback -- Structure: Same organization as public API; resource-specific files - -**fulfillment-service/proto/tests:** -- Purpose: Test-only protocol definitions -- Contains: Test objects and test service definitions -- Location: `osac/tests/v1/` - -**fulfillment-service/it:** -- Purpose: Integration tests -- Contains: Test suites, test fixtures, kind cluster setup -- Key files: `*_suite_test.go` (ginkgo suites), `crds/` (Kubernetes manifests for test setup) - -**fulfillment-service/charts:** -- Purpose: Helm chart templates for deployment -- Contains: Chart for the main service -- Subdirectories: `service/` (main service chart) - -**fulfillment-service/manifests:** -- Purpose: Kustomize-based deployment manifests -- Contains: Base and overlay configurations -- Structure: `base/` (component manifests), `overlays/{kind,openshift}/` (environment-specific overlays) - -**fulfillment-service/docs:** -- Purpose: Generated documentation -- Contains: Generated API docs, OpenAPI specifications -- Key files: Auto-generated during build - -**osac-operator/cmd:** -- Purpose: Operator binary entry points -- Contains: main.go with operator initialization and controller setup, plus console-proxy -- Key files: `main.go` (operator bootstrap), `console-proxy/` (console proxy binary) - -**osac-operator/api/v1alpha1:** -- Purpose: Kubernetes CRD definitions (ClusterOrder, ComputeInstance, Tenant, networking, IP resources, etc.) -- Contains: Go structs defining CustomResource types -- Key files: `clusterorder_types.go`, `computeinstance_types.go`, `tenant_types.go`, `virtualnetwork_types.go`, `subnet_types.go`, `securitygroup_types.go`, `publicip_types.go`, `publicippool_types.go`, `publicipattachment_types.go`, `externalip_types.go`, `externalippool_types.go`, `externalipattachment_types.go`, `natgateway_types.go`, `job_types.go` - -**osac-operator/internal/controller:** -- Purpose: Kubernetes controller implementations (flat structure, no subdirectories) -- Contains: Resource-specific controllers and feedback controllers for all resource types -- Resources: clusterorder, computeinstance, tenant, virtualnetwork, subnet, securitygroup, publicip, publicippool, publicipattachment, externalip, externalippool, externalipattachment, natgateway, baremetalinstance (feedback only), storage -- Key files: `*_controller.go` (main controller), `*_feedback_controller.go` (feedback from provisioning) - -**osac-operator/pkg/provisioning:** -- Purpose: Provisioning backend abstraction -- Contains: Provider interfaces, AAP provider, EDA webhook provider -- Key files: `provider.go` (interface), `aap_provider.go`, `eda_provider.go` - -**osac-operator/pkg/aap:** -- Purpose: Ansible Automation Platform integration -- Contains: AAP client, template resolution, job submission -- Key files: `client.go` - -**osac-operator/internal/consoleproxy:** -- Purpose: Console proxy implementation for VM console access -- Contains: Proxy server logic for the console-proxy binary - -**osac-operator/internal/migrations:** -- Purpose: CRD migration logic -- Contains: Migration helpers for CRD schema changes - -**osac-operator/config/crd:** -- Purpose: Custom Resource Definition manifests -- Contains: CRD YAML for all operator resources -- Subdirectories: `bases/` (CRD definitions), `fakes/` (test CRDs) - -**osac-operator/config/rbac:** -- Purpose: Role-based access control -- Contains: ClusterRole, ClusterRoleBinding, ServiceAccount manifests -- Generated from controller-gen markers in code - -**bare-metal-fulfillment-operator/cmd:** -- Purpose: Operator binary entry point -- Contains: main.go with operator initialization -- Key file: `main.go` - -**bare-metal-fulfillment-operator/api/v1alpha1:** -- Purpose: Kubernetes CRD definitions for bare metal resources -- Contains: Go structs defining `BareMetalInstance` and `BareMetalPool` types -- Key files: `baremetalinstance_types.go`, `baremetalpool_types.go` - -**bare-metal-fulfillment-operator/internal/controller:** -- Purpose: Kubernetes controller implementations for bare metal resources -- Contains: Controllers for BareMetalInstance and BareMetalPool reconciliation -- Key files: `baremetalinstance_controller.go`, `baremetalpool_controller.go` - -**bare-metal-fulfillment-operator/internal:** -- Purpose: Private implementation packages -- Contains: Controller logic, inventory management, profile handling, shared helpers -- Subdirectories: `controller/`, `inventory/`, `profile/`, `management/`, `helpers/`, `shared/` - -**osac-aap/collections/ansible_collections/massopencloud/esi:** -- Purpose: Ansible collection for infrastructure provisioning -- Contains: Roles and playbooks for VM and network provisioning -- Subdirectories: `roles/` (ansible roles for different provisioning tasks), `plugins/filter/` (custom ansible filters) -- Key roles: `host/`, `l2/`, `l3/`, `floating_ip/`, etc. - -**osac-installer:** -- Purpose: Installation and setup -- Contains: Deployment scripts, prerequisites, demo configurations -- Key files: Setup scripts, prerequisite checklists - -**osac-test-infra:** -- Purpose: Shared testing utilities -- Contains: Kind cluster helpers, test fixtures, common test setup -- Key files: Infrastructure setup and test utilities - -**enhancement-proposals:** -- Purpose: Design documents and RFCs -- Contains: Markdown documents describing features and architecture decisions -- Key files: `.md` design documents - -## Key File Locations - -**Entry Points:** -- `fulfillment-service/cmd/fulfillment-service/main.go`: Service binary entry -- `fulfillment-service/cmd/osac/main.go`: CLI binary entry -- `osac-operator/cmd/main.go`: Operator entry point - -**Configuration:** -- `fulfillment-service/CLAUDE.md`: Development guide with build commands -- `fulfillment-service/go.mod`: Go module and dependencies -- `fulfillment-service/buf.yaml`: Protocol Buffer linting and generation config -- `fulfillment-service/buf.gen.yaml`: Proto code generation rules -- `osac-operator/PROJECT`: Operator project metadata (for controller-gen) - -**Core Logic:** -- `fulfillment-service/internal/servers/clusters_server.go`: Cluster resource server -- `fulfillment-service/internal/servers/private_clusters_server.go`: Cluster private RPC implementation -- `fulfillment-service/internal/servers/generic_server.go`: Generic CRUD server base -- `fulfillment-service/internal/database/dao/generic_dao.go`: Generic data access object -- `fulfillment-service/internal/controllers/reconciler.go`: Resource reconciliation base -- `osac-operator/pkg/provisioning/provider.go`: Provisioning provider interface -- `osac-operator/pkg/aap/client.go`: AAP integration - -**Testing:** -- `fulfillment-service/it/*_suite_test.go`: Integration test suites -- `osac-operator/test/e2e/`: E2E test cases -- `fulfillment-service/internal/testing/`: Shared test utilities - -## Naming Conventions - -**Files:** -- `*_server.go`: Public gRPC server implementation -- `private_*_server.go`: Private gRPC server implementation (admin/controller) -- `*_server_test.go`: Unit tests for server -- `*_controller.go`: Kubernetes controller implementation -- `*_feedback_controller.go`: Feedback handler for provisioning events -- `*_type.proto`: Protocol Buffer message definitions -- `*s_service.proto`: Protocol Buffer service definitions (note: plural resource name + `s`) -- `*_test.go`: Unit test files - -**Directories:** -- `internal/`: Private implementation packages -- `cmd/`: Binary entry points -- `api/`: Public API definitions -- `config/`: Configuration manifests -- `proto/`: Protocol Buffer definitions -- `it/`: Integration tests - -## Where to Add New Code - -**New Resource Type (e.g., LoadBalancer):** - -1. **Proto Definitions:** - - `fulfillment-service/proto/public/osac/public/v1/loadbalancer_type.proto` - Message schema - - `fulfillment-service/proto/public/osac/public/v1/loadbalancers_service.proto` - RPC operations - - `fulfillment-service/proto/private/osac/private/v1/loadbalancer_type.proto` - Private schema (mirror) - - `fulfillment-service/proto/private/osac/private/v1/loadbalancers_service.proto` - Private operations - -2. **Server Implementation:** - - `fulfillment-service/internal/servers/loadbalancers_server.go` - Public server + builder - - `fulfillment-service/internal/servers/private_loadbalancers_server.go` - Private implementation - - `fulfillment-service/internal/servers/loadbalancers_server_test.go` - Server tests - -3. **Database:** - - `fulfillment-service/internal/database/migrations/NNNN_create_loadbalancers.up.sql` - Schema migration - - Auto-used by GenericDAO with appropriate type parameter - -4. **Controller (if operator-managed):** - - `osac-operator/api/v1alpha1/loadbalancer_types.go` - CRD definition - - `osac-operator/internal/controller/loadbalancer_controller.go` - Controller implementation - - `osac-operator/config/crd/bases/` - Generated CRD manifests - -5. **Tests:** - - `fulfillment-service/internal/servers/loadbalancers_server_test.go` - Server unit tests - - `fulfillment-service/it/loadbalancers_suite_test.go` - Integration tests (Ginkgo suite) - - `osac-operator/test/e2e/loadbalancer_test.go` - E2E tests (if operator-managed) - -**New Controller Reconciler (Fulfillment-service):** - -- `fulfillment-service/internal/controllers//reconciler.go` - Reconciliation implementation -- Use pattern from `internal/controllers/cluster/reconciler.go` or `virtualnetwork/reconciler.go` -- Inherit from base `Reconciler[O]` in `controllers/reconciler.go` - -**New Server Middleware/Interceptor:** - -- `fulfillment-service/internal/cmd/service/start/grpcserver/` - Register in gRPC server setup -- Follow pattern of existing interceptors (panic recovery, logging, metrics, auth) - -**Utilities/Helpers:** - -- Shared utilities: `fulfillment-service/internal/utils/`, `fulfillment-service/internal/text/`, `fulfillment-service/internal/json/` -- Kubernetes helpers: `fulfillment-service/internal/kubernetes/{labels,annotations,gvks}/` -- Networking helpers: `fulfillment-service/internal/network/` - -## Special Directories - -**fulfillment-service/internal/api/:** -- Purpose: Generated code (auto-generated by `buf generate`) -- Generated: Yes (do not edit manually) -- Committed: Yes (committed to git for reproducibility) -- Regenerate: After proto changes, run `buf generate` in fulfillment-service root - -**osac-operator/config/crd/bases/:** -- Purpose: Kubernetes CRD manifests (auto-generated) -- Generated: Yes (auto-generated by controller-gen) -- Committed: Yes (committed to git) -- Regenerate: After CRD type changes, run `make manifests` in osac-operator root - -**fulfillment-service/it/crds/:** -- Purpose: Kubernetes manifests used by integration tests -- Generated: No (manually maintained) -- Committed: Yes -- Usage: Loaded by integration tests to set up test cluster state - -**fulfillment-service/charts/service/templates/:** -- Purpose: Helm chart templates for service deployment -- Generated: No (maintained manually) -- Committed: Yes -- Usage: Helm renders templates during installation/upgrade - ---- - -## osac-ux (UI Reference) - -GitHub: [osac-project/osac-ux](https://github.com/osac-project/osac-ux) -pnpm workspace monorepo. Cloned read-only — no PRs from backend sessions. - -``` -osac-ux/ -├── apps/ -│ ├── app-frontend/ # @osac/app-frontend — React 19 SPA (Vite) -│ └── e2e/ # @osac/e2e — Cypress tests -├── libs/ -│ ├── types/ # @osac/types — Buf-generated TS from fulfillment-service protos -│ └── ui-components/ # @osac/ui-components — PatternFly 6 components + API hooks -│ └── src/ -│ ├── pages/ -│ │ ├── tenant/ # screens — tenant role -│ │ ├── admin/ # screens — admin + provider roles -│ │ └── provider/ # screens — provider role only -│ ├── components/ # real UI logic, grouped by domain (vm/, Cluster/, Network/…) -│ └── api/v1/ # TanStack Query hooks per resource -├── proxy/ # Go chi reverse proxy — OIDC auth + API forwarding -├── docs/ # api-query-arch.md, deployment guide -└── AGENTS.md # Authoritative UI coding spec (structure, PatternFly 6, API hooks, UX rules) -``` - -Key reference files for AI agents: -- `libs/ui-components/src/api/types.ts` — ApiRoute union + @temp-api annotations -- `libs/ui-components/src/api/v1/.ts` — @temp-api type definitions -- `apps/app-frontend/src/demo/mock-store.ts` — full mock data for all resources -- `AGENTS.md` — coding spec (PatternFly 6 rules, TypeScript conventions, component structure) - ---- - -*Structure analysis: 2026-07-27* diff --git a/reference/TESTING.md b/reference/TESTING.md deleted file mode 100644 index bfec265b..00000000 --- a/reference/TESTING.md +++ /dev/null @@ -1,431 +0,0 @@ -# Testing Patterns - -**Analysis Date:** 2026-07-27 - -## Test Framework - -**Runner:** -- Ginkgo v2.32.0 (`github.com/onsi/ginkgo/v2`) -- Config: No explicit config file; configuration via environment variables and code - -**Assertion Library:** -- Gomega (`github.com/onsi/gomega`) - -**Run Commands:** -```bash -# Run unit tests only (excludes integration tests in it/) -ginkgo run -r internal - -# Run specific package tests -ginkgo run internal/servers - -# Run tests matching a name pattern -ginkgo run -r internal --focus="CreateCluster" - -# Verbose output -ginkgo run -v internal/servers - -# Skip tests matching pattern -ginkgo run -r internal --skip="database" - -# Run integration tests (requires kind cluster and /etc/hosts entries) -ginkgo run it - -# Preserve kind cluster for debugging -IT_KEEP_KIND=true ginkgo run it - -# Run only setup phase -IT_KEEP_KIND=true ginkgo run --label-filter=setup it - -# Use kustomize instead of Helm for integration tests -IT_DEPLOY_MODE=kustomize ginkgo run it - -# Run all tests including integration -ginkgo run -r -``` - -## Test File Organization - -**Location:** -- Unit tests: Co-located with source files in same package -- Integration tests: Separate `it/` directory at project root -- Test utilities: `internal/testing/` package with shared fixtures - -**Naming:** -- Unit test files: `*_test.go` (e.g., `auth_context_test.go`) -- Suite setup files: `*_suite_test.go` (e.g., `auth_suite_test.go`) -- Integration test files: `it_*.go` (e.g., `it_access_test.go`) - -**Structure:** -``` -fulfillment-service/ -├── internal/ -│ ├── auth/ -│ │ ├── auth_context.go -│ │ ├── auth_context_test.go -│ │ ├── auth_suite_test.go -│ │ └── ... -│ ├── servers/ -│ │ ├── clusters_server.go -│ │ ├── clusters_server_test.go -│ │ ├── servers_suite_test.go -│ │ └── ... -│ └── testing/ -│ ├── database_server.go -│ ├── tool.go -│ └── ... -├── it/ -│ ├── it_suite_test.go -│ ├── it_access_test.go -│ └── ... -``` - -## Test Structure - -**Suite Organization:** - -```go -// In auth_suite_test.go -func TestAuth(t *testing.T) { - RegisterFailHandler(Fail) - RunSpecs(t, "Auth package") -} - -var logger *slog.Logger - -var _ = BeforeSuite(func() { - var err error - logger, err = logging.NewLogger(). - SetLevel(slog.LevelDebug.String()). - SetWriter(GinkgoWriter). - Build() - Expect(err).ToNot(HaveOccurred()) -}) -``` - -**Individual Test Patterns:** - -```go -// In auth_context_test.go -var _ = Describe("Subject inside context", func() { - It("Returns the same subject that was added", func() { - subject := &Subject{} - ctx := ContextWithSubject(context.Background(), subject) - extracted := SubjectFromContext(ctx) - Expect(extracted).To(BeIdenticalTo(subject)) - }) - - It("Panics if there is no subject", func() { - ctx := context.Background() - Expect(func() { - SubjectFromContext(ctx) - }).To(PanicWith("failed to get subject from context")) - }) -}) -``` - -**Test Patterns:** -- Setup: `BeforeEach` for test-local setup, `BeforeSuite` for package-wide setup -- Teardown: `DeferCleanup()` for cleanup operations (called in reverse order) -- Assertions: Gomega matchers with fluent `Expect(value).To(Matcher)` -- Grouping: `Describe` blocks for test suites, `It` blocks for individual tests -- Nesting: `Describe` blocks can be nested for logical organization - -## Mocking - -**Framework:** `go.uber.org/mock` (uber-go/mock) - -**Patterns:** - -```go -// In source file (interface definition) -//go:generate mockgen -destination=attribution_logic_mock.go -package=auth . AttributionLogic -type AttributionLogic interface { - DetermineAssignedCreators(ctx context.Context) (collections.Set[string], error) -} - -// Generated mock is created in attribution_logic_mock.go -// Use in tests with gomock.Controller -``` - -**Mock Usage in Tests:** - -```go -ctrl := gomock.NewController(t) -defer ctrl.Finish() - -mockAttribution := auth.NewMockAttributionLogic(ctrl) -mockAttribution.EXPECT(). - DetermineAssignedCreators(gomock.Any()). - Return(someSet, nil). - Times(1) - -// Pass mock to code under test -server, err := NewClustersServer(). - SetLogger(logger). - SetAttributionLogic(mockAttribution). - Build() -``` - -**What to Mock:** -- External service interfaces (auth logic, tenancy logic) -- Storage layer interfaces (DAO, database) -- Token sources (file, script, static) -- Infrastructure dependencies (Kubernetes client, OPA) - -**What NOT to Mock:** -- Concrete business logic implementations -- Value types (strings, numbers, protobuf messages) -- Standard library types -- Context (use real context.Background()) - -## Fixtures and Factories - -**Test Data:** - -```go -// Builder pattern for test proto messages -request := publicv1.ClustersListRequest_builder{}.Build() - -// Helper to create test subjects -subject := &Subject{ - ID: "test-user", - Type: SubjectTypeUser, - Authenticated: true, -} -``` - -**Location:** -- Shared test utilities: `internal/testing/` (e.g., `DatabaseServer`, `Tool`) -- Package-local test data: Defined inline in test files -- Database setup: `dao.CreateTables[T]()` creates schema dynamically - -**Common Fixtures:** - -```go -// In servers_suite_test.go - package-wide setup -var ( - logger *slog.Logger - server *DatabaseServer - attribution auth.AttributionLogic - tenancy auth.TenancyLogic -) - -// In test BeforeEach -ctx := context.Background() -db := server.MakeDatabase() -pool, err := pgxpool.New(ctx, db.MakeURL()) -tm, err := database.NewTxManager(). - SetLogger(logger). - SetPool(pool). - Build() -tx, err := tm.Begin(ctx) -ctx = database.TxIntoContext(ctx, tx) -err = dao.CreateTables[*privatev1.Cluster](ctx) -``` - -## Coverage - -**Requirements:** No explicit coverage targets enforced, but tests are comprehensive - -**View Coverage:** -- Manual: Run `ginkgo` with `-cover` flag (not commonly used in this project) -- Unit test coverage typically checked in CI/CD pipeline (not documented in CLAUDE.md) - -## Linting - -**fulfillment-service:** -- Go + proto lint: `uv run dev.py lint` (runs Go linter and `buf lint`) -- Proto only: `buf lint` -- Python only: `uv run ruff check` -- Pre-commit hooks run `buf lint` (proto) and Go lint automatically - -**osac-operator:** -- Go lint: `make lint` (golangci-lint) - -**osac-test-infra:** -- Python lint: `make lint` (ruff check + ruff format --check) -- Pre-commit: `pre-commit run --all-files` (yamllint, ansible-lint, standard hooks) - -## Test Types - -**Unit Tests:** -- Scope: Individual functions/methods within a package -- Location: `internal/*/` directories -- Approach: Isolated, use mocks for dependencies, test single behavior per case -- Command: `ginkgo run -r internal` -- Example: `auth_context_test.go` tests context insertion and extraction in isolation - -**Integration Tests:** -- Scope: Multiple components working together, requires infrastructure -- Location: `it/` directory at project root -- Approach: Full kind cluster with PostgreSQL, Keycloak, gRPC/REST endpoints -- Setup: Environment variables (IT_KEEP_KIND, IT_DEPLOY_MODE, Debug) -- Command: `ginkgo run it` -- Infrastructure requirements: - - Kind cluster named `fulfillment-service-it` - - `/etc/hosts` entries for DNS resolution: - - `127.0.0.1 keycloak.keycloak.svc.cluster.local` - - `127.0.0.1 fulfillment-api.osac.svc.cluster.local` - - `127.0.0.1 fulfillment-internal-api.osac.svc.cluster.local` - - Keycloak deployment for auth - - PostgreSQL for data storage - - Helm/Kustomize for application deployment -- Example: `it_access_test.go` tests API access control across public/private endpoints with real clients - -**E2E Tests (osac-test-infra):** -- Framework: pytest with pytest-xdist (parallel execution, default 4 workers) -- Location: `osac-test-infra/tests/` (separate repo) -- Test suites: `vmaas/` (ComputeInstance), `caas/` (ClusterOrder), `storage/` (tenant storage), `catalog/` (CatalogItem) -- Client abstractions: `tests/core/` (GRPCClient, K8sClient, OsacCLI, polling helpers) -- Session fixtures: gRPC client, K8s client, CLI wrapper, JWT tokens (`tests/conftest.py`) -- Two-kubeconfig design: hub cluster (CRs) + workload cluster (VMs via `OSAC_VM_KUBECONFIG`) -- Run commands: - - ```bash - cd osac-test-infra - uv sync - make test # All suites (parallel, 4 workers) - make test-vmaas # VMaaS suite - make test-caas # CaaS suite - make test-storage # Storage suite - TEST=test_compute_instance_lifecycle make test-vmaas # Single test filter - uv run pytest tests/vmaas/ -v --tb=short -n 0 # Sequential (debugging) - ``` - -- Lint: `make lint` (ruff check + ruff format --check) -- Dependencies: pytest>=8.0, pytest-xdist>=3.0, pyyaml>=6.0, websocket-client>=1.6 - -**osac-operator Tests:** -- Framework: Ginkgo + Gomega with envtest (real etcd + kube-apiserver) -- Unit tests: `make test` (runs `go test` excluding `test/integration/`, with coverage) -- Integration tests: `make test-integration` (runs test + test-kustomize + test-smoke) -- Kustomize validation: `make test-kustomize` (catches missing files in kustomization.yaml) -- Smoke test: `make test-smoke` (creates kind cluster `osac-test`, applies CRDs, verifies) -- Kind integration: `make test-integration-kind` (`go test ./test/integration/ -v -ginkgo.v`) -- Lint: `make lint` (golangci-lint) -- Clean up: `kind delete cluster --name osac-test` - -## Common Patterns - -**Async Testing:** - -```go -It("Handles async operations", func() { - done := make(chan bool) - - go func() { - // async work - done <- true - }() - - Eventually(done).Should(Receive()) -}) - -// Or use Ginkgo's built-in async support -Eventually(func() error { - // poll this - return err -}).Should(Succeed()) -``` - -**Error Testing:** - -```go -It("Returns not found error", func() { - _, err := client.Get(ctx, req) - Expect(err).To(MatchError(&dao.ErrNotFound{})) -}) - -It("Panics on missing context", func() { - Expect(func() { - SubjectFromContext(context.Background()) - }).To(PanicWith("failed to get subject from context")) -}) - -It("Fails on invalid build", func() { - server, err := NewClustersServer(). - // missing required fields - Build() - Expect(err).To(MatchError("logger is mandatory")) - Expect(server).To(BeNil()) -}) -``` - -**Database Testing Pattern:** - -```go -BeforeEach(func() { - ctx = context.Background() - db := server.MakeDatabase() - defer db.Close() - - pool, err := pgxpool.New(ctx, db.MakeURL()) - Expect(err).ToNot(HaveOccurred()) - DeferCleanup(pool.Close) - - tm, err := database.NewTxManager(). - SetLogger(logger). - SetPool(pool). - Build() - Expect(err).ToNot(HaveOccurred()) - - tx, err := tm.Begin(ctx) - Expect(err).ToNot(HaveOccurred()) - DeferCleanup(func() { - err := tm.End(ctx, tx) - Expect(err).ToNot(HaveOccurred()) - }) - ctx = database.TxIntoContext(ctx, tx) - - // Create tables dynamically - err = dao.CreateTables[*privatev1.Cluster](ctx) - Expect(err).ToNot(HaveOccurred()) -}) -``` - -**Client Testing (Integration):** - -```go -It("Works with real gRPC client", func() { - ctx := context.Background() - - // Get connection (public user auth) - client := publicv1.NewClustersClient(tool.UserConn()) - - // Make request using proto builder - req := publicv1.ClustersListRequest_builder{}.Build() - resp, err := client.List(ctx, req) - - Expect(err).ToNot(HaveOccurred()) - Expect(resp).ToNot(BeNil()) -}) -``` - -## Test Configuration - -**Environment Variables (Integration Tests):** -- `IT_KEEP_KIND=true` - Preserve kind cluster after tests -- `IT_KEEP_SERVICE=true` - Keep application deployment after tests -- `IT_DEPLOY_MODE=kustomize` - Use kustomize instead of Helm -- `Debug=true` - Enable debugger in container image - -**Suite Setup Pattern:** - -```go -type Config struct { - KeepKind bool `envconfig:"keep_kind" default:"false"` - KeepService bool `envconfig:"keep_service" default:"false"` - DeployMode string `envconfig:"deploy_mode" default:"helm"` - Debug bool `envconfig:"debug" default:"false"` -} - -var _ = BeforeSuite(func() { - err := envconfig.Process("IT", &config) - Expect(err).ToNot(HaveOccurred()) -}) -``` - ---- - -*Testing analysis: 2026-07-27*