Skip to content
134 changes: 134 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Architecture — Cross-Component

This is a hand-trimmed excerpt covering only cross-component concerns that
span multiple `osac/` components (`fulfillment-service`, `osac-operator`,
`bare-metal-fulfillment-operator`) and have no equivalent in any single
component's own `AGENTS.md`. See [README.md](README.md) for provenance and
maintenance notes. Component-specific architecture (code organization,
layers, patterns internal to one component) is authoritative in that
component's own `AGENTS.md` — see `fulfillment-service/AGENTS.md`'s
`## Architecture` section and `osac-operator/AGENTS.md`'s `## Architecture`
section.

## Data Flow

**Create Resource (Client to Fulfillment):**

1. Client calls gRPC/REST endpoint on public server (e.g., CreateCluster)
2. Public server maps request to private API representation
3. Private server applies tenancy/attribution metadata
4. Generic server validates and creates database record
5. Success response returned to client
6. Notifier broadcasts change event if configured

**Resource Provisioning (Fulfillment to Operator):**

1. Controller polls or watches fulfillment service via Reconciler
2. Reconciler calls GetCluster with filter/watch expressions
3. Fulfillment service returns resource with current status
4. Controller reconciles spec vs status (checks conditions)
5. If action needed, controller triggers provisioning provider
6. Provider (AAP) executes Ansible jobs or webhooks

**Provisioning Feedback (Operator to Fulfillment):**

1. Controller receives event from provisioning provider (job completion, status)
2. Feedback controller sends Signal RPC to fulfillment service private API
3. Private server updates resource status, conditions, and finalizers
4. Status change persists to database
5. Public server reflects updated status when queried

**State Management:**

- Desired state: Stored in resource Spec
- Observed state: Stored in resource Status (conditions, phase, last-observed-generation)
- Metadata: creation_timestamp, deletion_timestamp, labels, annotations
- Controller duty: Reconcile observed state toward desired state
- Database as source of truth: Controllers read from DB via gRPC, write back via Signal RPC
- Tenancy enforcement: see Cross-Cutting Concerns → Multi-tenancy, below

## Entry Points

**fulfillment-service binary:**
- Location: `fulfillment-service/cmd/fulfillment-service/main.go` → `internal/cmd/service/Root()`
- Triggers: Service start commands (grpc-server, rest-gateway, controller, console-proxy), dev/probe commands
- Responsibilities: Service initialization, subcommand routing, logging setup

**gRPC Server:**
- Location: `fulfillment-service/internal/cmd/service/start/grpcserver/`
- Triggers: `fulfillment-service start grpc-server` CLI command
- Responsibilities: Listen on gRPC port, register all service implementations, set up interceptors (panic recovery, metrics, logging, auth, transactions)

**REST Gateway:**
- Location: `fulfillment-service/internal/cmd/service/start/restgateway/`
- Triggers: `fulfillment-service start rest-gateway` CLI command
- Responsibilities: Translate HTTP/JSON to gRPC; proxy requests to gRPC server; serve OpenAPI specs

**Controller (Fulfillment):**
- Location: `fulfillment-service/internal/cmd/service/start/controller/`
- Triggers: `fulfillment-service start controller` CLI command
- Responsibilities: Run reconcilers for in-process resource monitoring and feedback
- Controllers: 18 resource-specific controllers in `internal/controllers/` covering baremetalinstance, cluster, computeinstance, externalip, externalipattachment, externalippool, identityprovider, natgateway, onboarding (tenant), project, projectmembership, role, rolebinding, securitygroup, subnet, tenant, user, virtualnetwork, plus shared finalizer management
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] technical-accuracy

The controller list includes 'onboarding (tenant)' alongside a separate 'tenant' entry. The parenthetical '(tenant)' could be misread as indicating onboarding IS the tenant controller, rather than that it operates ON tenant objects (verified: internal/controllers/onboarding/ and internal/controllers/tenant/ are distinct directories with different reconcilers).

Suggested fix: Consider rewording to 'onboarding' alone or 'onboarding (tenant onboarding)' to distinguish from the separate tenant controller.


**Console Proxy (Fulfillment):**
- Location: `fulfillment-service/internal/cmd/service/start/consoleproxy/`
- Triggers: `fulfillment-service start console-proxy` CLI command
- Responsibilities: Issues and validates console-session tickets, proxies gRPC/WebSocket console traffic toward the management cluster's console proxy (see `fulfillment-service/docs/VM_CONSOLE.md` for the full request path); runs as its own Kubernetes Deployment, separate from the gRPC/REST/controller processes

**osac-operator binary:**
- Location: `osac-operator/cmd/main.go`
- Triggers: Kubernetes operator deployment (helm/kustomize)
- Responsibilities: Initialize multicluster manager, register controllers, set up gRPC client to fulfillment service, start reconciliation loops

**Operator Controllers (osac-operator):**
- Location: `osac-operator/internal/controller/`
- Pattern: Most resource types have a primary controller and a feedback controller (e.g., `computeinstance_controller.go` + `computeinstance_feedback_controller.go`), with shared feedback logic in a generic `feedback_controller.go`. Exceptions: BaremetalInstance has a feedback controller only; ClusterOrder folds its status-patching directly into `clusterorder_controller.go` with no separate feedback controller; Storage has a standalone controller with no feedback pair.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
- See `osac-operator/AGENTS.md`'s "Resources Managed" and "Dual-Controller Pattern" sections for the per-resource-type controller list and behavior.
- Triggers: Resource created/updated in Kubernetes or fulfillment service
- Responsibilities: Reconcile spec vs status, trigger provisioning providers, send feedback signals

**CLI binary:**
- Location: `fulfillment-service/cmd/osac/main.go` → `internal/cmd/cli/Root()`
- Triggers: Manual CLI invocation for cluster/host/compute instance management
- Responsibilities: Provide kubectl-like CLI interface, call fulfillment service gRPC APIs

**Console Proxy (Operator):**
- Location: `osac-operator/cmd/console-proxy/main.go`
- Triggers: Deployed as a Kubernetes aggregated API server alongside the operator
- Responsibilities: Proxies KubeVirt VM console/VNC access; see `osac-operator/AGENTS.md`'s "Console Proxy" section for implementation details (auth, discovery, subresource routing, TLS config)

**bare-metal-fulfillment-operator:**
- Location: `bare-metal-fulfillment-operator/`
- CRD types: BareMetalInstance, BareMetalPool (`api/v1alpha1/`)
- Controllers: `baremetalinstance_controller.go`, `baremetalpool_controller.go` (`internal/controller/`)
- Triggers: BareMetalInstance/BareMetalPool resources created/updated in Kubernetes
- Responsibilities: Orchestrates bare-metal host provisioning via Metal3 integration; manages pool-based allocation of bare-metal hosts

## Error Handling

**Strategy:** Hierarchical error wrapping with context preservation; gRPC status codes mapped to domain errors.

**Patterns:**

- DAO errors: Wrapped with table/operation context (e.g., "failed to create cluster in table 'clusters'")
- Server errors: Translated to gRPC status codes (NotFound → Code.NOT_FOUND, AlreadyExists → Code.ALREADY_EXISTS)
- Controller errors: Logged with full context; reconciliation retried with exponential backoff
- Auth errors: Return Unauthenticated or PermissionDenied gRPC codes; tenancy violations logged
- Database transaction errors: Automatically rolled back; client receives clear error message

## Cross-Cutting Concerns

**Logging:** slog (Go's structured logging); each layer logs with context (request ID, resource ID, tenant ID); configuration via CLI flags (`--log-level`, `--log-format`)

**Validation:** Protocol Buffer field presence/constraints at message definition; server-side validation before storage (see `fulfillment-service/AGENTS.md`'s Database Layer section for CEL-based query filtering)

**Authentication:** JWT token extraction from gRPC metadata; OAuth2 token file reading for service-to-service auth; token verification delegated to Keycloak

**Multi-tenancy:** Tenant ID stored in resource annotations (`osac.openshift.io/tenant`); all database queries filtered by tenant automatically; OPA policies enforce isolation at authorization layer

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] technical-accuracy

The Cross-Cutting Concerns section states 'Logging: slog (Go's structured logging)' as a blanket statement across components. This is accurate for fulfillment-service, but osac-operator and bare-metal-fulfillment-operator use zap via controller-runtime's logr interface, and osac-metering also uses logr/zap. Since this document covers cross-component concerns, the claim should be qualified.

Suggested fix: Qualify the logging statement, e.g.: 'Logging: fulfillment-service uses slog; Kubernetes operators and metering use zap via controller-runtime's logr interface.'

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — valid, fixed in 90609f17:

  • [technical-accuracy] docs/ARCHITECTURE.md:143 — Agreed. Verified directly: osac-operator and bare-metal-fulfillment-operator both call controller-runtime's zap.New()/ctrl.SetLogger(), and osac-metering wraps zap via zapr.NewLogger() behind a logr.Logger interface — none of the three use slog. Reworded to attribute slog to fulfillment-service specifically and zap (via controller-runtime's/logr's interface) to the operators and metering.

**Observability:**
- Metrics: Prometheus instrumentation at gRPC interceptor level; controller reconciliation metrics tracked
- Health checks: gRPC health check protocol; Kubernetes liveness/readiness probes
- Events: Database change events published via Notifier; watch streams propagate changes to controllers

**Transactions:** gRPC interceptor wraps each RPC in database transaction; automatic rollback on error; ensures consistency across CRUD operations
20 changes: 20 additions & 0 deletions docs/CONVENTIONS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Conventions — Cross-Repo Dependencies

This is a hand-trimmed excerpt covering only the cross-repo dependency table
that has no equivalent in any single component's own `AGENTS.md`. See
[README.md](README.md) for provenance and maintenance notes.
Component-specific coding conventions (naming, style, imports, error
handling, logging, comments, function/module design, proto naming) are
authoritative in `fulfillment-service/AGENTS.md` and `osac-operator/AGENTS.md`.

## Cross-Repo Dependencies

When changing one repo, check all dependent repos in this table before submitting:

| Change in | Also check | Why |
|-----------|-----------|-----|
| `osac-operator` CRD types | `fulfillment-service` reconciler registration | New CRD types must be registered in the fulfillment-service reconciler (in-repo change, same PR) |
| `osac-operator` CRD spec changes | `osac-aap` roles that read CRD fields | Adding a field to `ClusterOrderSpec` requires the AAP playbook to extract and use it |
| `fulfillment-service` CLI flag changes | `osac-test-infra` test helpers | Adding `--pull-secret-file` required updating `OsacCLI.create_cluster` in the test infra (separate repo, separate PR) |

Evidence: MGMT-24226 eval scored 3/5 because the agent fixed `fulfillment-service` and `osac-aap` but missed updating `osac-installer`'s pinned image version — this was pre-mono-repo-merge (OSAC-1739), when these were still separate repos and image tags were pinned per-commit. That specific mechanism (a separate pinned tag needing a manual re-sync via `sync-image-tags.sh`) no longer exists post-mono-repo — `osac/osac-installer/scripts/sync-image-tags.sh` was removed upstream (`OSAC-3367`) and `osac/osac-installer/values/*/values.yaml` now leaves mono-repo-resident components' image tags unpinned. See the CRD-registration and CLI-flag rows above for what still needs a cross-file check today.
35 changes: 35 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Cross-Component Docs

Hand-trimmed, cross-component-only excerpts of what used to be
`osac-workspace/reference/ARCHITECTURE.md` and `reference/CONVENTIONS.md` —
not raw regenerated codebase-analysis snapshots. Per
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
[ADR 0001](https://github.com/osac-project/osac-workspace/blob/main/decisions/0001-dedicated-ai-skills-repo.md)'s
exit-criterion (e) (OSAC-4008): these two files ([ARCHITECTURE.md](ARCHITECTURE.md),
[CONVENTIONS.md](CONVENTIONS.md)) were trimmed down to only the content with
no equivalent in any single component's own `AGENTS.md`, and relocated
here, co-located with the code they analyze. The other five `reference/*.md`
siblings (`CONCERNS.md`, `INTEGRATIONS.md`, `STACK.md`, `STRUCTURE.md`,
`TESTING.md`) had zero live consumers and will be deleted (rather than
relocated) as part of the companion `osac-workspace` cleanup PR referenced
above.

`ARCHITECTURE.md` says "Cross-Component" and `CONVENTIONS.md` says
"Cross-Repo Dependencies" — used interchangeably for this directory's
scope (content spanning a component/repo boundary with no home in a
single `AGENTS.md`); this isn't a deliberate scope distinction between
the two files.

- **Component-specific architecture/conventions are authoritative in each
component's own `AGENTS.md`** — `fulfillment-service/AGENTS.md`'s and
`osac-operator/AGENTS.md`'s own `## Architecture` sections, not these
files, are the source of truth for anything scoped to one component.
- **`osac-workspace`'s `reference/ARCHITECTURE.md`/`CONVENTIONS.md` will
become symlinks into this directory** once `osac-workspace`'s companion
cleanup PR merges (OSAC-4008 Task 4, sequenced after this PR) — edit
here, not there, once that lands.
- **Regeneration tooling is unconfirmed.** These files originated from a
`.planning/codebase/`-tool-generated snapshot (exact tool/command not
located during OSAC-4008's investigation — TBD, confirm before relying
on any automated regeneration). Any future regeneration must re-apply
this trim (cross-component content only) rather than overwrite these
files wholesale with an unfiltered snapshot.
Loading