RHEcosystemAppEng · GuyZivRH · Apr 14, 2026 · Apr 14, 2026 · Apr 14, 2026 · Apr 14, 2026
diff --git a/.gitignore b/.gitignore
@@ -13,6 +13,7 @@
 Docs/plan_suggestions/
 Docs/commit_reviews/
 Docs/plan_reviews/
+Docs/ab_testing_generalization_plan.md
 
 # Python
 __pycache__/

diff --git a/Docs/ab_testing_generalization_plan.md b/Docs/ab_testing_generalization_plan.md
diff --git a/Docs/implementation_plan.md b/Docs/implementation_plan.md
@@ -6,7 +6,7 @@
 
 ## Overview
 
-Build an automated, Tekton-orchestrated pipeline on OpenShift that accepts skill submissions, validates them, scaffolds skilled/unskilled container variants, builds images, runs Harbor evaluations via a custom OpenShift backend, and produces statistical reports comparing skilled vs. unskilled performance.
+Build an automated, Tekton-orchestrated pipeline on OpenShift that accepts skill submissions, validates them, scaffolds treatment/control container variants, builds images, runs Harbor evaluations via a custom OpenShift backend, and produces statistical reports comparing treatment vs. control performance.
 
 ### Non-Goals
 
@@ -22,7 +22,7 @@ This pipeline spans two repositories:
 | **[ABEvalFlow](https://github.com/RHEcosystemAppEng/ABEvalFlow)** (this repo) | Pipeline definitions, scripts, templates, config | Tekton YAML, Python scripts, Jinja2 templates, Harbor backend |
 | **[agentic-collections](https://github.com/RHEcosystemAppEng/agentic-collections)** | Skills, tasks, tests (post-evaluation) | Persona-based plugins (`rh-sre`, `rh-developer`, `ocp-admin`, etc.), 100+ skills |
 
-The `tasks/` and `tasks-no-skills/` directories generated during scaffolding are **ephemeral workspace artifacts** — they exist only during a pipeline run, not as permanent directories in either repo.
+The `tasks-treatment/` and `tasks-control/` directories generated during scaffolding are **ephemeral workspace artifacts** — they exist only during a pipeline run, not as permanent directories in either repo.
 
 ### Harbor Fork
 
@@ -64,8 +64,7 @@ ABEvalFlow/
 │       ├── analyze-report.yaml     # Step 7
 │       └── publish-store.yaml      # Step 8
 ├── templates/                      # Jinja2 templates for scaffolding
-│   ├── Dockerfile.skilled.j2
-│   ├── Dockerfile.unskilled.j2
+│   ├── Dockerfile.j2
 │   ├── test.sh.j2
 │   └── task.toml.j2
 ├── scripts/                        # Python scripts used by pipeline tasks
@@ -192,8 +191,7 @@ Exit codes: `0` = pass, `1` = validation failure (with structured JSON error out
 
 **Goal:** Create templates that generate the correct Dockerfiles and supporting files.
 
-- [ ] `Dockerfile.skilled.j2` — COPYs `skills/`, `docs/`, `tests/`, `supportive/`, and `instruction.md`.
-- [ ] `Dockerfile.unskilled.j2` — COPYs `tests/`, `supportive/`, and `instruction.md` but **excludes** `skills/` and `docs/`.
+- [x] `Dockerfile.j2` — Unified template using `copy_pairs` loop; COPYs strategy-determined directories plus common files (`tests/`, `supportive/`, `instruction.md`).
 - [ ] `test.sh.j2` — Entry script that runs the agent, then executes `test_outputs.py` and optional `llm_judge.py`.
 - [ ] `task.toml.j2` — Harbor task configuration.
 
@@ -203,9 +201,9 @@ Exit codes: `0` = pass, `1` = validation failure (with structured JSON error out
 
 - Input: path to validated submission directory.
 - Output (ephemeral workspace artifacts, not permanent repo dirs):
-  - `tasks/<skill-name>/` — skilled variant with rendered Dockerfile, test.sh, task.toml.
-  - `tasks-no-skills/<skill-name>/` — unskilled variant.
-- Renders templates with context from `metadata.yaml` and directory inspection (presence of `supportive/`, `docs/`, etc.).
+  - `tasks-treatment/<skill-name>/` — treatment variant with rendered Dockerfile, test.sh, task.toml.
+  - `tasks-control/<skill-name>/` — control variant (baseline).
+- Renders templates with context from `metadata.yaml`, directory inspection, and experiment strategy (which determines copy specs per variant).
 
 ### 2.3 Scaffold Tekton Task (`pipeline/tasks/scaffold.yaml`)
 
@@ -214,22 +212,22 @@ Exit codes: `0` = pass, `1` = validation failure (with structured JSON error out
 
 ### 2.4 Definition of Done
 
-- [ ] Both variants produced with correct Dockerfile COPY directives.
-- [ ] Skilled variant includes skills/docs; unskilled excludes them.
-- [ ] `test.sh` and `task.toml` render correctly for both variants.
-- [ ] Unit tests pass for `scaffold.py`.
+- [x] Both variants produced with correct Dockerfile COPY directives via strategy-driven `copy_pairs`.
+- [x] Treatment variant includes strategy-determined dirs (e.g., skills/docs for skill experiments); control excludes them.
+- [x] `test.sh` and `task.toml` render correctly for both variants.
+- [x] Unit tests pass for `scaffold.py`.
 
 ---
 
 ## Phase 3 — Build & Push Images (Steps 4-5)
 
 ### 3.1 Build Task (`pipeline/tasks/build-push.yaml`)
 
-**Goal:** Build both skilled and unskilled images and push to registry.
+**Goal:** Build both treatment and control images and push to registry.
 
 - **Build tool constraint:** ADR Decision #5 specifies `docker buildx`. However, OpenShift clusters run CRI-O (not Docker) and do not provide a Docker daemon in pods. Using `docker buildx` inside unprivileged Tekton steps requires a Docker-in-Docker sidecar or socket mount, both of which require privileged access and contradict the security posture. **Buildah** (`buildah bud` + `buildah push`) is the standard rootless, daemonless alternative on OpenShift and runs in `ubi9` base images without privilege escalation. This constraint must be reconciled with ADR Decision #5 before implementation — likely by adopting Buildah for OpenShift.
 - Builds from the scaffolded directories.
-- Tags: `<registry>/<namespace>/<skill-name>:skilled-<commit-sha>` and `<registry>/<namespace>/<skill-name>:unskilled-<commit-sha>`.
+- Tags: `<registry>/<namespace>/<skill-name>:treatment-<commit-sha>` and `<registry>/<namespace>/<skill-name>:control-<commit-sha>`.
 - Push to **internal OpenShift registry** for evaluation (per ADR decision #6).
 - Quay promotion happens in Phase 6 (not here) to avoid double-push.
 
@@ -238,31 +236,31 @@ Exit codes: `0` = pass, `1` = validation failure (with structured JSON error out
 - [ ] Create image pull/push secrets for Quay.io.
 - [ ] Configure OpenShift internal registry access for pipeline ServiceAccount.
 - [ ] Define image retention policy (default: 30 days on Quay for reproducibility).
-- [ ] Add `latest-skilled` / `latest-unskilled` floating tags per skill for the monitoring pipeline. **Note:** Digest-based references remain the source of truth for reproducibility; floating tags are monitoring convenience only and may race under concurrent runs.
+- [ ] Add `latest-treatment` / `latest-control` floating tags per skill for the monitoring pipeline. **Note:** Digest-based references remain the source of truth for reproducibility; floating tags are monitoring convenience only and may race under concurrent runs.
 
 ### 3.3 Image Reference Handoff
 
 The `build-push` Tekton task must emit two **results** for downstream consumption:
 
-- `skilled-image-ref` — full digest-based reference (e.g., `registry/ns/skill@sha256:...`)
-- `unskilled-image-ref` — same format
+- `treatment-image-ref` — full digest-based reference (e.g., `registry/ns/skill@sha256:...`)
+- `control-image-ref` — same format
 
 The `pipeline.yaml` wires these to the `harbor-eval` task:
 
 ```yaml
 params:
-  - name: skilled-image
-    value: "$(tasks.build-push.results.skilled-image-ref)"
-  - name: unskilled-image
-    value: "$(tasks.build-push.results.unskilled-image-ref)"
+  - name: treatment-image
+    value: "$(tasks.build-push.results.treatment-image-ref)"
+  - name: control-image
+    value: "$(tasks.build-push.results.control-image-ref)"
 ```
 
 Use digest-based references (not mutable tags) between tasks to avoid tag mutation between push and eval.
 
 ### 3.4 Definition of Done
 
 - [ ] Both variants built and pushed to OpenShift internal registry.
-- [ ] `skilled-image-ref` and `unskilled-image-ref` emitted as Tekton results (digest-based).
+- [ ] `treatment-image-ref` and `control-image-ref` emitted as Tekton results (digest-based).
 - [ ] Push secrets functional.
 
 ---
@@ -314,8 +312,8 @@ Additional requirements:
 
 ### 4.4 Trial Execution Configuration
 
-- The `harbor-eval` Tekton task accepts `skilled-image-ref` and `unskilled-image-ref` as **params** wired from Phase 3 results.
-- N = 20 attempts per variant (skilled + unskilled = 40 total sessions).
+- The `harbor-eval` Tekton task accepts `treatment-image-ref` and `control-image-ref` as **params** wired from Phase 3 results.
+- N = configurable attempts per variant (default 20, treatment + control = 40 total sessions).
 - Configure resource requests/limits per trial Pod.
 - LLM endpoint configured via environment variable — backend is agnostic to whether it points to LiteLLM, a direct API, or a self-hosted model.
 - Trial Pod timeout: configurable, with a global evaluation timeout.
@@ -335,7 +333,7 @@ The pipeline ServiceAccount needs (prefer named Secrets for least-privilege wher
 
 ### 4.6 Definition of Done
 
-- [ ] 40 trial Pods complete (20 skilled + 20 unskilled).
+- [ ] Trial Pods complete (N per variant × 2 variants, default 40 total).
 - [ ] Cleanup verified — no stale Pods after evaluation.
 - [ ] Retry behavior validated for transient failures.
 - [ ] Unit tests pass with mocked K8s API.
@@ -350,8 +348,8 @@ The pipeline ServiceAccount needs (prefer named Secrets for least-privilege wher
 **Goal:** Consume Harbor output and produce a statistical report.
 
 Metrics to compute:
-- **Pass rate** per variant (skilled, unskilled).
-- **Skills uplift (gap):** `pass_rate_skilled - pass_rate_unskilled`.
+- **Pass rate** per variant (treatment, control).
+- **Uplift (gap):** `pass_rate_treatment - pass_rate_control`.
 - **Statistical significance:** p-value via Fisher's exact test or chi-squared.
 - **Heatmap generation:** matplotlib/seaborn figures saved as PNG.
 - **LLM judge scores** (when `llm_judge.py` is present): include a qualitative score summary section. Define a schema for `llm_judge.py` output (JSON with `score`, `rationale`) to ensure `analyze.py` can reliably parse it.
@@ -494,7 +492,7 @@ The pipeline and Harbor backend are agnostic — they pass LLM config as environ
 
 ### 8.5 Cost Controls & Observability
 
-A single evaluation run consumes 40 LLM sessions (N=20 x 2 variants). Cost management is a first-class concern:
+A single evaluation run consumes N × 2 LLM sessions (default N=20, 40 total). Cost management is a first-class concern:
 
 - [ ] Configure LiteLLM per-key budget limits (when using Vertex mode).
 - [ ] Implement pre-flight cost estimate: before launching Harbor, estimate token usage based on skill complexity and configured N. Log the estimate to the run summary to flag potential runaway cost before spend happens.

diff --git a/Docs/workstreams_roadmap.md b/Docs/workstreams_roadmap.md
@@ -0,0 +1,149 @@
+# Workstreams Roadmap
+
+> Last updated: 2026-04-14
+
+## Overview
+
+Four workstreams to complete the ABEvalFlow pipeline. WS1 is the critical path — it renames skilled/unskilled to treatment/control across the codebase and adds the A/B experiment framework.
+
+```mermaid
+flowchart LR
+    WS1["WS1: A/B Generalization\n(APPENG-4932)"] --> WS2["WS2: Build & Push\n(APPENG-4905)"]
+    WS2 --> WS3B["WS3B: harbor-eval.yaml\n(APPENG-4906)"]
+    WS1 -.-> WS3A["WS3A: Harbor Doc Update\n(APPENG-4906)"]
+    WS1 -.-> WS4["WS4: Trigger Doc Update"]
+```
+
+---
+
+## Current State (2026-04-14)
+
+| Item | Status |
+|------|--------|
+| PR #1 — Phase 1 validation (APPENG-4903) | Merged |
+| PR #2 — Tekton triggers + validate task (APPENG-4903) | Merged |
+| PR #3 — Phase 2 scaffolding (APPENG-4904) | Merged |
+| PR #4 — Rename to ABEvalFlow | Merged |
+| Branch `APPENG-4905/phase-3-build-push` | Stale — forked from `c98b547`, missing PRs #1-4. Abandoned. |
+| Harbor OpenShift backend (`skills_eval_corrections`) | Feature-complete in fork, unit tested |
+
+---
+
+## WS1: A/B Eval Flow Conversion (APPENG-4932)
+
+**Branch:** `APPENG-4932/ab-eval-flow-conversion`
+**Plan:** [ab_testing_generalization_plan.md](./ab_testing_generalization_plan.md)
+
+Refactor the pipeline from hardcoded "skilled vs unskilled" to a general "treatment vs control" A/B framework. Skills remain the default experiment type. Adds support for model, prompt, and custom experiment types via a strategy pattern.
+
+### Execution Steps
+
+See the detailed commit plan in [ab_testing_generalization_plan.md](./ab_testing_generalization_plan.md).
+
+| Step | Commit | Key files | Tests |
+|------|--------|-----------|-------|
+| 1 | Schema: `ExperimentConfig`, `VariantSpec`, `CopySpec` | `abevalflow/schemas.py` | `tests/test_validate.py` |
+| 2 | Strategy pattern | `abevalflow/experiment.py` (new) | `tests/test_experiment.py` (new) |
+| 3 | Scaffold refactor | `scripts/scaffold.py` | `tests/test_scaffold.py` |
+| 4 | Template unification | `templates/Dockerfile.j2` (new), delete old | existing tests cover |
+| 5 | Tekton YAML renames | `pipeline/tasks/scaffold.yaml` | N/A (YAML only) |
+| 6 | Docs + README terminology | `implementation_plan.md`, `README.md` | N/A |
+
+---
+
+## WS2: Build & Push (APPENG-4905)
+
+**Branch:** `APPENG-4905/build-push-treatment-control` (to be created after WS1 merges)
+**Depends on:** WS1 merged
+
+Recreate the build-push Tekton task from scratch on current `main` using treatment/control naming. The old `APPENG-4905/phase-3-build-push` branch is abandoned — it diverged from `c98b547` (before PRs #1-4) and would require a conflict-heavy rebase with no benefit.
+
+### What to build
+
+- `pipeline/tasks/build-push.yaml` — Buildah-based build and push for treatment/control images
+  - Params: `treatment-task-dir`, `control-task-dir`, `skill-name`, `commit-sha`, `registry-url`, `registry-namespace`
+  - Results: `treatment-image-ref`, `control-image-ref` (digest-based)
+  - Steps: `build-push-treatment`, `build-push-control` (rootless Buildah, `--storage-driver=vfs`)
+  - Image tags: `:treatment-<sha>`, `:control-<sha>`
+  - Namespace: `ab-eval-flow`
+- `config/rbac.yaml` — RoleBinding for `system:image-builder` in `ab-eval-flow` namespace
+- Update `Docs/implementation_plan.md` — Phase 3 checkboxes
+
+### Reference
+
+The old branch had a working `build-push.yaml` (109 lines) that can be used as a starting point, just with renamed params/results/steps and updated namespace.
+
+### Cleanup
+
+After this merges, delete the old `APPENG-4905/phase-3-build-push` branch (local and remote).
+
+---
+
+## WS3: Harbor Backend (APPENG-4906)
+
+### WS3A: Harbor Handoff Doc Update
+
+**Can run in parallel** with any workstream.
+
+Update [harbor_openshift_backend.md](./harbor_openshift_backend.md) to match the actual implementation in `src/harbor/environments/openshift.py`:
+
+| Doc says | Reality | Action |
+|----------|---------|--------|
+| File: `openshift_environment.py` | `openshift.py` | Fix filename |
+| `_build_and_push_image` is no-op only | Supports pre-built (`image_ref`) AND podman build | Document both modes |
+| `readOnlyRootFilesystem: true` | Intentionally unset (many workloads need writes) | Update security section |
+| RBAC: ConfigMaps, Secrets, ImageStreams | Only Pods + exec used | Narrow RBAC table |
+| skilled/unskilled terminology | treatment/control after WS1 | Update naming |
+
+### WS3B: harbor-eval.yaml Tekton Task
+
+**Depends on:** WS2 merged (needs image ref handoff)
+
+New `pipeline/tasks/harbor-eval.yaml` in ABEvalFlow:
+- Params: `treatment-image-ref`, `control-image-ref`, `n-trials`, `namespace`
+- Runs `harbor run --env openshift --ek image_ref=<ref> --ek namespace=<ns>`
+- Collects results to workspace/PVC
+
+### Investigation needed before WS3B
+
+Verify Harbor's CLI interface for trial count: `--runs`, `--n-trials`, config-based, or passed via `task.toml`. Check `harbor/cli/tasks.py` for the `--ek` help text.
+
+---
+
+## WS4: Trigger Doc Update
+
+**Can run in parallel** with any workstream.
+
+Update [trigger_models_and_experiment_types.md](./trigger_models_and_experiment_types.md) based on discussion with Daniele Martinoli (2026-04-14):
+
+### Changes
+
+1. **Option 1 stays primary** — standalone submission repo (to be created, e.g. `RHEcosystemAppEng/ab-eval-submissions`)
+2. **Clarify "ephemeral"** — the git submission is persistent as a git artifact, but it's not the final destination; it's an evaluation request. For skills there is code to contribute, but for agent/model/MCP comparisons the output is a decision (env var change, configuration), not a code contribution
+3. **Option 2 enhancement — hybrid approach** — GH Action triggered by a PR label from admins, calling the same pipeline. The eval platform just needs the gitops submission repo to run, no matter how it's created. Skill-admins control their own trigger policy
+4. **Admin-gating** — not all developers should trigger evaluations; admin label/approval gates the pipeline. This is on the skill-owner side, separate from the eval platform
+5. **Two-role separation** — skill-admin (controls trigger policy, labels PRs) vs eval-platform (runs the pipeline from the submission repo)
+6. **Non-code experiments** — agent compare, model compare, MCP eval are env-var/config changes, not repo contributions — reinforces why Option 1 is the natural universal fit
+
+### Scope
+
+Documentation update only. No code changes. The submission repo trigger (Option 1) is what we build. The PR-based hybrid (Option 2) is acknowledged and designed for, but not implemented now.
+
+---
+
+## Execution Order
+
+1. **WS1** — A/B generalization (branch exists, start coding)
+2. **WS4** — trigger doc update (can be done during WS1 PR review)
+3. **WS3A** — harbor doc update (can be done during WS1 PR review)
+4. **WS2** — build-push with treatment/control naming (after WS1 merges)
+5. **WS3B** — harbor-eval.yaml task (after WS2 merges)
+
+## Jira Tickets
+
+| Ticket | Workstream | Status |
+|--------|------------|--------|
+| APPENG-4932 | WS1: A/B Eval Flow Conversion | In progress |
+| APPENG-4905 | WS2: Build & Push Images | Blocked on WS1 |
+| APPENG-4906 | WS3: Harbor OpenShift Backend | Partially done (fork complete, doc + Tekton task remain) |
+| (none yet) | WS4: Trigger Doc Update | Not started |
diff --git a/README.md b/README.md
@@ -6,12 +6,12 @@ Automated Tekton-orchestrated pipeline on OpenShift for evaluating AI skill subm
 
 1. **Submit** — Push a skill directory to the submissions repo; a Tekton EventListener triggers the pipeline.
 2. **Validate** — Checks structure, compiles test files, validates `metadata.yaml` schema.
-3. **Scaffold** — Generates two container variants via Jinja2 templates:
-   - **Skilled** — includes the skill and reference docs.
-   - **Unskilled** — excludes them (baseline).
+3. **Scaffold** — Generates two container variants via Jinja2 templates and an experiment strategy:
+   - **Treatment** — includes the experimental material (e.g., skills and reference docs for a skill experiment).
+   - **Control** — baseline without the experimental material.
 4. **Build & Push** — Builds both images and pushes to the OpenShift internal registry.
-5. **Evaluate** — Harbor runs N=20 attempts per variant (40 total) using a custom OpenShift backend.
-6. **Analyze** — Computes pass rates, skills uplift (gap), statistical significance (p-value), and generates heatmaps.
+5. **Evaluate** — Harbor runs N attempts per variant (default N=20, 40 total) using a custom OpenShift backend.
+6. **Analyze** — Computes pass rates, uplift (gap), statistical significance (p-value), and generates heatmaps.
 7. **Publish** — Stores reports, promotes passing images to Quay.io, and opens a PR to [agentic-collections](https://github.com/RHEcosystemAppEng/agentic-collections).
 
 ## Repository Structure