diff --git a/Docs/trigger_guide.md b/Docs/trigger_guide.md new file mode 100644 index 00000000..fbb9d671 --- /dev/null +++ b/Docs/trigger_guide.md @@ -0,0 +1,140 @@ +# Trigger Guide — SkillsEvalFlow Pipeline + +## How the Pipeline is Triggered + +The pipeline is triggered automatically when a skill submission is pushed to a +repository configured with a GitHub webhook pointing at the EventListener. + +``` +git push (submissions/my-skill/) + → GitHub webhook (POST) + → EventListener (skills-submission-listener) + → CEL interceptor filters for submissions/ path changes + → TriggerBinding extracts repo URL, revision, skill directory + → TriggerTemplate creates a PipelineRun + → Pipeline executes: validate → scaffold → build → evaluate → report +``` + +## Submission Contract + +A valid skill submission must follow this structure: + +``` +submissions// +├── metadata.yaml # Required — name is the only mandatory field +├── instruction.md # Required (manual mode) — task description +├── skills/ +│ └── SKILL.md # Required — canonical skill file +├── tests/ +│ ├── test_outputs.py # Required (manual mode) — pytest verification +│ └── llm_judge.py # Optional — LLM-based evaluation +├── docs/ # Optional — reference documentation +├── scripts/ # Optional — helper scripts +└── supportive/ # Optional — mock MCPs, data files (<50MB) +``` + +See `examples/sample_skill/` for a minimal working example. + +## Webhook Configuration + +Configure a GitHub webhook on the submissions repository: + +| Setting | Value | +|--------------|----------------------------------------------------------| +| Payload URL | `https:///` | +| Content type | `application/json` | +| Events | **Just the push event** | +| Secret | Shared secret (configure in EventListener if needed) | + +The EventListener route is created automatically when the EventListener is +deployed. To find it: + +```bash +oc get route -n skills-eval-flow -l eventlistener=skills-submission-listener +``` + +## Manual Trigger (for Testing) + +You can bypass the webhook and trigger the pipeline directly. + +### Option 1: `tkn` CLI + +```bash +tkn pipeline start skills-eval-pipeline \ + -p repo-url=https://github.com/RHEcosystemAppEng/agentic-collections.git \ + -p revision=main \ + -p skill-dir=my-skill \ + -w name=shared-workspace,volumeClaimTemplateFile=pipeline/triggers/pvc-template.yaml \ + -n skills-eval-flow +``` + +### Option 2: PipelineRun YAML + +```yaml +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + generateName: skills-eval-manual- + namespace: skills-eval-flow +spec: + pipelineRef: + name: skills-eval-pipeline + params: + - name: repo-url + value: https://github.com/RHEcosystemAppEng/agentic-collections.git + - name: revision + value: main + - name: skill-dir + value: my-skill + workspaces: + - name: shared-workspace + volumeClaimTemplate: + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +``` + +Apply with: + +```bash +oc create -f pipelinerun.yaml -n skills-eval-flow +``` + +## How the CEL Interceptor Works + +The EventListener uses an inline CEL interceptor to: + +1. **Filter** — only fires when at least one commit touches a file under + `submissions/`: + + ```cel + body.commits.exists(c, + c.added.exists(f, f.startsWith('submissions/')) || + c.modified.exists(f, f.startsWith('submissions/')) + ) + ``` + +2. **Extract** — pulls the skill directory name from the first matching file + path (e.g., `submissions/my-skill/SKILL.md` → `my-skill`): + + ```cel + body.commits.map(c, c.added + c.modified) + .flatten() + .filter(f, f.startsWith('submissions/')) + [0].split('/')[1] + ``` + +> **Note:** Single-skill-per-push is assumed. If multiple skills are pushed in +> one commit, only the first one detected is evaluated. + +## Tekton Components + +| Component | File | Purpose | +|-----------|------|---------| +| EventListener | `pipeline/triggers/event-listener.yaml` | Receives webhooks, filters, extracts skill dir | +| TriggerBinding | `pipeline/triggers/trigger-binding.yaml` | Maps webhook payload to pipeline params | +| TriggerTemplate | `pipeline/triggers/trigger-template.yaml` | Creates PipelineRun from params | +| Validate Task | `pipeline/tasks/validate.yaml` | Validates submission structure and schema | diff --git a/examples/sample_skill/instruction.md b/examples/sample_skill/instruction.md new file mode 100644 index 00000000..0d448048 --- /dev/null +++ b/examples/sample_skill/instruction.md @@ -0,0 +1,11 @@ +# Sample Task + +You are given a Python project. Your goal is to create a `greeting.py` module +that provides a `greet(name: str) -> str` function returning a personalized +greeting message. + +## Requirements + +- The function must accept a single `name` argument +- Return format: `"Hello, {name}! Welcome aboard."` +- Handle empty string input by returning `"Hello, stranger! Welcome aboard."` diff --git a/examples/sample_skill/metadata.yaml b/examples/sample_skill/metadata.yaml new file mode 100644 index 00000000..d346a9f2 --- /dev/null +++ b/examples/sample_skill/metadata.yaml @@ -0,0 +1,7 @@ +name: sample-skill +description: A minimal sample skill for demonstrating the submission contract +persona: rh-developer +version: "0.1.0" +tags: + - sample + - demo diff --git a/examples/sample_skill/skills/SKILL.md b/examples/sample_skill/skills/SKILL.md new file mode 100644 index 00000000..e089340a --- /dev/null +++ b/examples/sample_skill/skills/SKILL.md @@ -0,0 +1,8 @@ +# Greeting Module Skill + +When asked to create a greeting module, follow these guidelines: + +- Use a single function `greet(name: str) -> str` +- Default to `"stranger"` when the name is empty +- Keep the output friendly and professional +- Use f-strings for formatting diff --git a/examples/sample_skill/tests/test_outputs.py b/examples/sample_skill/tests/test_outputs.py new file mode 100644 index 00000000..66221984 --- /dev/null +++ b/examples/sample_skill/tests/test_outputs.py @@ -0,0 +1,24 @@ +"""Verification tests for the greeting task.""" + +import importlib +import sys +from pathlib import Path + + +def _load_greeting(): + workspace = Path("/workspace") + if workspace.exists(): + sys.path.insert(0, str(workspace)) + return importlib.import_module("greeting") + + +def test_greet_with_name(): + mod = _load_greeting() + result = mod.greet("Alice") + assert result == "Hello, Alice! Welcome aboard." + + +def test_greet_empty_string(): + mod = _load_greeting() + result = mod.greet("") + assert result == "Hello, stranger! Welcome aboard." diff --git a/pipeline/tasks/.gitkeep b/pipeline/tasks/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/pipeline/tasks/validate.yaml b/pipeline/tasks/validate.yaml new file mode 100644 index 00000000..52611721 --- /dev/null +++ b/pipeline/tasks/validate.yaml @@ -0,0 +1,91 @@ +apiVersion: tekton.dev/v1 +kind: Task +metadata: + name: validate-submission + namespace: skills-eval-flow +spec: + description: >- + Validates a skill submission directory against the submission contract. + Checks instruction.md, SKILL.md, test compilation, metadata schema, and + supportive/ size limits. Outputs structured JSON results. + params: + - name: skill-dir + type: string + description: Path to the skill directory within the workspace (e.g. my-skill) + - name: max-workspace-mb + type: string + default: "500" + description: Maximum allowed workspace size in MB (guards against oversized submissions) + - name: pipeline-repo-url + type: string + default: "https://github.com/RHEcosystemAppEng/SkillsEvalFlow.git" + description: URL of the SkillsEvalFlow pipeline repository + - name: pipeline-repo-revision + type: string + default: "main" + description: Branch or SHA of the pipeline repo to use for scripts + workspaces: + - name: source + description: Workspace containing the cloned submissions repository + results: + - name: validation-result + description: JSON object with valid (bool) and errors (array) + - name: skill-name + description: The validated skill name (from metadata.yaml) + steps: + - name: check-workspace-size + image: registry.access.redhat.com/ubi9/python-311:latest + script: | + #!/usr/bin/env bash + set -euo pipefail + # Resource guard: reject oversized submissions (e.g. accidental binary + # uploads) before they consume PVC storage and slow down the pipeline. + # Minimum content checks (required files) are handled by validate.py. + WORKSPACE_DIR="$(workspaces.source.path)" + MAX_MB=$(params.max-workspace-mb) + SIZE_KB=$(du -sk "$WORKSPACE_DIR" | cut -f1) + SIZE_MB=$((SIZE_KB / 1024)) + if [ "$SIZE_MB" -gt "$MAX_MB" ]; then + echo "Workspace size ${SIZE_MB}MB exceeds limit ${MAX_MB}MB" + exit 1 + fi + echo "Workspace size: ${SIZE_MB}MB (limit: ${MAX_MB}MB)" + + - name: clone-pipeline-repo + image: registry.access.redhat.com/ubi9/python-311:latest + script: | + #!/usr/bin/env bash + set -euo pipefail + PIPELINE_DIR="$(workspaces.source.path)/_pipeline" + git clone --depth 1 --branch "$(params.pipeline-repo-revision)" \ + "$(params.pipeline-repo-url)" "$PIPELINE_DIR" + echo "Cloned pipeline repo at $(params.pipeline-repo-revision)" + + - name: validate + image: registry.access.redhat.com/ubi9/python-311:latest + script: | + #!/usr/bin/env bash + set -euo pipefail + PIPELINE_DIR="$(workspaces.source.path)/_pipeline" + SKILL_PATH="$(workspaces.source.path)/submissions/$(params.skill-dir)" + + cd "$PIPELINE_DIR" + pip install --quiet --no-cache-dir pydantic pyyaml + + python scripts/validate.py "$SKILL_PATH" \ + | tee /tmp/validation-output.json + + cat /tmp/validation-output.json | tr -d '\n' \ + > "$(results.validation-result.path)" + + VALID=$(python -c "import json; print(json.load(open('/tmp/validation-output.json'))['valid'])") + if [ "$VALID" = "True" ]; then + python -c " + import yaml, sys + meta = yaml.safe_load(open(sys.argv[1])) + print(meta['name'], end='') + " "$SKILL_PATH/metadata.yaml" > "$(results.skill-name.path)" + else + echo "INVALID" > "$(results.skill-name.path)" + exit 1 + fi diff --git a/pipeline/triggers/.gitkeep b/pipeline/triggers/.gitkeep deleted file mode 100644 index e69de29b..00000000 diff --git a/pipeline/triggers/event-listener.yaml b/pipeline/triggers/event-listener.yaml new file mode 100644 index 00000000..6bf66e6c --- /dev/null +++ b/pipeline/triggers/event-listener.yaml @@ -0,0 +1,37 @@ +apiVersion: triggers.tekton.dev/v1beta1 +kind: EventListener +metadata: + name: skills-submission-listener + namespace: skills-eval-flow +spec: + serviceAccountName: pipeline + triggers: + - name: skills-push-trigger + interceptors: + - ref: + name: "cel" + params: + - name: "filter" + # Only fire when at least one commit touches submissions/ + value: >- + body.commits.exists(c, + c.added.exists(f, f.startsWith('submissions/')) || + c.modified.exists(f, f.startsWith('submissions/')) + ) + - name: "overlays" + value: + - key: skill_directory + # Extract the first subdirectory under submissions/ + # e.g. submissions/my-skill/SKILL.md -> my-skill + # NOTE: single-skill-per-push assumed; multi-skill pushes + # only evaluate the first skill found. + expression: >- + body.commits.map(c, + c.added + c.modified + ).flatten().filter(f, + f.startsWith('submissions/') + )[0].split('/')[1] + bindings: + - ref: skills-submission-binding + template: + ref: skills-submission-template diff --git a/pipeline/triggers/trigger-binding.yaml b/pipeline/triggers/trigger-binding.yaml new file mode 100644 index 00000000..8a984196 --- /dev/null +++ b/pipeline/triggers/trigger-binding.yaml @@ -0,0 +1,15 @@ +apiVersion: triggers.tekton.dev/v1beta1 +kind: TriggerBinding +metadata: + name: skills-submission-binding + namespace: skills-eval-flow +spec: + params: + - name: repo-url + value: $(body.repository.clone_url) + - name: revision + value: $(body.after) + - name: repo-name + value: $(body.repository.full_name) + - name: skill-dir + value: $(extensions.skill_directory) diff --git a/pipeline/triggers/trigger-template.yaml b/pipeline/triggers/trigger-template.yaml new file mode 100644 index 00000000..36a7d304 --- /dev/null +++ b/pipeline/triggers/trigger-template.yaml @@ -0,0 +1,39 @@ +apiVersion: triggers.tekton.dev/v1beta1 +kind: TriggerTemplate +metadata: + name: skills-submission-template + namespace: skills-eval-flow +spec: + params: + - name: repo-url + description: The git repository URL + - name: revision + description: The git commit SHA + - name: repo-name + description: The full repository name (org/repo) + - name: skill-dir + description: The skill directory path under submissions/ + resourcetemplates: + - apiVersion: tekton.dev/v1 + kind: PipelineRun + metadata: + generateName: skills-eval- + spec: + pipelineRef: + name: skills-eval-pipeline + params: + - name: repo-url + value: $(tt.params.repo-url) + - name: revision + value: $(tt.params.revision) + - name: skill-dir + value: $(tt.params.skill-dir) + workspaces: + - name: shared-workspace + volumeClaimTemplate: + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi