Appeng 4903/tekton triggers and validate task

Docs/trigger_guide.md

@@ -0,0 +1,140 @@
+# Trigger Guide — SkillsEvalFlow Pipeline
+
+## How the Pipeline is Triggered
+
+The pipeline is triggered automatically when a skill submission is pushed to a
+repository configured with a GitHub webhook pointing at the EventListener.
+
+```
+git push (submissions/my-skill/)
+    → GitHub webhook (POST)
+    → EventListener (skills-submission-listener)
+    → CEL interceptor filters for submissions/ path changes
+    → TriggerBinding extracts repo URL, revision, skill directory
+    → TriggerTemplate creates a PipelineRun
+    → Pipeline executes: validate → scaffold → build → evaluate → report
+```
+
+## Submission Contract
+
+A valid skill submission must follow this structure:
+
+```
+submissions/<skill-name>/
+├── metadata.yaml              # Required — name is the only mandatory field
+├── instruction.md             # Required (manual mode) — task description
+├── skills/
+│   └── SKILL.md               # Required — canonical skill file
+├── tests/
+│   ├── test_outputs.py        # Required (manual mode) — pytest verification
+│   └── llm_judge.py           # Optional — LLM-based evaluation
+├── docs/                      # Optional — reference documentation
+├── scripts/                   # Optional — helper scripts
+└── supportive/                # Optional — mock MCPs, data files (<50MB)
+```
+
+See `examples/sample_skill/` for a minimal working example.
+
+## Webhook Configuration
+
+Configure a GitHub webhook on the submissions repository:
+
+| Setting      | Value                                                    |
+|--------------|----------------------------------------------------------|
+| Payload URL  | `https://<eventlistener-route>/`                         |
+| Content type | `application/json`                                       |
+| Events       | **Just the push event**                                  |
+| Secret       | Shared secret (configure in EventListener if needed)     |
+
+The EventListener route is created automatically when the EventListener is
+deployed. To find it:
+
+```bash
+oc get route -n skills-eval-flow -l eventlistener=skills-submission-listener
+```
+
+## Manual Trigger (for Testing)
+
+You can bypass the webhook and trigger the pipeline directly.
+
+### Option 1: `tkn` CLI
+
+```bash
+tkn pipeline start skills-eval-pipeline \
+  -p repo-url=https://github.com/RHEcosystemAppEng/agentic-collections.git \
+  -p revision=main \
+  -p skill-dir=my-skill \
+  -w name=shared-workspace,volumeClaimTemplateFile=pipeline/triggers/pvc-template.yaml \
+  -n skills-eval-flow
+```
+
+### Option 2: PipelineRun YAML
+
+```yaml
+apiVersion: tekton.dev/v1
+kind: PipelineRun
+metadata:
+  generateName: skills-eval-manual-
+  namespace: skills-eval-flow
+spec:
+  pipelineRef:
+    name: skills-eval-pipeline
+  params:
+    - name: repo-url
+      value: https://github.com/RHEcosystemAppEng/agentic-collections.git
+    - name: revision
+      value: main
+    - name: skill-dir
+      value: my-skill
+  workspaces:
+    - name: shared-workspace
+      volumeClaimTemplate:
+        spec:
+          accessModes:
+            - ReadWriteOnce
+          resources:
+            requests:
+              storage: 1Gi
+```
+
+Apply with:
+
+```bash
+oc create -f pipelinerun.yaml -n skills-eval-flow
+```
+
+## How the CEL Interceptor Works
+
+The EventListener uses an inline CEL interceptor to:
+
+1. **Filter** — only fires when at least one commit touches a file under
+   `submissions/`:
+
+   ```cel
+   body.commits.exists(c,
+     c.added.exists(f, f.startsWith('submissions/')) ||
+     c.modified.exists(f, f.startsWith('submissions/'))
+   )
+   ```
+
+2. **Extract** — pulls the skill directory name from the first matching file
+   path (e.g., `submissions/my-skill/SKILL.md` → `my-skill`):
+
+   ```cel
+   body.commits.map(c, c.added + c.modified)
+     .flatten()
+     .filter(f, f.startsWith('submissions/'))
+     [0].split('/')[1]
+   ```
+
+> **Note:** Single-skill-per-push is assumed. If multiple skills are pushed in
+> one commit, only the first one detected is evaluated.
+
+## Tekton Components
+
+| Component | File | Purpose |
+|-----------|------|---------|
+| EventListener | `pipeline/triggers/event-listener.yaml` | Receives webhooks, filters, extracts skill dir |
+| TriggerBinding | `pipeline/triggers/trigger-binding.yaml` | Maps webhook payload to pipeline params |
+| TriggerTemplate | `pipeline/triggers/trigger-template.yaml` | Creates PipelineRun from params |
+| Validate Task | `pipeline/tasks/validate.yaml` | Validates submission structure and schema |

examples/sample_skill/instruction.md

@@ -0,0 +1,11 @@
+# Sample Task
+
+You are given a Python project. Your goal is to create a `greeting.py` module
+that provides a `greet(name: str) -> str` function returning a personalized
+greeting message.
+
+## Requirements
+
+- The function must accept a single `name` argument
+- Return format: `"Hello, {name}! Welcome aboard."`
+- Handle empty string input by returning `"Hello, stranger! Welcome aboard."`

examples/sample_skill/metadata.yaml

@@ -0,0 +1,7 @@
+name: sample-skill
+description: A minimal sample skill for demonstrating the submission contract
+persona: rh-developer
+version: "0.1.0"
+tags:
+  - sample
+  - demo

examples/sample_skill/skills/SKILL.md

@@ -0,0 +1,8 @@
+# Greeting Module Skill
+
+When asked to create a greeting module, follow these guidelines:
+
+- Use a single function `greet(name: str) -> str`
+- Default to `"stranger"` when the name is empty
+- Keep the output friendly and professional
+- Use f-strings for formatting

examples/sample_skill/tests/test_outputs.py

@@ -0,0 +1,24 @@
+"""Verification tests for the greeting task."""
+
+import importlib
+import sys
+from pathlib import Path
+
+
+def _load_greeting():
+    workspace = Path("/workspace")
+    if workspace.exists():
+        sys.path.insert(0, str(workspace))
+    return importlib.import_module("greeting")
+
+
+def test_greet_with_name():
+    mod = _load_greeting()
+    result = mod.greet("Alice")
+    assert result == "Hello, Alice! Welcome aboard."
+
+
+def test_greet_empty_string():
+    mod = _load_greeting()
+    result = mod.greet("")
+    assert result == "Hello, stranger! Welcome aboard."

pipeline/tasks/validate.yaml

@@ -0,0 +1,91 @@
+apiVersion: tekton.dev/v1
+kind: Task
+metadata:
+  name: validate-submission
+  namespace: skills-eval-flow
+spec:
+  description: >-
+    Validates a skill submission directory against the submission contract.
+    Checks instruction.md, SKILL.md, test compilation, metadata schema, and
+    supportive/ size limits. Outputs structured JSON results.
+  params:
+    - name: skill-dir
+      type: string
+      description: Path to the skill directory within the workspace (e.g. my-skill)
+    - name: max-workspace-mb
+      type: string
+      default: "500"
+      description: Maximum allowed workspace size in MB (guards against oversized submissions)
+    - name: pipeline-repo-url
+      type: string
+      default: "https://github.com/RHEcosystemAppEng/SkillsEvalFlow.git"
+      description: URL of the SkillsEvalFlow pipeline repository
+    - name: pipeline-repo-revision
+      type: string
+      default: "main"
+      description: Branch or SHA of the pipeline repo to use for scripts
+  workspaces:
+    - name: source
+      description: Workspace containing the cloned submissions repository
+  results:
+    - name: validation-result
+      description: JSON object with valid (bool) and errors (array)
+    - name: skill-name
+      description: The validated skill name (from metadata.yaml)
+  steps:
+    - name: check-workspace-size
+      image: registry.access.redhat.com/ubi9/python-311:latest
+      script: |
+        #!/usr/bin/env bash
+        set -euo pipefail
+        # Resource guard: reject oversized submissions (e.g. accidental binary
+        # uploads) before they consume PVC storage and slow down the pipeline.
+        # Minimum content checks (required files) are handled by validate.py.
+        WORKSPACE_DIR="$(workspaces.source.path)"
+        MAX_MB=$(params.max-workspace-mb)
+        SIZE_KB=$(du -sk "$WORKSPACE_DIR" | cut -f1)
+        SIZE_MB=$((SIZE_KB / 1024))
+        if [ "$SIZE_MB" -gt "$MAX_MB" ]; then
+          echo "Workspace size ${SIZE_MB}MB exceeds limit ${MAX_MB}MB"
+          exit 1
+        fi
+        echo "Workspace size: ${SIZE_MB}MB (limit: ${MAX_MB}MB)"
+
+    - name: clone-pipeline-repo
+      image: registry.access.redhat.com/ubi9/python-311:latest
+      script: |
+        #!/usr/bin/env bash
+        set -euo pipefail
+        PIPELINE_DIR="$(workspaces.source.path)/_pipeline"
+        git clone --depth 1 --branch "$(params.pipeline-repo-revision)" \
+          "$(params.pipeline-repo-url)" "$PIPELINE_DIR"
+        echo "Cloned pipeline repo at $(params.pipeline-repo-revision)"
+
+    - name: validate
+      image: registry.access.redhat.com/ubi9/python-311:latest
+      script: |
+        #!/usr/bin/env bash
+        set -euo pipefail
+        PIPELINE_DIR="$(workspaces.source.path)/_pipeline"
+        SKILL_PATH="$(workspaces.source.path)/submissions/$(params.skill-dir)"
+
+        cd "$PIPELINE_DIR"
+        pip install --quiet --no-cache-dir pydantic pyyaml
+
+        python scripts/validate.py "$SKILL_PATH" \
+          | tee /tmp/validation-output.json
+
+        cat /tmp/validation-output.json | tr -d '\n' \
+          > "$(results.validation-result.path)"
+
+        VALID=$(python -c "import json; print(json.load(open('/tmp/validation-output.json'))['valid'])")
+        if [ "$VALID" = "True" ]; then
+          python -c "
+        import yaml, sys
+        meta = yaml.safe_load(open(sys.argv[1]))
+        print(meta['name'], end='')
+        " "$SKILL_PATH/metadata.yaml" > "$(results.skill-name.path)"
+        else
+          echo "INVALID" > "$(results.skill-name.path)"
+          exit 1
+        fi

pipeline/triggers/event-listener.yaml

@@ -0,0 +1,37 @@
+apiVersion: triggers.tekton.dev/v1beta1
+kind: EventListener
+metadata:
+  name: skills-submission-listener
+  namespace: skills-eval-flow
+spec:
+  serviceAccountName: pipeline
+  triggers:
+    - name: skills-push-trigger
+      interceptors:
+        - ref:
+            name: "cel"
+          params:
+            - name: "filter"
+              # Only fire when at least one commit touches submissions/
+              value: >-
+                body.commits.exists(c,
+                  c.added.exists(f, f.startsWith('submissions/')) ||
+                  c.modified.exists(f, f.startsWith('submissions/'))
+                )
+            - name: "overlays"
+              value:
+                - key: skill_directory
+                  # Extract the first subdirectory under submissions/
+                  # e.g. submissions/my-skill/SKILL.md -> my-skill
+                  # NOTE: single-skill-per-push assumed; multi-skill pushes
+                  # only evaluate the first skill found.
+                  expression: >-
+                    body.commits.map(c,
+                      c.added + c.modified
+                    ).flatten().filter(f,
+                      f.startsWith('submissions/')
+                    )[0].split('/')[1]
+      bindings:
+        - ref: skills-submission-binding
+      template:
+        ref: skills-submission-template

pipeline/triggers/trigger-binding.yaml

@@ -0,0 +1,15 @@
+apiVersion: triggers.tekton.dev/v1beta1
+kind: TriggerBinding
+metadata:
+  name: skills-submission-binding
+  namespace: skills-eval-flow
+spec:
+  params:
+    - name: repo-url
+      value: $(body.repository.clone_url)
+    - name: revision
+      value: $(body.after)
+    - name: repo-name
+      value: $(body.repository.full_name)
+    - name: skill-dir
+      value: $(extensions.skill_directory)

pipeline/triggers/trigger-template.yaml

@@ -0,0 +1,39 @@
+apiVersion: triggers.tekton.dev/v1beta1
+kind: TriggerTemplate
+metadata:
+  name: skills-submission-template
+  namespace: skills-eval-flow
+spec:
+  params:
+    - name: repo-url
+      description: The git repository URL
+    - name: revision
+      description: The git commit SHA
+    - name: repo-name
+      description: The full repository name (org/repo)
+    - name: skill-dir
+      description: The skill directory path under submissions/
+  resourcetemplates:
+    - apiVersion: tekton.dev/v1
+      kind: PipelineRun
+      metadata:
+        generateName: skills-eval-
+      spec:
+        pipelineRef:
+          name: skills-eval-pipeline
+        params:
+          - name: repo-url
+            value: $(tt.params.repo-url)
+          - name: revision
+            value: $(tt.params.revision)
+          - name: skill-dir
+            value: $(tt.params.skill-dir)
+        workspaces:
+          - name: shared-workspace
+            volumeClaimTemplate:
+              spec:
+                accessModes:
+                  - ReadWriteOnce
+                resources:
+                  requests:
+                    storage: 1Gi