Import backends from RHAPSODY

.github/README.md

@@ -0,0 +1,127 @@
+# AsyncFlow Examples CI System
+
+This directory contains the CI/CD configuration for automatically testing AsyncFlow examples across different Python versions and backend configurations.
+
+## Overview
+
+The CI system provides:
+
+- **Automated Testing**: Examples are tested on every push/PR with manual dispatch option
+- **Multi-Backend Support**: Tests examples with different execution backends (concurrent, dask, radical_pilot)
+- **Configuration-Driven**: Uses `examples-config.yml` to specify per-example settings
+- **Comprehensive Validation**: Includes syntax checking, dependency management, and output validation
+
+## Configuration Format
+
+The `examples-config.yml` file defines how each example should be tested:
+
+```yaml
+examples:
+  example-name:
+    script: "path/to/example.py"          # Path to example script
+    backend: "concurrent"                 # Backend to use: noop, concurrent, dask, radical.pilot
+    timeout_sec: 120                      # Timeout in seconds
+    dependencies:                         # Additional pip packages
+      - "numpy>=1.20"
+      - "matplotlib"
+    min_output_lines: 5                   # Minimum expected output lines
+    skip_python:                          # Python versions to skip
+      - "3.8"
+```
+
+### Configuration Options
+
+- **`script`**: Path to the example Python file (defaults to `examples/{example-name}.py`)
+- **`backend`**: Execution backend to use (default: `concurrent`)
+- **`timeout_sec`**: Maximum execution time in seconds (default: 120)
+- **`dependencies`**: List of additional pip packages to install
+- **`min_output_lines`**: Minimum number of output lines expected (default: 1)
+- **`skip_python`**: List of Python versions to skip for this example
+
+## Workflows
+
+### Examples CI (`examples.yml`)
+
+Triggered on:
+
+- Push to main/master/develop branches (when relevant files change)
+- Pull requests (when relevant files change)
+- Manual workflow dispatch
+
+**What it tests:**
+
+- Only examples affected by code changes
+- Multiple Python versions (3.9, 3.11, 3.12)
+- Fast feedback for development
+- When manually dispatched, can test all examples with custom Python versions
+
+## GitHub Action (`run-example`)
+
+The reusable action handles:
+
+1. **Environment Setup**: Python installation and caching
+2. **Dependency Management**: Core package + example-specific + backend dependencies
+3. **Configuration Loading**: Parsing YAML config and applying example settings
+4. **Execution**: Running examples with proper timeout and error handling
+5. **Validation**: Checking output for errors and minimum content requirements
+6. **Artifact Collection**: Saving outputs for debugging failures
+
+## Local Testing
+
+You can validate the configuration locally:
+
+```bash
+# Install dependencies
+pip install pyyaml
+
+# Run validation script
+python .github/bin/validate_examples.py
+```
+
+This script checks:
+
+- Configuration file syntax and structure
+- Example script existence and syntax
+- Basic configuration validation
+
+## Adding New Examples
+
+1. **Create the example file** in the `examples/` directory
+2. **Add configuration** to `examples-config.yml`:
+   ```yaml
+   examples:
+     my-new-example:
+       script: "examples/my-new-example.py"
+       backend: "concurrent"
+       timeout_sec: 60
+       dependencies:
+         - "requests"
+   ```
+3. **Test locally** using the validation script
+4. **Commit changes** - CI will automatically test the new example
+
+## Backend-Specific Configuration
+
+### Concurrent Backend
+
+- **Description**: Built-in Python `concurrent.futures` backend
+- **Dependencies**: None (included with Python)
+- **Use Case**: General examples, I/O-bound tasks
+
+### Dask Backend
+
+- **Description**: Distributed computing backend
+- **Dependencies**: `dask[complete]`
+- **Use Case**: CPU-intensive parallel tasks
+
+### RADICAL-Pilot Backend
+
+- **Description**: HPC-focused execution backend
+- **Dependencies**: `radical.pilot`
+- **Use Case**: HPC environments, large-scale computing
+
+### NoOp Backend
+
+- **Description**: No-operation backend for testing
+- **Dependencies**: None
+- **Use Case**: Testing workflow logic without execution

.github/actionlint.yaml

@@ -0,0 +1,21 @@
+# Configuration for actionlint
+# https://github.com/rhysd/actionlint/blob/main/docs/config.md
+
+# Disable specific checks if needed
+# self-hosted-runner:
+#   # Labels of self-hosted runner in array of strings
+#   labels:
+#     - linux.2xlarge
+#     - windows-latest-xl
+#   # Repository path of self-hosted runner configurations
+#   config-file: path/to/runner-config.json
+
+# Schema validation config
+# config-variables:
+#   # Array of known custom configuration variable names
+#   vars: []
+
+# Format configuration
+# format:
+#   # Available formats: sarif, json
+#   output-format: ""

.github/actions/run-example/action.yml

@@ -0,0 +1,202 @@
+name: "Run single AsyncFlow example"
+description: "Runs a single AsyncFlow example with a specific Python version and backend"
+
+inputs:
+  example:
+    description: "Example key in .github/examples-config.yml"
+    required: true
+  python-version:
+    description: "Python version, e.g., 3.11"
+    required: true
+
+outputs:
+  success:
+    description: "Whether the example ran successfully"
+    value: ${{ steps.run.outputs.success }}
+
+runs:
+  using: "composite"
+  steps:
+    - name: Setup Python
+      uses: actions/setup-python@v5
+      id: setup_py
+      with:
+        python-version: ${{ inputs.python-version }}
+
+    - name: Show Python details
+      shell: bash
+      run: |
+        PY="${{ steps.setup_py.outputs.python-path }}"
+        echo "python path: $PY"
+        "$PY" -V
+        "$PY" -c "import sys, sysconfig; print('include:', sysconfig.get_paths()['include'])"
+
+    - name: Cache pip
+      uses: actions/cache@v4
+      with:
+        path: ~/.cache/pip
+        key: pip-asyncflow-${{ runner.os }}-${{ inputs.python-version }}-${{ hashFiles('**/pyproject.toml') }}
+        restore-keys: |
+          pip-asyncflow-${{ runner.os }}-${{ inputs.python-version }}-
+
+    - name: Ensure yq present
+      shell: bash
+      run: |
+        if ! command -v yq >/dev/null 2>&1; then
+          sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
+          sudo chmod +x /usr/local/bin/yq
+        fi
+
+    - name: Load example configuration
+      shell: bash
+      id: cfg
+      run: |
+        set -euo pipefail
+        CFG=".github/examples-config.yml"
+        EXAMPLE="${{ inputs.example }}"
+
+        if [ ! -f "$CFG" ]; then
+          echo "❌ Missing $CFG"
+          exit 2
+        fi
+
+        if ! yq -e ".examples.$EXAMPLE" "$CFG" >/dev/null; then
+          echo "❌ Example '$EXAMPLE' not found in $CFG"
+          yq '.examples | keys' "$CFG" || echo "No examples found"
+          exit 2
+        fi
+
+        script=$(yq -r ".examples.$EXAMPLE.script" "$CFG")
+        timeout_sec=$(yq -r ".examples.$EXAMPLE.timeout_sec" "$CFG")
+        backend=$(yq -r ".examples.$EXAMPLE.backend // \"concurrent\"" "$CFG")
+        min_output_lines=$(yq -r ".examples.$EXAMPLE.min_output_lines // 3" "$CFG")
+
+        # Handle arrays properly - return empty array if null
+        skip_python_raw=$(yq -r ".examples.$EXAMPLE.skip_python" "$CFG")
+        if [ "$skip_python_raw" = "null" ]; then
+          skip_python="[]"
+        else
+          skip_python=$(yq -o json ".examples.$EXAMPLE.skip_python" "$CFG" | jq -c '.')
+        fi
+
+        dependencies_raw=$(yq -r ".examples.$EXAMPLE.dependencies" "$CFG")
+        if [ "$dependencies_raw" = "null" ]; then
+          dependencies="[]"
+        else
+          dependencies=$(yq -o json ".examples.$EXAMPLE.dependencies" "$CFG" | jq -c '.')
+        fi
+
+        echo "script=$script" >> "$GITHUB_OUTPUT"
+        echo "timeout_sec=$timeout_sec" >> "$GITHUB_OUTPUT"
+        echo "backend=$backend" >> "$GITHUB_OUTPUT"
+        echo "min_output_lines=$min_output_lines" >> "$GITHUB_OUTPUT"
+        echo "skip_python=$skip_python" >> "$GITHUB_OUTPUT"
+        echo "dependencies=$dependencies" >> "$GITHUB_OUTPUT"
+
+        # Check if this Python version should be skipped
+        if echo "$skip_python" | jq -e "index(\"${{ inputs.python-version }}\")" > /dev/null; then
+          echo "skip=true" >> "$GITHUB_OUTPUT"
+          echo "⏭️ Skipping example on Python ${{ inputs.python-version }}"
+        else
+          echo "skip=false" >> "$GITHUB_OUTPUT"
+        fi
+
+    - name: Skip if configured
+      if: steps.cfg.outputs.skip == 'true'
+      shell: bash
+      run: |
+        echo "Example ${{ inputs.example }} configured to skip Python ${{ inputs.python-version }}"
+        exit 0
+
+    - name: Install AsyncFlow and dependencies
+      if: steps.cfg.outputs.skip != 'true'
+      shell: bash
+      run: |
+        PY="${{ steps.setup_py.outputs.python-path }}"
+        "$PY" -m pip install --upgrade pip
+        "$PY" -m pip install -e .
+
+        # Install example-specific dependencies
+        DEPS='${{ steps.cfg.outputs.dependencies }}'
+        if [ "$DEPS" != "[]" ]; then
+          echo "Installing dependencies: $DEPS"
+          echo "$DEPS" | jq -r '.[]' | while read dep; do
+            echo "Installing: $dep"
+            "$PY" -m pip install "$dep"
+          done
+        fi
+
+        # Install backend-specific dependencies
+        BACKEND='${{ steps.cfg.outputs.backend }}'
+        case "$BACKEND" in
+          "dask")
+            echo "Installing Dask backend dependencies via AsyncFlow optional dependencies"
+            "$PY" -m pip install -e '.[dask]' || echo "Dask backend installation failed"
+            ;;
+          "radical_pilot")
+            echo "Installing RADICAL-Pilot backend dependencies via AsyncFlow optional dependencies"
+            "$PY" -m pip install -e '.[radicalpilot]' || echo "RADICAL-Pilot backend installation failed"
+            ;;
+          "concurrent"|"noop")
+            echo "Using built-in backend: $BACKEND"
+            ;;
+        esac
+
+    - name: Run example script
+      if: steps.cfg.outputs.skip != 'true'
+      id: run
+      shell: bash
+      run: |
+        set -euo pipefail
+        PY="${{ steps.setup_py.outputs.python-path }}"
+        SCRIPT="${{ steps.cfg.outputs.script }}"
+        TIMEOUT="${{ steps.cfg.outputs.timeout_sec }}"
+        BACKEND="${{ steps.cfg.outputs.backend }}"
+
+        # Set backend environment variable
+        export RADICAL_ASYNCFLOW_BACKEND="$BACKEND"
+
+        echo "🚀 Running: $SCRIPT (timeout: ${TIMEOUT}s, backend: $BACKEND)"
+
+        # Create output directory and run with output capture
+        mkdir -p "example-outputs"
+        OUTPUT_FILE="example-outputs/${{ inputs.example }}-py${{ inputs.python-version }}.log"
+
+        if timeout "${TIMEOUT}s" "$PY" "$SCRIPT" > "$OUTPUT_FILE" 2>&1; then
+          echo "success=true" >> "$GITHUB_OUTPUT"
+          echo "✅ Example completed successfully"
+
+          # Show output summary
+          LINES=$(wc -l < "$OUTPUT_FILE")
+          echo "Output: $LINES lines"
+
+          # Check minimum output requirement
+          MIN_LINES=${{ steps.cfg.outputs.min_output_lines }}
+          if [ "$LINES" -lt "$MIN_LINES" ]; then
+            echo "⚠️ Output below minimum ($LINES < $MIN_LINES lines) - possible crash"
+            echo "Full output:"
+            cat "$OUTPUT_FILE"
+          else
+            echo "First 10 lines of output:"
+            head -10 "$OUTPUT_FILE"
+          fi
+        else
+          echo "success=false" >> "$GITHUB_OUTPUT"
+          echo "❌ Example failed"
+          echo "Last 20 lines of output:"
+          tail -20 "$OUTPUT_FILE"
+          exit 1
+        fi
+
+    - name: Validate output patterns
+      if: steps.cfg.outputs.skip != 'true' && steps.run.outputs.success == 'true'
+      shell: bash
+      run: |
+        OUTPUT_FILE="example-outputs/${{ inputs.example }}-py${{ inputs.python-version }}.log"
+
+        # Check for error patterns (but allow expected ones)
+        if grep -i "traceback\|exception\|error" "$OUTPUT_FILE" | grep -v "expected\|handled\|graceful"; then
+          echo "⚠️ Found potential errors in output (but example still succeeded)"
+        fi
+
+        echo "✅ Output validation completed"