-
Notifications
You must be signed in to change notification settings - Fork 0
Migrating Azure pipelines to GHA workflows #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
65 commits
Select commit
Hold shift + click to select a range
9cf7a7a
Migrating from azure pipelines to GHA with minor updates for Ruff and…
lamkina 1d0a80e
Fix strategy matrix for mypy workflow
anthony-ozdemir b3cfd71
Update .github/workflows/fprettify.yaml
lamkina 8db7f40
Update .github/workflows/clang_format.yaml
lamkina 88c8ce4
Removing reference to azure in root README
lamkina ec760cc
Removing pypi workflow
lamkina 88e9fd6
Adding specific ruff workflow and removing old format and lint workflow
lamkina ff3a1e4
updating ruff yaml extension
lamkina 34c5431
Fixing indentation
lamkina e484d0c
Updating ruff workflow to fix gha error
lamkina f280eb3
Testing ruff pwd
lamkina 042b4eb
Fixing ruff directory issues
lamkina d98f177
Debugging ruff formatting path
lamkina ea96700
Trying to fix path issues with gha and ruff
lamkina 5ac78f5
Fixing an argument bug
lamkina 3f51459
Trying to fix build bug with paths
lamkina b6f5096
Making the branch name check a callable workflow
lamkina 342d9dc
Fixing bug with branch workflow
lamkina c08da3d
Removing pylint
lamkina 25b908f
Improving docstrings
lamkina 5baa589
Changing all .yml to .yaml for consistency
lamkina e7fb61b
Adding docker login action instead of command
lamkina 94e3293
Removing isort workflow and config in favor of ruff
lamkina f8a702e
Adding import sorting to the select
lamkina da811a1
Refactoring build script and adding intel, removing standalone mypy w…
lamkina b20cc71
Adding explicit tapenade version
lamkina 80d874b
Updating readme for new workflow setup
lamkina 4da3a78
Bumping clang format to version 20
lamkina 053c384
Bumping actions checkout to v6
lamkina 3f327d9
Removing MDO Lab codeowners
lamkina 2ac4151
Removing import sorting by default from Ruff config and adding sortin…
lamkina 05cfcee
Updating readme, refactoring build workflow, and adding mypy to build…
lamkina 582951c
Minor update to the build workflow for turning intel/gcc on or off
lamkina 7d7ef20
Fixing env bug
lamkina 30a1c97
Improving the logic checks for the build workflow for intel/gcc
lamkina a2db2c1
Fixing mypy
lamkina 08d7ff2
Creating composite actions for build workflow to split jobs
lamkina 930109c
Fixing .yml to .yaml
lamkina 502dadf
Adding a checkout step for composite actions workflow
lamkina 8d0b709
Removing composite actions and trying yaml anchor approach
lamkina 0a22772
Fixing yaml anchor
lamkina 8b24add
Fixing yaml anchors pt 2
lamkina 4adc7d8
Addressing new round of comments pt 1
lamkina b6571d6
Restoring fprettify shell script
lamkina 8217298
Fixing secrets docs in readme
lamkina 97e5837
Updating workflow options section
lamkina fd0814e
Readme update and accounting for local/global ruff toml configs
lamkina 3286a73
Fixing ruff config again
lamkina f328a25
Adding a standalone mypy workflow
lamkina ddbaa7d
Updating readme with mypy workflow
lamkina f993ecc
Removed flag for intel/gcc and will just build/test both always
lamkina 8e3663a
Fixing secrets examples in readme to use inherit
lamkina 80baaf7
Reverting path change
lamkina e0af275
Removing unecessary boolean flags
lamkina 413d615
Adding clang-format script back
lamkina 29b2168
Adding main as an acceptable branch for the client name checks
lamkina b0abb63
Adding timeout for ruff workflow
lamkina e6295bd
Adding refactored docker workflows
lamkina 96987c6
Fixing yaml extensions
lamkina 60e667f
Targeting the PR branch for the composite actions
lamkina f166200
Fixing env variables for docker-related workflows
lamkina c5b27e1
Refacotring the env setup to the composite action
lamkina 9c45fc1
Another tweak to the environment setup for docker composite action
lamkina ced74f8
Set composite actions branch to main
lamkina e37e17e
Documentation update for composite actions
lamkina File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
File renamed without changes.
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| name: Docker cleanup | ||
|
|
||
| description: Stop and remove the shared Docker container. | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Cleanup | ||
| if: always() | ||
| shell: bash | ||
| run: | | ||
| # Stop and remove the container only if it exists to avoid noisy errors. | ||
| if docker ps -a --format '{{.Names}}' | grep -qx app; then | ||
| docker stop app && docker rm app || true | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| name: Docker setup | ||
|
|
||
| description: Shared steps to prepare repo environment and start container. | ||
|
|
||
| inputs: | ||
| SCRITICAL_HOMEDIR: | ||
| description: Base home directory for container paths | ||
| required: false | ||
| default: /home/scriticaluser | ||
| BASHRC: | ||
| description: Path to bashrc for container environment | ||
| required: false | ||
| default: /home/scriticaluser/.bashrc_scritical | ||
| DOCKER_USER: | ||
| description: Docker registry username | ||
| required: true | ||
| DOCKER_OAT: | ||
| description: Docker registry Organization Access Token | ||
| required: true | ||
| DOCKER_TAG: | ||
| description: Docker image tag to use | ||
| required: true | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Set repo environment variables | ||
| shell: bash | ||
| run: | | ||
| scritical_homedir="${{ inputs.SCRITICAL_HOMEDIR }}" | ||
| echo "SCRITICAL_HOMEDIR=$scritical_homedir" >> "$GITHUB_ENV" | ||
| echo "BASHRC=${{ inputs.BASHRC }}" >> "$GITHUB_ENV" | ||
| repo_name="${{ github.event.repository.name }}" | ||
| echo "REPO_NAME=$repo_name" >> "$GITHUB_ENV" | ||
| echo "DOCKER_WORKING_DIR=$scritical_homedir/repos/$repo_name" >> "$GITHUB_ENV" | ||
| echo "DOCKER_MOUNT_DIR=$scritical_homedir/mount/$repo_name" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Login to Docker Hub | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| username: ${{ inputs.DOCKER_USER }} | ||
| password: ${{ inputs.DOCKER_OAT }} | ||
|
|
||
| - name: Start Docker container | ||
| shell: bash | ||
| run: | | ||
| docker run -t -d --name app \ | ||
| --mount "type=bind,src=${{ github.workspace }},target=${{ env.DOCKER_MOUNT_DIR }}" \ | ||
| scritical/private-dev:${{ inputs.DOCKER_TAG }} /bin/bash | ||
|
|
||
| # Copy repo to working directory | ||
| docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" | ||
|
|
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,250 @@ | ||
| # GitHub Actions Workflows | ||
|
|
||
| Reusable GitHub Actions workflows for Supercritical repositories. These workflows are called from individual repositories using the `workflow_call` trigger. | ||
|
|
||
| ## Available Workflows | ||
|
|
||
| | Workflow | Description | | ||
| | :------- | :---------- | | ||
| | `build.yaml` | Build and test code in Docker container | | ||
| | `tapenade.yaml` | Tapenade automatic differentiation checks | | ||
| | `clang_format.yaml` | C/C++ formatting checks | | ||
| | `fprettify.yaml` | Fortran 90 formatting checks | | ||
| | `ruff.yaml` | Python formatting and linting with Ruff | | ||
| | `mypy.yaml` | Python type checking with MyPy | | ||
| | `branch-name-check.yaml` | Enforce branch naming conventions | | ||
|
|
||
| --- | ||
|
|
||
| ## Composite Actions | ||
|
|
||
| Shared composite actions used by workflows and repository CI files. | ||
|
|
||
| | Action | Description | | ||
| | :----- | :---------- | | ||
| | `docker-setup` | Set Docker env vars, login, and start the container | | ||
| | `docker-cleanup` | Stop and remove the container | | ||
|
|
||
| ### docker-setup | ||
|
|
||
| | Name | Type | Default | Description | | ||
| | :--- | :--- | :------ | :---------- | | ||
| | `SCRITICAL_HOMEDIR` | string | `/home/scriticaluser` | Base home directory for container paths | | ||
| | `BASHRC` | string | `/home/scriticaluser/.bashrc_scritical` | Bashrc for container environment | | ||
| | `DOCKER_USER` | string | n/a | Docker registry username | | ||
| | `DOCKER_OAT` | string | n/a | Docker registry Organization Access Token | | ||
| | `DOCKER_TAG` | string | n/a | Docker image tag to use | | ||
|
|
||
| ### docker-cleanup | ||
|
|
||
| No inputs. | ||
|
|
||
| --- | ||
|
|
||
| ## Workflow Options | ||
|
|
||
| Configuration inheritance/overrides are documented in each workflow section below. | ||
|
|
||
| ### build.yaml | ||
|
|
||
| Docker-based build and test workflow using the `scritical/private-dev` image. | ||
| Runs GCC and/or Intel jobs based on `GCC` and `INTEL` input flags. | ||
|
|
||
| | Name | Type | Default | Description | | ||
| | :--- | :--- | :------ | :---------- | | ||
| | `TIMEOUT` | number | `120` | Runtime allowed for the job, in minutes | | ||
| | `GCC_CONFIG` | string | `""` | Path to GCC configuration file (from repository root) | | ||
lamkina marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| | `INTEL_CONFIG` | string | `""` | Path to Intel configuration file (from repository root) | | ||
lamkina marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| | `BUILD_SCRIPT` | string | `.github/build_real.sh` | Path to build script. Empty string skips this step | | ||
| | `TEST_SCRIPT` | string | `.github/test_real.sh` | Path to test script. Empty string skips this step | | ||
|
|
||
| **Required Secrets:** | ||
| | Name | Description | | ||
| | :--- | :---------- | | ||
| | `DOCKER_USER` | Docker registry username | | ||
| | `DOCKER_OAT` | Docker registry Organization Access Token | | ||
lamkina marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| If these secrets are configured at the organization level, callers can use `secrets: inherit` instead of listing each secret. | ||
|
|
||
| --- | ||
|
|
||
| ### ruff.yaml | ||
|
|
||
| Python formatting and linting using Ruff. | ||
| The workflow checks out the org-wide Ruff configuration from `scritical/.github` and uses it for format, lint, and isort checks. | ||
|
|
||
| | Name | Type | Default | Description | | ||
| | :--- | :--- | :------ | :---------- | | ||
| | `MCCABE` | boolean | `false` | Enable McCabe complexity check (pass/fail, max complexity = 10) | | ||
| | `ISORT` | boolean | `false` | Enable import sorting check (pass/fail) | | ||
|
|
||
| **Configuration Override:** Create a `ruff.toml` in your repo with: | ||
| ```toml | ||
| extend = "~/.config/ruff/ruff.toml" | ||
|
|
||
| # Local overrides here | ||
| [lint] | ||
| ignore = ["N802"] | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ### mypy.yaml | ||
|
|
||
| Runs MyPy type checking in the GCC OpenMPI Docker image. | ||
|
|
||
| | Name | Type | Default | Description | | ||
| | :--- | :--- | :------ | :---------- | | ||
| | `TIMEOUT` | number | `30` | Runtime allowed for the job, in minutes | | ||
|
|
||
| **Required Secrets:** | ||
| | Name | Description | | ||
| | :--- | :---------- | | ||
| | `DOCKER_USER` | Docker registry username | | ||
| | `DOCKER_OAT` | Docker registry Organization Access Token | | ||
|
|
||
| If these secrets are configured at the organization level, callers can use `secrets: inherit` instead of listing each secret. | ||
|
|
||
| --- | ||
|
|
||
| ### tapenade.yaml | ||
|
|
||
| Run Tapenade automatic differentiation and check for uncommitted changes. | ||
|
|
||
| | Name | Type | Default | Description | | ||
| | :--- | :--- | :------ | :---------- | | ||
| | `TIMEOUT` | number | `10` | Runtime allowed for the job, in minutes | | ||
| | `TAPENADE_SCRIPT` | string | `.github/build_tapenade.sh` | Path to Tapenade build script | | ||
|
|
||
| Uses Tapenade version 3.16 from tapenade_3.16-v2-723-ge8da61555.tar. Using a different version will create a diff. | ||
|
|
||
| Here is the link to our tapenade version: https://gitlab.inria.fr/tapenade/tapenade/-/package_files/112870/download | ||
|
|
||
| --- | ||
|
|
||
| ### clang_format.yaml | ||
|
|
||
| C/C++ code formatting checks using clang-format. | ||
|
|
||
| | Name | Type | Default | Description | | ||
| | :--- | :--- | :------ | :---------- | | ||
| | `TIMEOUT` | number | `10` | Runtime allowed for the job, in minutes | | ||
|
|
||
| **Configuration Override:** Create a `.clang-format` file in your repo root. | ||
|
|
||
| --- | ||
|
|
||
| ### fprettify.yaml | ||
|
|
||
| Fortran 90 code formatting checks using fprettify. | ||
|
|
||
| | Name | Type | Default | Description | | ||
| | :--- | :--- | :------ | :---------- | | ||
| | `TIMEOUT` | number | `10` | Runtime allowed for the job, in minutes | | ||
|
|
||
| **Configuration Override:** Create a `.fprettify.rc` file in your repo root. | ||
|
|
||
| --- | ||
|
|
||
| ### branch-name-check.yaml | ||
|
|
||
| Enforces branch naming conventions for pull requests: | ||
|
|
||
| - For PRs targeting `main`, source branches must start with `feature-`, `bugfix-`, or `hotfix-`. | ||
| - For PRs targeting `client-*`, source branches must start with `feature-`, `bugfix-`, or `hotfix-`. | ||
lamkina marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## Setting Up Workflows | ||
|
|
||
| ### Step 1: Create Workflow File | ||
|
|
||
| Create `.github/workflows/ci.yaml` in your repository: | ||
|
|
||
| ```yaml | ||
| name: CI | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| branches: [main] | ||
|
|
||
| jobs: | ||
| build: | ||
| uses: scritical/.github/.github/workflows/build.yaml@main | ||
| with: | ||
| GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk # If necessary | ||
| secrets: inherit | ||
|
|
||
| ruff: | ||
| uses: scritical/.github/.github/workflows/ruff.yaml@main | ||
|
|
||
| clang-format: | ||
| uses: scritical/.github/.github/workflows/clang_format.yaml@main | ||
|
|
||
| fprettify: | ||
| uses: scritical/.github/.github/workflows/fprettify.yaml@main | ||
| ``` | ||
|
|
||
| ### Step 2: Write Build Script | ||
|
|
||
| Create `.github/build_real.sh` in your repository: | ||
|
|
||
| ```bash | ||
| #!/bin/bash | ||
| set -e | ||
| cp $CONFIG_FILE config/config.mk | ||
| make | ||
| pip install . | ||
| ``` | ||
|
|
||
| ### Step 3: Write Test Script | ||
|
|
||
| Create `.github/test_real.sh` in your repository: | ||
|
|
||
| ```bash | ||
| #!/bin/bash | ||
| set -e | ||
| ./input_files/get-input-files.sh | ||
| testflo -v . -n 1 | ||
| ``` | ||
|
|
||
| ### Step 4: Add Secrets | ||
|
|
||
| In your orginaization settings, add the required secrets: | ||
| - `DOCKER_USER` - Docker organization name | ||
lamkina marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - `DOCKER_OAT` - Docker Organization Access Token for pulling private images | ||
|
|
||
| --- | ||
|
|
||
| ## Complex Builds | ||
|
|
||
| For repositories requiring both real and complex builds: | ||
|
|
||
| ```yaml | ||
| jobs: | ||
| build-real: | ||
| uses: scritical/.github/.github/workflows/build.yaml@main | ||
| with: | ||
| GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk | ||
| BUILD_SCRIPT: .github/build_real.sh | ||
| TEST_SCRIPT: .github/test_real.sh | ||
| secrets: inherit | ||
|
|
||
| build-complex: | ||
| uses: scritical/.github/.github/workflows/build.yaml@main | ||
| with: | ||
| GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk | ||
| BUILD_SCRIPT: .github/build_complex.sh | ||
| TEST_SCRIPT: .github/test_complex.sh | ||
| secrets: inherit | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Adding Status Badge | ||
|
|
||
| Add a status badge to your README: | ||
|
|
||
| ```markdown | ||
|  | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.