Skip to content

chore(ci): more tweaks to the release process #361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mckornfield merged 3 commits into from
Apr 17, 2026
Merged

chore(ci): more tweaks to the release process #361

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
47d9975
chore(ci): more tweaks to the release process
mckornfield Apr 6, 2026
6b9c0b6
address cr
mckornfield Apr 14, 2026
d3478e1
Merge branch 'main' into more-release-tweaks/mck
mckornfield Apr 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 9 additions & 111 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ All workflows that use `.github/actions/setup-python-env` now default to the ver
| [gpu-tests.yml](gpu-tests.yml) | Nightly , manual | GPU smoke tests (required) and E2E tests |
| [conventional-commit.yml](conventional-commit.yml) | PRs | Validates PR titles follow conventional commit format |
| [docs.yml](docs.yml) | Push to `main` (docs paths) | Builds and deploys documentation to GitHub Pages |
| [release.yml](release.yml) | Manual dispatch | Builds and publishes package to Test PyPI or PyPI (production) |
| [release.yml](release.yml) | Push tags to `v*` | Builds and publishes package to Test PyPI or PyPI (production) |

Copilot AI Apr 16, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The release workflow now publishes to both Test PyPI and PyPI on every v* tag push, but this table entry still says “Test PyPI or PyPI”. Update the wording to match the current behavior (e.g., “Test PyPI then PyPI”).

Suggested change
| [release.yml](release.yml) | Push tags to `v*` | Builds and publishes package to Test PyPI or PyPI (production) |
| [release.yml](release.yml) | Push tags to `v*` | Builds and publishes package to Test PyPI then PyPI |

Copilot uses AI. Check for mistakes.
| [secrets-detector.yml](secrets-detector.yml) | PRs | Scans for accidentally committed secrets |

## Pull Request Testing (copy-pr-bot)
Expand Down Expand Up @@ -93,15 +93,13 @@ flowchart LR
push --> ci & gpu
cpb --> gpu
pr --> ci & conventional & secrets
manual --> release
tag[Tag push v[0-9]*] --> internalRelease
tag[Tag push v[0-9]*] --> release

Copilot AI Apr 16, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workflow diagram still includes an “Internal Release” subgraph, but there is no internal-release.yml workflow in this repository (and the Internal Release section was removed below). Either remove the Internal Release nodes from the diagram or add back the corresponding workflow/docs so the diagram matches reality.

Copilot uses AI. Check for mistakes.

buildWheel --> publishPyPI --> ghRelease --> slackNotify
buildWheelInt --> publishArtifactory

conventional -.->|reuses| FW-CI-templates
secrets -.->|reuses| FW-CI-templates
release -.->|reuses| FW-CI-templates
```
Comment on lines 101 to 103

Copilot AI Apr 16, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diagram removes the release reuse link, but the later “Reusable Workflows” section still states that “All compliance and release workflows reuse templates …” and lists _release_library.yml. Since release.yml is now implemented inline, update that section (or the diagram) to avoid misleading readers.

Copilot uses AI. Check for mistakes.

## CI Checks Workflow
Expand Down Expand Up @@ -197,123 +195,23 @@ Or comment on the PR: `I have read the DCO Document and I hereby sign the DCO`

Scans PRs for accidentally committed secrets. False positives can be added to `.github/workflows/config/.secrets.baseline`.

## Internal Release Workflow

The `internal-release.yml` workflow builds a wheel and publishes it to NVIDIA Artifactory or PyPI.

### Triggers

**Tag push (automatic):** Pushing a `v[0-9]*` tag (e.g. `git tag v0.2.0 && git push --tags`) automatically builds and publishes to Artifactory. This is the primary release mechanism.

**Manual dispatch:** Go to Actions > Internal Release and run with:

- `release-ref`: Branch, tag, or commit SHA to build (defaults to `main`)
- `publish-target`: `artifactory` (default) or `pypi`

### How to Publish Internally

Tag-based (recommended):

```bash
git tag v0.2.0
git push --tags
```

This triggers the workflow automatically and publishes to Artifactory.

Via GitHub Actions (manual):

1. Go to Actions > Internal Release
2. Click Run workflow
3. Enter the branch, tag, or commit SHA to build
4. Select publish target (`artifactory` or `pypi`)

Requires `ARTIFACTORY_USERNAME`, `ARTIFACTORY_TOKEN`, and `ARTIFACTORY_INTERNAL_URL` secrets for Artifactory; `TWINE_USERNAME` and `TWINE_PASSWORD` for PyPI.

Locally (via Makefile):

Add the required env vars to your `.env.local` (git-ignored, auto-loaded by mise):

```bash
TWINE_REPOSITORY_URL=<artifactory-repo-url>
TWINE_USERNAME=<your-username>
TWINE_PASSWORD=<your-api-key>
```

Then run:

```bash
# Build wheel only
make build-wheel

# Build and publish to Artifactory
make publish-internal
```

## Release Workflow (Production)

The production release workflow uses the [FW-CI-templates `_release_library.yml`](https://github.com/NVIDIA-NeMo/FW-CI-templates) reusable workflow to publish to PyPI.
The production release workflow publishes to test PyPI and regular PyPI. It also creates release notes

Copilot AI Apr 16, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is missing punctuation and is a bit unclear. Consider ending it with a period and clarifying that GitHub Release notes are auto-generated (via gh release create --generate-notes).

Suggested change
The production release workflow publishes to test PyPI and regular PyPI. It also creates release notes
The production release workflow publishes to Test PyPI and PyPI. It also creates GitHub Release notes automatically via `gh release create --generate-notes`.

Copilot uses AI. Check for mistakes.

### How to Release

this is placeholder information until we do a real release. will update then.

1. Go to Actions > Release NeMo Safe Synthesizer
2. Click Run workflow
3. Fill in the required inputs:

- `release-ref`: Full SHA or tag of the commit to release
- `dry-run`: Set to `false` for production release (publishes to PyPI)
- `create-gh-release`: Whether to create a GitHub release
- `version-bump-branch`: Branch to push the version bump PR (usually `main`)
1. Push a tag to the repository (start with a release candidate like `v0.0.5rc0` for big changes)
2. Monitor the release pipeline to see it makes its way to Test PyPI/PyPI.

### Release Process

The workflow performs the following steps:

1. Dry-run build - Validates the wheel can be built
2. Version bump - Creates a PR to bump the version in `package_info.py`
3. Build wheel - Builds the production wheel
4. Publish to PyPI - Uploads to PyPI (or test PyPI for dry runs)
5. Create GitHub release - Creates a tagged release with changelog
6. Notify - Sends Slack notification

### Version Management

Version is managed in `[src/nemo_safe_synthesizer/package_info.py](../../src/nemo_safe_synthesizer/package_info.py)`:

```python
MAJOR = 0
MINOR = 1
PATCH = 0
PRE_RELEASE = ""
BUILD = 1
DEV_RELEASE = False
```

The release workflow automatically bumps the PATCH version (or PRE_RELEASE for release candidates).

## Required Secrets

The following secrets must be configured in GitHub repository settings:

| Secret | Purpose |
| --------------------------- | ---------------------------- |
| `TWINE_USERNAME` | PyPI username |
| `TWINE_PASSWORD` | PyPI API token |
| `SLACK_WEBHOOK_ADMIN` | Slack admin notifications |
| `SLACK_RELEASE_ENDPOINT` | Slack release notifications |
| `PAT` | GitHub Personal Access Token |
| `SSH_KEY` | GPG signing key |
| `SSH_PWD` | GPG key passphrase |
| `BOT_KEY` | GitHub App private key |
| `ARTIFACTORY_USERNAME` | NVIDIA Artifactory username |
| `ARTIFACTORY_TOKEN` | NVIDIA Artifactory API key |
| `ARTIFACTORY_INTERNAL_URL` | NVIDIA Artifactory repo URL |

| Variable | Purpose |
| -------- | ------------- |
| `BOT_ID` | GitHub App ID |
1. Build wheel - Builds the production wheel
2. Push to test PyPI
3. Publish to PyPI - Uploads to PyPI
4. Create GitHub release

## Reusable Workflows

Expand Down
27 changes: 5 additions & 22 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,9 @@
name: "Release NeMo Safe Synthesizer"

on:
workflow_dispatch:
inputs:
release-ref:
description: Full SHA (not short) or tag of the commit to build
required: true
type: string
dry-run:
description: "Dry Run: only publish to Test PyPI"
required: true
default: true
type: boolean
create-gh-release:
description: Create a GitHub release
required: true
default: true
type: boolean
push:
tags:
- 'v*'
Comment thread
mckornfield marked this conversation as resolved.
Comment thread
mckornfield marked this conversation as resolved.

defaults:
run:
Expand All @@ -48,15 +35,14 @@ permissions:

jobs:
publish-wheel:
name: Publish wheel (${{ inputs.dry-run && 'Test PyPI' || 'PyPI' }})
name: Publish wheel Test PyPI and PyPI
runs-on: linux-amd64-cpu4
outputs:
version: ${{ steps.build.outputs.version }}
steps:
- name: Checkout at release ref
uses: actions/checkout@v6
with:
ref: ${{ inputs.release-ref }}
fetch-depth: 0
fetch-tags: true

Expand Down Expand Up @@ -98,7 +84,6 @@ jobs:
dist/*.whl

- name: Publish to PyPI

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes the production PyPI publish unconditional for every v* tag. That increases the blast radius of an accidental/incorrect tag push (including RC tags if you intend those to be test-only). If RC tags should not publish to production PyPI, add an explicit condition (e.g., based on tag naming) and/or gate the PyPI publish behind a protected environment approval.

Suggested change
- name: Publish to PyPI
- name: Publish to PyPI
if: ${{ !contains(github.ref_name, 'rc') && !contains(github.ref_name, 'alpha') && !contains(github.ref_name, 'beta') && !contains(github.ref_name, '.dev') }}

Copilot uses AI. Check for mistakes.
if: ${{ !inputs.dry-run }}
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_PERSONAL_TOKEN }}
Comment on lines 86 to 89

Copilot AI Apr 16, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this workflow always publishes to PyPI on tag pushes, it’s hard to recover from partial failures by re-running the job: PyPI rejects re-uploading the same file, and the upload command here doesn’t use --skip-existing (unlike the Test PyPI step). Consider adding --skip-existing or handling the “already exists” error so reruns can proceed to GitHub release creation.

Copilot uses AI. Check for mistakes.
Expand All @@ -111,13 +96,11 @@ jobs:
create-gh-release:
name: Create GitHub release
needs: publish-wheel
if: ${{ inputs.create-gh-release }}
runs-on: ubuntu-latest
steps:
- name: Checkout at release ref
uses: actions/checkout@v6
with:
ref: ${{ inputs.release-ref }}
fetch-depth: 0

- name: Download wheel artifact
Expand All @@ -139,5 +122,5 @@ jobs:
gh release create "v${VERSION}" \
dist/*.whl \
--title "v${VERSION}" \
--notes-file CHANGELOG.md \
--generate-notes \
$PRERELEASE_FLAG
Comment on lines 122 to 126

Copilot AI Apr 6, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gh release create "v${VERSION}" derives the tag name from the built wheel version, but the workflow is triggered by a pushed tag. If the pushed tag and computed VERSION diverge, this can create a release (and possibly a new tag) that doesn’t match the triggering ref. Consider using the triggering tag (github.ref_name) as the release tag and failing fast if it doesn’t match the wheel-derived version.

Copilot uses AI. Check for mistakes.
Loading