Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
da6982f
chore: final touches
mikasenghaas Jun 19, 2026
175600b
docs: don't use auto-tracked num_turns as the @vf.metric example
mikasenghaas Jun 19, 2026
fea7e8b
docs: move the load_tasks example into the Loading tasks section
mikasenghaas Jun 19, 2026
a73e7aa
docs: show full GSM8KTask construction in the load_tasks example
mikasenghaas Jun 19, 2026
5c6ac61
feat: allow @reward to return dict[str, float]
mikasenghaas Jun 19, 2026
b175492
docs: drop the score/score_group override note
mikasenghaas Jun 19, 2026
4c46f82
docs: livelier scoring examples + reframe the notes heading
mikasenghaas Jun 19, 2026
c5ed34b
docs: restructure the trace-reading table and reframe in-runtime scoring
mikasenghaas Jun 19, 2026
e5e3381
docs: separate scoring outputs from read inputs; say "host" not "eval…
mikasenghaas Jun 19, 2026
1607fb0
docs: drop validate from the lifecycle-hooks table
mikasenghaas Jun 19, 2026
bb8e299
docs: give info its own section, contrasted with state
mikasenghaas Jun 19, 2026
6ce36f2
docs: note that a Toolset wraps an MCP server
mikasenghaas Jun 19, 2026
2f0374b
docs: show the self-launching __main__ line in the tool/user examples
mikasenghaas Jun 19, 2026
e61da2a
docs: merge own-host/own-sandbox into a single "own runtime" placemen…
mikasenghaas Jun 19, 2026
571bbbc
docs: add the missing reference examples
mikasenghaas Jun 19, 2026
4366186
docs: reframe the custom-harness intro and list mini-swe-agent/kimi-code
mikasenghaas Jun 19, 2026
32d2bcb
docs: surface the NEEDS_CONTAINER taskset capability flag
mikasenghaas Jun 19, 2026
fcac141
refactor: rename harness capability flag SUPPORTS_TASK_TOOLS -> SUPPO…
mikasenghaas Jun 19, 2026
83d2cdf
docs: harness example configures via CLI args, drops max_steps
mikasenghaas Jun 19, 2026
a2d1e76
docs: split the harness "Writing one" section into subsections
mikasenghaas Jun 19, 2026
edcc52e
docs: correct the tau2-bench-v1 example description
mikasenghaas Jun 19, 2026
19fa340
chore: drop internal bench/ scripts and COMPARE.md from v1
mikasenghaas Jun 19, 2026
4e7981c
ci: restore taskset/harness PyPI publish workflows; bump to 0.2.0
mikasenghaas Jun 19, 2026
01c0dd4
docs: drop the experimental flag from the renderer client
mikasenghaas Jun 19, 2026
27c435d
chore: drop stale v1-runtime dependency comment
mikasenghaas Jun 19, 2026
31a7bea
chore: tidy verifiers pyproject
mikasenghaas Jun 19, 2026
db46601
ci: exclude v1 envs from the auto env-publish for now
mikasenghaas Jun 19, 2026
2f977db
ci: fix version-mismatch message to reference pyproject, not __init__.py
mikasenghaas Jun 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/publish-envs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ jobs:
- name: Set matrix
id: set-matrix
run: |
# Get all environment directories
ENV_IDS=$(ls -d environments/*/ 2>/dev/null | xargs -n1 basename | jq -R . | jq -sc .)
# Auto-publish only the classic (v0) envs for now; v1 envs (the *_v1 packages
# and the `compact` harness example) aren't hub-published yet.
ENV_IDS=$(ls -d environments/*/ 2>/dev/null | xargs -n1 basename | { grep -vE '_v1$|^compact$' || true; } | jq -R . | jq -sc .)

if [ "$ENV_IDS" = "[]" ] || [ "$ENV_IDS" = "null" ]; then
echo "has_envs=false" >> $GITHUB_OUTPUT
Expand Down
210 changes: 210 additions & 0 deletions .github/workflows/publish-harnesses.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,210 @@
name: Publish harnesses

on:
workflow_dispatch:
inputs:
tag:
description: 'Existing tag to release (e.g. harnesses-v0.1.1)'
required: true
type: string
push:
branches:
- main
tags:
- "harnesses-v*"
Comment on lines +12 to +14

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low workflows/publish-harnesses.yml:12

The push event includes a tags filter for harnesses-v* (lines 13-14), so when auto-tag-on-main pushes a new tag (line 65), the tag push triggers a second workflow run. That second run executes build-tagpublish-tag and attempts to upload the same version to PyPI again, which fails with a 409 Conflict because PyPI rejects duplicate uploads. Consider removing the tags filter from the push event so tag pushes do not start a redundant run.

      - main
-    tags:
-      - "harnesses-v*"
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @.github/workflows/publish-harnesses.yml around lines 12-14:

The `push` event includes a `tags` filter for `harnesses-v*` (lines 13-14), so when `auto-tag-on-main` pushes a new tag (line 65), the tag push triggers a second workflow run. That second run executes `build-tag` → `publish-tag` and attempts to upload the same version to PyPI again, which fails with a 409 Conflict because PyPI rejects duplicate uploads. Consider removing the `tags` filter from the `push` event so tag pushes do not start a redundant run.


jobs:
auto-tag-on-main:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
contents: write
outputs:
created: ${{ steps.tag.outputs.created }}
tag: ${{ steps.tag.outputs.tag }}
version: ${{ steps.tag.outputs.version }}
steps:
- name: Checkout main
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Create release tag for untagged version
id: tag
run: |
echo "created=false" >> "$GITHUB_OUTPUT"

VERSION=$(python - <<'PY'
import tomllib
from pathlib import Path
import sys

data = tomllib.loads(Path("packages/harnesses/pyproject.toml").read_text())
version = data.get("project", {}).get("version")
if not version:
sys.exit("Could not find [project].version in packages/harnesses/pyproject.toml")
print(version)
PY
)

TAG="harnesses-v${VERSION}"

if git rev-parse -q --verify "refs/tags/${TAG}" >/dev/null; then
echo "Tag ${TAG} already exists locally; skipping."
exit 0
fi

if git ls-remote --exit-code --tags origin "refs/tags/${TAG}" >/dev/null 2>&1; then
echo "Tag ${TAG} already exists on origin; skipping."
exit 0
fi

git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git tag -a "$TAG" -m "Release $TAG"
git push origin "$TAG"

echo "created=true" >> "$GITHUB_OUTPUT"
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"

build-from-auto-tag:
needs: auto-tag-on-main
if: github.event_name == 'push' && github.ref == 'refs/heads/main' && needs.auto-tag-on-main.outputs.created == 'true'
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
tag: ${{ needs.auto-tag-on-main.outputs.tag }}
version: ${{ needs.auto-tag-on-main.outputs.version }}
steps:
- name: Checkout auto-created tag
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: refs/tags/${{ needs.auto-tag-on-main.outputs.tag }}

- name: Install uv
uses: astral-sh/setup-uv@v7

- name: Build harnesses
run: uv build packages/harnesses

- name: Upload dist artifacts
uses: actions/upload-artifact@v4
with:
name: harnesses-dist
path: packages/harnesses/dist/
if-no-files-found: error
retention-days: 7

publish-from-auto-tag:
needs: build-from-auto-tag
runs-on: ubuntu-latest
environment: pypi-prod
permissions:
id-token: write
steps:
- name: Download dist artifacts
uses: actions/download-artifact@v4
with:
name: harnesses-dist
path: dist/

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate PyPI publish on release

Medium Severity

When auto-tag-on-main creates and pushes a release tag, the same workflow run publishes to PyPI via publish-from-auto-tag, and the tag push starts a second run whose publish-tag job uploads the same version again. PyPI typically rejects the duplicate, so the tag-triggered workflow fails after a successful release.

Additional Locations (2)
Fix in Cursor Fix in Web

Triggered by project rule: BugBot Instructions

Reviewed by Cursor Bugbot for commit 2f977db. Configure here.


build-tag:
if: github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/harnesses-v')
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
tag: ${{ steps.release.outputs.tag }}
version: ${{ steps.release.outputs.version }}
steps:
- name: Checkout tagged release (dispatch)
if: github.event_name == 'workflow_dispatch'
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: refs/tags/${{ inputs.tag }}

- name: Checkout tagged release (push)
if: github.event_name != 'workflow_dispatch'
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Resolve release tag
id: release
env:
EVENT_NAME: ${{ github.event_name }}
PUSHED_REF: ${{ github.ref_name }}
INPUT_TAG: ${{ github.event_name == 'workflow_dispatch' && inputs.tag || '' }}
run: |
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
TAG="$INPUT_TAG"
else
TAG="$PUSHED_REF"
fi

case "$TAG" in
harnesses-v*) ;;
*)
echo "Release tags must be prefixed with 'harnesses-v' (received '$TAG')" >&2
exit 1
;;
esac

VERSION="${TAG#harnesses-v}"
FILE_VERSION=$(python - <<'PY'
import tomllib
from pathlib import Path
import sys

data = tomllib.loads(Path("packages/harnesses/pyproject.toml").read_text())
version = data.get("project", {}).get("version")
if not version:
sys.exit("Could not find [project].version in packages/harnesses/pyproject.toml")
print(version)
PY
)

if [ "$FILE_VERSION" != "$VERSION" ]; then
echo "Version mismatch: tag requests '$VERSION' but packages/harnesses/pyproject.toml defines '$FILE_VERSION'" >&2
exit 1
fi

echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"

- name: Install uv
uses: astral-sh/setup-uv@v7

- name: Build harnesses
run: uv build packages/harnesses

- name: Upload dist artifacts
uses: actions/upload-artifact@v4
with:
name: harnesses-dist
path: packages/harnesses/dist/
if-no-files-found: error
retention-days: 7

publish-tag:
needs: build-tag
runs-on: ubuntu-latest
environment: pypi-prod
permissions:
id-token: write
steps:
- name: Download dist artifacts
uses: actions/download-artifact@v4
with:
name: harnesses-dist
path: dist/

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
Loading
Loading