Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CI on macosx-arm64 #1734

Merged
merged 39 commits into from
Apr 8, 2024
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
4c7e96d
Build conda package locally
eberrigan Mar 25, 2024
76f1dbc
Try 2.8.4
eberrigan Mar 27, 2024
77549b5
Merge develop into branch to fix dependencies
eberrigan Mar 29, 2024
f820228
Change tensorflow version to 2.7.4 in where conda packages are used
eberrigan Mar 29, 2024
165eae0
Make tensorflow requirements in pypi looser
eberrigan Mar 29, 2024
4541ac7
Merge branch 'develop' into elizabeth/change-TensorFlow-version-in-co…
talmo Mar 29, 2024
b2d4ece
Conda package has TensorFlow 2.7.0 and h5py and numpy installed via c…
eberrigan Apr 1, 2024
ef5521e
Change tensorflow version in `environment_no_cuda.yml` to test using CI
eberrigan Apr 1, 2024
8972dd7
Test new sleap/tensorflow package
talmo Apr 7, 2024
2de2de3
Reset build number
talmo Apr 8, 2024
5cbcf1f
Bump version
talmo Apr 8, 2024
9aa66a4
Update mac deps
talmo Apr 8, 2024
5285ad4
Update to Arm64 Mac runners
talmo Apr 8, 2024
059326d
pin `importlib-metadata`
eberrigan Apr 8, 2024
2972027
Merge remote-tracking branch 'origin/elizabeth/change-TensorFlow-vers…
eberrigan Apr 8, 2024
a5de8a6
Pin more stuff on mac
talmo Apr 8, 2024
a7349a5
constrain `opencv` version due to new qt dependencies
eberrigan Apr 8, 2024
ff266e0
Update more mac stuff
talmo Apr 8, 2024
f0f8462
Patches to get to green
talmo Apr 8, 2024
1960cf8
More mac skipping
talmo Apr 8, 2024
1992c80
Re-enable mac tests
talmo Apr 8, 2024
266530c
Handle GPU re-init
talmo Apr 8, 2024
6a88476
Fix mac build CI
talmo Apr 8, 2024
d23b274
Widen tolerance for movenet correctness test
talmo Apr 8, 2024
47c51f2
Merge branch 'develop' into talmo/fix-mac-v140
talmo Apr 8, 2024
3dfcb30
Fix build ci
talmo Apr 8, 2024
c0db085
Try for manual build without upload
talmo Apr 8, 2024
4c0b7ca
Try to reduce training CI time
talmo Apr 8, 2024
1ace179
Rework actions
talmo Apr 8, 2024
09e3f71
Fix miniforge usage
talmo Apr 8, 2024
570f35b
Tweaks
talmo Apr 8, 2024
a048b7c
Fix build ci
talmo Apr 8, 2024
cf0369e
Disable manual build
talmo Apr 8, 2024
4ff9371
Try merging CI coverage
talmo Apr 8, 2024
cd52a8a
GPU/CPU usage in tests
talmo Apr 8, 2024
3e7f03d
Lint
talmo Apr 8, 2024
08c06aa
Clean up
talmo Apr 8, 2024
160309a
Fix test skip condition
talmo Apr 8, 2024
bc845e4
Remove scratch test
talmo Apr 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .conda/condarc.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
channels:
- conda-forge
- nvidia
# - https://conda.anaconda.org/sleap/label/dev
- https://conda.anaconda.org/sleap/label/dev
- sleap
- anaconda
3 changes: 1 addition & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,7 @@ jobs:
strategy:
fail-fast: false
matrix:
# os: ["ubuntu-22.04", "windows-2022", "macos-14"]
os: ["ubuntu-22.04", "windows-2022"] # TODO: Fix mac tests
os: ["ubuntu-22.04", "windows-2022", "macos-14"]
# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrixinclude
include:
# Use this condarc as default
Expand Down
27 changes: 15 additions & 12 deletions .github/workflows/build_ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ jobs:
# Use this condarc as default
- condarc: .conda/condarc.yaml
- wheel_name: sleap-wheel-linux
- pyver: "3.7"
steps:
# Setup
- uses: actions/checkout@v2
Expand All @@ -39,15 +40,16 @@ jobs:
path: ~/conda_pkgs_dir
key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{ hashFiles('environment_build.yml', 'pyproject.toml') }}

- name: Setup Miniconda for Build
# https://github.com/conda-incubator/setup-miniconda
uses: conda-incubator/[email protected]
- name: Setup Miniconda
uses: conda-incubator/[email protected]
with:
python-version: 3.7
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
environment-file: environment_build.yml
miniforge-version: latest
condarc-file: ${{ matrix.condarc }}
python-version: ${{ matrix.pyver }}
environment-file: environment_build.yml
activate-environment: sleap_ci
conda-solver: "libmamba"
# use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!

- name: Print build environment info
shell: bash -l {0}
Expand Down Expand Up @@ -77,8 +79,7 @@ jobs:
strategy:
fail-fast: false
matrix:
# os: ["ubuntu-22.04", "windows-2022", "macos-14"]
os: ["ubuntu-22.04", "windows-2022"]
os: ["ubuntu-22.04", "windows-2022", "macos-14"]
# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstrategymatrixinclude
include:
# Default values
Expand All @@ -90,9 +91,11 @@ jobs:
pip install '$wheel_path'[dev]
- test_args: pytest --durations=-1 tests/
- condarc: .conda/condarc.yaml
- pyver: "3.7"
# Use special condarc if macos
# - os: "macos-14"
# condarc: .conda_mac/condarc.yaml
- os: "macos-14"
condarc: .conda_mac/condarc.yaml
pyver: "3.10"
# Ubuntu specific values
- os: ubuntu-22.04
# Otherwise core dumped in github actions
Expand All @@ -109,10 +112,10 @@ jobs:
- name: Checkout repo
uses: actions/checkout@v3

- name: Set up Python 3.7
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.7
python-version: ${{ matrix.pyver }}

# Download wheel
- name: Download wheel artifact
Expand Down
110 changes: 59 additions & 51 deletions .github/workflows/build_manual.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ on:
paths:
- '.conda/meta.yaml'
- '.conda_mac/meta.yaml'
- '.github/workflows/build_manual.yml'
branches:
# - develop
- fakebranch
# - talmo/fix-mac-v140

jobs:
build:
Expand All @@ -24,9 +26,11 @@ jobs:
include:
# Use this condarc as default
- condarc: .conda/condarc.yaml
- pyver: "3.7"
# Use special condarc if macos
- os: "macos-14"
condarc: .conda_mac/condarc.yaml
pyver: "3.9"
steps:
# Setup
- uses: actions/checkout@v2
Expand All @@ -38,20 +42,24 @@ jobs:
with:
path: ~/conda_pkgs_dir
key: ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{ hashFiles('environment_build.yml', 'requirements.txt') }}

- name: Setup Miniconda
# https://github.com/conda-incubator/setup-miniconda
uses: conda-incubator/[email protected]
uses: conda-incubator/[email protected]
with:
python-version: 3.7
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
environment-file: environment_build.yml
miniforge-version: latest
condarc-file: ${{ matrix.condarc }}
python-version: ${{ matrix.pyver }}
environment-file: environment_build.yml
activate-environment: sleap_ci
conda-solver: "libmamba"
# use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!

- name: Print environment info
shell: bash -l {0}
run: |
which python
conda info
conda list

# Build pip wheel (Not Windows)
- name: Build pip wheel (Not Windows)
Expand Down Expand Up @@ -90,55 +98,55 @@ jobs:
run: |
conda build .conda_mac --output-folder build

# Login to conda (Ubuntu)
- name: Login to Anaconda (Ubuntu)
if: matrix.os == 'ubuntu-22.04'
env:
ANACONDA_LOGIN: ${{ secrets.ANACONDA_LOGIN }}
shell: bash -l {0}
run: |
yes 2>/dev/null | anaconda login --username sleap --password "$ANACONDA_LOGIN" || true
# # Login to conda (Ubuntu)
# - name: Login to Anaconda (Ubuntu)
# if: matrix.os == 'ubuntu-22.04'
# env:
# ANACONDA_LOGIN: ${{ secrets.ANACONDA_LOGIN }}
# shell: bash -l {0}
# run: |
# yes 2>/dev/null | anaconda login --username sleap --password "$ANACONDA_LOGIN" || true

# Login to conda (Windows)
- name: Login to Anaconda (Windows)
if: matrix.os == 'windows-2022'
env:
ANACONDA_LOGIN: ${{ secrets.ANACONDA_LOGIN }}
shell: powershell
run: |
echo "yes" | anaconda login --username sleap --password "$env:ANACONDA_LOGIN"
# # Login to conda (Windows)
# - name: Login to Anaconda (Windows)
# if: matrix.os == 'windows-2022'
# env:
# ANACONDA_LOGIN: ${{ secrets.ANACONDA_LOGIN }}
# shell: powershell
# run: |
# echo "yes" | anaconda login --username sleap --password "$env:ANACONDA_LOGIN"

# Login to conda (Mac)
- name: Login to Anaconda (Mac)
if: matrix.os == 'macos-14'
env:
ANACONDA_LOGIN: ${{ secrets.ANACONDA_LOGIN }}
shell: bash -l {0}
run: |
yes 2>/dev/null | anaconda login --username sleap --password "$ANACONDA_LOGIN" || true
# # Login to conda (Mac)
# - name: Login to Anaconda (Mac)
# if: matrix.os == 'macos-14'
# env:
# ANACONDA_LOGIN: ${{ secrets.ANACONDA_LOGIN }}
# shell: bash -l {0}
# run: |
# yes 2>/dev/null | anaconda login --username sleap --password "$ANACONDA_LOGIN" || true

# Upload conda package (Windows)
- name: Upload conda package (Windows/dev)
if: matrix.os == 'windows-2022'
shell: powershell
run: |
anaconda -v upload "build\win-64\*.tar.bz2" --label dev
# # Upload conda package (Windows)
# - name: Upload conda package (Windows/dev)
# if: matrix.os == 'windows-2022'
# shell: powershell
# run: |
# anaconda -v upload "build\win-64\*.tar.bz2" --label dev

# Upload conda package (Ubuntu)
- name: Upload conda package (Ubuntu/dev)
if: matrix.os == 'ubuntu-22.04'
shell: bash -l {0}
run: |
anaconda -v upload build/linux-64/*.tar.bz2 --label dev
# # Upload conda package (Ubuntu)
# - name: Upload conda package (Ubuntu/dev)
# if: matrix.os == 'ubuntu-22.04'
# shell: bash -l {0}
# run: |
# anaconda -v upload build/linux-64/*.tar.bz2 --label dev

# Upload conda package (Mac)
- name: Upload conda package (Mac/dev)
if: matrix.os == 'macos-14'
shell: bash -l {0}
run: |
anaconda -v upload build/osx-arm64/*.tar.bz2 --label dev
# # Upload conda package (Mac)
# - name: Upload conda package (Mac/dev)
# if: matrix.os == 'macos-14'
# shell: bash -l {0}
# run: |
# anaconda -v upload build/osx-arm64/*.tar.bz2 --label dev

- name: Logout from Anaconda
shell: bash -l {0}
run: |
anaconda logout
# - name: Logout from Anaconda
# shell: bash -l {0}
# run: |
# anaconda logout
75 changes: 28 additions & 47 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,37 +11,20 @@ on:
- "environment_no_cuda.yml"
- "requirements.txt"
- "dev_requirements.txt"
push:
branches:
- master
- develop
paths:
- "sleap/**"
- "tests/**"
- ".github/workflows/ci.yml"
- "environment_no_cuda.yml"
- "requirements.txt"
- "dev_requirements.txt"
# push:
# branches:
# - main
# - develop
# paths:
# - "sleap/**"
# - "tests/**"
# - ".github/workflows/ci.yml"
# - "environment_no_cuda.yml"
# - "requirements.txt"
# - "dev_requirements.txt"

jobs:
type_check:
name: Type Check
runs-on: "ubuntu-22.04"
steps:
- name: Checkout repo
uses: actions/checkout@v3
- name: Set up Python 3.7
uses: actions/setup-python@v4
with:
python-version: 3.7
- name: Install Dependencies
run: |
pip install mypy
- name: Run MyPy
# TODO: remove this once all MyPy errors get fixed
continue-on-error: true
run: |
mypy --follow-imports=skip --ignore-missing-imports sleap tests
# Lint
lint:
name: Lint
runs-on: "ubuntu-22.04"
Expand All @@ -59,47 +42,45 @@ jobs:
- name: Run Black
run: |
black --check sleap tests

# Tests
tests:
name: Tests (${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
# os: ["ubuntu-22.04", "windows-2022", "macos-14"] # TODO: Fix Mac tests
os: ["ubuntu-22.04", "windows-2022"]
os: ["ubuntu-22.04", "windows-2022", "macos-14"]
include:
# Default values
- env_file: environment_no_cuda.yml
- test_args: --durations=-1 tests/
- test_args: --cov=sleap --cov-report=xml --durations=-1 tests/
# Mac specific values
# - os: macos-14
# env_file: environment_mac.yml
- os: macos-14
env_file: environment_mac.yml
# Ubuntu specific values
- os: ubuntu-22.04
test_args: --cov=sleap --cov-report=xml --durations=-1 tests/
steps:
- name: Checkout repo
uses: actions/checkout@v3

- name: Setup Micromamba
# https://github.com/mamba-org/setup-micromamba
uses: mamba-org/setup-micromamba@v1
- name: Setup Conda
uses: conda-incubator/[email protected]
with:
micromamba-version: '1.4.6-0'
miniforge-version: latest
conda-solver: "libmamba"
environment-file: ${{ matrix.env_file }}
environment-name: sleap_ci
init-shell: >-
bash
powershell
post-cleanup: all
activate-environment: sleap_ci
# use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!

# Print environment info
- name: Print environment info
shell: bash -l {0}
run: |
which python
micromamba info
micromamba list
conda info
conda list
pip freeze

# Test environment
Expand All @@ -110,8 +91,8 @@ jobs:

# Upload coverage
- name: Upload coverage
uses: codecov/codecov-action@v1
if: matrix.os == 'ubuntu-22.04'
uses: codecov/codecov-action@v4
# if: matrix.os == 'ubuntu-22.04'
with:
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: true
Expand Down
Loading
Loading