Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flatten builds, use rattler-build #99

Merged
merged 6 commits into from
Jan 16, 2025
Merged

Conversation

minrk
Copy link
Member

@minrk minrk commented Jan 9, 2025

  • switch to rattler-build
  • flatten build matrix, so duplicate libdolfinx isn't built (fewer, longer jobs because the python matrix isn't parallelized anymore)
  • windows still (temporarily) skipped

@conda-forge-admin

This comment was marked as outdated.

file: test-dolfinx
env:
scalar: ${{ scalar }}
target_platform: ${{ target_platform }}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These shouldn't be required, but are for now to workaround prefix-dev/rattler-build#1317

@minrk minrk mentioned this pull request Jan 9, 2025
@minrk
Copy link
Member Author

minrk commented Jan 13, 2025

With flatter builds, loads of time is saved because 80% of libdolfin builds are redundant (assuming a matrix of five Pythons). But the actual time-to-completion is still longer because of conda-forge's parallelism. But it does mean we are hogging a lot of resources, especially for the 40 arm/ppc builds, which take much longer than the rest, especially in the Python test phase.

Some summary of times:

flat (no python matrix) split (python matrix) conda-build (python matrix)
builds 20 100 100
wall time 93 min 63 min 86 min
serialized CI time 18 hours 32 hours 47 hours
shortest build 18 min 6 min 14 min
longest build 93 min 41 min 45 min

So if everything's going smoothly, a lot of resources are saved by flattening builds like this. But if/when things start failing (e.g. the frequent intermittent network issues we've been seeing over the last week or so), the flatter builds significantly impact the turnaround time and cost of restarted builds. Notably, the shortest merged builds are only slightly slower than the shortest single-python builds with conda-build, so maybe this isn't so bad.

@minrk
Copy link
Member Author

minrk commented Jan 13, 2025

@conda-forge-admin please lint

@minrk
Copy link
Member Author

minrk commented Jan 14, 2025

I'll merge this one once conda-smithy releases the lint fixes

@conda-forge-admin
Copy link
Contributor

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/recipe.yaml) and found it was in an excellent condition.

@minrk minrk marked this pull request as ready for review January 16, 2025 10:00
@minrk minrk merged commit 973010e into conda-forge:main Jan 16, 2025
23 checks passed
@minrk minrk deleted the rattler-flat branch January 16, 2025 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants