[vision] Every CI job maps to a locally-runnable pixi command

[jleibs]

Background

• Local-first + generated CI #4148 provides more details of a concrete vision of how we can make our CI job-graph much easier to develop and maintain.

This is an incremental step toward that vision specifically focused on the maintenance and execution of individual jobs.

Non-goals

We will not try to solve:

• Job-dependency-based graph execution
• Cross-compilation

Goal

Any user can produce the same results as the jobs that are run in CI by instead running a sequence of 1 or more pixi commands in the correct order, on machines that are compatible with the job.

Proposal

• Every CI job maps to a single pixi command
• Any parameterizable job/workflow details are specified via CLI arguments to that job
• Any artifacts consumed by or produced by that job are mediated via gcloud artifacts using an environment key + commit-sha
  • To keep local testing from leaking into proper CI builds, by default, jobs run in CI use a "CI" environment key and jobs run locally use a "dev" environment key.
  • If the required artifacts for the job aren't found in gcloud, the job prints out an error with the missing artifact and where it expects to find them.
  • Every job should always store its output artifacts at a location based on the name of the job. That way when we capture the inputs a job depends on, we are implicitly capturing the jobs that must be run before in the job graph. (Again, no magic execution here, but it's a helpful breadcrumb to enable the user to figure out what else needs to run if their local job is missing inputs).
  • Optional: abstract the gcloud upload/download so that we can set an environment variable to use a local folder instead of gcloud.
• Every job is designed to be safely run repeatedly. If its output is already present it should gracefully not run. If passed a "--force" flag, it should run and re-upload the artifact.
• The CI jobs now just use the prefix-dev/setup-pixi action followed by a single run command such as:
  - run: pixi run job-name --param1 ${{ inputs.PARAM1}}