Skip to content

Split Build job into Build Packages and Build rest of artifacts#30173

Merged
scbedd merged 13 commits intoAzure:mainfrom
scbedd:feature/split-build-job
Jun 12, 2023
Merged

Split Build job into Build Packages and Build rest of artifacts#30173
scbedd merged 13 commits intoAzure:mainfrom
scbedd:feature/split-build-job

Conversation

@scbedd
Copy link
Copy Markdown
Member

@scbedd scbedd commented May 1, 2023

Resolves #30038

Job layout before this change

Build
  -> Analyze
  -> PythonVersionX+PlatformY
  -> PythonVersionX+PlatformY
  -> ..

Where Build contains Create wheel/sdist, Create APIStub, Create docs.

Job layout after this change

Build (wheel/sdist)
  -> Build Extended (Docs, APIStubGen)
  -> Analyze
  -> PythonVersionX+PlatformY
  -> PythonVersionX+PlatformY
  -> ..

Two primary benefits after this change:

  1. Tests will start faster as we won't be waiting for more than what is actually necessary to properly invoke those jobs. (we only actually need the whl/sdist, the rest can run and publish separately.)
  2. This also really simplifies implementation of Enable non-universal wheel assembly #28211 as we no longer need to condition the apistubgen and docs steps to only run on the ubuntu agent!

image

For core, we start testing 12 minutes earlier due to this change. Packages like storage or ml will heavily benefit as well!

@kdestin initially raised this as a possibility, and I'm super happy with how it turned out. Thanks Kevin!

@scbedd scbedd requested review from benbp and weshaggard as code owners May 1, 2023 21:18
@scbedd scbedd self-assigned this May 1, 2023
@scbedd scbedd requested a review from mccoyp May 1, 2023 23:36
Copy link
Copy Markdown
Member

@mccoyp mccoyp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! This is great idea @kdestin, and thanks @scbedd for the implementation!

@scbedd
Copy link
Copy Markdown
Member Author

scbedd commented May 2, 2023

I had a chat with @praveenkuttappan to verify.

This PR is not ready to merge as-is. We broke APIStubGen. Before, the step would inject a json output for the package in the packages artifact alongside the wheel/sdist.

<artifact staging root>/
    azure-core/
        azure-core.whl
        azure-core.zip
        azure-core_python.json

but now that we don't run APIStubGen as part of the assembly of the packages artifact, we need to pull from a different place when we run the APIStubGen verify step in Analyze.

To address this, we will make Analyze job depend on Build Extended (which will need to publish this APIStubGen artifact) and download the artifact from that job instead of Build.

@scbedd
Copy link
Copy Markdown
Member Author

scbedd commented May 3, 2023

I think this PR is ready to go in.

However, I'm going to wait for @praveenkuttappan to return and do a final check.

@scbedd
Copy link
Copy Markdown
Member Author

scbedd commented May 26, 2023

/azp run python - core - ci

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@scbedd
Copy link
Copy Markdown
Member Author

scbedd commented Jun 12, 2023

/azp run python - core - ci

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@azure-sdk
Copy link
Copy Markdown
Collaborator

API change check

APIView has identified API level changes in this PR and created following API reviews.

azure-template

@scbedd
Copy link
Copy Markdown
Member Author

scbedd commented Jun 12, 2023

/check-enforcer evaluate

@scbedd scbedd enabled auto-merge (squash) June 12, 2023 23:23
@scbedd
Copy link
Copy Markdown
Member Author

scbedd commented Jun 12, 2023

/check-enforcer evaluate

@scbedd scbedd merged commit 102fa52 into Azure:main Jun 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

Divide build into two sections packages + everything else

6 participants