Skip to content

Partial CI workflows: take 2#60294

Merged
frenck merged 2 commits intodevfrom
frenck-2021-3072
Nov 24, 2021
Merged

Partial CI workflows: take 2#60294
frenck merged 2 commits intodevfrom
frenck-2021-3072

Conversation

@frenck
Copy link
Copy Markdown
Member

@frenck frenck commented Nov 24, 2021

Proposed change

Second iteration on getting CI to run partially on PRs when possible, in order to speed up our CI.

Improvements in this PR:

  • Extended the changes jobs, that now determines all states used throughout the suite.
  • Removed separate coverage combining jobs, handled by Codecov directly now
  • Single jobs handling both fully & partially runs (prevents seeing "skipped" jobs in PR results)
  • Split runs for: Black, Bandit, Pylint, flake8, mypy 🚀
  • When run partially, a test group is created for each integration. This allows coverage to be limited to the package of the integration in the results carried forward by Codecov.
  • Coverage results and progress per integration in partial runs
  • Should run the average PR in under 5 minutes.

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • The code has been formatted using Black (black --fast homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
  • Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

  • No score or internal
  • 🥈 Silver
  • 🥇 Gold
  • 🏆 Platinum

To help with the load of incoming pull requests:

@probot-home-assistant probot-home-assistant bot added new-feature small-pr PRs with less than 30 lines. labels Nov 24, 2021
Comment on lines +27 to +28
# test_full_suite: 'true'
test_full_suite: ${{ steps.info.outputs.test_full_suite }}
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If something is fishy, this is the kill switch.

@frenck frenck marked this pull request as ready for review November 24, 2021 17:01
if: needs.changes.outputs.test_full_suite == 'false'
run: |
. venv/bin/activate
pylint homeassistant/components/${{ needs.changes.outputs.integrations_glob }}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 👍

Comment on lines +71 to +99
if [[ "${{ steps.integrations.outputs.changes }}" != "[]" ]];
then
# Create a file glob for the integrations
integrations_glob=$(echo '${{ steps.integrations.outputs.changes }}' | jq -cSr '. | join(",")')
[[ "${integrations_glob}" == *","* ]] && integrations_glob="{${integrations_glob}}"

# Create list of testable integrations
possible_integrations=$(echo '${{ steps.integrations.outputs.changes }}' | jq -cSr '.[]')
tests=$(
for integration in ${possible_integrations};
do
if [[ -d "tests/components/${integration}" ]]; then
echo -n "\"${integration}\",";
fi;
done
)

[[ ! -z "${tests}" ]] && tests="${tests::-1}"
tests="[${tests}]"
test_groups="${tests}"
# Test group count should be 1, we don't split partial tests
test_group_count=1

# Create a file glob for the integrations tests
tests_glob=$(echo "${tests}" | jq -cSr '. | join(",")')
[[ "${tests_glob}" == *","* ]] && tests_glob="{${tests_glob}}"

test_full_suite="false"
fi
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future (not for this PR) would be nicer if this was generated by a python script so this would be easier to test. Reading this gives me a bit of a headache.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me its actually the opposite 😄

Copy link
Copy Markdown
Member

@bdraco bdraco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is going to be hard to test except in production.

Since we can just as easily revert (and it is not actually going to break the code), lets do it.

@frenck frenck merged commit fb40a5c into dev Nov 24, 2021
@frenck frenck deleted the frenck-2021-3072 branch November 24, 2021 17:14
@github-actions github-actions bot locked and limited conversation to collaborators Nov 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla-signed new-feature small-pr PRs with less than 30 lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants