Skip to content

Move downgrade/upgrade tests to run new Python breeze #26828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Oct 3, 2022
Merged

Move downgrade/upgrade tests to run new Python breeze #26828

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/build-images.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ permissions:
contents: read
pull-requests: read
env:
MOUNT_SELECTED_LOCAL_SOURCES: "false"
ANSWER: "yes"
DB_RESET: "true"
VERBOSE: "true"
Expand Down
87 changes: 70 additions & 17 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ permissions:
contents: read
packages: read
env:
MOUNT_SELECTED_LOCAL_SOURCES: "false"
ANSWER: "yes"
DB_RESET: "true"
VERBOSE: "true"
Expand Down Expand Up @@ -963,7 +962,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
needs: [build-info, wait-for-ci-images]
env:
RUNS_ON: ${{ needs.build-info.outputs.runs-on }}
MOUNT_SELECTED_LOCAL_SOURCES: "true"
TEST_TYPES: "Helm"
BACKEND: ""
DB_RESET: "false"
Expand Down Expand Up @@ -1055,10 +1053,28 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
run: breeze ci-image pull --tag-as-latest
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade"
run: ./scripts/ci/testing/run_downgrade_test.sh
- name: "Test Offline SQL generation"
run: ./scripts/ci/testing/run_offline_sql_test.sh
- name: "Test downgrade migration file ${{ env.BACKEND }}"
run: >
breeze shell "airflow db reset --skip-init -y &&
airflow db upgrade --to-revision heads &&
airflow db downgrade -r e959f08ac86c -y &&
airflow db upgrade"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade ORM ${{ env.BACKEND }}"
run: >
breeze shell "airflow db reset -y &&
airflow db upgrade &&
airflow db downgrade -r e959f08ac86c -y &&
airflow db upgrade"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test Offline SQL generation ${{ env.BACKEND }}"
run: >
breeze shell "airflow db upgrade --from-version 2.0.0 -r heads --show-sql-only &&
airflow db downgrade --to-version 2.0.0 --show-sql-only -y"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Tests: ${{needs.build-info.outputs.test-types}}"
run: breeze testing tests --run-in-parallel
env:
Expand Down Expand Up @@ -1129,8 +1145,22 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
run: breeze ci-image pull --tag-as-latest
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade"
run: ./scripts/ci/testing/run_downgrade_test.sh
- name: "Test downgrade migration file ${{ env.BACKEND }}"
run: >
breeze shell "airflow db reset --skip-init -y &&
airflow db upgrade --to-revision heads &&
airflow db downgrade -r e959f08ac86c -y &&
airflow db upgrade"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade ORM ${{ env.BACKEND }}"
run: >
breeze shell "airflow db reset -y &&
airflow db upgrade &&
airflow db downgrade -r e959f08ac86c -y &&
airflow db upgrade"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Tests: ${{needs.build-info.outputs.test-types}}"
run: breeze testing tests --run-in-parallel
env:
Expand Down Expand Up @@ -1201,8 +1231,22 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
run: breeze ci-image pull --tag-as-latest
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade"
run: ./scripts/ci/testing/run_downgrade_test.sh
- name: "Test downgrade migration file ${{ env.BACKEND }}"
run: >
breeze shell "airflow db reset --skip-init -y &&
airflow db upgrade --to-revision heads &&
airflow db downgrade -r e959f08ac86c -y &&
airflow db upgrade"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade ORM ${{ env.BACKEND }}"
run: >
breeze shell "airflow db reset -y &&
airflow db upgrade &&
airflow db downgrade -r e959f08ac86c -y &&
airflow db upgrade"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Tests: ${{needs.build-info.outputs.test-types}}"
run: breeze testing tests --run-in-parallel
env:
Expand Down Expand Up @@ -1271,8 +1315,22 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
run: breeze ci-image pull --tag-as-latest
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade"
run: ./scripts/ci/testing/run_downgrade_test.sh
- name: "Test downgrade migration file ${{ env.BACKEND }}"
run: >
breeze shell "airflow db reset --skip-init -y &&
airflow db upgrade --to-revision heads &&
airflow db downgrade -r e959f08ac86c -y &&
airflow db upgrade"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Test downgrade ORM ${{ env.BACKEND }}"
run: >
breeze shell "airflow db reset -y &&
airflow db upgrade &&
airflow db downgrade -r e959f08ac86c -y &&
airflow db upgrade"
env:
IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
- name: "Tests: ${{needs.build-info.outputs.test-types}}"
run: breeze testing tests --run-in-parallel
env:
Expand Down Expand Up @@ -1448,11 +1506,6 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- run: ./scripts/ci/install_breeze.sh
- name: "Free space"
run: breeze ci free-space
- name: "Cache virtualenv environment"
uses: actions/cache@v3
with:
path: '.build/.docker_venv'
key: ${{ runner.os }}-docker-venv-${{ hashFiles('scripts/ci/images/ci_run_docker_tests.py') }}
- name: Wait for PROD images ${{ env.PYTHON_VERSIONS }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}
# We wait for the images to be available either from "build-images.yml' run as pull_request_target
# or from build-prod-images above.
Expand Down
60 changes: 8 additions & 52 deletions CI.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -487,7 +487,8 @@ the sources are not mapped from your host machine.
Depending whether the scripts are run locally via `Breeze <BREEZE.rst>`_ or whether they
are run in ``Build Images`` or ``Tests`` workflows they can take different values.

You can use those variables when you try to reproduce the build locally.
You can use those variables when you try to reproduce the build locally (alternatively you can pass
those via command line flags passed to ``breeze`` command.

+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Variable | Local | Build Images | CI | Comment |
Expand All @@ -505,29 +506,7 @@ You can use those variables when you try to reproduce the build locally.
| | | | | it requires to perform manual init/reset |
| | | | | if you stop the environment. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Mount variables |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``MOUNT_SELECTED_LOCAL_SOURCES`` | true | false | false | Determines whether local sources are |
| | | | | mounted to inside the container. Useful for |
| | | | | local development, as changes you make |
| | | | | locally can be immediately tested in |
| | | | | the container. We mount only selected, |
| | | | | important folders. We do not mount the whole |
| | | | | project folder in order to avoid accidental |
| | | | | use of artifacts (such as ``egg-info`` |
| | | | | directories) generated locally on the |
| | | | | host during development. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``MOUNT_ALL_LOCAL_SOURCES`` | false | false | false | Determines whether all local sources are |
| | | | | mounted to inside the container. Useful for |
| | | | | local development when you need to access .git |
| | | | | folders and other folders excluded when |
| | | | | ``MOUNT_SELECTED_LOCAL_SOURCES`` is true. |
| | | | | You might need to manually delete egg-info |
| | | | | folder when you enter breeze and the folder was |
| | | | | generated using different Python versions. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Force variables |
| Forcing answer |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``ANSWER`` | | yes | yes | This variable determines if answer to questions |
| | | | | during the build process should be |
Expand All @@ -550,7 +529,7 @@ You can use those variables when you try to reproduce the build locally.
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``COMMIT_SHA`` | | GITHUB_SHA | GITHUB_SHA | SHA of the commit of the build is run |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Initialization |
| In container environment initialization |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``SKIP_ENVIRONMENT_INITIALIZATION`` | false\* | false\* | false\* | Skip initialization of test environment |
| | | | | |
Expand All @@ -560,35 +539,12 @@ You can use those variables when you try to reproduce the build locally.
| | | | | |
| | | | | \* set to true in GitHub CodeSpaces |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Verbosity variables |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``PRINT_INFO_FROM_SCRIPTS`` | true\* | true\* | true\* | Allows to print output to terminal from running |
| | | | | scripts. It prints some extra outputs if true |
| | | | | including what the commands do, results of some |
| | | | | operations, summary of variable values, exit |
| | | | | status from the scripts, outputs of failing |
| | | | | commands. If verbose is on it also prints the |
| | | | | commands executed by docker, kind, helm, |
| | | | | kubectl. Disabled in pre-commit checks. |
| | | | | |
| | | | | \* set to false in pre-commits |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``VERBOSE`` | false | true | true | Determines whether docker, helm, kind, |
| | | | | kubectl commands should be printed before |
| | | | | execution. This is useful to determine |
| | | | | what exact commands were executed for |
| | | | | debugging purpose as well as allows |
| | | | | to replicate those commands easily by |
| | | | | copy&pasting them from the output. |
| | | | | requires ``PRINT_INFO_FROM_SCRIPTS`` set to |
| | | | | true. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| ``VERBOSE_COMMANDS`` | false | false | false | Determines whether every command |
| | | | | executed in bash should also be printed |
| | | | | executed in docker should also be printed |
| | | | | before execution. This is a low-level |
| | | | | debugging feature of bash (set -x) and |
| | | | | it should only be used if you are lost |
| | | | | at where the script failed. |
| | | | | debugging feature of bash (set -x) enabled in |
| | | | | entrypoint and it should only be used if you |
| | | | | need to debug the bash scripts in container. |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
| Image build variables |
+-----------------------------------------+-------------+--------------+------------+-------------------------------------------------+
Expand Down
26 changes: 1 addition & 25 deletions dev/README_RELEASE_AIRFLOW.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,31 +358,7 @@ Run script to re-tag images from the ``main`` branch to the ``vX-Y-test`` branc

### Update default branches

#### In the legacy, bash breeze (to be removed when the bash breeze is entirely gone)

In ``./scripts/ci/libraries/_intialization.sh`` update branches to reflect the new branch:

```bash
export DEFAULT_BRANCH=${DEFAULT_BRANCH="main"}
export DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH="constraints-main"}
```

should become this, where ``X-Y`` is your new branch version:

```bash
export DEFAULT_BRANCH=${DEFAULT_BRANCH="vX-Y-test"}
export DEFAULT_CONSTRAINTS_BRANCH=${DEFAULT_CONSTRAINTS_BRANCH="constraints-X-Y"}
```

In ``./scripts/ci/libraries/_build_images.sh`` add branch to preload packages from (replace X and Y in
values for comparison and regexp):

```bash
elif [[ ${AIRFLOW_VERSION} =~ v?X\.Y* ]]; then
AIRFLOW_BRANCH_FOR_PYPI_PRELOADING="vX-Y-stable"
```

#### In the new breeze
#### In Breeze

In ``./dev/breeze/src/airflow_breeze/branch_defaults.py`` update branches to reflect the new branch:

Expand Down
1 change: 0 additions & 1 deletion dev/breeze/src/airflow_breeze/utils/docker_command_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -589,7 +589,6 @@ def update_expected_environment_variables(env: dict[str, str]) -> None:
set_value_to_default_if_not_set(env, 'LOAD_DEFAULT_CONNECTIONS', "false")
set_value_to_default_if_not_set(env, 'LOAD_EXAMPLES', "false")
set_value_to_default_if_not_set(env, 'PACKAGE_FORMAT', ALLOWED_PACKAGE_FORMATS[0])
set_value_to_default_if_not_set(env, 'PRINT_INFO_FROM_SCRIPTS', "true")
set_value_to_default_if_not_set(env, 'PYTHONDONTWRITEBYTECODE', "true")
set_value_to_default_if_not_set(env, 'RUN_SYSTEM_TESTS', "false")
set_value_to_default_if_not_set(env, 'RUN_TESTS', "false")
Expand Down
3 changes: 0 additions & 3 deletions scripts/ci/constraints/ci_branch_constraints.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,6 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"

if [[ ${GITHUB_REF} == 'refs/heads/main' ]]; then
echo "::set-output name=branch::constraints-main"
elif [[ ${GITHUB_REF} =~ refs/heads/v([0-9\-]*)\-(test|stable) ]]; then
Expand Down
3 changes: 0 additions & 3 deletions scripts/ci/constraints/ci_commit_constraints.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,6 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
# shellcheck source=scripts/ci/libraries/_script_init.sh
. "$( dirname "${BASH_SOURCE[0]}" )/../libraries/_script_init.sh"

cp -v ./files/constraints-*/constraints*.txt repo/
cd repo || exit 1
git config --local user.email "[email protected]"
Expand Down
1 change: 0 additions & 1 deletion scripts/ci/docker-compose/_docker.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ MYSQL_VERSION
NUM_RUNS
PACKAGE_FORMAT
POSTGRES_VERSION
PRINT_INFO_FROM_SCRIPTS
PYTHONDONTWRITEBYTECODE
PYTHON_MAJOR_MINOR_VERSION
RUN_TESTS
Expand Down
Loading