Locally build PROD image contains local source for airflow but pypi code for providers
#37815
Closed
1 task done
Labels
kind:meta
High-level information important to the community
Comments
|
cc: cc: @sudiptob2 and @o-nikolas @jedcunningham - recorded the issue with PROD image/ Kubernetes here - I hope with #37796 merged, we will be able to solve it quickly. |
potiuk
added a commit
to potiuk/airflow
that referenced
this issue
Mar 3, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. It fixes a number of problems the `--user` installation created us in the past and does it in fully backwards compatible way. This improves both "production" use for end user as well as local iteration on the PROD image during tests - but also for CI. Improvements for "end user": * user does not have to use `pip install --user` to install new packages any more and it is not enabled by default with PIP_USER flag. * users can use uv to install packages when they extend the image (but it's not obligatory - pip continues working as it did) * users can use `uv` to build custom production image, which gives 40%-50% saving for image build time compring to `pip`. * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used) - so we have full compatibility with previous images. Improvements for development: * when image is built from sources (no --use-docker-context-files are specified), airflow is installed in --editable mode, which means that airflow + all providers are installed locally from airflow sources, not from packages - which means that both airflow and providers have the latest version inside the prod image. * when local sources changes and you want to run k8s tests locally, it is now WAY faster (several minutes) to iterate with your changes because you do not have to rebuild the base image - the only thing needed is to copy sources to the PROD image to "/opt/airflow" which is where editable installlation is done from. You only need to rebuild the image if dependencies change. * By default `uv` is used for local source build for k8s tests so even if you have to rebuild it, it is way faster (60%-80%) during iterating with the image. CI/DEV tooling improvements: * this PR switches to use `uv` by default for most prod images we build in CI, but it adds a check if the image still builds with `pip`. * we also switch to more PEP standard way of installing packages from local filesystem (package-name @ file:///FILE) Fixes: apache#37785 Fixes: apache#37815
potiuk
added a commit
that referenced
this issue
Mar 3, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. It fixes a number of problems the `--user` installation created us in the past and does it in fully backwards compatible way. This improves both "production" use for end user as well as local iteration on the PROD image during tests - but also for CI. Improvements for "end user": * user does not have to use `pip install --user` to install new packages any more and it is not enabled by default with PIP_USER flag. * users can use uv to install packages when they extend the image (but it's not obligatory - pip continues working as it did) * users can use `uv` to build custom production image, which gives 40%-50% saving for image build time compring to `pip`. * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used) - so we have full compatibility with previous images. Improvements for development: * when image is built from sources (no --use-docker-context-files are specified), airflow is installed in --editable mode, which means that airflow + all providers are installed locally from airflow sources, not from packages - which means that both airflow and providers have the latest version inside the prod image. * when local sources changes and you want to run k8s tests locally, it is now WAY faster (several minutes) to iterate with your changes because you do not have to rebuild the base image - the only thing needed is to copy sources to the PROD image to "/opt/airflow" which is where editable installlation is done from. You only need to rebuild the image if dependencies change. * By default `uv` is used for local source build for k8s tests so even if you have to rebuild it, it is way faster (60%-80%) during iterating with the image. CI/DEV tooling improvements: * this PR switches to use `uv` by default for most prod images we build in CI, but it adds a check if the image still builds with `pip`. * we also switch to more PEP standard way of installing packages from local filesystem (package-name @ file:///FILE) Fixes: #37785 Fixes: #37815
potiuk
added a commit
that referenced
this issue
Mar 5, 2024
This PR introduces a joint way to treat the .local (--user) folder as both - venv and `--user` package installation. It fixes a number of problems the `--user` installation created us in the past and does it in fully backwards compatible way. This improves both "production" use for end user as well as local iteration on the PROD image during tests - but also for CI. Improvements for "end user": * user does not have to use `pip install --user` to install new packages any more and it is not enabled by default with PIP_USER flag. * users can use uv to install packages when they extend the image (but it's not obligatory - pip continues working as it did) * users can use `uv` to build custom production image, which gives 40%-50% saving for image build time compring to `pip`. * python -m venv --system-site-packages continues to use the .local packages from the .local installation (and not uses them if --system-site-packages is not used) - so we have full compatibility with previous images. Improvements for development: * when image is built from sources (no --use-docker-context-files are specified), airflow is installed in --editable mode, which means that airflow + all providers are installed locally from airflow sources, not from packages - which means that both airflow and providers have the latest version inside the prod image. * when local sources changes and you want to run k8s tests locally, it is now WAY faster (several minutes) to iterate with your changes because you do not have to rebuild the base image - the only thing needed is to copy sources to the PROD image to "/opt/airflow" which is where editable installlation is done from. You only need to rebuild the image if dependencies change. * By default `uv` is used for local source build for k8s tests so even if you have to rebuild it, it is way faster (60%-80%) during iterating with the image. CI/DEV tooling improvements: * this PR switches to use `uv` by default for most prod images we build in CI, but it adds a check if the image still builds with `pip`. * we also switch to more PEP standard way of installing packages from local filesystem (package-name @ file:///FILE) Fixes: #37785 Fixes: #37815 Update contributing-docs/testing/k8s_tests.rst Co-authored-by: Niko Oliveira <[email protected]> Update contributing-docs/testing/k8s_tests.rst Co-authored-by: Niko Oliveira <[email protected]> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <[email protected]> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <[email protected]> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <[email protected]> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <[email protected]> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <[email protected]> Update scripts/docker/install_airflow.sh Co-authored-by: Niko Oliveira <[email protected]> Update docs/docker-stack/changelog.rst Co-authored-by: Niko Oliveira <[email protected]> Update docs/docker-stack/build.rst Co-authored-by: Niko Oliveira <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Body
The PROD image built using
breeze prod-image buildafter #36536 and after changes inpipandsetuptoolsmade it impossible to install--editableinstallation in--userinstallation is broken - see pypa/pip#7953 for details.It contains only airflow sources copied from local code, but provider s are installed from
PyPIwhich makes it impossible to iterate locally on the PROD image changes and makes it difficult to test any local changes - for example incncf.kubernetesprovider.This does not affect
CItests - because CI tests build PROD image by pre-building dev packages for all PROD-image providers and airflow and install both Airflow and Providers from such pre-built packages copied todocker-context-files, so our CI still test what is in the PR.This isssue can be solved once we merge #37796 - there we switch PROD image to use
venvrather than--userflag and that should allow to fix local instalation to revert back to use--editablebuild and copy all airflow sources to the image including all providers, allowing to iteratively test it. Also, by doing that we should be able to add better ways of sharing the code in the future (for example using TelepresenceCommitter
The text was updated successfully, but these errors were encountered: