Updates installation page organization and contents#3542
Updates installation page organization and contents#3542melissawm wants to merge 1 commit intoAI-Hypercomputer:mainfrom
Conversation
melissawm
requested review from
A9isha,
NicoGrande,
NuojCheng,
RissyRan,
SurbhiJainUSC,
aireenmei,
bvandermoon,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
igorts-git,
jacoguzo,
jesselu-google,
jiangjy1982,
khatwanimohit,
richjames0,
shralex,
suexu1025 and
vipannalla
as code owners
|
|
||
| ```bash | ||
| # 1. Install uv, a fast Python package installer | ||
| pip install uv |
There was a problem hiding this comment.
Using pip install uv is a bad idea here, since it will install uv under the global python env and may create problems later. I modified this to include a link to the official uv docs so users can install the tool globally and then use it to install MaxText. Let me know if you have better ideas.
There was a problem hiding this comment.
The official documentation also suggests to do pip install uv: https://docs.astral.sh/uv/getting-started/installation/#pypi. So, what is the differecne?
There was a problem hiding this comment.
It definitely is supported, but the installation method we are suggesting here sounds a bit convoluted to me. At least in my own system (Linux) I see the following when running pip install uv:
➜ maxtext git:(install-maxtext) pip install uv
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
python-xyz', where xyz is the package you are trying to
install.
If you wish to install a non-Arch-packaged Python package,
create a virtual environment using 'python -m venv path/to/venv'.
Then use path/to/venv/bin/python and path/to/venv/bin/pip.
If you wish to install a non-Arch packaged Python application,
it may be easiest to use 'pipx install xyz', which will manage a
virtual environment for you. Make sure you have python-pipx
installed via pacman.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
➜ maxtext git:(install-maxtext)
If I do activate a venv before pip installing uv, it defeats the purpose of having the maxtext/uv-managed env at all. So I wonder if we could just recommend folks to install uv independently, and not potentially run into this kind of warning.
There's actually another problem here, in that the PyPI workflow as described will only work if folks clone the repo first, since they would need the maxtext_env environment that is checked in to version control. So I think all these instructions need to be reviewed.
My suggestion would be to
- Remove the
maxtext_venvuse for PyPI installations; or - Mention that in all cases we need to clone the repo first.
Since all instructions in this page include using the helper scripts for downloading github dependencies right now, I'll go with #2.
Now there is another issue that users might face: the repo has dependencies for the latest development version; if we try installing the stable version with pip using the maxtext_venv from latest, this will probably lead to conflicts in the installed packages, no?
| install_maxtext_tpu_github_deps | ||
| Choose a single installation option from this list to fit your use case. For | ||
| each installation option, a different helper script is necessary to complete | ||
| the installation of required dependencies that are not yet available on PyPI. |
There was a problem hiding this comment.
I have split these into multiple code blocks since this will make it easier for readers to copy each of them individually (since they will only execute one of the options)
|
The docs build here will pass once #3247 is merged. |
| 1. **Modify Base Requirements**: Update the desired dependencies in `base_requirements/requirements.txt` or the hardware-specific files (`base_requirements/tpu-base-requirements.txt`, `base_requirements/gpu-base-requirements.txt`). | ||
| 2. **Generate New Files**: Run the `seed-env` CLI tool to generate new, fully-pinned requirements files based on your changes. | ||
| 3. **Update Project Files**: Copy the newly generated files into the `generated_requirements/` directory. | ||
| 4. **Handle GitHub Dependencies**: Move any dependencies that are installed directly from GitHub from the generated files to `src/install_maxtext_extra_deps/extra_deps_from_github.txt`. |
There was a problem hiding this comment.
Change src/install_maxtext_extra_deps/extra_deps_from_github.txt to src/dependencies/github_deps
|
|
||
| ```bash | ||
| seed-env \ | ||
| --local-requirements=src/dependencies/requirements/base_requirements/cuda12-base-requirements.txt \ |
There was a problem hiding this comment.
nit: gpu-base-requirements.txt
|
|
||
| Finally, test that the new dependencies install correctly and that MaxText runs as expected. | ||
|
|
||
| 1. **Create a clean environment:** It's best to start with a fresh Python virtual environment. |
There was a problem hiding this comment.
Remove installation steps (1. and 2.) and add a link to install_maxtext documentation: https://maxtext.readthedocs.io/en/latest/install_maxtext.html#from-source
| 1. **Modify Base Requirements**: Update the desired dependencies in `base_requirements/requirements.txt` or the hardware-specific files (`base_requirements/tpu-base-requirements.txt`, `base_requirements/gpu-base-requirements.txt`). | ||
| 2. **Generate New Files**: Run the `seed-env` CLI tool to generate new, fully-pinned requirements files based on your changes. | ||
| 3. **Update Project Files**: Copy the newly generated files into the `generated_requirements/` directory. | ||
| 4. **Handle GitHub Dependencies**: Move any dependencies that are installed directly from GitHub from the generated files to `src/install_maxtext_extra_deps/extra_deps_from_github.txt`. |
There was a problem hiding this comment.
I don't think seed-env tool generates requirements in github deps. Maybe we can remove this.
cc: @bvandermoon
There was a problem hiding this comment.
Sorry - I did not update this page, just moved it from the installation page. Will make sure to review in my next commit.
| - Move `generated_tpu_artifacts/tpu-requirements.txt` to `generated_requirements/tpu-requirements.txt`. | ||
| - Move `generated_gpu_artifacts/cuda12-requirements.txt` to `generated_requirements/cuda12-requirements.txt`. | ||
|
|
||
| 2. **Update `extra_deps_from_github.txt` (if necessary):** |
There was a problem hiding this comment.
Same comment as above.
|
|
||
| ## Step 1: Install seed-env | ||
|
|
||
| First, you need to install the `seed-env` command-line tool by running `pip install seed-env uv`. Or follow the instructions in the |
There was a problem hiding this comment.
In another doc in this PR you commented that pip install uv is not a good idea.
There was a problem hiding this comment.
Sorry - I did not update this page, just moved it from the installation page. Will make sure to review in my next commit.
| source maxtext_venv/bin/activate | ||
| ```bash | ||
| uv pip install maxtext[cuda12]==0.2.1 --resolution=lowest | ||
| install_maxtext_cuda12_github_dep |
There was a problem hiding this comment.
I believe this should have an "s" suffix: "install_maxtext_cuda12_github_deps". It appears several times in this doc.
|
|
||
| ```bash | ||
| # 1. Install uv, a fast Python package installer | ||
| pip install uv |
There was a problem hiding this comment.
The official documentation also suggests to do pip install uv: https://docs.astral.sh/uv/getting-started/installation/#pypi. So, what is the differecne?
3 participants
Description
Updates installation page contents and structure, with focus on complying with the Google Technical Writing Guidelines and team feedback.
Also splits the "How to update requirements" to a separate page under a "development" folder, containing documentation on how to contribute.
Tests
Documentation builds locally.
Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.