Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pixi over setup scripts on CI + local dev #4302

Merged
merged 25 commits into from
Nov 23, 2023
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
4bbbe7e
ensure toolchain is installed on ci_docker
jprochazk Nov 21, 2023
06f16dc
remove `setup_web.sh` usage from `ci_docker`
jprochazk Nov 21, 2023
0c08660
delete `start_server.sh`
jprochazk Nov 21, 2023
b9b7931
add `binaryen` for `wasm-opt`
jprochazk Nov 21, 2023
af79d20
add `nox` (without `pipx`)
jprochazk Nov 21, 2023
1708155
remove usage of `pngcrush`
jprochazk Nov 21, 2023
bf438fd
add lib deps from `setup_dev.sh` to `troubleshooting.md`
jprochazk Nov 21, 2023
00b8e01
fix lint
jprochazk Nov 21, 2023
54667b6
remove `setup_web.sh` usage from build/upload rerun-cli
jprochazk Nov 22, 2023
8d8bccc
Merge branch 'main' into jan/obliterate-setup-scripts
jprochazk Nov 22, 2023
c72ba35
update cspell
jprochazk Nov 22, 2023
a61d65a
caching is slow
jprochazk Nov 22, 2023
b8f8350
use pixi to build rerun-cli
jprochazk Nov 22, 2023
42daff3
remove unused task
jprochazk Nov 22, 2023
f55bb22
fix get-sha jobs to use pr head ref
jprochazk Nov 22, 2023
4df2c73
use pixi to build wheel
jprochazk Nov 22, 2023
e303b48
build rerun cli by calling cargo directly via pixi
jprochazk Nov 22, 2023
0b6cc92
don't cache pixi anywhere
jprochazk Nov 22, 2023
2670bc0
i don't know bash
jprochazk Nov 22, 2023
57c9f32
test `pixi run` in `build_web_demo`
jprochazk Nov 22, 2023
a24f75a
use pixi in publish web
jprochazk Nov 22, 2023
110a569
reduce the number of dependencies installed in `build-web-demo`
jprochazk Nov 22, 2023
13f888e
remove `source_link_commit_override` in favor of `get-sha` step
jprochazk Nov 22, 2023
29d8ca5
upgrade `timm` to fix python3.11 compatibility
jprochazk Nov 22, 2023
472fd7d
Merge branch 'main' into jan/obliterate-setup-scripts
jprochazk Nov 23, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 5 additions & 47 deletions .github/workflows/reusable_build_and_upload_rerun_cli.yml
Original file line number Diff line number Diff line change
Expand Up @@ -135,56 +135,13 @@ jobs:
service_account: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}
targets: ${{ needs.set-config.outputs.TARGET }}

# The pip-cache setup logic doesn't work in the ubuntu docker container
# That's probably fine since we bake these deps into the container already
- name: Setup python
if: ${{ inputs.PLATFORM != 'linux' }}
uses: actions/setup-python@v4
- uses: prefix-dev/[email protected]
with:
python-version: ${{ env.PYTHON_VERSION }}
cache: "pip"
cache-dependency-path: "scripts/ci/requirements.txt"

# These should already be in the docker container, but run for good measure. A no-op install
# should be fast, and this way things don't break if we add new packages without rebuilding
# docker
- run: pip install -r scripts/ci/requirements.txt

# We have a nice script for that: ./scripts/setup_web.sh
# Unfortunately, we can't run that on Windows, because Windows doesn't come with
# a package manager like grown-up OSes do (at least not the CI version of Windows).
# Also we can't run it on linux because the 20.04 Docker container will install
# an old version of binaryen/wasm-opt that barfs on the `--fast-math` flag
# So we only run the script on macos, and then on Windows we do the parts of the script manually.
# On ubuntu, the correct packages are pre-installed in our docker container.

- name: Install prerequisites for building the web-viewer Wasm (non-Windows)
if: (inputs.PLATFORM == 'macos-intel') || (inputs.PLATFORM == 'macos-arm')
shell: bash
run: ./scripts/setup_web.sh
pixi-version: v0.6.0

# The first steps of setup_web.sh, for Windows:
- name: Install wasm32 cargo target for building the web-viewer Wasm on windows
if: inputs.PLATFORM == 'windows'
- name: Build rerun-cli
shell: bash
run: rustup target add wasm32-unknown-unknown

# The last step of setup_web.sh, for Windows.
# Since 'winget' is not available within the GitHub runner, we download the package directly:
# See: https://github.com/marketplace/actions/engineerd-configurator
- name: Install binaryen for building the web-viewer Wasm on windows
if: inputs.PLATFORM == 'windows'
uses: engineerd/[email protected]
with:
name: "wasm-opt.exe"
url: "https://github.com/WebAssembly/binaryen/releases/download/version_111/binaryen-version_111-x86_64-windows.tar.gz"
pathInArchive: "binaryen-version_111/bin/wasm-opt.exe"

- name: Build rerun-cli (release)
uses: actions-rs/cargo@v1
with:
command: build
args: --locked -p rerun-cli --no-default-features --features native_viewer,web_viewer --release --target ${{ needs.set-config.outputs.TARGET }}
run: pixi run build-rerun-cli --target ${{ needs.set-config.outputs.TARGET }}

- id: "auth"
uses: google-github-actions/auth@v1
Expand Down Expand Up @@ -212,3 +169,4 @@ jobs:
path: "./target/${{ needs.set-config.outputs.TARGET }}/release/${{ needs.set-config.outputs.BIN_NAME }}"
destination: "rerun-builds/adhoc/${{inputs.ADHOC_NAME}}/rerun-cli/${{ inputs.PLATFORM }}"
parent: false

10 changes: 6 additions & 4 deletions ci_docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,12 @@ RUN set -eux; \
cargo --version; \
rustc --version;

# Install stuff from `rust-toolchain`
ADD rust-toolchain rust-toolchain
# `show` is currently the only way to install whatever is in `rust-toolchain`
# This is subject to change, but there is no alternative: https://github.com/rust-lang/rustup/issues/1397
RUN rustup show

# Install some cargo tools we know we'll always need
# We can't do this until after we've installed rust / cargo above
# Latest cargo-deny requires a rust update!
Expand All @@ -97,10 +103,6 @@ RUN pip install -r requirements-build.txt
ADD rerun_py/requirements-lint.txt requirements-lint.txt
RUN pip install -r requirements-lint.txt

# Install tools from setup_web.sh
ADD scripts/setup_web.sh setup_web.sh
# Installing binaryen is going to cause this script to fail, but that's ok -- we install it manually
RUN ./setup_web.sh || true
# Note: We need a more modern binaryen than ships on ubuntu-20.4, so we pull it from github
RUN curl -L https://github.com/WebAssembly/binaryen/releases/download/version_112/binaryen-version_112-x86_64-linux.tar.gz | tar xzk --strip-components 1

Expand Down
35 changes: 32 additions & 3 deletions docs/content/getting-started/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,44 @@ or [join our Discord](https://discord.gg/Gcm8BbTaAj).
## Running on Linux
Rerun should work out-of-the-box on Mac and Windows, but on Linux you need to first run:

`sudo apt-get install -y libclang-dev libgtk-3-dev libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev libxkbcommon-dev libssl-dev`
```sh
sudo apt-get -y install \
libclang-dev \
libatk-bridge2.0 \
libfontconfig1-dev \
libfreetype6-dev \
libglib2.0-dev \
libgtk-3-dev \
libssl-dev \
libxcb-render0-dev \
libxcb-shape0-dev \
libxcb-xfixes0-dev \
libxkbcommon-dev \
patchelf
```

On Fedora Rawhide you need to run:

`dnf install clang clang-devel clang-tools-extra libxkbcommon-devel pkg-config openssl-devel libxcb-devel`
```sh
sudo dnf install \
clang \
clang-devel \
clang-tools-extra \
libxcb-devel \
libxkbcommon-devel \
openssl-devel \
pkg-config
```

On WSL2, in addition to the above packages for Linux, you also need to run:

`sudo apt-get install -y libvulkan1 libxcb-randr0 mesa-vulkan-drivers adwaita-icon-theme-full`
```sh
sudo apt-get -y install \
libvulkan1 \
libxcb-randr0 \
mesa-vulkan-drivers \
adwaita-icon-theme-full
```

[TODO(#1250)](https://github.com/rerun-io/rerun/issues/1250): Running with the wayland window manager
sometimes causes Rerun to crash. Try setting `WINIT_UNIX_BACKEND=x11` as a workaround.
Expand Down
5 changes: 5 additions & 0 deletions docs/cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,11 @@
"Larsson",
"lavapipe",
"lerp",
"libatk",
"libclang",
"libfontconfig",
"libfreetype",
"libglib",
"libgtk",
"librerun",
"libvulkan",
Expand Down Expand Up @@ -214,6 +218,7 @@
"pablovela",
"pacman",
"parametrizations",
"patchelf",
"pathlib",
"Pautrat",
"Pavlakos",
Expand Down
Loading
Loading