Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue: binary assets for everyone and everything #3942

Closed
4 tasks done
teh-cmc opened this issue Oct 20, 2023 · 0 comments · Fixed by #3998
Closed
4 tasks done

Tracking issue: binary assets for everyone and everything #3942

teh-cmc opened this issue Oct 20, 2023 · 0 comments · Fixed by #3998
Assignees
@teh-cmc
Labels
🧑‍💻 dev experience developer experience (excluding CI) 💬 discussion ⛴ release Related to shipping or publishing
Milestone
0.11

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Oct 20, 2023 •
edited
Loading

Context

We want our (pre-)releases to contain pre-compiled per-platform per-arch binary assets for all things Rerun:

  • the Rerun Viewer itself,
  • our python wheels,
  • our C bindings (rerun_c),
  • our C++ bindings (rerun_cpp),
  • our all-in-one crossplatform C/C++ package (rerun_sdk.zip),
  • anything else that fit.

These release assets are the most reliable and scalable way to distribute Rerun, no matter how complex the end user's build system situation happens to be: they integrate well with even the most complex workflows.

Additionally, a pre-built viewer makes spawn() in Rust & C++ possible, and allows us to not embed the full viewer in our python wheel if we don't want to.
For cargo users, it also makes installing the Rerun Viewer via cargo binstall possible.

Our contributor workflow is currently built on top of our cloud storage (build.rerun.io): all binary artifacts are neatly shipped and versioned in there, and CI actions simply copy stuff in and out as needed.
This is quite nice, but we cannot expect end-users to dig through our internal build systems to get what they need: only contributors should ever have to deal with build.rerun.io URLs.

End users wanting to get their hands on our release assets should need to go no further than the well standardized path provided by Github, which neatly integrates with a million tools out there.

Proposal

  • Keep everything as-is for contributors (build.rerun.io for everything).
  • Implement a new script and associated action that automatically uploads release assets from build.rerun.io to their associated github (pre)release.
    • That way everything is still driven by build.rerun.io internally, we just copy things back-and-forth at the last minute.
  • Implement an action that builds and uploads the Rerun Viewer for all platforms.
    • Make sure cargo binstall works.
  • (Re)implement spawn() in all languages using $PATH approach.
    • Use spawn() for all examples in all languages.

Pending questions

  • What about arrow-cpp artifacts?

Related issues
@teh-cmc teh-cmc added 💬 discussion 🧑‍💻 dev experience developer experience (excluding CI) ⛴ release Related to shipping or publishing labels Oct 20, 2023
@teh-cmc teh-cmc self-assigned this Oct 20, 2023
@teh-cmc teh-cmc mentioned this issue Oct 23, 2023
Merged
4 tasks
teh-cmc added a commit that referenced this issue Oct 23, 2023
@teh-cmc
Automatically synchronize build.rerun.io & release assets (#3945)
aec440c 
This introduces new scripts and workflows to synchronize the contents of
our cloud storage (`build.rerun.io`) with our release assets.

This can either be run manually from a dev machine, e.g.:
```sh
python scripts/ci/sync_release_assets.py --github-release prerelease --github-token <GITHUB_TOKEN> --remove --update
```
or manually dispatched through the Actions interface, e.g.
[here](https://github.com/rerun-io/rerun/actions/runs/6613455304/job/17961328298#step:7:25).

It will run automatically as part of the pre-release process on
push-to-main.

---

Here's how that looks today:

![image](https://github.com/rerun-io/rerun/assets/2910679/9fdc4664-ade3-40f6-bcdb-bdef76e2effe)


---

- Part of #3942 
- Fixes #2107
@teh-cmc teh-cmc mentioned this issue Oct 23, 2023
Merged
4 tasks
teh-cmc added a commit that referenced this issue Oct 23, 2023
@teh-cmc
CI: Rerun CLI as a release asset (#3959)
dbf83c4 
Builds Rerun CLI for all supported platforms, and automatically adds
them to release assets.

```sh
$ cargo binstall --force rerun-cli
 INFO resolve: Resolving package: 'rerun-cli'
 INFO This will install the following binaries:
 INFO   - rerun (bin-rerun-cli-x86_64-unknown-linux-gnu-GhCrateMeta -> /home/cmc/.cargo/bin/rerun)
 INFO Installing binaries...
 INFO Done in 2.849511699s

$ which rerun
/home/cmc/.cargo/bin/rerun

$ file $(which rerun)
/home/cmc/.cargo/bin/rerun: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=77170f12c868bb5dd3858bbe0303f210c0edcc26, for GNU/Linux 3.2.0, with debug_info, not stripped

$ rerun --version  # the version will not match 0.9.1 because I manually uploaded a tip binary to the 0.9.1 release
rerun-cli 0.10.0-alpha.7+dev [rustc 1.72.1 (d5c2e9c34 2023-09-13), LLVM 16.0.5] x86_64-unknown-linux-gnu 3959/merge 0c83403, built 2023-10-23T16:18:52Z

```

---

- Part of #3942 
- Fixes #3901
@Wumpf Wumpf added this to the 0.11 C++ polish milestone Oct 24, 2023
@teh-cmc teh-cmc mentioned this issue Oct 25, 2023
Merged
4 tasks
teh-cmc added a commit that referenced this issue Oct 26, 2023
@teh-cmc
Spawn via $PATH 3: C and C++ implementations (#3998)
2f0fce7 
- Introduces standalone `rr_spawn(rr_spawn_options)` C API that allows
one to start a Rerun Viewer process ready to listen for TCP connections,
as well as the associated `rr_recording_stream` integration.
- Introduces standalone `rerun::spawn()` C++ API that allows one to
start a Rerun Viewer process ready to listen for TCP connections, as
well as the associated `RecordingStream` integration.



https://github.com/rerun-io/rerun/assets/2910679/8ae5003a-78b2-4e75-91d3-6dc4b8dd22ac



---

Spawn via `$PATH` series:
- #3996
- #3997
- #3998

---

- Fixes #3757 
- Fixes #3942
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@teh-cmc teh-cmc

Labels
🧑‍💻 dev experience developer experience (excluding CI) 💬 discussion ⛴ release Related to shipping or publishing
Projects
None yet
Milestone
0.11
Development

Successfully merging a pull request may close this issue.

Spawn via $PATH 3: C and C++ implementations rerun-io/rerun
2 participants
@Wumpf @teh-cmc