Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make arrow the default storage #823

Merged
merged 6 commits into from
Jan 20, 2023
Merged

Make arrow the default storage #823

merged 6 commits into from
Jan 20, 2023

Conversation

jleibs
Copy link
Member

@jleibs jleibs commented Jan 18, 2023

With this PR we are switching our default communication and storage implementation to one based on https://arrow.apache.org/. This will make for a solid foundation going forward, making it easier to add new data types, implement the SDK in morel languages, and more.

In the short term, we have attempted to maintain near feature-parity in all logging APIs and UI components. However, as this is a ground-up rewrite of many components, there may be edge cases we have missed. If you are experiencing issues or regressions, as always, please file an issue. That said, if you are blocked, we have left an escape hatch in the form of the RERUN_STORE environment variable. For the time being, you can continue using the old store by setting: RERUN_STORE=classic.

We will begin deprecation and removal of the classic store in the coming weeks, so stay tuned for more updates.

Blockers

Non-blocking issues

  • Optimize point clouds #850
  • log_points logs one MsgId per point. See for instance nyud demo.
  • Clicking the skeleton in examples/mp_pose/main.py used to bring up the Annotation Context mapping (label Person etc)

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I've added a line to CHANGELOG.md (if this is a big enough change to warrant it)

@jleibs jleibs marked this pull request as ready for review January 18, 2023 17:32
@emilk emilk self-requested a review January 18, 2023 17:35
@emilk
Copy link
Member

emilk commented Jan 19, 2023

A blocker: just py-build && RUST_LOG=debug RERUN_STORE=arrow examples/car/main.py spams infinite amounts of

Screen Shot 2023-01-19 at 12 01 50

Someone is logging something we absolutely shouldn't

@emilk emilk marked this pull request as draft January 19, 2023 11:29
@jleibs jleibs marked this pull request as ready for review January 20, 2023 16:24
Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LET'S GO

Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LET'S GO (again)

@jleibs jleibs merged commit 70600a2 into main Jan 20, 2023
@jleibs jleibs deleted the jleibs/arrow_default branch January 20, 2023 18:52
@emilk emilk mentioned this pull request Jan 23, 2023
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants