-
Notifications
You must be signed in to change notification settings - Fork 393
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make arrow the default storage #823
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LET'S GO
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Found one more issue:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LET'S GO (again)
With this PR we are switching our default communication and storage implementation to one based on https://arrow.apache.org/. This will make for a solid foundation going forward, making it easier to add new data types, implement the SDK in morel languages, and more.
In the short term, we have attempted to maintain near feature-parity in all logging APIs and UI components. However, as this is a ground-up rewrite of many components, there may be edge cases we have missed. If you are experiencing issues or regressions, as always, please file an issue. That said, if you are blocked, we have left an escape hatch in the form of the
RERUN_STORE
environment variable. For the time being, you can continue using the old store by setting:RERUN_STORE=classic
.We will begin deprecation and removal of the classic store in the coming weeks, so stay tuned for more updates.
Blockers
RUST_LOG=debug
spam (see below)RERUN_STORE=arrow examples/deep_sdf/main.py --save ../deep_sdf_arrow.rrd
->Tried to log to classic store when in mode Arrow
examples/nyud/main.py
acts differently, and worse, in Arrow. It creates a lot of weird 3D views.f16
np.array
and see it in the viewerexamples/stable_diffusion/main.py
panics deserializingTextEntry
Non-blocking issues
log_points
logs oneMsgId
per point. See for instancenyud
demo.examples/mp_pose/main.py
used to bring up the Annotation Context mapping (label Person
etc)Checklist
CHANGELOG.md
(if this is a big enough change to warrant it)