Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auto-Select via SAM #7051

Merged
merged 83 commits into from
May 17, 2023
Merged

Auto-Select via SAM #7051

merged 83 commits into from
May 17, 2023

Conversation

fm3
Copy link
Member

@fm3 fm3 commented May 4, 2023 •
edited by philippotto
Loading

TODO

  • Backend
    • Request data from datastore
    • Send data to SAM server
    • Forward returned data to frontend
    • Include elementClass in sam server request
    • Double-check bbox/mag combination (does the datastore interpret the bbox as mag1 or as in-mag? do we need to mag-align the bbox?)
    • Double-check what happens if the bbox is outside of the layer? Is the data black? What would be best for SAM?
    • Before merging, set segmentAnythingEnabled = false in application.conf and refresh snapshots with that
  • Frontend
    • integrate
    • add switch between old and new quick select strategy
    • support different mags
    • support other viewports
    • optimize performance
      • prefetch embedding and session
      • misc optimizations
    • fix bugs
      • incorrect strides for yz viewport
      • correct alignment of bboxes
      • invisible geometries
      • incorrect cache look up due to empty bboxes
      • bbox too large error

URL of deployed dev instance (used for testing):

Steps to test:

  • activate "quick select tool"
  • use it as usual (without activating the AI button)
  • activate the AI button and use it similar to the old tool via dnd

Issues:

  • Updated changelog
  • Updated documentation if applicable
  • Removed dev-only changes like prints and application.conf edits
    • Backend/application.conf
    • Frontend
  • Considered common edge cases
  • Needs datastore update after deployment

fm3 and others added 23 commits May 4, 2023 13:59
@fm3
load data from datastore
f069f52
@fm3
send element class to sam server
6604240
@fm3
mag handling
3df1b35
@philippotto
rough sam integration into frontend
6d985ef
@philippotto
Merge branch 'sam' of github.com:scalableminds/webknossos into sam
1ab28af
@philippotto
use embedding route from backend instead of fetching precomputed embe…
86b7466 
…dding
@philippotto
small unrelated bug fix for canceling poll under certain race condition
1992858
@philippotto
switch for dynamic vs hardcoded embedding
7d82f92
@philippotto
better error handling and disable fetching of hardcoded embedding
46bf263
@philippotto
only use user-requested bbox when applying results for better perform…
3077639 
…ance
@philippotto
request embedding for actual user position
14c40d9
@philippotto
implement caching and reuse of embedding
d3c4fe9
@philippotto
temporarily disable most CI checks
3f28c99
@philippotto
restore application.conf setting for slick and swagger; add another c…
d87c3c3 
…omment to explain impact on dev instances
@philippotto
make it work in arbitrary mags
4f1d606
@philippotto
integrate mag into embedding cache
c5d8d7a
@philippotto
show busy indicator when loading embedding
e88bc82
@philippotto
remove the entire heuristic-based quickselect code
ff85c21
@philippotto
refactor and clean up so that ML and heuristic based quick select are…
26ef93e 
… both usable
@philippotto
allow to select AI or not for quick select in UI
289af62
@philippotto
fix inverted style
8d53d74
@philippotto
only allow quick-select with SAM on wkorg
fe9b8f2
@philippotto
make sam select compatible with other viewports
bacc9fc
daniel-wer
daniel-wer approved these changes May 15, 2023
View reviewed changes
Copy link
Member

@daniel-wer daniel-wer left a comment •
edited by philippotto
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, works very well! 🎉

  • Minor: The center dot of the quick select rectangle is distorted in the YZ and XZ viewports if the scale is not uniform

frontend/javascripts/oxalis/model/sagas/quick_select_ml_saga.ts Show resolved Hide resolved
fm3 and others added 16 commits May 16, 2023 10:45
@fm3
include intensity min/max and element class in byte array sent to sam…
1eb5585 
… server
@philippotto
Merge branch 'master' of github.com:scalableminds/webknossos into sam
04b6479
@fm3
assert min/max intensity is supplied if element class is float or double
6b25b11
@fm3
fix frontend typecheck
deb3461
@philippotto
send intensity range to embedding endpoint if layer is not uint8
910495c
@philippotto
Merge branch 'sam' of github.com:scalableminds/webknossos into sam
6c1c50f
@fm3
backend error messages
a81efb1
@fm3
use 4 bytes each for min/max of intensity range, not 8
fd85947
@philippotto
only show center marker when using old quick-select mode; fix scale o…
b19c0cf 
…f center marker
@philippotto
Merge branch 'sam' of github.com:scalableminds/webknossos into sam
9990703
@philippotto
add comment
d28e105
@fm3
send metadata with correct endianness
40aec28
@fm3
Merge branch 'sam' of github.com:scalableminds/webknossos into sam
1ae7278
@philippotto
take layer name into account when caching embedding
1967637
@philippotto
Merge branch 'sam' of github.com:scalableminds/webknossos into sam
a28d084
@fm3
Merge branch 'master' into sam
0c137d1
daniel-wer
daniel-wer approved these changes May 17, 2023
View reviewed changes
Copy link
Member

@daniel-wer daniel-wer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Frontend code LGTM 👍 🎉

@philippotto philippotto enabled auto-merge (squash) May 17, 2023 09:03
@philippotto philippotto merged commit 155e94e into master May 17, 2023
@philippotto philippotto deleted the sam branch May 17, 2023 09:06
hotzenklotz added a commit that referenced this pull request May 17, 2023
@hotzenklotz
Merge branch 'master' of github.com:scalableminds/webknossos into emp…
790a5bf 
…ty-list-drawings

* 'master' of github.com:scalableminds/webknossos: (23 commits)
  fix scrolling in organization switcher in terms-of-services modal (#7083)
  Auto-Select via SAM (#7051)
  Build STL in chunks when exporting them (#7074)
  Improve performance for large (oversegmentation) meshes (#7077)
  Fix display of used storage in power plan (#7057)
  Fix user limits for invites (#7078)
  adds fileSize to voxelytics workflow list (#7071)
  Improve error logging for bucket requests (#7053)
  Fix zarr streaming datasource-properties.json generation for non-wkw/zarr datasets (#7065)
  Min length for layer names is one (#7064)
  Include voxelytics workflow name in tab title (#7070)
  Fix local to global layer index look up (#7066)
  Merge editable mappings (#7026)
  store correct artifacts for wkorg nightly (#7060)
  Team edit modal (#7043)
  Fix voxel offset in chunk name for unsharded neuroglancer precomputed datasets (#7062)
  Fix offset when loading non-aligned buckets for zarr/n5/precomputed (#7058)
  fixes wallTimes query for workflow reports (#7059)
  Handle Remote Dataset Edge Cases: compressed content-encoding, float voxel size (#7041)
  Release 23.05.2 (#7056)
  ...
hotzenklotz added a commit that referenced this pull request May 17, 2023
@hotzenklotz
Merge branch 'master' of github.com:scalableminds/webknossos into rem…
b5a788f 
…ove_wkconnect

* 'master' of github.com:scalableminds/webknossos: (33 commits)
  Fix bug in fallback rendering when layer has missing mags (#7082)
  Fix duplicate on volumes with no fallback layer (#7085)
  fix scrolling in organization switcher in terms-of-services modal (#7083)
  Auto-Select via SAM (#7051)
  Build STL in chunks when exporting them (#7074)
  Improve performance for large (oversegmentation) meshes (#7077)
  Fix display of used storage in power plan (#7057)
  Fix user limits for invites (#7078)
  adds fileSize to voxelytics workflow list (#7071)
  Improve error logging for bucket requests (#7053)
  Fix zarr streaming datasource-properties.json generation for non-wkw/zarr datasets (#7065)
  Min length for layer names is one (#7064)
  Include voxelytics workflow name in tab title (#7070)
  Fix local to global layer index look up (#7066)
  Merge editable mappings (#7026)
  store correct artifacts for wkorg nightly (#7060)
  Team edit modal (#7043)
  Fix voxel offset in chunk name for unsharded neuroglancer precomputed datasets (#7062)
  Fix offset when loading non-aligned buckets for zarr/n5/precomputed (#7058)
  fixes wallTimes query for workflow reports (#7059)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@valentin-pinkau valentin-pinkau valentin-pinkau left review comments

@philippotto philippotto philippotto left review comments

@normanrz normanrz normanrz left review comments

@daniel-wer daniel-wer daniel-wer approved these changes

Assignees

@fm3 fm3

@philippotto philippotto

@valentin-pinkau valentin-pinkau

Labels
backend frontend new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Integrate ML model for "quick segment" tool
5 participants
@fm3 @philippotto @normanrz @daniel-wer @valentin-pinkau