Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Apply-Agglomerate Performance Increase #2 #4706

Merged
merged 22 commits into from
Aug 11, 2020
Merged

Apply-Agglomerate Performance Increase #2 #4706

merged 22 commits into from
Aug 11, 2020

Conversation

youri-k
Copy link
Contributor

@youri-k youri-k commented Jul 7, 2020 •
edited
Loading

Open ToDos:

  • find fitting config values for reader block size, cache size, max reader range
    -> reader block size should be adapted to file block size, cache size & max reader range should be as large as possible
  • experiment with harder or softer locking, i.e. multi-threaded vs better caching

URL of deployed dev instance (used for testing):

  • https://___.webknossos.xyz

Steps to test:

  • try loading different agglomerates
  • cumsum optimization is used when a cumsum.json is in the agglomerates folder

Issues:

Youri K added 15 commits May 28, 2020 13:22
load blocks of 10 ids instead of one and remove unnecessary async code
5877fb4
Merge branch 'master' of github.com:scalableminds/webknossos into agg…
ddc277a 
…lo-perf
use hdf dataset for caching and synchronize cache correctly
ae7547b
log time of handling uncached keys and create new config value for bl…
f3cc589 
…ock size
Merge branch 'master' of github.com:scalableminds/webknossos into agg…
ad8c9e7 
…lo-perf
Merge branch 'master' of github.com:scalableminds/webknossos into use…
caf3f02 
…-cumsum
add cumsum support to access data in a blockwise fashion
2b3db4d
Merge branch 'master' of github.com:scalableminds/webknossos into use…
ab18623 
…-cumsum
minor refactoring
11b1679
shorten range for long cumsum ranges
7dbc00a
fix bb traversal
59f0746
add logging
9f70577
calc global cuboid for processing
ccbfcfe
use correct sorting
9ad9337
cleanup code
f01bd05
@youri-k youri-k added the backend label Jul 7, 2020
@youri-k youri-k self-assigned this Jul 7, 2020
@youri-k youri-k requested a review from fm3 July 13, 2020 12:22
@youri-k youri-k marked this pull request as ready for review July 13, 2020 12:22
fm3
fm3 requested changes Jul 13, 2020
View reviewed changes
Copy link
Member

@fm3 fm3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply the list of small refactorings we discussed in person :)

fm3
fm3 approved these changes Jul 16, 2020
View reviewed changes
Copy link
Member

@fm3 fm3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the time to work on this! 🎉

@youri-k youri-k merged commit 8eb3982 into master Aug 11, 2020
@fm3 fm3 mentioned this pull request Feb 27, 2021
Merged
1 task
@fm3 fm3 deleted the use-cumsum branch November 21, 2023 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fm3 fm3 fm3 approved these changes

Assignees

@youri-k youri-k

Labels
backend
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

agglomerate mapping too slow for large amounts of segments
2 participants
@youri-k @fm3