Implement viewing sharded neuroglancer precomputed datasets #6920
Conversation
frcroth
changed the title
…ogram sampled positions
|
I managed to improve the performance :) The main problem was that Array[Int] does not work as a cache key, because java arrays are not automatically equal if they have the same content. Sorry for suggesting it in the first place 🙈 I changed it to string again now. In the process I also increased parallelization by fetching the source chunks for a given read requests in parallel (Future.sequence rather than Fox.serialCombined) and I also noticed that the histogram sampled positions aren’t aligned with the 32 grid, so they used way more source chunks than intended. I added a commit adressing all these things. I’ll add a review for the rest of the code in the coming days :) |
| for { | ||
| bytes <- Fox.option2Fox(shardPath.readBytes(Some(shardIndexRange))) | ||
| } yield bytes |
There was a problem hiding this comment.
| for { | |
| bytes <- Fox.option2Fox(shardPath.readBytes(Some(shardIndexRange))) | |
| } yield bytes | |
| Fox.option2Fox(shardPath.readBytes(Some(shardIndexRange))) |
should be identical
| if (header.isSharded) { | ||
| for { | ||
| chunkData: Array[Byte] <- readShardedChunk(chunkIndex) | ||
| chunkShape = header.chunkSizeAtIndex(chunkIndex) | ||
| multiArray <- chunkReader.parseChunk(chunkData, chunkShape) | ||
| } yield multiArray | ||
| } else { | ||
| val chunkFilename = getChunkFilename(chunkIndex) | ||
| val chunkFilePath = relativePath.resolve(chunkFilename) | ||
| val storeKey = chunkFilePath.storeKey | ||
| val chunkShape = header.chunkSizeAtIndex(chunkIndex) | ||
| chunkReader.read(storeKey, chunkShape) |
There was a problem hiding this comment.
I’m wondering whether we can get the two branches here into one. As I understand, currently, both branches have their own version of reading from the store, decompressing, then typing. Could it be unified? Maybe the sharding implementation could just return the chunk path plus byte range to be passed to the existing chunkReader? (With non-sharding returning None for the range)
…come-toast * 'master' of github.com:scalableminds/webknossos: Log all details on deleting annotation layer (#6950) fix typo Rename demo instance to wkorg instance (#6941) Add LOD mesh support for frontend (#6909) Fix layout of view mode switcher and move it (#6949) VaultPath no longer extends nio.Path (#6942) Release 23.04.0 (#6945) Use new zip.js version to allow zip64 uploads (#6939) Implement viewing sharded neuroglancer precomputed datasets (#6920) Reject dataset uploads if organization storage quota is exceeded (#6893) Refactor deprecated antd Dropdown menus (#6898)
…wings * 'master' of github.com:scalableminds/webknossos: updates docs for docker installation (#6963) Fix misc stuff when viewing tasks/annotations of another user (#6957) Remove segment from list and add undo/redo for segments (#6944) Log all details on deleting annotation layer (#6950) fix typo Rename demo instance to wkorg instance (#6941) Add LOD mesh support for frontend (#6909) Fix layout of view mode switcher and move it (#6949) VaultPath no longer extends nio.Path (#6942) Release 23.04.0 (#6945) Use new zip.js version to allow zip64 uploads (#6939) Implement viewing sharded neuroglancer precomputed datasets (#6920) Reject dataset uploads if organization storage quota is exceeded (#6893) Refactor deprecated antd Dropdown menus (#6898)
URL of deployed dev instance (used for testing):
Steps to test:
Issues:
(Please delete unneeded items, merge only when none are left open)