Fix intial agglomerate loading

[youri-k]

URL of deployed dev instance (used for testing):

• https://fixagglomeratecache.webknossos.xyz

Steps to test:

• load a new agglomerate mapping
• no black buckets should appear

Issues:

• fixes Backend reports existing buckets as missing when activating an agglomerate file mapping #4599

---

• Updated (unreleased) changelog
• [ ] Updated (unreleased) migration guide if applicable
• [ ] Updated documentation if applicable
• [ ] Adapted wk-connect if datastore API changes
• Needs datastore update after deployment
• Ready for review

[fm3]

Cool! Seems to do the trick, but could you explain a bit what was going on here? I’m not sure I get the change. Did the old code not wait for finishAccess? If so, why was that causing the bug? 🧐

[daniel-wer]

Thanks for the quick fix @youri-k 🙏

[youri-k]

@fm3 Tbh I'm not entirely sure, why the different position of the finishAccess() caused the bug. However, my theory is the following:
The first bucket with a new mapping is requested and the agglomerate file is not in cache. The agglomerate file is loading and the Future is not completed. When the Future is completed, there are other buckets that request the same future, which is now completed, so the backend responds to those immediately. When all other requests are finished, the original first requests get processed. However, when they try to read from the hdf5 file, it is already closed because the other reads were finished and the finishAccess() call was too early.

The only thing, I don't know is, why the file was closed after the last read because the cache should be large enough to retain it and not call removeElement().

[fm3]

The first bucket with a new mapping is requested and the agglomerate file is not in cache. The agglomerate file is loading and the Future is not completed. When the Future is completed, there are other buckets that request the same future, which is now completed, so the backend responds to those immediately. When all other requests are finished, the original first requests get processed. However, when they try to read from the hdf5 file, it is already closed because the other reads were finished and the finishAccess() call was too early.

Okay, that sounds about right. Good catch!

The only thing, I don't know is, why the file was closed after the last read because the cache should be large enough to retain it and not call removeElement().

Maybe we can add some logging to see if items get removed from the cache earlier than we expect? This could have performance implications. That’s independent of this PR, though.