Transactional Batch Saving

[fm3]

URL of deployed dev instance (used for testing):

• https://savebatchtransactions.webknossos.xyz

Steps to test:

• drag large NML into wK
• save, leave page before it reaches 100%
• reload page, should get old or new tracing (not something in between)
• kill backend during saving
• retry should either continue seamlessly or show conflict and the upcoming reload should give the old (consistent) tracing – (which of these two depends on the timing of the backend kill)
• the only way to still get inconsistent tracings, i think, is to kill the backend during the committing phase when it received the final request of a transaction
• when having a tracing open in two tabs, one (A) saving a large NML, and one (B) saving a small change after A has already successfully sent some requests, the small change (B) should still win (commits first) and the batched large saving process from A should fail when committing (refresh should then yield the small change from B)
• normal saving should also still work (both skeleton + volume)
• normal saving combined with a large transaction should also work

Issues:

• fixes saving skeletons: ensure that update action batches are saved transactionally #3694

TODO

• sync batch cache to disk (redis?)
• ignore duplicate requests properly again (see Silently ignore duplicate tracing save requests #3767 )
• clean up code, extract functions?
• remove redundant request id
• add config for redis address
• in health check, try redis
• redis error handling?
• update readme (redis is now required)
• discuss if progress percentage should be adapted (committing requests are significantly slower on the server) (→for now, let’s leave it as is)
• fix front-end tests
• include redis in docker files
• adapt infrastructure setup to include redis
• test health on a dev instance
• test all the scenarios
• after testing, remove (some of) the debug output

---

• Updated changelog
• Updated migration guide if applicable
• [ ] Updated documentation if applicable
• Needs datastore update after deployment
• Ready for review (backend)

[fm3]

@jstriebel could I ask you to look into adding redis to the docker files so that we can eventually use this in production? :) I’d also be grateful for a first review/look at the backend changes. I realize the code is rather complex and I’d be happy to discuss it further

@philippotto Could you have a look at the frontend tests? :)

[fm3]

@jstriebel I think all checkboxes are ticked now, would you like to go for a final review round? :)

[jstriebel]

@fm3 I had another quick look, LGTM so far, just left some minor comments 👍

In the backend code I only checked RedisTemporaryStore.scala in detail as it changed with the withExceptionHandler. Is there anything else that I should review once more in detail? And I guess somebody should also have a look at the frontend changes.

[jstriebel]

👍 everything fine from my side

[jstriebel]

👍 everything fine from my side

[daniel-wer]

Frontend looks good 👍