Store editable mappings in multiple fossildb columns+keys (#6903)

* WIP: Store editable mappings in multiple fossildb columns+keys

* wip: propagate new format

* agglomerate skeleton

* no longer apply updates lazily

* WIP: store segment to agglomerate

* puzzle together segment to agglomerate from chunks

* use proto for remote segment to agglomerate lookup

* WIP: update chunked segment to agglomerate

* save segment to agglomerate

* fix saving segment to agglomerate chunk

* cleanup fetching segmentToAgglomerate; count up versions

* WIP: pass buffers

* WIP: EditableMappingUpdater with buffers

* use said buffers to store updated keys before flushing after update group

* duplicate route

* WIP: move to lazy applying of updates again

* fix merge action endless loop

* rename proto file

* Add FossilDB Migration

* measure time, add more caching

* add agglomerate to graph cache

* migration perf

* More caching, jgrapht connected components, remove dead code

* cleanup

* remove debug logging

* remove time logging from migration script

* list versions instead of getMultipleVersions

* separate migrating history from rest

* pr feedback

* undo dev changes

* add args to migrate script

* cleanup logging

* migration guide

* pr feedback on comment + migration guide wording

* do not cache Fox.failures, add logging to split actions

* add segment id assertions

* more logging

* warning if edge is absent

* make actions with invalid segment ids no-ops

* log more warnings, tryo around mapping in datastore, make zero segment split/merge no-op

* remove unused import

* don't retry bucket look up on data store when there's an editable mapping

* remove dev ci skips

---------

Co-authored-by: Philipp Otto <[email protected]>

MIGRATIONS.unreleased.md

@@ -7,5 +7,7 @@ User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 
 ## Unreleased
 [Commits](https://github.com/scalableminds/webknossos/compare/23.05.0...HEAD)
+- FossilDB needs to be opened with new additional column families editableMappingsInfo, editableMappingsAgglomerateToGraph, editableMappingsSegmentToAgglomerate.
+ - For instances with existing editable mapping (a.k.a supervoxel proofreading) annotations: To keep those annotations alive, a python migration has to be run with access to your tracingstore’s FossilDB. It is recommended to do this during a webknossos downtime to avoid data loss. It needs python 3.8+ and the pip packages installable by `pip install grpcio-tools grpcio-health-checking`. Run it with `python tools/migrate-editable-mappings/migrate-editable-mappings.py -v -w -o localhost:7155`. Omit -o for a faster migration but no access to older versions of the editable mappings. The migration is idempotent.
 
 ### Postgres Evolutions:

app/models/annotation/AnnotationMutexService.scala

@@ -34,8 +34,9 @@ class AnnotationMutexService @Inject()(val lifecycle: ApplicationLifecycle,
   override protected def tickerInterval: FiniteDuration = 1 hour
 
   override protected def tick(): Unit = {
-    logger.info("Cleaning up expired annotation mutexes...")
-    annotationMutexDAO.deleteExpired()
+    for {
+      deleteCount <- annotationMutexDAO.deleteExpired()
+    } yield logger.info(s"Cleaned up $deleteCount expired annotation mutexes.")
     ()
   }
 
@@ -111,9 +112,7 @@ class AnnotationMutexDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionC
                    """.asUpdate)
     } yield ()
 
-  def deleteExpired(): Fox[Unit] =
-    for {
-      _ <- run(q"DELETE FROM webknossos.annotation_mutexes WHERE expiry < NOW()".asUpdate)
-    } yield ()
+  def deleteExpired(): Fox[Int] =
+    run(q"DELETE FROM webknossos.annotation_mutexes WHERE expiry < NOW()".asUpdate)
 
 }

conf/application.conf

@@ -175,7 +175,7 @@ datastore {
     address = "localhost"
     port = 6379
   }
-  agglomerateSkeleton.maxEdges = 100000
+  agglomerateSkeleton.maxEdges = 1000000
 }
 
 # Proxy some routes to prefix + route (only if features.isWkorgInstance, route "/" only if logged out)

conf/messages

@@ -111,7 +111,7 @@ dataVault.setup.failed=Failed to set up remote file system
 dataVault.getPath.failed=Failed to get remote path
 
 
-dataSource.notFound=Datasource not found on datastore server
+dataSource.notFound=Datasource not found on datastore server. Might still be initializing.
 
 dataStore.list.failed=Failed to retrieve list of data stores.
 dataStore.notFound=DataStore not found.

fossildb/run.sh

@@ -14,6 +14,7 @@ if [ ! -f "$JAR" ] || [ ! "$CURRENT_VERSION" == "$VERSION" ]; then
   wget -q --show-progress -O "$JAR" "$URL"
 fi
 
-COLLECTIONS="skeletons,skeletonUpdates,volumes,volumeData,volumeUpdates,editableMappings,editableMappingUpdates"
+# Note that the editableMappings column is no longer used by wk. Still here for backwards compatibility.
+COLLECTIONS="skeletons,skeletonUpdates,volumes,volumeData,volumeUpdates,editableMappings,editableMappingUpdates,editableMappingsInfo,editableMappingsAgglomerateToGraph,editableMappingsSegmentToAgglomerate"
 
 exec java -jar "$JAR" -c "$COLLECTIONS" -d "$FOSSILDB_HOME/data" -b "$FOSSILDB_HOME/backup"

fossildb/version

@@ -1 +1 @@
-0.1.19
+0.1.22

frontend/javascripts/oxalis/model/bucket_data_handling/wkstore_adapter.ts

@@ -108,13 +108,17 @@ export async function requestWithFallback(
   const requestUrl = shouldUseDataStore ? getDataStoreUrl() : getTracingStoreUrl();
   const bucketBuffers = await requestFromStore(requestUrl, layerInfo, batch, maybeVolumeTracing);
   const missingBucketIndices = getNullIndices(bucketBuffers);
-  // The request will only be retried for
-  // volume annotations with a fallback layer, for buckets that are missing
-  // on the tracing store (buckets that were not annotated)
+
+  // If buckets could not be found on the tracing store (e.g. this happens when the buckets
+  // were not annotated yet), they are instead looked up in the fallback layer
+  // on the tracing store.
+  // This retry mechanism is only active for volume tracings with fallback layers without
+  // editable mappings (aka proofreading).
   const retry =
     missingBucketIndices.length > 0 &&
     maybeVolumeTracing != null &&
-    maybeVolumeTracing.fallbackLayer != null;
+    maybeVolumeTracing.fallbackLayer != null &&
+    !maybeVolumeTracing.mappingIsEditable;
 
   if (!retry) {
     return bucketBuffers;