Merge branch 'master' of github.com:scalableminds/webknossos into drawings

* 'master' of github.com:scalableminds/webknossos:
  updates docs for docker installation (#6963)
  Fix misc stuff when viewing tasks/annotations of another user (#6957)
  Remove segment from list and add undo/redo for segments (#6944)
  Log all details on deleting annotation layer (#6950)
  fix typo
  Rename demo instance to wkorg instance (#6941)
  Add LOD mesh support for frontend (#6909)
  Fix layout of view mode switcher and move it (#6949)
  VaultPath no longer extends nio.Path (#6942)
  Release 23.04.0 (#6945)
  Use new zip.js version to allow zip64 uploads (#6939)
  Implement viewing sharded neuroglancer precomputed datasets (#6920)
  Reject dataset uploads if organization storage quota is exceeded (#6893)
  Refactor deprecated antd Dropdown menus (#6898)

Author: hotzenklotz

CHANGELOG.released.md

@@ -7,6 +7,32 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 For upgrade instructions, please check the [migration guide](MIGRATIONS.released.md).
 
+## [23.04.0](https://github.com/scalableminds/webknossos/releases/tag/23.04.0) - 2023-03-27
+[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...23.04.0)
+
+### Highlights
+- Added email notifications for WK worker jobs. [#6918](https://github.com/scalableminds/webknossos/pull/6918)
+- Added support for viewing sharded neuroglancer precomputed datasets. [#6920](https://github.com/scalableminds/webknossos/pull/6920)
+
+### Added
+- Added support for datasets where layers are transformed individually (with an affine matrix). Transformations can be specified via datasource-properties.json or via JS API (will be ephemeral, then). [#6748](https://github.com/scalableminds/webknossos/pull/6748)
+- Added list of all respective team members to the administration page for teams. [#6915](https://github.com/scalableminds/webknossos/pull/6915)
+- Added support for uploading zip64 files. [#6939](https://github.com/scalableminds/webknossos/pull/6939)
+
+### Changed
+- Interpolation during rendering is now more performance intensive, since the rendering approach was changed. Therefore, interpolation is disabled by default. On the flip side, the rendered quality is often higher than it used to be. [#6748](https://github.com/scalableminds/webknossos/pull/6748)
+- Updated the styling of the "welcome" screen for new users to be in line with the new branding. [#6904](https://github.com/scalableminds/webknossos/pull/6904)
+- Improved Terms-of-Service modal (e.g., allow to switch organization even when modal was blocking the remaining usage of WEBKNOSSOS). [#6930](https://github.com/scalableminds/webknossos/pull/6930)
+- Uploads are now blocked when the organization’s storage quota is exceeded. [#6893](https://github.com/scalableminds/webknossos/pull/6893)
+
+### Fixed
+- Fixed an issue with text hints not being visible on the logout page for dark mode users. [#6916](https://github.com/scalableminds/webknossos/pull/6916)
+- Fixed creating task types with a selected preferred mode. [#6928](https://github.com/scalableminds/webknossos/pull/6928)
+- Fixed support for rendering of negative floats. [#6895](https://github.com/scalableminds/webknossos/pull/6895)
+- Fixed caching issues with webworkers. [#6932](https://github.com/scalableminds/webknossos/pull/6932)
+- Fixed download button for annotations which was disabled in some cases. [#6931](https://github.com/scalableminds/webknossos/pull/6931)
+- Fixed antd deprecation warning for Dropdown menus. [#6898](https://github.com/scalableminds/webknossos/pull/6898)
+
 ## [23.03.1](https://github.com/scalableminds/webknossos/releases/tag/23.03.1) - 2023-03-14
 [Commits](https://github.com/scalableminds/webknossos/compare/23.03.0...23.03.1)
 

CHANGELOG.unreleased.md

@@ -8,25 +8,20 @@ and this project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MIC
 For upgrade instructions, please check the [migration guide](MIGRATIONS.released.md).
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...HEAD)
+[Commits](https://github.com/scalableminds/webknossos/compare/23.04.0...HEAD)
 
 ### Added
-- Added support for datasets where layers are transformed individually (with an affine matrix). Transformations can be specified via datasource-properties.json or via JS API (will be ephemeral, then). [#6748](https://github.com/scalableminds/webknossos/pull/6748)
-- Added list of all respective team members to the administration page for teams. [#6915](https://github.com/scalableminds/webknossos/pull/6915)
-- Added email notifications for WK worker jobs. [#6918](https://github.com/scalableminds/webknossos/pull/6918)
+- Added rendering precomputed meshes with level of detail depending on the zoom of the 3D viewport. This feature only works with version 3 mesh files. [#6909](https://github.com/scalableminds/webknossos/pull/6909)
+- Segments can now be removed from the segment list via the context menu. [#6944](https://github.com/scalableminds/webknossos/pull/6944)
+- Editing the meta data of segments (e.g., the name) is now undoable. [#6944](https://github.com/scalableminds/webknossos/pull/6944)
 - Added more icons and small redesigns for various pages in line with the new branding. [#6938](https://github.com/scalableminds/webknossos/pull/6938)
 
+
 ### Changed
-- Interpolation during rendering is now more performance intensive, since the rendering approach was changed. Therefore, interpolation is disabled by default. On the flip side, the rendered quality is often higher than it used to be. [#6748](https://github.com/scalableminds/webknossos/pull/6748)
-- Updated the styling of the "welcome" screen for new users to be in line with the new branding. [#6904](https://github.com/scalableminds/webknossos/pull/6904)
-- Improved Terms-of-Service modal (e.g., allow to switch organization even when modal was blocking the remaining usage of WEBKNOSSOS). [#6930](https://github.com/scalableminds/webknossos/pull/6930)
+- Moved the view mode selection in the toolbar next to the position field. [#6949](https://github.com/scalableminds/webknossos/pull/6949)
 
 ### Fixed
-- Fixed an issue with text hints not being visible on the logout page for dark mode users. [#6916](https://github.com/scalableminds/webknossos/pull/6916)
-- Fixed creating task types with a selected preferred mode. [#6928](https://github.com/scalableminds/webknossos/pull/6928)
-- Fixed support for rendering of negative floats. [#6895](https://github.com/scalableminds/webknossos/pull/6895)
-- Fixed caching issues with webworkers. [#6932](https://github.com/scalableminds/webknossos/pull/6932)
-- Fixed download button for annotations which was disabled in some cases. [#6931](https://github.com/scalableminds/webknossos/pull/6931)
+- Fixed incorrect initial tab when clicking "Show Annotations" for a user in the user list. Also, the datasets tab was removed from that page as it was the same as the datasets table from the main dashboard. [#6957](https://github.com/scalableminds/webknossos/pull/6957)
 
 ### Removed
 

MIGRATIONS.released.md

@@ -1,10 +1,17 @@
 # Migration Guide (Released)
-All migrations of WEBKNOSOSS are documented in this file.
+
+All migrations of WEBKNOSSOS are documented in this file.
 See `MIGRATIONS.unreleased.md` for the changes which are not yet part of an official release.
 
 This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 
+## [23.04.0](https://github.com/scalableminds/webknossos/releases/tag/23.04.0) - 2023-03-27
+[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...23.04.0)
+
+### Postgres Evolutions:
+None.
+
 ## [23.03.1](https://github.com/scalableminds/webknossos/releases/tag/23.03.1) - 2023-03-14
 [Commits](https://github.com/scalableminds/webknossos/compare/23.03.0...23.03.1)
 - WEBKNOSSOS now requires at least Java 11 (up from Java 8). [#6869](https://github.com/scalableminds/webknossos/pull/6869)

MIGRATIONS.unreleased.md

@@ -6,6 +6,8 @@ This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...HEAD)
+[Commits](https://github.com/scalableminds/webknossos/compare/23.04.0...HEAD)
+
+ - The config key features.isDemoInstance was renamed to features.isWkorgInstance (only needs to be adapted for the main wkorg instance). [#6941](https://github.com/scalableminds/webknossos/pull/6941/files)
 
 ### Postgres Evolutions:

app/RequestHandler.scala

@@ -1,5 +1,5 @@
 import com.typesafe.scalalogging.LazyLogging
-import controllers.{Assets, DemoProxyController, SitemapController}
+import controllers.{Assets, WkorgProxyController, SitemapController}
 import javax.inject.Inject
 import play.api.OptionalDevContext
 import play.api.http.{DefaultHttpRequestHandler, HttpConfiguration, HttpErrorHandler, HttpFilters}
@@ -13,7 +13,7 @@ class RequestHandler @Inject()(webCommands: WebCommands,
                                router: Router,
                                errorHandler: HttpErrorHandler,
                                httpConfiguration: HttpConfiguration,
-                               demoProxyController: DemoProxyController,
+                               wkorgProxyController: WkorgProxyController,
                                filters: HttpFilters,
                                conf: WkConf,
                                assets: Assets,
@@ -35,11 +35,11 @@ class RequestHandler @Inject()(webCommands: WebCommands,
     } else if (request.uri.matches("^(/assets/).*$")) {
       val path = request.path.replaceFirst("^(/assets/)", "")
       Some(assets.at(path = "/public", file = path))
-    } else if (request.uri.matches("""^/sitemap.xml$""") && conf.Features.isDemoInstance) {
+    } else if (request.uri.matches("""^/sitemap.xml$""") && conf.Features.isWkorgInstance) {
       Some(sitemapController.getSitemap(conf.Http.uri))
-    } else if (request.uri.matches("^/sw\\.(.*)\\.js$") && conf.Features.isDemoInstance) {
+    } else if (request.uri.matches("^/sw\\.(.*)\\.js$") && conf.Features.isWkorgInstance) {
       Some(Action { Ok("").as("text/javascript") })
     } else if (request.uri == "/favicon.ico") {
       Some(Action { NotFound })
-    } else Some(demoProxyController.proxyPageOrMainView)
+    } else Some(wkorgProxyController.proxyPageOrMainView)
 }

app/controllers/AnnotationController.scala

@@ -237,11 +237,12 @@ class AnnotationController @Inject()(
         _ <- bool2Fox(AnnotationType.Explorational.toString == typ) ?~> "annotation.deleteLayer.explorationalsOnly"
         annotation <- provider.provideAnnotation(typ, id, request.identity)
         _ <- bool2Fox(annotation._user == request.identity._id) ?~> "notAllowed" ~> FORBIDDEN
-        _ <- annotation.annotationLayers.find(annotationLayer => annotationLayer.name == layerName) ?~> Messages(
+        layer <- annotation.annotationLayers.find(annotationLayer => annotationLayer.name == layerName) ?~> Messages(
           "annotation.layer.notFound",
           layerName)
         _ <- bool2Fox(annotation.annotationLayers.length != 1) ?~> "annotation.deleteLayer.onlyLayer"
-        _ = logger.info(s"Deleting annotation layer $layerName for annotation $id")
+        _ = logger.info(
+          s"Deleting annotation layer $layerName (tracing id ${layer.tracingId}, typ ${layer.typ}) for annotation $id")
         _ <- annotationService.deleteAnnotationLayer(annotation, layerName)
       } yield Ok
     }

app/controllers/AuthenticationController.scala

@@ -144,7 +144,7 @@ class AuthenticationController @Inject()(
       _ <- Fox.runIf(inviteBox.isDefined)(Fox.runOptional(inviteBox.toOption)(i =>
         inviteService.deactivateUsedInvite(i)(GlobalAccessContext)))
       brainDBResult <- Fox.runIf(registerBrainDB)(brainTracing.registerIfNeeded(user, password.getOrElse("")))
-      _ = if (conf.Features.isDemoInstance) {
+      _ = if (conf.Features.isWkorgInstance) {
         mailchimpClient.registerUser(user, multiUser, tag = MailchimpTag.RegisteredAsUser)
       } else {
         Mailer ! Send(defaultMails.newUserMail(user.name, email, brainDBResult.flatten, autoActivate))
@@ -597,7 +597,7 @@ class AuthenticationController @Inject()(
                       defaultMails.newOrganizationMail(organization.displayName,
                                                        email.toLowerCase,
                                                        request.headers.get("Host").getOrElse("")))
-                    if (conf.Features.isDemoInstance) {
+                    if (conf.Features.isWkorgInstance) {
                       mailchimpClient.registerUser(user, multiUser, MailchimpTag.RegisteredAsAdmin)
                     }
                     Ok

app/controllers/MeshController.scala

@@ -11,6 +11,9 @@ import javax.inject.Inject
 
 import scala.concurrent.ExecutionContext
 
+// Note that this wk-side controller deals with user-uploaded meshes stored in postgres
+// Not to be confused with the DSMeshController that deals with on-disk meshfiles
+
 class MeshController @Inject()(meshDAO: MeshDAO,
                                annotationDAO: AnnotationDAO,
                                sil: Silhouette[WkEnv],

app/controllers/WKRemoteDataStoreController.scala

@@ -60,6 +60,9 @@ class WKRemoteDataStoreController @Inject()(
           organization <- organizationDAO.findOneByName(uploadInfo.organization)(GlobalAccessContext) ?~> Messages(
             "organization.notFound",
             uploadInfo.organization) ~> NOT_FOUND
+          usedStorageBytes <- organizationDAO.getUsedStorage(organization._id)
+          _ <- Fox.runOptional(organization.includedStorageBytes)(includedStorage =>
+            bool2Fox(usedStorageBytes <= includedStorage)) ?~> "dataSet.upload.storageExceeded" ~> FORBIDDEN
           _ <- bool2Fox(organization._id == user._organization) ?~> "notAllowed" ~> FORBIDDEN
           _ <- dataSetService.assertValidDataSetName(uploadInfo.name)
           _ <- dataSetService.assertNewDataSetName(uploadInfo.name, organization._id) ?~> "dataSet.name.alreadyTaken"

app/controllers/DemoProxyController.scala renamed to app/controllers/WkorgProxyController.scala

@@ -14,7 +14,7 @@ import utils.WkConf
 import scala.concurrent.ExecutionContext
 import scala.util.matching.Regex
 
-class DemoProxyController @Inject()(ws: WSClient, conf: WkConf, sil: Silhouette[WkEnv], multiUserDAO: MultiUserDAO)(
+class WkorgProxyController @Inject()(ws: WSClient, conf: WkConf, sil: Silhouette[WkEnv], multiUserDAO: MultiUserDAO)(
     implicit ec: ExecutionContext)
     extends Controller {
 
@@ -30,7 +30,7 @@ class DemoProxyController @Inject()(ws: WSClient, conf: WkConf, sil: Silhouette[
   }
 
   private def matchesProxyPage(request: UserAwareRequest[WkEnv, AnyContent]): Boolean =
-    conf.Features.isDemoInstance && conf.Proxy.routes
+    conf.Features.isWkorgInstance && conf.Proxy.routes
       .exists(route => matchesPageWithWildcard(route, request.path)) && (request.identity.isEmpty || request.uri != "/")
 
   private def matchesPageWithWildcard(routeWithWildcard: String, actualRequest: String): Boolean = {

app/models/binary/credential/CredentialDAO.scala

@@ -2,7 +2,7 @@ package models.binary.credential
 
 import com.scalableminds.util.tools.Fox
 import com.scalableminds.webknossos.datastore.storage.{
-  FileSystemCredential,
+  DataVaultCredential,
   GoogleServiceAccountCredential,
   HttpBasicAuthCredential,
   S3AccessKeyCredential
@@ -79,14 +79,14 @@ class CredentialDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContex
                        values(${_id}, ${CredentialType.GoogleServiceAccount}, ${credential.name}, ${credential.secretJson.toString}, ${credential.user}, ${credential.organization})""".asUpdate)
     } yield ()
 
-  def findOne(id: ObjectId): Fox[FileSystemCredential] =
+  def findOne(id: ObjectId): Fox[DataVaultCredential] =
     for {
       r <- run(q"select $columns from webknossos.credentials_ where _id = $id".as[CredentialsRow])
       firstRow <- r.headOption.toFox
       parsed <- parseAnyCredential(firstRow)
     } yield parsed
 
-  private def parseAnyCredential(r: CredentialsRow): Fox[FileSystemCredential] =
+  private def parseAnyCredential(r: CredentialsRow): Fox[DataVaultCredential] =
     for {
       typeParsed <- CredentialType.fromString(r.`type`).toFox
       parsed <- typeParsed match {

app/models/binary/credential/CredentialService.scala

@@ -2,7 +2,7 @@ package models.binary.credential
 
 import com.scalableminds.util.tools.Fox
 import com.scalableminds.webknossos.datastore.storage.{
-  FileSystemCredential,
+  DataVaultCredential,
   DataVaultsHolder,
   GoogleServiceAccountCredential,
   HttpBasicAuthCredential,
@@ -22,7 +22,7 @@ class CredentialService @Inject()(credentialDAO: CredentialDAO) {
                           credentialIdentifier: Option[String],
                           credentialSecret: Option[String],
                           userId: ObjectId,
-                          organizationId: ObjectId): Option[FileSystemCredential] =
+                          organizationId: ObjectId): Option[DataVaultCredential] =
     uri.getScheme match {
       case DataVaultsHolder.schemeHttps | DataVaultsHolder.schemeHttp =>
         credentialIdentifier.map(
@@ -45,7 +45,7 @@ class CredentialService @Inject()(credentialDAO: CredentialDAO) {
         } yield GoogleServiceAccountCredential(uri.toString, secretJson, userId.toString, organizationId.toString)
     }
 
-  def insertOne(credential: FileSystemCredential)(implicit ec: ExecutionContext): Fox[ObjectId] = {
+  def insertOne(credential: DataVaultCredential)(implicit ec: ExecutionContext): Fox[ObjectId] = {
     val _id = ObjectId.generate
     for {
       _ <- credential match {

app/models/binary/explore/ExploreRemoteLayerService.scala

@@ -10,6 +10,7 @@ import com.scalableminds.webknossos.datastore.dataformats.precomputed.{
 import com.scalableminds.webknossos.datastore.dataformats.zarr._
 import com.scalableminds.webknossos.datastore.datareaders.n5.N5Header
 import com.scalableminds.webknossos.datastore.datareaders.zarr._
+import com.scalableminds.webknossos.datastore.datavault.VaultPath
 import com.scalableminds.webknossos.datastore.models.datasource._
 import com.scalableminds.webknossos.datastore.storage.{DataVaultsHolder, RemoteSourceDescriptor}
 import com.typesafe.scalalogging.LazyLogging
@@ -21,7 +22,6 @@ import oxalis.security.WkEnv
 import play.api.libs.json.{Json, OFormat}
 
 import java.net.URI
-import java.nio.file.Path
 import javax.inject.Inject
 import scala.collection.mutable.ListBuffer
 import scala.concurrent.ExecutionContext
@@ -162,8 +162,8 @@ class ExploreRemoteLayerService @Inject()(credentialService: CredentialService)
                                                             requestingUser._id,
                                                             requestingUser._organization)
       remoteSource = RemoteSourceDescriptor(uri, credentialOpt)
-      credentialId <- Fox.runOptional(credentialOpt)(c => credentialService.insertOne(c)) ?~> "remoteFileSystem.credential.insert.failed"
-      remotePath <- DataVaultsHolder.getVaultPath(remoteSource) ?~> "remoteFileSystem.setup.failed"
+      credentialId <- Fox.runOptional(credentialOpt)(c => credentialService.insertOne(c)) ?~> "dataVault.credential.insert.failed"
+      remotePath <- DataVaultsHolder.getVaultPath(remoteSource) ?~> "dataVault.setup.failed"
       layersWithVoxelSizes <- exploreRemoteLayersForRemotePath(
         remotePath,
         credentialId.map(_.toString),
@@ -186,7 +186,7 @@ class ExploreRemoteLayerService @Inject()(credentialService: CredentialService)
     else uri
 
   private def exploreRemoteLayersForRemotePath(
-      remotePath: Path,
+      remotePath: VaultPath,
       credentialId: Option[String],
       reportMutable: ListBuffer[String],
       explorers: List[RemoteLayerExplorer])(implicit ec: ExecutionContext): Fox[List[(DataLayer, Vec3Double)]] =

app/models/binary/explore/N5ArrayExplorer.scala

@@ -5,19 +5,19 @@ import com.scalableminds.webknossos.datastore.dataformats.MagLocator
 import com.scalableminds.webknossos.datastore.dataformats.n5.{N5DataLayer, N5Layer, N5SegmentationLayer}
 import com.scalableminds.webknossos.datastore.datareaders.AxisOrder
 import com.scalableminds.webknossos.datastore.datareaders.n5.N5Header
+import com.scalableminds.webknossos.datastore.datavault.VaultPath
 import com.scalableminds.webknossos.datastore.models.datasource.Category
 
-import java.nio.file.Path
 import scala.concurrent.ExecutionContext.Implicits.global
 
 class N5ArrayExplorer extends RemoteLayerExplorer {
 
   override def name: String = "N5 Array"
 
-  override def explore(remotePath: Path, credentialId: Option[String]): Fox[List[(N5Layer, Vec3Double)]] =
+  override def explore(remotePath: VaultPath, credentialId: Option[String]): Fox[List[(N5Layer, Vec3Double)]] =
     for {
-      headerPath <- Fox.successful(remotePath.resolve(N5Header.FILENAME_ATTRIBUTES_JSON))
-      name <- guessNameFromPath(remotePath)
+      headerPath <- Fox.successful(remotePath / N5Header.FILENAME_ATTRIBUTES_JSON)
+      name = guessNameFromPath(remotePath)
       n5Header <- parseJsonFromPath[N5Header](headerPath) ?~> s"failed to read n5 header at $headerPath"
       elementClass <- n5Header.elementClass ?~> "failed to read element class from n5 header"
       guessedAxisOrder = AxisOrder.asZyxFromRank(n5Header.rank)