Merge branch 'master' of github.com:scalableminds/webknossos into wel…

…come-toast * 'master' of github.com:scalableminds/webknossos: Log all details on deleting annotation layer (#6950) fix typo Rename demo instance to wkorg instance (#6941) Add LOD mesh support for frontend (#6909) Fix layout of view mode switcher and move it (#6949) VaultPath no longer extends nio.Path (#6942) Release 23.04.0 (#6945) Use new zip.js version to allow zip64 uploads (#6939) Implement viewing sharded neuroglancer precomputed datasets (#6920) Reject dataset uploads if organization storage quota is exceeded (#6893) Refactor deprecated antd Dropdown menus (#6898)
scalableminds · Apr 3, 2023 · 6d77a39 · 6d77a39
2 parents b1ab410 + 9cf4b0f
commit 6d77a39
Show file tree

Hide file tree

Showing 102 changed files with 2,504 additions and 2,059 deletions.
diff --git a/CHANGELOG.released.md b/CHANGELOG.released.md
@@ -7,6 +7,32 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 For upgrade instructions, please check the [migration guide](MIGRATIONS.released.md).
 
+## [23.04.0](https://github.com/scalableminds/webknossos/releases/tag/23.04.0) - 2023-03-27
+[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...23.04.0)
+
+### Highlights
+- Added email notifications for WK worker jobs. [#6918](https://github.com/scalableminds/webknossos/pull/6918)
+- Added support for viewing sharded neuroglancer precomputed datasets. [#6920](https://github.com/scalableminds/webknossos/pull/6920)
+
+### Added
+- Added support for datasets where layers are transformed individually (with an affine matrix). Transformations can be specified via datasource-properties.json or via JS API (will be ephemeral, then). [#6748](https://github.com/scalableminds/webknossos/pull/6748)
+- Added list of all respective team members to the administration page for teams. [#6915](https://github.com/scalableminds/webknossos/pull/6915)
+- Added support for uploading zip64 files. [#6939](https://github.com/scalableminds/webknossos/pull/6939)
+
+### Changed
+- Interpolation during rendering is now more performance intensive, since the rendering approach was changed. Therefore, interpolation is disabled by default. On the flip side, the rendered quality is often higher than it used to be. [#6748](https://github.com/scalableminds/webknossos/pull/6748)
+- Updated the styling of the "welcome" screen for new users to be in line with the new branding. [#6904](https://github.com/scalableminds/webknossos/pull/6904)
+- Improved Terms-of-Service modal (e.g., allow to switch organization even when modal was blocking the remaining usage of WEBKNOSSOS). [#6930](https://github.com/scalableminds/webknossos/pull/6930)
+- Uploads are now blocked when the organization’s storage quota is exceeded. [#6893](https://github.com/scalableminds/webknossos/pull/6893)
+
+### Fixed
+- Fixed an issue with text hints not being visible on the logout page for dark mode users. [#6916](https://github.com/scalableminds/webknossos/pull/6916)
+- Fixed creating task types with a selected preferred mode. [#6928](https://github.com/scalableminds/webknossos/pull/6928)
+- Fixed support for rendering of negative floats. [#6895](https://github.com/scalableminds/webknossos/pull/6895)
+- Fixed caching issues with webworkers. [#6932](https://github.com/scalableminds/webknossos/pull/6932)
+- Fixed download button for annotations which was disabled in some cases. [#6931](https://github.com/scalableminds/webknossos/pull/6931)
+- Fixed antd deprecation warning for Dropdown menus. [#6898](https://github.com/scalableminds/webknossos/pull/6898)
+
 ## [23.03.1](https://github.com/scalableminds/webknossos/releases/tag/23.03.1) - 2023-03-14
 [Commits](https://github.com/scalableminds/webknossos/compare/23.03.0...23.03.1)
 

diff --git a/CHANGELOG.unreleased.md b/CHANGELOG.unreleased.md
@@ -8,24 +8,15 @@ and this project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MIC
 For upgrade instructions, please check the [migration guide](MIGRATIONS.released.md).
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...HEAD)
+[Commits](https://github.com/scalableminds/webknossos/compare/23.04.0...HEAD)
 
 ### Added
-- Added support for datasets where layers are transformed individually (with an affine matrix). Transformations can be specified via datasource-properties.json or via JS API (will be ephemeral, then). [#6748](https://github.com/scalableminds/webknossos/pull/6748)
-- Added list of all respective team members to the administration page for teams. [#6915](https://github.com/scalableminds/webknossos/pull/6915)
-- Added email notifications for WK worker jobs. [#6918](https://github.com/scalableminds/webknossos/pull/6918)
+- Added rendering precomputed meshes with level of detail depending on the zoom of the 3D viewport. This feature only works with version 3 mesh files. [#6909](https://github.com/scalableminds/webknossos/pull/6909)
 
 ### Changed
-- Interpolation during rendering is now more performance intensive, since the rendering approach was changed. Therefore, interpolation is disabled by default. On the flip side, the rendered quality is often higher than it used to be. [#6748](https://github.com/scalableminds/webknossos/pull/6748)
-- Updated the styling of the "welcome" screen for new users to be in line with the new branding. [#6904](https://github.com/scalableminds/webknossos/pull/6904)
-- Improved Terms-of-Service modal (e.g., allow to switch organization even when modal was blocking the remaining usage of WEBKNOSSOS). [#6930](https://github.com/scalableminds/webknossos/pull/6930)
+- Moved the view mode selection in the toolbar next to the position field. [#6949](https://github.com/scalableminds/webknossos/pull/6949)
 
 ### Fixed
-- Fixed an issue with text hints not being visible on the logout page for dark mode users. [#6916](https://github.com/scalableminds/webknossos/pull/6916)
-- Fixed creating task types with a selected preferred mode. [#6928](https://github.com/scalableminds/webknossos/pull/6928)
-- Fixed support for rendering of negative floats. [#6895](https://github.com/scalableminds/webknossos/pull/6895)
-- Fixed caching issues with webworkers. [#6932](https://github.com/scalableminds/webknossos/pull/6932)
-- Fixed download button for annotations which was disabled in some cases. [#6931](https://github.com/scalableminds/webknossos/pull/6931)
 
 ### Removed
 

diff --git a/MIGRATIONS.released.md b/MIGRATIONS.released.md
@@ -1,10 +1,17 @@
 # Migration Guide (Released)
-All migrations of WEBKNOSOSS are documented in this file.
+
+All migrations of WEBKNOSSOS are documented in this file.
 See `MIGRATIONS.unreleased.md` for the changes which are not yet part of an official release.
 
 This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 
+## [23.04.0](https://github.com/scalableminds/webknossos/releases/tag/23.04.0) - 2023-03-27
+[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...23.04.0)
+
+### Postgres Evolutions:
+None.
+
 ## [23.03.1](https://github.com/scalableminds/webknossos/releases/tag/23.03.1) - 2023-03-14
 [Commits](https://github.com/scalableminds/webknossos/compare/23.03.0...23.03.1)
 - WEBKNOSSOS now requires at least Java 11 (up from Java 8). [#6869](https://github.com/scalableminds/webknossos/pull/6869)

diff --git a/MIGRATIONS.unreleased.md b/MIGRATIONS.unreleased.md
@@ -6,6 +6,8 @@ This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...HEAD)
+[Commits](https://github.com/scalableminds/webknossos/compare/23.04.0...HEAD)
+
+ - The config key features.isDemoInstance was renamed to features.isWkorgInstance (only needs to be adapted for the main wkorg instance). [#6941](https://github.com/scalableminds/webknossos/pull/6941/files)
 
 ### Postgres Evolutions:
diff --git a/app/RequestHandler.scala b/app/RequestHandler.scala
@@ -1,5 +1,5 @@
 import com.typesafe.scalalogging.LazyLogging
-import controllers.{Assets, DemoProxyController, SitemapController}
+import controllers.{Assets, WkorgProxyController, SitemapController}
 import javax.inject.Inject
 import play.api.OptionalDevContext
 import play.api.http.{DefaultHttpRequestHandler, HttpConfiguration, HttpErrorHandler, HttpFilters}
@@ -13,7 +13,7 @@ class RequestHandler @Inject()(webCommands: WebCommands,
                                router: Router,
                                errorHandler: HttpErrorHandler,
                                httpConfiguration: HttpConfiguration,
-                               demoProxyController: DemoProxyController,
+                               wkorgProxyController: WkorgProxyController,
                                filters: HttpFilters,
                                conf: WkConf,
                                assets: Assets,
@@ -35,11 +35,11 @@ class RequestHandler @Inject()(webCommands: WebCommands,
     } else if (request.uri.matches("^(/assets/).*$")) {
       val path = request.path.replaceFirst("^(/assets/)", "")
       Some(assets.at(path = "/public", file = path))
-    } else if (request.uri.matches("""^/sitemap.xml$""") && conf.Features.isDemoInstance) {
+    } else if (request.uri.matches("""^/sitemap.xml$""") && conf.Features.isWkorgInstance) {
       Some(sitemapController.getSitemap(conf.Http.uri))
-    } else if (request.uri.matches("^/sw\\.(.*)\\.js$") && conf.Features.isDemoInstance) {
+    } else if (request.uri.matches("^/sw\\.(.*)\\.js$") && conf.Features.isWkorgInstance) {
       Some(Action { Ok("").as("text/javascript") })
     } else if (request.uri == "/favicon.ico") {
       Some(Action { NotFound })
-    } else Some(demoProxyController.proxyPageOrMainView)
+    } else Some(wkorgProxyController.proxyPageOrMainView)
 }
diff --git a/app/controllers/AnnotationController.scala b/app/controllers/AnnotationController.scala
@@ -237,11 +237,12 @@ class AnnotationController @Inject()(
         _ <- bool2Fox(AnnotationType.Explorational.toString == typ) ?~> "annotation.deleteLayer.explorationalsOnly"
         annotation <- provider.provideAnnotation(typ, id, request.identity)
         _ <- bool2Fox(annotation._user == request.identity._id) ?~> "notAllowed" ~> FORBIDDEN
-        _ <- annotation.annotationLayers.find(annotationLayer => annotationLayer.name == layerName) ?~> Messages(
+        layer <- annotation.annotationLayers.find(annotationLayer => annotationLayer.name == layerName) ?~> Messages(
           "annotation.layer.notFound",
           layerName)
         _ <- bool2Fox(annotation.annotationLayers.length != 1) ?~> "annotation.deleteLayer.onlyLayer"
-        _ = logger.info(s"Deleting annotation layer $layerName for annotation $id")
+        _ = logger.info(
+          s"Deleting annotation layer $layerName (tracing id ${layer.tracingId}, typ ${layer.typ}) for annotation $id")
         _ <- annotationService.deleteAnnotationLayer(annotation, layerName)
       } yield Ok
     }

diff --git a/app/controllers/AuthenticationController.scala b/app/controllers/AuthenticationController.scala
@@ -144,7 +144,7 @@ class AuthenticationController @Inject()(
       _ <- Fox.runIf(inviteBox.isDefined)(Fox.runOptional(inviteBox.toOption)(i =>
         inviteService.deactivateUsedInvite(i)(GlobalAccessContext)))
       brainDBResult <- Fox.runIf(registerBrainDB)(brainTracing.registerIfNeeded(user, password.getOrElse("")))
-      _ = if (conf.Features.isDemoInstance) {
+      _ = if (conf.Features.isWkorgInstance) {
         mailchimpClient.registerUser(user, multiUser, tag = MailchimpTag.RegisteredAsUser)
       } else {
         Mailer ! Send(defaultMails.newUserMail(user.name, email, brainDBResult.flatten, autoActivate))
@@ -597,7 +597,7 @@ class AuthenticationController @Inject()(
                       defaultMails.newOrganizationMail(organization.displayName,
                                                        email.toLowerCase,
                                                        request.headers.get("Host").getOrElse("")))
-                    if (conf.Features.isDemoInstance) {
+                    if (conf.Features.isWkorgInstance) {
                       mailchimpClient.registerUser(user, multiUser, MailchimpTag.RegisteredAsAdmin)
                     }
                     Ok

diff --git a/app/controllers/MeshController.scala b/app/controllers/MeshController.scala
@@ -11,6 +11,9 @@ import javax.inject.Inject
 
 import scala.concurrent.ExecutionContext
 
+// Note that this wk-side controller deals with user-uploaded meshes stored in postgres
+// Not to be confused with the DSMeshController that deals with on-disk meshfiles
+
 class MeshController @Inject()(meshDAO: MeshDAO,
                                annotationDAO: AnnotationDAO,
                                sil: Silhouette[WkEnv],

diff --git a/app/controllers/WKRemoteDataStoreController.scala b/app/controllers/WKRemoteDataStoreController.scala
@@ -60,6 +60,9 @@ class WKRemoteDataStoreController @Inject()(
           organization <- organizationDAO.findOneByName(uploadInfo.organization)(GlobalAccessContext) ?~> Messages(
             "organization.notFound",
             uploadInfo.organization) ~> NOT_FOUND
+          usedStorageBytes <- organizationDAO.getUsedStorage(organization._id)
+          _ <- Fox.runOptional(organization.includedStorageBytes)(includedStorage =>
+            bool2Fox(usedStorageBytes <= includedStorage)) ?~> "dataSet.upload.storageExceeded" ~> FORBIDDEN
           _ <- bool2Fox(organization._id == user._organization) ?~> "notAllowed" ~> FORBIDDEN
           _ <- dataSetService.assertValidDataSetName(uploadInfo.name)
           _ <- dataSetService.assertNewDataSetName(uploadInfo.name, organization._id) ?~> "dataSet.name.alreadyTaken"

diff --git a/app/controllers/DemoProxyController.scala → app/controllers/WkorgProxyController.scala b/app/controllers/DemoProxyController.scala → app/controllers/WkorgProxyController.scala
@@ -14,7 +14,7 @@ import utils.WkConf
 import scala.concurrent.ExecutionContext
 import scala.util.matching.Regex
 
-class DemoProxyController @Inject()(ws: WSClient, conf: WkConf, sil: Silhouette[WkEnv], multiUserDAO: MultiUserDAO)(
+class WkorgProxyController @Inject()(ws: WSClient, conf: WkConf, sil: Silhouette[WkEnv], multiUserDAO: MultiUserDAO)(
     implicit ec: ExecutionContext)
     extends Controller {
 
@@ -30,7 +30,7 @@ class DemoProxyController @Inject()(ws: WSClient, conf: WkConf, sil: Silhouette[
   }
 
   private def matchesProxyPage(request: UserAwareRequest[WkEnv, AnyContent]): Boolean =
-    conf.Features.isDemoInstance && conf.Proxy.routes
+    conf.Features.isWkorgInstance && conf.Proxy.routes
       .exists(route => matchesPageWithWildcard(route, request.path)) && (request.identity.isEmpty || request.uri != "/")
 
   private def matchesPageWithWildcard(routeWithWildcard: String, actualRequest: String): Boolean = {

diff --git a/app/models/binary/credential/CredentialDAO.scala b/app/models/binary/credential/CredentialDAO.scala
@@ -2,7 +2,7 @@ package models.binary.credential
 
 import com.scalableminds.util.tools.Fox
 import com.scalableminds.webknossos.datastore.storage.{
-  FileSystemCredential,
+  DataVaultCredential,
   GoogleServiceAccountCredential,
   HttpBasicAuthCredential,
   S3AccessKeyCredential
@@ -79,14 +79,14 @@ class CredentialDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContex
                        values(${_id}, ${CredentialType.GoogleServiceAccount}, ${credential.name}, ${credential.secretJson.toString}, ${credential.user}, ${credential.organization})""".asUpdate)
     } yield ()
 
-  def findOne(id: ObjectId): Fox[FileSystemCredential] =
+  def findOne(id: ObjectId): Fox[DataVaultCredential] =
     for {
       r <- run(q"select $columns from webknossos.credentials_ where _id = $id".as[CredentialsRow])
       firstRow <- r.headOption.toFox
       parsed <- parseAnyCredential(firstRow)
     } yield parsed
 
-  private def parseAnyCredential(r: CredentialsRow): Fox[FileSystemCredential] =
+  private def parseAnyCredential(r: CredentialsRow): Fox[DataVaultCredential] =
     for {
       typeParsed <- CredentialType.fromString(r.`type`).toFox
       parsed <- typeParsed match {

diff --git a/app/models/binary/credential/CredentialService.scala b/app/models/binary/credential/CredentialService.scala
@@ -2,7 +2,7 @@ package models.binary.credential
 
 import com.scalableminds.util.tools.Fox
 import com.scalableminds.webknossos.datastore.storage.{
-  FileSystemCredential,
+  DataVaultCredential,
   DataVaultsHolder,
   GoogleServiceAccountCredential,
   HttpBasicAuthCredential,
@@ -22,7 +22,7 @@ class CredentialService @Inject()(credentialDAO: CredentialDAO) {
                           credentialIdentifier: Option[String],
                           credentialSecret: Option[String],
                           userId: ObjectId,
-                          organizationId: ObjectId): Option[FileSystemCredential] =
+                          organizationId: ObjectId): Option[DataVaultCredential] =
     uri.getScheme match {
       case DataVaultsHolder.schemeHttps | DataVaultsHolder.schemeHttp =>
         credentialIdentifier.map(
@@ -45,7 +45,7 @@ class CredentialService @Inject()(credentialDAO: CredentialDAO) {
         } yield GoogleServiceAccountCredential(uri.toString, secretJson, userId.toString, organizationId.toString)
     }
 
-  def insertOne(credential: FileSystemCredential)(implicit ec: ExecutionContext): Fox[ObjectId] = {
+  def insertOne(credential: DataVaultCredential)(implicit ec: ExecutionContext): Fox[ObjectId] = {
     val _id = ObjectId.generate
     for {
       _ <- credential match {

diff --git a/app/models/binary/explore/ExploreRemoteLayerService.scala b/app/models/binary/explore/ExploreRemoteLayerService.scala
@@ -10,6 +10,7 @@ import com.scalableminds.webknossos.datastore.dataformats.precomputed.{
 import com.scalableminds.webknossos.datastore.dataformats.zarr._
 import com.scalableminds.webknossos.datastore.datareaders.n5.N5Header
 import com.scalableminds.webknossos.datastore.datareaders.zarr._
+import com.scalableminds.webknossos.datastore.datavault.VaultPath
 import com.scalableminds.webknossos.datastore.models.datasource._
 import com.scalableminds.webknossos.datastore.storage.{DataVaultsHolder, RemoteSourceDescriptor}
 import com.typesafe.scalalogging.LazyLogging
@@ -21,7 +22,6 @@ import oxalis.security.WkEnv
 import play.api.libs.json.{Json, OFormat}
 
 import java.net.URI
-import java.nio.file.Path
 import javax.inject.Inject
 import scala.collection.mutable.ListBuffer
 import scala.concurrent.ExecutionContext
@@ -162,8 +162,8 @@ class ExploreRemoteLayerService @Inject()(credentialService: CredentialService)
                                                             requestingUser._id,
                                                             requestingUser._organization)
       remoteSource = RemoteSourceDescriptor(uri, credentialOpt)
-      credentialId <- Fox.runOptional(credentialOpt)(c => credentialService.insertOne(c)) ?~> "remoteFileSystem.credential.insert.failed"
-      remotePath <- DataVaultsHolder.getVaultPath(remoteSource) ?~> "remoteFileSystem.setup.failed"
+      credentialId <- Fox.runOptional(credentialOpt)(c => credentialService.insertOne(c)) ?~> "dataVault.credential.insert.failed"
+      remotePath <- DataVaultsHolder.getVaultPath(remoteSource) ?~> "dataVault.setup.failed"
       layersWithVoxelSizes <- exploreRemoteLayersForRemotePath(
         remotePath,
         credentialId.map(_.toString),
@@ -186,7 +186,7 @@ class ExploreRemoteLayerService @Inject()(credentialService: CredentialService)
     else uri
 
   private def exploreRemoteLayersForRemotePath(
-      remotePath: Path,
+      remotePath: VaultPath,
       credentialId: Option[String],
       reportMutable: ListBuffer[String],
       explorers: List[RemoteLayerExplorer])(implicit ec: ExecutionContext): Fox[List[(DataLayer, Vec3Double)]] =

diff --git a/app/models/binary/explore/N5ArrayExplorer.scala b/app/models/binary/explore/N5ArrayExplorer.scala
@@ -5,19 +5,19 @@ import com.scalableminds.webknossos.datastore.dataformats.MagLocator
 import com.scalableminds.webknossos.datastore.dataformats.n5.{N5DataLayer, N5Layer, N5SegmentationLayer}
 import com.scalableminds.webknossos.datastore.datareaders.AxisOrder
 import com.scalableminds.webknossos.datastore.datareaders.n5.N5Header
+import com.scalableminds.webknossos.datastore.datavault.VaultPath
 import com.scalableminds.webknossos.datastore.models.datasource.Category
 
-import java.nio.file.Path
 import scala.concurrent.ExecutionContext.Implicits.global
 
 class N5ArrayExplorer extends RemoteLayerExplorer {
 
   override def name: String = "N5 Array"
 
-  override def explore(remotePath: Path, credentialId: Option[String]): Fox[List[(N5Layer, Vec3Double)]] =
+  override def explore(remotePath: VaultPath, credentialId: Option[String]): Fox[List[(N5Layer, Vec3Double)]] =
     for {
-      headerPath <- Fox.successful(remotePath.resolve(N5Header.FILENAME_ATTRIBUTES_JSON))
-      name <- guessNameFromPath(remotePath)
+      headerPath <- Fox.successful(remotePath / N5Header.FILENAME_ATTRIBUTES_JSON)
+      name = guessNameFromPath(remotePath)
       n5Header <- parseJsonFromPath[N5Header](headerPath) ?~> s"failed to read n5 header at $headerPath"
       elementClass <- n5Header.elementClass ?~> "failed to read element class from n5 header"
       guessedAxisOrder = AxisOrder.asZyxFromRank(n5Header.rank)