Merge branch 'master' into mobile

* master: (43 commits) update screenshots (#4038) docker: don't set sbt/ivy cache (#4034) Slow down brush size change for small sizes (#4012) voulme tool bar now depends only on the active tool (#4029) Allow user to adapt GPU memory consumption to allow for better quality (#4015) Updates the Docs (#4020) hiding pricing and features in iframe (#4019) Fix sitemap for publication detail view (#4024) Add sitemap.xml (#4006) Allow empty trees (#4010) Prefer annotation zoom over dataset zoom (#3992) Add details view for publications (#3994) remove enzyme test (#3997) Tensorflow segmentation (#3461) ensure that max zoom step is not exceeded when changing viewport area (#3996) allow isosurfaces for hybrid tracings when setting window.allowIsosurfaces (#3998) re-fetch buckets if fetching them failed (#3999) Only re-compute bounding sphere of skeletons if something changed (#3995) Fix dropdown login padding (#3988) Persist tree visibility (#3942) ...
scalableminds · Apr 25, 2019 · f9bd0f4 · f9bd0f4
2 parents a053abb + 685355a
commit f9bd0f4
Show file tree

Hide file tree

Showing 184 changed files with 7,674 additions and 8,229 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,20 +7,55 @@ and this project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MIC
 For upgrade instructions, please check the [migration guide](MIGRATIONS.md).
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos/compare/19.03.0...HEAD)
+[Commits](https://github.com/scalableminds/webknossos/compare/19.04.0...HEAD)
+
+### Added
+- 
+
+### Changed
+- Improved performance for large tracings. [#3995](https://github.com/scalableminds/webknossos/pull/3995)
+- Improved how the rendering quality can be adapted in the settings. The setting can now be used to tune the quality to your hardware specification. [#4015](https://github.com/scalableminds/webknossos/pull/4015)
+- Empty trees in skeleton tracings are now allowed. [#4010](https://github.com/scalableminds/webknossos/pull/4010)
+
+### Fixed
+- Fixed a missing redirect after registering for an existing organization (with autoVerify=true) via the onboarding flow. [#3984](https://github.com/scalableminds/webknossos/pull/3984)
+- Fixed rendering artifacts which could occur under certain conditions. [#4015](https://github.com/scalableminds/webknossos/pull/4015)
+
+### Removed
+- 
+
+
+## [19.04.0](https://github.com/scalableminds/webknossos/releases/tag/19.04.0) - 2019-04-01
+[Commits](https://github.com/scalableminds/webknossos/compare/19.03.0...19.04.0)
+
+### Highlights
+This release multiple new interactions are expanding webKnossos:
+- Added merger mode for skeleton and hybrid tracings. It allows to merge segments from e.g. generated oversegmentations. [#3619](https://github.com/scalableminds/webknossos/pull/3619)
+- Added a shortcut (Q) and button to screenshot the tracing views. [#3834](https://github.com/scalableminds/webknossos/pull/3834)
+- Rendered isosurfaces in the 3D viewport can now be interacted with. Shift+Click on an isosurface will jump exactly to where you clicked. Also, hovering over an isosurface will highlight that cell in all viewports. [#3858](https://github.com/scalableminds/webknossos/pull/3858)
+- Neuroglancer precomputed datasets can now be added to webKnossos using the webknossos-connect service. [#3843](https://github.com/scalableminds/webknossos/pull/3843)
+
+Also the data viewing and tracing workflow is smoothed further:
+- Different loading strategies are now supported ("best quality first" and "progressive quality"). Additionally, the rendering can use different magnifications as a fallback. [#3801](https://github.com/scalableminds/webknossos/pull/3801)
+- Performance improvements :racing_car: [#3880](https://github.com/scalableminds/webknossos/pull/3880) & [#3902](https://github.com/scalableminds/webknossos/pull/3902)
 
 ### Added
 - Rendered isosurfaces in the 3D viewport can now be interacted with. Shift+Click on an isosurface will jump exactly to where you clicked. Also, hovering over an isosurface will highlight that cell in all viewports. [#3858](https://github.com/scalableminds/webknossos/pull/3858)
 - webKnossos now comes with a list of sample datasets that can be automatically downloaded and imported from the menu. [#3725](https://github.com/scalableminds/webknossos/pull/3725)
 - Added a shortcut (Q) and button in the actions dropdown to screenshot the tracing views. The screenshots will contain everything that is visible in the tracing views, so feel free to disable the crosshairs in the settings or toggle the tree visibility using the (1) and (2) shortcuts before triggering the screenshot. [#3834](https://github.com/scalableminds/webknossos/pull/3834)
 - Neuroglancer precomputed datasets can now be added to webKnossos using the webknossos-connect (wk-connect) service. To setup a wk-connect datastore follow the instructions in the [Readme](https://github.com/scalableminds/webknossos-connect). Afterwards, datasets can be added through "Add Dataset" - "Add Dataset via wk-connect". [#3843](https://github.com/scalableminds/webknossos/pull/3843)
+- Added support for mappings for 8-bit and 16-bit segmentation layers. [#3953](https://github.com/scalableminds/webknossos/pull/3953)
 - The dataset settings within the tracing view allow to select between different loading strategies now ("best quality first" and "progressive quality"). Additionally, the rendering can use different magnifications as a fallback (instead of only one magnification). [#3801](https://github.com/scalableminds/webknossos/pull/3801)
-- The mapping selection dropbown is now sorted alphabetically. [#3864](https://github.com/scalableminds/webknossos/pull/3864)
+- The mapping selection dropdown is now sorted alphabetically. [#3864](https://github.com/scalableminds/webknossos/pull/3864)
 - Added the possibility to filter datasets in the dashboard according to their availability. By default, datasets which are missing on disk (e.g., when the datastore was deleted) are not shown anymore. This behavior can be configured via the settings icon next to the search box in the dashboard. [#3883](https://github.com/scalableminds/webknossos/pull/3883)
 - Added merger mode for skeleton and hybrid tracings. It allows to merge segments from e.g. generated segmentations. [#3619](https://github.com/scalableminds/webknossos/pull/3619)
 - The HTML template now includes SEO tags for demo instances and hides internal instances from search engines.
+- Segmentation ID mappings can now be used in volume and hybrid tracings. [#3949](https://github.com/scalableminds/webknossos/pull/3949)
 - A maximize-button was added to the viewports in the annotation view. Maximization can also be toggled with the `.` shortcut. [#3876](https://github.com/scalableminds/webknossos/pull/3876)
 - [webknossos-connect](https://github.com/scalableminds/webknossos-connect) now starts with webKnossos on local and development instances by default. [#3913](https://github.com/scalableminds/webknossos/pull/3913)
+- The visibilities of trees in a skeleton tracing is now persisted across page loads. [#3942](https://github.com/scalableminds/webknossos/pull/3942)
+- Added a button for each color layer to enable/disable the layer. [#3943](https://github.com/scalableminds/webknossos/pull/3943)
+- Paginated routes now send a `X-Total-Count` HTTP header which shows how many entries were found in total. [#3899](https://github.com/scalableminds/webknossos/pull/3899)
 
 ### Changed
 - Improved the flight mode performance for tracings with very large trees (>80.000 nodes). [#3880](https://github.com/scalableminds/webknossos/pull/3880)

diff --git a/Dockerfile.dev b/Dockerfile.dev
@@ -1,4 +1,4 @@
-FROM scalableminds/sbt:master__104
+FROM scalableminds/sbt:master__113
 ARG VERSION_NODE="8.x"
 
 ENV DEBIAN_FRONTEND noninteractive

diff --git a/MIGRATIONS.md b/MIGRATIONS.md
@@ -5,13 +5,21 @@ This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 User-facing changes are documented in the [changelog](CHANGELOG.md).
 
 ## Unreleased
+- 
+
+### Postgres Evolutions:
+- 
+
+
+## [19.04.0](https://github.com/scalableminds/webknossos/releases/tag/19.04.0) - 2019-04-01
 - Redis is now needed for the tracingstore module. Make sure to install redis in your setup and adapt the config keys `tracingstore.redis.address` and `tracingstore.redis.port`.
 - To ensure that the existing behavior for loading data is preserved ("best quality first" as opposed to the new "progressive quality" default) execute: `update webknossos.user_datasetconfigurations set configuration = configuration || jsonb '{"loadingStrategy":"BEST_QUALITY_FIRST"}'`. See [#3801](https://github.com/scalableminds/webknossos/pull/3801) for additional context.
 - The config parameter `application.title` has been added. Make sure to set a title for your instance.
 - The assets URLs now include `assets/` again, if you link to assets directly, please update the paths (e.g. in postgres `organizations.logoUrl`)
 
 ### Postgres Evolutions:
 - [041-add-datastore-isconnector.sql](conf/evolutions/041-add-datastore-isconnector.sql)
+- [042-add-json-object-constraints.sql](conf/evolutions/042-add-json-object-constraints.sql)
 
 
 ## [19.03.0](https://github.com/scalableminds/webknossos/releases/tag/19.03.0) - 2019-03-04

diff --git a/app/RequestHandler.scala b/app/RequestHandler.scala
@@ -1,5 +1,5 @@
 import com.typesafe.scalalogging.LazyLogging
-import controllers.Assets
+import controllers.{Assets, SitemapController}
 import javax.inject.Inject
 import play.api.Environment
 import play.api.http.{DefaultHttpRequestHandler, HttpConfiguration, HttpErrorHandler, HttpFilters}
@@ -13,7 +13,8 @@ class RequestHandler @Inject()(router: Router,
                                filters: HttpFilters,
                                conf: WkConf,
                                assets: Assets,
-                               env: Environment)
+                               env: Environment,
+                               sitemapController: SitemapController)
     extends DefaultHttpRequestHandler(
       router,
       errorHandler,
@@ -27,8 +28,10 @@ class RequestHandler @Inject()(router: Router,
     if (request.uri.matches("^(/api/|/data/|/tracings/).*$")) {
       super.routeRequest(request)
     } else if (request.uri.matches("^(/assets/).*$")) {
-      val path = request.path.split('/').filter(_ != "assets").mkString("/")
+      val path = request.path.replaceFirst("^(/assets/)", "")
       Some(assets.at(path = "/public", file = path))
+    } else if (request.uri.matches("""^/sitemap.xml$""") && conf.Features.isDemoInstance) {
+      Some(sitemapController.getSitemap(Some(request.host)))
     } else {
       Some(Action { Ok(views.html.main(conf)) })
     }

diff --git a/app/controllers/ProjectController.scala b/app/controllers/ProjectController.scala
@@ -10,6 +10,7 @@ import net.liftweb.common.Empty
 import oxalis.security.WkEnv
 import com.mohiva.play.silhouette.api.Silhouette
 import com.mohiva.play.silhouette.api.actions.{SecuredRequest, UserAwareRequest}
+import com.scalableminds.util.tools.DefaultConverters.BoolToOption
 import play.api.i18n.{Messages, MessagesApi}
 import play.api.libs.json.Json
 import utils.ObjectId
@@ -120,16 +121,25 @@ class ProjectController @Inject()(projectService: ProjectService,
       Ok(js)
     }
 
-  def tasksForProject(projectName: String, limit: Option[Int] = None, pageNumber: Option[Int] = None) =
+  def tasksForProject(projectName: String,
+                      limit: Option[Int] = None,
+                      pageNumber: Option[Int] = None,
+                      includeTotalCount: Option[Boolean]) =
     sil.SecuredAction.async { implicit request =>
       for {
         project <- projectDAO.findOneByName(projectName) ?~> Messages("project.notFound", projectName) ~> NOT_FOUND
         _ <- Fox.assertTrue(userService.isTeamManagerOrAdminOf(request.identity, project._team)) ?~> "notAllowed" ~> FORBIDDEN
         tasks <- taskDAO.findAllByProject(project._id, limit.getOrElse(Int.MaxValue), pageNumber.getOrElse(0))(
           GlobalAccessContext)
+        taskCount <- Fox.runOptional(includeTotalCount.flatMap(BoolToOption.convert))(_ =>
+          taskDAO.countAllByProject(project._id)(GlobalAccessContext))
         js <- Fox.serialCombined(tasks)(task => taskService.publicWrites(task))
       } yield {
-        Ok(Json.toJson(js))
+        val result = Ok(Json.toJson(js))
+        taskCount match {
+          case Some(count) => result.withHeaders("X-Total-Count" -> count.toString)
+          case None        => result
+        }
       }
     }
 

diff --git a/app/controllers/SitemapController.scala b/app/controllers/SitemapController.scala
@@ -0,0 +1,23 @@
+package controllers
+
+import akka.stream.scaladsl.Source
+import com.google.inject.Inject
+import com.mohiva.play.silhouette.api.Silhouette
+import oxalis.security.WkEnv
+import play.api.libs.iteratee.streams.IterateeStreams
+import utils.SitemapWriter
+
+import scala.concurrent.ExecutionContext
+
+class SitemapController @Inject()(sitemapWriter: SitemapWriter, sil: Silhouette[WkEnv])(implicit ec: ExecutionContext)
+    extends Controller {
+
+  def getSitemap(prefix: Option[String] = None) = sil.UserAwareAction { implicit request =>
+    val downloadStream = sitemapWriter.toSitemapStream(prefix)
+
+    Ok.chunked(Source.fromPublisher(IterateeStreams.enumeratorToPublisher(downloadStream)))
+      .as("application/xml")
+      .withHeaders(CONTENT_DISPOSITION ->
+        """sitemap.xml""")
+  }
+}
diff --git a/app/controllers/UserController.scala b/app/controllers/UserController.scala
@@ -52,32 +52,49 @@ class UserController @Inject()(userService: UserService,
     }
   }
 
-  def annotations(isFinished: Option[Boolean], limit: Option[Int], pageNumber: Option[Int] = None) =
+  def annotations(isFinished: Option[Boolean],
+                  limit: Option[Int],
+                  pageNumber: Option[Int] = None,
+                  includeTotalCount: Option[Boolean] = None) =
     sil.SecuredAction.async { implicit request =>
       for {
         annotations <- annotationDAO.findAllFor(request.identity._id,
                                                 isFinished,
                                                 AnnotationType.Explorational,
                                                 limit.getOrElse(defaultAnnotationLimit),
                                                 pageNumber.getOrElse(0))
+        annotationCount <- Fox.runOptional(includeTotalCount.flatMap(BoolToOption.convert))(_ =>
+          annotationDAO.countAllFor(request.identity._id, isFinished, AnnotationType.Explorational))
         jsonList <- Fox.serialCombined(annotations)(a => annotationService.compactWrites(a))
       } yield {
-        Ok(Json.toJson(jsonList))
+        val result = Ok(Json.toJson(jsonList))
+        annotationCount match {
+          case Some(count) => result.withHeaders("X-Total-Count" -> count.toString)
+          case None        => result
+        }
       }
     }
 
-  def tasks(isFinished: Option[Boolean], limit: Option[Int], pageNumber: Option[Int] = None) = sil.SecuredAction.async {
-    implicit request =>
-      for {
-        annotations <- annotationDAO.findAllFor(request.identity._id,
-                                                isFinished,
-                                                AnnotationType.Task,
-                                                limit.getOrElse(defaultAnnotationLimit),
-                                                pageNumber.getOrElse(0))
-        jsonList <- Fox.serialCombined(annotations)(a => annotationService.publicWrites(a, Some(request.identity)))
-      } yield {
-        Ok(Json.toJson(jsonList))
+  def tasks(isFinished: Option[Boolean],
+            limit: Option[Int],
+            pageNumber: Option[Int] = None,
+            includeTotalCount: Option[Boolean] = None) = sil.SecuredAction.async { implicit request =>
+    for {
+      annotations <- annotationDAO.findAllFor(request.identity._id,
+                                              isFinished,
+                                              AnnotationType.Task,
+                                              limit.getOrElse(defaultAnnotationLimit),
+                                              pageNumber.getOrElse(0))
+      annotationCount <- Fox.runOptional(includeTotalCount.flatMap(BoolToOption.convert))(_ =>
+        annotationDAO.countAllFor(request.identity._id, isFinished, AnnotationType.Task))
+      jsonList <- Fox.serialCombined(annotations)(a => annotationService.publicWrites(a, Some(request.identity)))
+    } yield {
+      val result = Ok(Json.toJson(jsonList))
+      annotationCount match {
+        case Some(count) => result.withHeaders("X-Total-Count" -> count.toString)
+        case None        => result
       }
+    }
   }
 
   def userLoggedTime(userId: String) = sil.SecuredAction.async { implicit request =>
@@ -132,7 +149,11 @@ class UserController @Inject()(userService: UserService,
       .map(loggedTime => Ok(Json.toJson(loggedTime)))
   }
 
-  def userAnnotations(userId: String, isFinished: Option[Boolean], limit: Option[Int], pageNumber: Option[Int] = None) =
+  def userAnnotations(userId: String,
+                      isFinished: Option[Boolean],
+                      limit: Option[Int],
+                      pageNumber: Option[Int] = None,
+                      includeTotalCount: Option[Boolean] = None) =
     sil.SecuredAction.async { implicit request =>
       for {
         userIdValidated <- ObjectId.parse(userId) ?~> "user.id.invalid"
@@ -143,13 +164,23 @@ class UserController @Inject()(userService: UserService,
                                                 AnnotationType.Explorational,
                                                 limit.getOrElse(defaultAnnotationLimit),
                                                 pageNumber.getOrElse(0))
+        annotationCount <- Fox.runOptional(includeTotalCount.flatMap(BoolToOption.convert))(_ =>
+          annotationDAO.countAllFor(userIdValidated, isFinished, AnnotationType.Explorational))
         jsonList <- Fox.serialCombined(annotations)(a => annotationService.compactWrites(a))
       } yield {
-        Ok(Json.toJson(jsonList))
+        val result = Ok(Json.toJson(jsonList))
+        annotationCount match {
+          case Some(count) => result.withHeaders("X-Total-Count" -> count.toString)
+          case None        => result
+        }
       }
     }
 
-  def userTasks(userId: String, isFinished: Option[Boolean], limit: Option[Int], pageNumber: Option[Int] = None) =
+  def userTasks(userId: String,
+                isFinished: Option[Boolean],
+                limit: Option[Int],
+                pageNumber: Option[Int] = None,
+                includeTotalCount: Option[Boolean] = None) =
     sil.SecuredAction.async { implicit request =>
       for {
         userIdValidated <- ObjectId.parse(userId) ?~> "user.id.invalid"
@@ -160,9 +191,15 @@ class UserController @Inject()(userService: UserService,
                                                 AnnotationType.Task,
                                                 limit.getOrElse(defaultAnnotationLimit),
                                                 pageNumber.getOrElse(0))
+        annotationCount <- Fox.runOptional(includeTotalCount.flatMap(BoolToOption.convert))(_ =>
+          annotationDAO.countAllFor(userIdValidated, isFinished, AnnotationType.Task))
         jsonList <- Fox.serialCombined(annotations)(a => annotationService.publicWrites(a, Some(request.identity)))
       } yield {
-        Ok(Json.toJson(jsonList))
+        val result = Ok(Json.toJson(jsonList))
+        annotationCount match {
+          case Some(count) => result.withHeaders("X-Total-Count" -> count.toString)
+          case None        => result
+        }
       }
     }