Clean up config (#5208)

* [WIP] clean up config

* [WIP] rename more keys

* [WIP] move more config keys

* [WIP adapt to new conf

* adapt remaining places to new conf

* include module info in buildinfo

* ID to Id

* prepare for master merge

* slackNotification.uri

* re-introduced http.port. warn if old keys are present. migration guide

* remove tracingstore pingIntervalMinutes

* add missing close brace

CHANGELOG.unreleased.md

@@ -23,6 +23,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Meshes that are imported by the user in the meshes tab are now rendered the same way as generated isosurface meshes. [#5326](https://github.com/scalableminds/webknossos/pull/5326)
 - In the new REST API version 4, projects are no longer referenced by name, but instead by id. [#5334](https://github.com/scalableminds/webknossos/pull/5334)
 - The layout of the tracing view was revamped. Most notably, the layout now has two well-behaved sidebars (left and right) which can be collapsed and expanded while the remaining space is used for the main data view. Additionally, a status bar was added which shows important information, such as the currently rendered magnification and useful mouse controls. [#5279](https://github.com/scalableminds/webknossos/pull/5279)
+- The deployment configuration of webKnossos was cleaned up. If you host your own webKnossos instance, be sure to update your config according to the migration guide. [#5208](https://github.com/scalableminds/webknossos/pull/5208)
 
 ### Fixed
 - Fixed a bug where some values in the project list were displayed incorrectly after pausing/unpausing the project. [#5339](https://github.com/scalableminds/webknossos/pull/5339)

MIGRATIONS.unreleased.md

@@ -6,7 +6,57 @@ This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
 User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 
 ## Unreleased
-- 
+- The config keys in application.conf were restructured. If you overwrite any of them for your config, please adapt to the new structure, according to the table below. If you run any stand-alone datastores or tracingstores, make sure to update their config files as well.
+
+old key | new key | notes
+--------|--------|-------
+`http.address` | removed | used by play, default is 0.0.0.0, you can still overwrite it if necessary
+`actor.defaultTimeout` | removed | was already unused
+`js.defaultTimeout` | removed | was already unused
+`akka.loggers` | removed | was already unused
+`application.name` | removed | was already unused
+`application.branch` | removed | was already unused
+`application.version` | removed | was already unused
+`application.title` | `webKnossos.tabTitle` |
+`application.insertInitialData` | `webKnossos.sampleOrganization.enabled` |
+`application.insertLocalConnectDatastore` | removed | feature removed, insert manually instead
+`application.authentication.defaultuser.email` | `webKnossos.sampleOrganization.user.email` |
+`application.authentication.defaultUser.password` | `webKnossos.sampleOrganization.user.password` |
+`application.authentication.defaultUser.token` | `webKnossos.sampleOrganization.user.token` |
+`application.authentication.defaultUser.isSuperUser` | `webKnossos.sampleOrganization.user.isSuperUser` |
+`application.authentication.ssoKey` | `webKnossos.user.ssoKey` |
+`application.authentication.inviteExpiry` | `webKnossos.user.inviteExpiry` |
+`webKnossos.user.time.tracingPauseInSeconds` | `webKnossos.user.time.tracingPause` | **type changed from Int to FiniteDuration, add ` seconds`**
+`webKnossos.query.maxResults` | removed | was already unused
+`user.cacheTimeoutInMinutes` | `webKnossos.cache.user.timeout` | **type changed from Int to FiniteDuration, add ` minutes`**
+`tracingstore.enabled` | removed | info contained in `play.modules.enabled`
+`datastore.enabled` | removed | info contained in `play.modules.enabled`
+`datastore.webKnossos.pingIntervalMinutes` | `datastore.webKnossos.pingInterval` | **type changed from Int to FiniteDuration, add ` minutes`**
+`braingames.binary.cacheMaxSize` | `datastore.cache.dataCube.maxEntries` |
+`braingames.binary.mappingCacheMaxSize` | `datastore.cache.mapping.maxEntries` |
+`braingames.binary.agglomerateFileCacheMaxSize` | `datastore.cache.agglomerateFile.maxFileHandleEntries` |
+`braingames.binary.agglomerateCacheMaxSize` | `datastore.cache.agglomerateFile.maxSegmentIdEntries` |
+`braingames.binary.agglomerateStandardBlockSize` | `datastore.cache.agglomerateFile.blockSize` |
+`braingames.binary.agglomerateMaxReaderRange` | `datastore.cache.agglomerateFile.cumsumMaxReaderRange` |
+`braingames.binary.loadTimeout` | removed | was already unused
+`braingames.binary.saveTimeout` | removed | was already unused
+`braingames.binary.isosurfaceTimeout` | `datastore.isosurface.timeout` |  **type changed from Int to FiniteDuration, add ` seconds`**
+`braingames.binary.isosurfaceActorPoolSize` | `datastore.isosurface.actorPoolSize` |
+`braingames.binary.baseFolder` | `datastore.baseFolder`
+`braingames.binary.agglomerateSkeletonEdgeLimit` | `datastore.agglomerateSkeleton.maxEdges`
+`braingames.binary.changeHandler.enabled` | `datastore.watchFileSystem.enabled`
+`braingames.binary.tickerInterval` | `datastore.watchFileSystem.interval` |  **type changed from Int to FiniteDuration, add ` minutes`**
+`mail.enabled` | removed | now enabled if `mail.host` is non-empty
+`jobs.username` | `jobs.user` |
+`braintracing.active` | `braintracing.enabled`
+`braintracing.url` | `braintracing.uri`
+`airbrake.apiKey` | removed | was already unused
+`airbrake.ssl` | removed | was already unused
+`airbrake.enabled` | removed | was already unused
+`airbrake.endpoint` | removed | was already unused
+`slackNotifications.url` | `slackNotifications.uri` |
+`google.analytics.trackingId` | `googleAnalytics.trackingId` |
+`operatorData` | `webKnossos.operatorData`
 
 ### Postgres Evolutions:
 - [070-dark-theme.sql](conf/evolutions/070-dark-theme.sql)

app/Startup.scala

@@ -33,6 +33,9 @@ class Startup @Inject()(actorSystem: ActorSystem,
     extends LazyLogging {
 
   logger.info("Executing Startup")
+
+  conf.warnIfOldKeysPresent()
+
   startActors(actorSystem)
 
   private val tokenAuthenticatorService = wkSilhouetteEnvironment.combinedAuthenticatorService.tokenAuthenticatorService
@@ -101,7 +104,6 @@ class Startup @Inject()(actorSystem: ActorSystem,
 
   private def startActors(actorSystem: ActorSystem) {
     val mailerConf = MailerConfig(
-      conf.Mail.enabled,
       conf.Mail.logToStdout,
       conf.Mail.Smtp.host,
       conf.Mail.Smtp.port,

app/controllers/Application.scala

@@ -13,14 +13,15 @@ import oxalis.security.WkEnv
 import play.api.libs.json.{JsObject, Json}
 import play.api.mvc.{Action, AnyContent, PlayBodyParsers}
 import slick.jdbc.PostgresProfile.api._
-import utils.{SQLClient, SimpleSQLDAO, WkConf}
+import utils.{SQLClient, SimpleSQLDAO, StoreModules, WkConf}
 
 import scala.concurrent.ExecutionContext
 
 class Application @Inject()(multiUserDAO: MultiUserDAO,
                             analyticsService: AnalyticsService,
                             releaseInformationDAO: ReleaseInformationDAO,
                             conf: WkConf,
+                            storeModules: StoreModules,
                             sil: Silhouette[WkEnv],
                             rpc: RPC)(implicit ec: ExecutionContext, bodyParsers: PlayBodyParsers)
     extends Controller {
@@ -35,7 +36,9 @@ class Application @Inject()(multiUserDAO: MultiUserDAO,
           "webknossos" -> webknossos.BuildInfo.toMap.mapValues(_.toString),
           "webknossos-wrap" -> webknossoswrap.BuildInfo.toMap.mapValues(_.toString),
           "schemaVersion" -> schemaVersion.toOption,
-          "token" -> token
+          "token" -> token,
+          "localDataStoreEnabled" -> storeModules.localDataStoreEnabled,
+          "localTracingStoreEnabled" -> storeModules.localTracingStoreEnabled
         ))
     }
   }
@@ -63,24 +66,20 @@ class Application @Inject()(multiUserDAO: MultiUserDAO,
 
   def health: Action[AnyContent] = sil.UserAwareAction.async { implicit request =>
     def checkDatastoreHealthIfEnabled: Fox[Unit] =
-      if (conf.Datastore.enabled) {
+      if (storeModules.localDataStoreEnabled) {
         for {
           response <- rpc(s"http://localhost:${conf.Http.port}/data/health").get
           if response.status == 200
         } yield ()
-      } else {
-        Fox.successful(())
-      }
+      } else Fox.successful(())
 
     def checkTracingstoreHealthIfEnabled: Fox[Unit] =
-      if (conf.Tracingstore.enabled) {
+      if (storeModules.localTracingStoreEnabled) {
         for {
           response <- rpc(s"http://localhost:${conf.Http.port}/tracings/health").get
           if response.status == 200
         } yield ()
-      } else {
-        Fox.successful(())
-      }
+      } else Fox.successful(())
 
     for {
       _ <- checkDatastoreHealthIfEnabled ?~> "dataStore.unavailable"

app/controllers/AuthenticationController.scala

@@ -60,7 +60,7 @@ class AuthenticationController @Inject()(
     actorSystem.actorSelection("/user/mailActor")
 
   private lazy val ssoKey =
-    conf.Application.Authentication.ssoKey
+    conf.WebKnossos.User.ssoKey
 
   def register: Action[AnyContent] = Action.async { implicit request =>
     signUpForm.bindFromRequest.fold(

app/controllers/InitialDataController.scala

@@ -15,7 +15,7 @@ import net.liftweb.common.{Box, Full}
 import org.joda.time.DateTime
 import oxalis.security._
 import play.api.libs.json.Json
-import utils.{ObjectId, WkConf}
+import utils.{ObjectId, StoreModules, WkConf}
 import javax.inject.Inject
 import models.organization.{Organization, OrganizationDAO}
 import play.api.mvc.{Action, AnyContent}
@@ -47,14 +47,15 @@ class InitialDataService @Inject()(userService: UserService,
                                    projectDAO: ProjectDAO,
                                    publicationDAO: PublicationDAO,
                                    organizationDAO: OrganizationDAO,
+                                   storeModules: StoreModules,
                                    conf: WkConf)(implicit ec: ExecutionContext)
     extends FoxImplicits
     with LazyLogging {
   implicit val ctx: GlobalAccessContext.type = GlobalAccessContext
 
-  private val defaultUserEmail = conf.Application.Authentication.DefaultUser.email
-  private val defaultUserPassword = conf.Application.Authentication.DefaultUser.password
-  private val defaultUserToken = conf.Application.Authentication.DefaultUser.token
+  private val defaultUserEmail = conf.WebKnossos.SampleOrganization.User.email
+  private val defaultUserPassword = conf.WebKnossos.SampleOrganization.User.password
+  private val defaultUserToken = conf.WebKnossos.SampleOrganization.User.token
   private val additionalInformation = """**Sample Organization**
 
 Sample Street 123
@@ -77,7 +78,7 @@ Samplecountry
     multiUserId,
     defaultUserEmail,
     userService.createPasswordInfo(defaultUserPassword),
-    isSuperUser = conf.Application.Authentication.DefaultUser.isSuperUser,
+    isSuperUser = conf.WebKnossos.SampleOrganization.User.isSuperUser,
   )
   private val defaultUser = User(
     userId,
@@ -109,7 +110,6 @@ Samplecountry
       _ <- updateLocalTracingStorePublicUri()
       _ <- insertLocalDataStoreIfEnabled()
       _ <- insertLocalTracingStoreIfEnabled()
-      _ <- insertConnectDataStoreIfEnabled()
       _ <- assertInitialDataEnabled
       _ <- assertNoOrganizationsPresent
       _ <- insertOrganization()
@@ -123,7 +123,7 @@ Samplecountry
 
   private def assertInitialDataEnabled: Fox[Unit] =
     for {
-      _ <- bool2Fox(conf.Application.insertInitialData) ?~> "initialData.notEnabled"
+      _ <- bool2Fox(conf.WebKnossos.SampleOrganization.enabled) ?~> "initialData.notEnabled"
     } yield ()
 
   def assertNoOrganizationsPresent: Fox[Unit] =
@@ -133,18 +133,22 @@ Samplecountry
     } yield ()
 
   private def insertDefaultUser(): Fox[Unit] =
-    userService.defaultUser.futureBox.flatMap {
-      case Full(_) => Fox.successful(())
-      case _ =>
-        for {
-          _ <- multiUserDAO.insertOne(defaultMultiUser)
-          _ <- userDAO.insertOne(defaultUser)
-          _ <- userExperiencesDAO.updateExperiencesForUser(defaultUser, Map("sampleExp" -> 10))
-          _ <- userTeamRolesDAO.insertTeamMembership(defaultUser._id,
-                                                     TeamMembership(organizationTeam._id, isTeamManager = true))
-          _ = logger.info("Inserted default user scmboy")
-        } yield ()
-    }.toFox
+    userService
+      .userFromMultiUserEmail(defaultUserEmail)
+      .futureBox
+      .flatMap {
+        case Full(_) => Fox.successful(())
+        case _ =>
+          for {
+            _ <- multiUserDAO.insertOne(defaultMultiUser)
+            _ <- userDAO.insertOne(defaultUser)
+            _ <- userExperiencesDAO.updateExperiencesForUser(defaultUser, Map("sampleExp" -> 10))
+            _ <- userTeamRolesDAO.insertTeamMembership(defaultUser._id,
+                                                       TeamMembership(organizationTeam._id, isTeamManager = true))
+            _ = logger.info("Inserted default user scmboy")
+          } yield ()
+      }
+      .toFox
 
   private def insertToken(): Fox[Unit] = {
     val expiryTime = conf.Silhouette.TokenAuthenticator.authenticatorExpiry.toMillis
@@ -199,7 +203,7 @@ Samplecountry
   private def insertProject(): Fox[Unit] =
     projectDAO.findAll.flatMap { projects =>
       if (projects.isEmpty) {
-        userService.defaultUser.flatMap { user =>
+        userService.userFromMultiUserEmail(defaultUserEmail).flatMap { user =>
           val project = Project(ObjectId.generate,
                                 organizationTeam._id,
                                 user._id,
@@ -220,37 +224,26 @@ Samplecountry
   }
 
   def insertLocalDataStoreIfEnabled(): Fox[Unit] =
-    if (conf.Datastore.enabled) {
+    if (storeModules.localDataStoreEnabled) {
       dataStoreDAO.findOneByUrl(conf.Http.uri).futureBox.flatMap { maybeStore =>
         if (maybeStore.isEmpty) {
           logger.info("Inserting local datastore")
           dataStoreDAO.insertOne(
-            DataStore("localhost",
+            DataStore(conf.Datastore.name,
                       conf.Http.uri,
                       conf.Datastore.publicUri.getOrElse(conf.Http.uri),
                       conf.Datastore.key))
         } else Fox.successful(())
       }
     } else Fox.successful(())
 
-  private def insertConnectDataStoreIfEnabled(): Fox[Unit] =
-    if (conf.Application.insertLocalConnectDatastore) {
-      dataStoreDAO.findOneByName("connect").futureBox.flatMap { maybeStore =>
-        if (maybeStore.isEmpty) {
-          logger.info("Inserting connect datastore")
-          dataStoreDAO.insertOne(
-            DataStore("connect", "http://localhost:8000", "http://localhost:8000", "secret-key", isConnector = true))
-        } else Fox.successful(())
-      }
-    } else Fox.successful(())
-
   private def insertLocalTracingStoreIfEnabled(): Fox[Unit] =
-    if (conf.Tracingstore.enabled) {
+    if (storeModules.localTracingStoreEnabled) {
       tracingStoreDAO.findOneByUrl(conf.Http.uri).futureBox.flatMap { maybeStore =>
         if (maybeStore.isEmpty) {
           logger.info("Inserting local tracingstore")
           tracingStoreDAO.insertOne(
-            TracingStore("localhost",
+            TracingStore(conf.Tracingstore.name,
                          conf.Http.uri,
                          conf.Tracingstore.publicUri.getOrElse(conf.Http.uri),
                          conf.Tracingstore.key))
@@ -259,7 +252,7 @@ Samplecountry
     } else Fox.successful(())
 
   private def updateLocalDataStorePublicUri(): Fox[Unit] =
-    if (conf.Datastore.enabled) {
+    if (storeModules.localDataStoreEnabled) {
       dataStoreDAO.findOneByUrl(conf.Http.uri).futureBox.flatMap { storeOpt: Box[DataStore] =>
         storeOpt match {
           case Full(store) =>
@@ -273,7 +266,7 @@ Samplecountry
     } else Fox.successful(())
 
   private def updateLocalTracingStorePublicUri(): Fox[Unit] =
-    if (conf.Tracingstore.enabled) {
+    if (storeModules.localTracingStoreEnabled) {
       tracingStoreDAO.findOneByUrl(conf.Http.uri).futureBox.flatMap { storeOpt: Box[TracingStore] =>
         storeOpt match {
           case Full(store) =>

app/controllers/JobsController.scala

@@ -207,7 +207,7 @@ class JobService @Inject()(wkConf: WkConf,
     } yield job
 
   private def flowerRpc(route: String): RPCRequest =
-    rpc(wkConf.Jobs.Flower.uri + route).withBasicAuth(wkConf.Jobs.Flower.username, wkConf.Jobs.Flower.password)
+    rpc(wkConf.Jobs.Flower.uri + route).withBasicAuth(wkConf.Jobs.Flower.user, wkConf.Jobs.Flower.password)
 
   def assertTiffExportBoundingBoxLimits(bbox: String): Fox[Unit] =
     for {

app/controllers/OrganizationController.scala

@@ -76,7 +76,7 @@ class OrganizationController @Inject()(organizationDAO: OrganizationDAO,
   }
 
   def getOperatorData: Action[AnyContent] = Action {
-    Ok(Json.toJson(conf.operatorData))
+    Ok(Json.toJson(conf.WebKnossos.operatorData))
   }
 
   def update(organizationName: String): Action[JsValue] = sil.SecuredAction.async(parse.json) { implicit request =>

app/models/binary/DataSetService.scala

@@ -206,7 +206,7 @@ class DataSetService @Inject()(organizationDAO: OrganizationDAO,
     }.futureBox
 
   private def publicationForFirstDataset: Fox[Option[ObjectId]] =
-    if (conf.Application.insertInitialData) {
+    if (conf.WebKnossos.SampleOrganization.enabled) {
       dataSetDAO.isEmpty.map { isEmpty =>
         if (isEmpty)
           Some(ObjectId("5c766bec6c01006c018c7459"))

app/models/task/TaskService.scala

@@ -27,8 +27,6 @@ class TaskService @Inject()(conf: WkConf,
                             projectDAO: ProjectDAO)(implicit ec: ExecutionContext)
     extends FoxImplicits {
 
-  private val MAX_OPEN_TASKS: Int = conf.WebKnossos.Tasks.maxOpenPerUser
-
   def publicWrites(task: Task)(implicit ctx: DBAccessContext): Fox[JsObject] =
     for {
       annotationBase <- annotationBaseFor(task._id)
@@ -65,7 +63,7 @@ class TaskService @Inject()(conf: WkConf,
     if (user.isAdmin) return teamDAO.findAllIdsByOrganization(user._organization)
     for {
       numberOfOpen <- countOpenNonAdminTasks(user)
-      teams <- if (numberOfOpen < MAX_OPEN_TASKS) userService.teamIdsFor(user._id)
+      teams <- if (numberOfOpen < conf.WebKnossos.Tasks.maxOpenPerUser) userService.teamIdsFor(user._id)
       else userService.teamManagerTeamIdsFor(user._id)
       _ <- bool2Fox(teams.nonEmpty) ?~> Messages("task.tooManyOpenOnes")
     } yield teams

app/models/user/Invite.scala

@@ -55,7 +55,7 @@ class InviteService @Inject()(conf: WkConf,
         tokenValue,
         organizationID,
         autoActivate,
-        new DateTime(System.currentTimeMillis() + conf.Application.Authentication.inviteExpiry.toMillis)
+        new DateTime(System.currentTimeMillis() + conf.WebKnossos.User.inviteExpiry.toMillis)
       )
 
   private def sendInviteMail(recipient: String, sender: User, invite: Invite)(
@@ -69,7 +69,7 @@ class InviteService @Inject()(conf: WkConf,
     } yield ()
 
   def removeExpiredInvites(): Fox[Unit] =
-    inviteDAO.deleteAllExpired
+    inviteDAO.deleteAllExpired()
 
   def deactivateUsedInvite(invite: Invite)(implicit ctx: DBAccessContext): Fox[Unit] =
     inviteDAO.deleteOne(invite._id)

app/models/user/UserCache.scala

@@ -11,7 +11,7 @@ class UserCache @Inject()(userDAO: UserDAO, conf: WkConf, cache: SyncCacheApi) {
     s"user.${id.toString}"
 
   def findUser(id: ObjectId): Fox[User] =
-    cache.getOrElseUpdate(cacheKeyForUser(id), conf.User.cacheTimeoutInMinutes) {
+    cache.getOrElseUpdate(cacheKeyForUser(id), conf.WebKnossos.Cache.User.timeout) {
       userDAO.findOne(id)(GlobalAccessContext)
     }
 

app/models/user/UserService.scala

@@ -48,9 +48,6 @@ class UserService @Inject()(conf: WkConf,
   private lazy val Mailer =
     actorSystem.actorSelection("/user/mailActor")
 
-  def defaultUser: Fox[User] =
-    userFromMultiUserEmail(conf.Application.Authentication.DefaultUser.email)(GlobalAccessContext)
-
   def userFromMultiUserEmail(email: String)(implicit ctx: DBAccessContext): Fox[User] =
     for {
       multiUser <- multiUserDAO.findOneByEmail(email)(GlobalAccessContext)