Skip to content

Commit

Permalink
Merge branch 'master' of github.com:scalableminds/webknossos into emp…
Browse files Browse the repository at this point in the history
…ty-list-drawings

* 'master' of github.com:scalableminds/webknossos: (25 commits)
  Fix issues with styling in dark mode on login page (#7052)
  Fix nightly by setting missing token (#7048)
  Release 23.05.1 (#7042)
  DRY types in update_actions.ts (#7036)
  Remove some spammy logging from backend (#7039)
  Use zarr string fill values (#7017)
  Fix voxel offset for Neuroglancer Precomputed datasets (#7019)
  Log when user is activated (#7027)
  Fix exception in applying UpdateTreeGroupVisibility skeleton action (#7037)
  Fix organization storage layouting (#7034)
  Update docker compose commands + dev install readme (#7002)
  Add segment groups (#6966)
  Add screenshot nightly test for wkorg (#7030)
  Workaround for WebGL crash for datasets with many segmentation layers (#6995)
  Fix download of public annotation, include access ctx in user cache key (#7025)
  Fix that changing a segment color could lead to a crash (#7000)
  Add more error chaining to annotation download (#7023)
  Guard against NaNs in shader (#7018)
  Store editable mappings in multiple fossildb columns+keys (#6903)
  Context action to move tree to group (#7005)
  ...
  • Loading branch information
hotzenklotz committed May 4, 2023
2 parents 43894da + 802cf84 commit c9ee334
Show file tree
Hide file tree
Showing 158 changed files with 4,285 additions and 1,550 deletions.
32 changes: 32 additions & 0 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -291,6 +291,27 @@ jobs:
- store_artifacts:
path: frontend/javascripts/test/snapshots/type-check

wkorg_nightly:
docker:
- image: scalableminds/puppeteer:master
resource_class: large
steps:
- checkout
- run:
name: Install dependencies
command: |
yarn install --frozen-lockfile
- run:
name: Run screenshot-tests
command: |
# CircleCI cancels the job after 60 minutes. To ensure that screenshots are still
# uploaded as artifacts, we define a timeout of 50 minutes for the screenshot tests.
timeout 3000 \
yarn test-wkorg-screenshot
- store_artifacts:
path: frontend/javascripts/test/screenshots

workflows:
version: 2
circleci_build:
Expand All @@ -312,3 +333,14 @@ workflows:
branches:
only:
- master
circleci_wkorg_nightly:
jobs:
- wkorg_nightly
triggers:
- schedule:
# 03:15 AM UTC
cron: "15 3 * * *"
filters:
branches:
only:
- master
37 changes: 37 additions & 0 deletions CHANGELOG.released.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,43 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
For upgrade instructions, please check the [migration guide](MIGRATIONS.released.md).

## [23.05.1](https://github.com/scalableminds/webknossos/releases/tag/23.05.1) - 2023-05-02
[Commits](https://github.com/scalableminds/webknossos/compare/23.05.0...23.05.1)

### Fixed
- Fixed rendering issues on some affected systems that led to "black holes". [#7018](https://github.com/scalableminds/webknossos/pull/7018)
- Added a workaround for a WebGL crash which could appear when a dataset contained many segmentation layers. [#6995](https://github.com/scalableminds/webknossos/pull/6995)

## [23.05.0](https://github.com/scalableminds/webknossos/releases/tag/23.05.0) - 2023-04-25
[Commits](https://github.com/scalableminds/webknossos/compare/23.04.2...23.05.0)

### Highlights
- Segments can now be removed from the segment list via the context menu. [#6944](https://github.com/scalableminds/webknossos/pull/6944)
- Added support for viewing neuroglancer precomputed segmentations using "compressed segmentation" compression. [#6947](https://github.com/scalableminds/webknossos/pull/6947)

### Added
- WEBKNOSSOS screenshots now include a logo in the lower left corner. Users with a paid license can disable this feature in the viewport options in the settings tab on the left side of the screen. [#6910](https://github.com/scalableminds/webknossos/pull/6910)
- Added support for datasets with mixed magnifications (e.g., one layer contains mag 2-2-2 and another contains 2-2-1). [#6943](https://github.com/scalableminds/webknossos/pull/6943)
- Added rendering precomputed meshes with level of detail depending on the zoom of the 3D viewport. This feature only works with version 3 mesh files. [#6909](https://github.com/scalableminds/webknossos/pull/6909)
- Editing the meta data of segments (e.g., the name) is now undoable. [#6944](https://github.com/scalableminds/webknossos/pull/6944)
- Added more icons and small redesigns for various pages in line with the new branding. [#6938](https://github.com/scalableminds/webknossos/pull/6938)
- Added more graphics and branding to job notification emails. [#6994](https://github.com/scalableminds/webknossos/pull/6994)
- Added action button in Team Admin page that links to the User Admin page to edit team members there. [#6958](https://github.com/scalableminds/webknossos/pull/6999)

### Changed
- Moved the view mode selection in the toolbar next to the position field. [#6949](https://github.com/scalableminds/webknossos/pull/6949)
- Redesigned welcome toast for new, annonymous users with new branding. [#6961](https://github.com/scalableminds/webknossos/pull/6961)
- When saving annotations, the URL of the webknossos instance is stored in the resulting NML file. [#6964](https://github.com/scalableminds/webknossos/pull/6964)

### Fixed
- Fixed unintended dependencies between segments of different volume layers which used the same segment id. Now, using the same segment id for segments in different volume layers should work without any problems. [#6960](https://github.com/scalableminds/webknossos/pull/6960)
- Fixed incorrect initial tab when clicking "Show Annotations" for a user in the user list. Also, the datasets tab was removed from that page as it was the same as the datasets table from the main dashboard. [#6957](https://github.com/scalableminds/webknossos/pull/6957)
- Fixed that unsaved changes were shown when opening an annotation, although there weren't any. [#6972](https://github.com/scalableminds/webknossos/pull/6972)
- Fixed misleading email about successful dataset upload, which was in some cases sent even for unusable datasets. [#6977](https://github.com/scalableminds/webknossos/pull/6977)
- Fixed upload of skeleton annotations with no trees, only bounding boxes, being incorrectly rejected. [#6985](https://github.com/scalableminds/webknossos/pull/6985)
- Fixed that Google Cloud Storage URLs with bucket names containing underscores could not be parsed. [#6998](https://github.com/scalableminds/webknossos/pull/6998)
- Fixed regression that caused public datasets to crash when not being logged in. [#7010](https://github.com/scalableminds/webknossos/pull/7010)

## [23.04.2](https://github.com/scalableminds/webknossos/releases/tag/23.04.2) - 2023-04-14
[Commits](https://github.com/scalableminds/webknossos/compare/23.04.1...23.04.2)

Expand Down
28 changes: 10 additions & 18 deletions CHANGELOG.unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,30 +8,22 @@ and this project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MIC
For upgrade instructions, please check the [migration guide](MIGRATIONS.released.md).

## Unreleased
[Commits](https://github.com/scalableminds/webknossos/compare/23.04.2...HEAD)
[Commits](https://github.com/scalableminds/webknossos/compare/23.05.1...HEAD)

### Added
- WEBKNOSSOS screenshots now include a logo in the lower left corner. Users with a paid license can disable this feature in the viewport options in the settings tab on the left side of the screen. [#6910](https://github.com/scalableminds/webknossos/pull/6910)
- Added support for datasets with mixed magnifications (e.g., one layer contains mag 2-2-2 and another contains 2-2-1). [#6943](https://github.com/scalableminds/webknossos/pull/6943)
- Added rendering precomputed meshes with level of detail depending on the zoom of the 3D viewport. This feature only works with version 3 mesh files. [#6909](https://github.com/scalableminds/webknossos/pull/6909)
- Segments can now be removed from the segment list via the context menu. [#6944](https://github.com/scalableminds/webknossos/pull/6944)
- Editing the meta data of segments (e.g., the name) is now undoable. [#6944](https://github.com/scalableminds/webknossos/pull/6944)
- Added more icons and small redesigns for various pages in line with the new branding. [#6938](https://github.com/scalableminds/webknossos/pull/6938)
- Added support for viewing neuroglancer precomputed segmentations using "compressed segmentation" compression. [#6947](https://github.com/scalableminds/webknossos/pull/6947)
- Added more graphics and branding to job notification emails. [#6994](https://github.com/scalableminds/webknossos/pull/6994)
- Added segment groups so that segments can be organized in a hierarchy (similar to skeletons). [#6966](https://github.com/scalableminds/webknossos/pull/6966)
- In addition to drag and drop, the selected tree(s) in the Skeleton tab can also be moved into another group by right-clicking the target group and selecting "Move selected tree(s) here". [#7005](https://github.com/scalableminds/webknossos/pull/7005)

### Changed
- Moved the view mode selection in the toolbar next to the position field. [#6949](https://github.com/scalableminds/webknossos/pull/6949)
- Redesigned welcome toast for new, annonymous users with new branding. [#6961](https://github.com/scalableminds/webknossos/pull/6961)
- When saving annotations, the URL of the webknossos instance is stored in the resulting NML file. [#6964](https://github.com/scalableminds/webknossos/pull/6964)

### Fixed
- Fixed unintended dependencies between segments of different volume layers which used the same segment id. Now, using the same segment id for segments in different volume layers should work without any problems. [#6960](https://github.com/scalableminds/webknossos/pull/6960)
- Fixed incorrect initial tab when clicking "Show Annotations" for a user in the user list. Also, the datasets tab was removed from that page as it was the same as the datasets table from the main dashboard. [#6957](https://github.com/scalableminds/webknossos/pull/6957)
- Fixed that unsaved changes were shown when opening an annotation, although there weren't any. [#6972](https://github.com/scalableminds/webknossos/pull/6972)
- Fixed misleading email about successful dataset upload, which was in some cases sent even for unusable datasets. [#6977](https://github.com/scalableminds/webknossos/pull/6977)
- Fixed upload of skeleton annotations with no trees, only bounding boxes, being incorrectly rejected. [#6985](https://github.com/scalableminds/webknossos/pull/6985)
- Fixed that Google Cloud Storage URLs with bucket names containing underscores could not be parsed. [#6998](https://github.com/scalableminds/webknossos/pull/6998)
- Fixed that changing a segment color could lead to a crash. [#7000](https://github.com/scalableminds/webknossos/pull/7000)
- The fill_value property of zarr dataset is now used when it is not a number. [#7017](https://github.com/scalableminds/webknossos/pull/7017)
- Fixed a bug that made downloads of public annotations fail occasionally. [#7025](https://github.com/scalableminds/webknossos/pull/7025)
- Fixed layouting of used storage space on the organization page. [#7034](https://github.com/scalableminds/webknossos/pull/7034)
- Fixed a bug where updating skeleton annotation tree group visibility would break the annotation. [#7037](https://github.com/scalableminds/webknossos/pull/7037)
- Fixed importing Neuroglancer Precomputed datasets that have a voxel offset in their header. [#7019](https://github.com/scalableminds/webknossos/pull/7019)
- Fixed an bug where invalid email addresses were not readable in dark mode on the login/signup pages. [#7052](https://github.com/scalableminds/webknossos/pull/7052)

### Removed

Expand Down
69 changes: 42 additions & 27 deletions DEV_INSTALL.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
## Development installation
### Docker
This is only recommended for local testing. Docker 17+ and Docker Compose 1.18+ are required.
# Development installation

## Docker

This is only recommended for local testing. Docker 19.03.0+ and Docker Compose 2.+ are required.

```bash
git clone -b master --depth=1 [email protected]:scalableminds/webknossos.git
cd webknossos
docker-compose pull webknossos
docker compose pull webknossos
./start-docker.sh
```

Expand All @@ -16,7 +18,7 @@ See the wiki for [instructions on updating](https://github.com/scalableminds/web

For non-localhost deployments, check out the [installation guide in the documentation](https://docs.webknossos.org/webknossos/installation.html).

### Dependencies
## Dependencies

* [Oracle JDK 11 to 14](http://www.oracle.com/technetwork/java/javase/downloads/index.html) or [Open JDK 11 to 14](http://openjdk.java.net/) (full JDK, JRE is not enough)
* [sbt](http://www.scala-sbt.org/)
Expand All @@ -27,7 +29,10 @@ For non-localhost deployments, check out the [installation guide in the document
* [yarn package manager](https://yarnpkg.com/)
* [git](http://git-scm.com/downloads)

### MacOS
* For some development tasks like refreshing snapshots, Docker 19.03.0+ and Docker Compose 2.+ are required

## MacOS

```bash
# WEBKNOSSOS needs to be run from x86_64 environment (only applicable for arm64-based Macs)
arch -x86_64 /bin/zsh
Expand Down Expand Up @@ -60,80 +65,90 @@ psql -c "GRANT ALL PRIVILEGES ON DATABASE webknossos TO postgres;"

# Checkout the WEBKNOSSOS git repository
git clone [email protected]:scalableminds/webknossos.git


```

Note: On arm64-based Macs (e.g. M1), you need to run WEBKNOSSOS in an x86_64 environment (Rosetta 2). In case you accidentally started WEBKNOSSOS in an arm64 environment, it is advisable to delete several caches `~/.m2`, `~/ivy2`, `~/.sbt`, `~/.yarn-cache` and run `./clean`. Since Postgres and Redis are isolated processes, they can be run either from arm64 or x86_64 environments.


### Ubuntu 20.04 LTS
## Ubuntu 22.04 LTS

```bash
sudo apt install -y curl ca-certificates wget
# Adding repositories for nodejs, sbt and yarn
curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | sudo tee /etc/apt/sources.list.d/sbt.list
echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | sudo tee /etc/apt/sources.list.d/sbt_old.list
curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo apt-key add
curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo -H gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/scalasbt-release.gpg --import
sudo chmod 644 /etc/apt/trusted.gpg.d/scalasbt-release.gpg
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

sudo apt update
sudo apt install -y nodejs git postgresql postgresql-client scala sbt openjdk-14-jdk yarn redis-server build-essential libblosc1
sudo apt install -y nodejs git postgresql postgresql-client scala sbt openjdk-11-jdk yarn redis-server build-essential libblosc1

# Assign a password to PostgreSQL user
sudo -u postgres psql -c "ALTER USER postgres WITH ENCRYPTED PASSWORD 'postgres';"
# Clone the git repo to the current directory
git clone -b master --depth=1 https://github.com/scalableminds/webknossos.git
```

If you already have a different Java version installed, set the default version to Java 14:
- run `sudo update-alternatives --config java`
- when prompted, select the desired version
If you already have a different Java version installed, set the default version to Java 11:

* run `sudo update-alternatives --config java`
* when prompted, select the desired version

On older Ubuntu distributions: Please make sure to have the correct versions of node, PostgreSQL and java installed.

### Manual Installation
## Manual Installation

### Java

##### Java
* Install Java JDK 14 (from Oracle or OpenJDK)
* make sure `JAVA_HOME` and `JDK_HOME` are set and `PATH` contains the path to JDK

##### sbt
See: http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html
### sbt

* See: [http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html](http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html)

##### PostgreSQL
* Install PostgreSQL from https://www.postgresql.org/download/
### PostgreSQL

* Install PostgreSQL from [https://www.postgresql.org/download/](https://www.postgresql.org/download/)
* PostgreSQL version **10+ is required**

##### Redis
* Install Redis from https://redis.io/download
### Redis

* Install Redis from [https://redis.io/download](https://redis.io/download)

### node.js & yarn

##### node.js & yarn
* Install node from http://nodejs.org/download/
* Install node from [http://nodejs.org/download/](http://nodejs.org/download/)
* node version **16+ is required**
* Install yarn package manager: `npm install -g yarn`

### Run locally
## Run locally

First, install all frontend dependencies via

```bash
yarn install
```

Note: During this installation step, it might happen that the module `gl` cannot be installed correctly. As this module is only used for testing WEBKNOSSOS, you can safely ignore this error.

To start WEBKNOSSOS, use

```bash
yarn start
```

This will fetch all Scala, Java and node dependencies and run the application on Port 9000.
Make sure that the PostgreSQL and Redis services are running before you start the application.

## Upgrades

For upgrades, please check the [changelog](CHANGELOG.released.md) & [migration guide](MIGRATIONS.released.md).

## Tests

```bash
# Frontend linting
yarn run lint
Expand All @@ -151,5 +166,5 @@ yarn flow
yarn test-verbose

# End-to-end tests
docker-compose run e2e-tests
docker compose run e2e-tests
```
15 changes: 14 additions & 1 deletion MIGRATIONS.released.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,19 @@ See `MIGRATIONS.unreleased.md` for the changes which are not yet part of an offi
This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
User-facing changes are documented in the [changelog](CHANGELOG.released.md).

## [23.05.1](https://github.com/scalableminds/webknossos/releases/tag/23.05.1) - 2023-05-02
[Commits](https://github.com/scalableminds/webknossos/compare/23.05.0...23.05.1)

### Postgres Evolutions:
None.

## [23.05.0](https://github.com/scalableminds/webknossos/releases/tag/23.05.0) - 2023-04-25
[Commits](https://github.com/scalableminds/webknossos/compare/23.04.2...23.05.0)
- The config key features.isDemoInstance was renamed to features.isWkorgInstance (only needs to be adapted for the main wkorg instance). [#6941](https://github.com/scalableminds/webknossos/pull/6941/files)

### Postgres Evolutions:
None.

## [23.04.2](https://github.com/scalableminds/webknossos/releases/tag/23.04.2) - 2023-04-14
[Commits](https://github.com/scalableminds/webknossos/compare/23.04.1...23.04.2)

Expand All @@ -22,7 +35,7 @@ None.
[Commits](https://github.com/scalableminds/webknossos/compare/23.03.1...23.04.0)

### Postgres Evolutions:
None.
- [101-coordinate-transformations.sql](conf/evolutions/101-coordinate-transformations.sql)

## [23.03.1](https://github.com/scalableminds/webknossos/releases/tag/23.03.1) - 2023-03-14
[Commits](https://github.com/scalableminds/webknossos/compare/23.03.0...23.03.1)
Expand Down
5 changes: 3 additions & 2 deletions MIGRATIONS.unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
User-facing changes are documented in the [changelog](CHANGELOG.released.md).

## Unreleased
[Commits](https://github.com/scalableminds/webknossos/compare/23.04.2...HEAD)
- The config key features.isDemoInstance was renamed to features.isWkorgInstance (only needs to be adapted for the main wkorg instance). [#6941](https://github.com/scalableminds/webknossos/pull/6941/files)
[Commits](https://github.com/scalableminds/webknossos/compare/23.05.1...HEAD)
- FossilDB needs to be opened with new additional column families editableMappingsInfo, editableMappingsAgglomerateToGraph, editableMappingsSegmentToAgglomerate.
- For instances with existing editable mapping (a.k.a supervoxel proofreading) annotations: To keep those annotations alive, a python migration has to be run with access to your tracingstore’s FossilDB. It is recommended to do this during a webknossos downtime to avoid data loss. It needs python 3.8+ and the pip packages installable by `pip install grpcio-tools grpcio-health-checking`. Run it with `python tools/migrate-editable-mappings/migrate-editable-mappings.py -v -w -o localhost:7155`. Omit -o for a faster migration but no access to older versions of the editable mappings. The migration is idempotent.

### Postgres Evolutions:
Loading

0 comments on commit c9ee334

Please sign in to comment.