Implement compressed segmentation (#6947)

CHANGELOG.unreleased.md

@@ -15,7 +15,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Segments can now be removed from the segment list via the context menu. [#6944](https://github.com/scalableminds/webknossos/pull/6944)
 - Editing the meta data of segments (e.g., the name) is now undoable. [#6944](https://github.com/scalableminds/webknossos/pull/6944)
 - Added more icons and small redesigns for various pages in line with the new branding. [#6938](https://github.com/scalableminds/webknossos/pull/6938)
-
+- Added support for viewing neuroglancer precomputed segmentations using "compressed segmentation" compression. [#6947](https://github.com/scalableminds/webknossos/pull/6947)
 
 ### Changed
 - Moved the view mode selection in the toolbar next to the position field. [#6949](https://github.com/scalableminds/webknossos/pull/6949)

docs/data_formats.md

@@ -11,7 +11,6 @@ WEBKNOSSOS natively supports loading and streaming data in the following formats
 - webKnossos-wrap (WKW)
 - Zarr ([OME NGFF v0.4+ spec](https://ngff.openmicroscopy.org/latest/))
 - Neuroglancer `precomputed` stored on Google Cloud
-- BossDB
 - N5
 
 See the page on [datasets](./datasets.md) for uploading and configuring datasets.
@@ -28,7 +27,7 @@ In particular, the following file formats are supported:
 - Single-file images (tif, czi, nifti, raw)
 - KNOSSOS file hierarchy 
 
-Note, for datasets in the Zarr, N5, Neuroglancer `Pre-Computed` or BossDB formats uploading and automatic conversion are not supported.
+Note, for datasets in the Zarr, N5 and Neuroglancer Precomputed formats uploading and automatic conversion are not supported.
 Instead, they can be directly streamed from an HTTP server or the cloud.
 See the page on [datasets](./datasets.md) for uploading and configuring these formats.
 

docs/datasets.md

@@ -70,25 +70,6 @@ Note that data streaming may incur costs and count against any usage limits or m
 Hint: If you happen to have any Zarr dataset locally that you would like to view in WEBKNOSSOS, consider running an HTTP server locally to serve the dataset. 
 Then WEBKNOSSOS can easily stream the data.
 
-### Working with Neuroglancer and BossDB datasets on webknossos.org
-webknossos.org supports loading and remotely streaming datasets in the [Neuroglancer precomputed format](https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed) stored in the Google Cloud or datasets served from [BossDB](https://bossdb.org).
-
-To import these datasets:
-
-1. From the *Datasets* tab in the user dashboard, click the *Add Dataset* button.
-2. Select the *Add Neuroglancer Dataset* or *Add BossDB Dataset* tab
-3. Provide some metadata information:  
-  - a *dataset name* 
-  - a URL or domain/collection identifier to locate the dataset on the remote service
-  - authentication credentials for accessing the resources on the remote service (optional)
-4. Click the *Add* button
-
-WEBKNOSSOS will NOT download/copy any data from these third-party data providers. 
-Rather, any data viewed in WEBKNOSSOS will be streamed read-only and directly from the remote source. 
-Any other WEBKNOSSOS feature, e.g., annotations, and access rights, will be stored in WEBKNOSSOS and do not affect these services. 
-
-Note that data streaming may count against any usage limits or minutes as defined by these third-party services. Check with the service provider or dataset owner.
-
 ### Uploading through the Python API
 For those wishing to automate dataset upload or to do it programmatically, check out the WEBKNOSSOS [Python library](https://github.com/scalableminds/webknossos-libs). It allows you to create, manage and upload datasets as well. 
 

docs/tooling.md

@@ -18,12 +18,6 @@ We provide several free, open-source libraries and tools alongside WEBKNOSSOS to
 - [Read more about the support data formats](./data_formats.md)
 
 
-## WEBKNOSSOS Connect
-- [https://github.com/scalableminds/webknossos-connect](https://github.com/scalableminds/webknossos-connect)
-- A WEBKNOSSOS compatible data connector written in Python
-- WEBKNOSSOS-connect serves as an adapter between the WEBKNOSSOS data store interface and other alternative data storage servers (e.g., BossDB) or static files hosted on Cloud Storage (e.g. Neuroglancer Precomputed)
-
-
 ## webKnossos Wrap Data Format (wkw)
 - [webknossos-wrap](https://github.com/scalableminds/webknossos-wrap)
 - Library for low-level read and write operations to wkw datasets

test/backend/CompressedSegmentationTestSuite.scala

@@ -0,0 +1,58 @@
+package backend
+
+import com.scalableminds.util.geometry.Vec3Int
+import com.scalableminds.webknossos.datastore.datareaders.precomputed.compressedsegmentation.CompressedSegmentation64
+import org.scalatestplus.play.PlaySpec
+
+import java.nio.{ByteBuffer, ByteOrder}
+
+class CompressedSegmentationTestSuite extends PlaySpec {
+
+  "Compressed segmentation" when {
+
+    /*
+    # The compressed test data was created with this python code:
+    import compressed_segmentation as cseg
+    import numpy as np
+
+    input = np.array([[[5, 5, 4],
+            [4, 1, 7],
+            [5, 3, 5]],
+
+           [[3, 8, 5],
+            [9, 8, 4],
+            [4, 9, 3]],
+
+           [[6, 1, 9],
+            [3, 7, 8],
+            [8, 0, 9]]], dtype=np.uint64)
+
+    compressed = cseg.compress(input)
+
+    # Compressed output
+    # 1, 0, 0, 0, 66, 0, 0, 4, 2, 0, 0, 0, 36, 5, 0, 0, 131, 2, 0, 0, 52, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 116, 1, 0, 0, 113, 6, 0, 0, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 67, 8, 0, 0, 54, 7, 0, 0, 36, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 9, 0, 0, 0, 0, 0, 0, 0
+     */
+
+    val compressed = Array[Byte](1, 0, 0, 0, 66, 0, 0, 4, 2, 0, 0, 0, 36, 5, 0, 0, -125, 2, 0, 0, 52, 7, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 116, 1, 0, 0, 113, 6, 0, 0, -126, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 67, 8, 0, 0, 54, 7, 0, 0, 36, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0,
+      0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0,
+      0, 0, 0, 0, 9, 0, 0, 0, 0, 0, 0, 0)
+
+    "datatype uint64" should {
+
+      "decompress compressed array" in {
+        val decompressedBytes =
+          CompressedSegmentation64.decompress(compressed, volumeSize = Array(3, 3, 3), blockSize = Vec3Int(8, 8, 8))
+        val arr = new Array[Long](27)
+        ByteBuffer.wrap(decompressedBytes).order(ByteOrder.LITTLE_ENDIAN).asLongBuffer().get(arr)
+        assert(arr.sameElements(Array(5, 3, 6, 4, 9, 3, 5, 4, 8, 5, 8, 1, 1, 8, 7, 3, 9, 0, 4, 5, 9, 7, 4, 8, 5, 3, 9)))
+      }
+    }
+  }
+}

webknossos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/ChunkReader.scala

@@ -4,7 +4,7 @@ import com.scalableminds.webknossos.datastore.datavault.VaultPath
 import com.typesafe.scalalogging.LazyLogging
 import ucar.ma2.{Array => MultiArray, DataType => MADataType}
 
-import java.io.{ByteArrayInputStream, ByteArrayOutputStream, IOException}
+import java.io.{ByteArrayInputStream, IOException}
 import javax.imageio.stream.MemoryCacheImageInputStream
 import scala.collection.immutable.NumericRange
 import scala.concurrent.Future
@@ -38,14 +38,9 @@ class ChunkReader(val header: DatasetHeader, val vaultPath: VaultPath, val chunk
   // and chunk shape (optional, only for data formats where each chunk reports its own shape, e.g. N5)
   protected def readChunkBytesAndShape(path: String,
                                        range: Option[NumericRange[Long]]): Option[(Array[Byte], Option[Array[Int]])] =
-    Using.Manager { use =>
-      (vaultPath / path).readBytes(range).map { bytes =>
-        val is = use(new ByteArrayInputStream(bytes))
-        val os = use(new ByteArrayOutputStream())
-        header.compressorImpl.uncompress(is, os)
-        (os.toByteArray, None)
-      }
-    }.get
+    (vaultPath / path).readBytes(range).map { bytes =>
+      (header.compressorImpl.decompress(bytes), None)
+    }
 }
 
 abstract class ChunkTyper {

webknossos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/Compressor.scala

@@ -1,5 +1,12 @@
 package com.scalableminds.webknossos.datastore.datareaders
 
+import com.scalableminds.util.geometry.Vec3Int
+import com.scalableminds.webknossos.datastore.datareaders.precomputed.PrecomputedDataType
+import com.scalableminds.webknossos.datastore.datareaders.precomputed.PrecomputedDataType.PrecomputedDataType
+import com.scalableminds.webknossos.datastore.datareaders.precomputed.compressedsegmentation.{
+  CompressedSegmentation32,
+  CompressedSegmentation64
+}
 import com.sun.jna.ptr.NativeLongByReference
 import org.apache.commons.compress.compressors.gzip.{
   GzipCompressorInputStream,
@@ -15,7 +22,7 @@ import java.nio.ByteBuffer
 import java.util
 import java.util.zip.{Deflater, DeflaterOutputStream, Inflater, InflaterInputStream}
 import javax.imageio.ImageIO
-import javax.imageio.ImageIO.{createImageInputStream}
+import javax.imageio.ImageIO.createImageInputStream
 import javax.imageio.stream.ImageInputStream
 
 sealed trait CompressionSetting
@@ -49,16 +56,18 @@ abstract class Compressor {
   def toString: String
 
   @throws[IOException]
-  def compress(is: InputStream, os: OutputStream): Unit
+  def compress(input: Array[Byte]): Array[Byte]
 
   @throws[IOException]
-  def uncompress(is: InputStream, os: OutputStream): Unit
+  def decompress(input: Array[Byte]): Array[Byte]
 
   @throws[IOException]
   def passThrough(is: InputStream, os: OutputStream): Unit = {
     val bytes = new Array[Byte](4096)
     var read = is.read(bytes)
-    while ({ read >= 0 }) {
+    while ({
+      read >= 0
+    }) {
       if (read > 0)
         os.write(bytes, 0, read)
       read = is.read(bytes)
@@ -73,10 +82,10 @@ class NullCompressor extends Compressor {
   override def toString: String = getId
 
   @throws[IOException]
-  override def compress(is: InputStream, os: OutputStream): Unit = passThrough(is, os)
+  override def compress(input: Array[Byte]): Array[Byte] = input
 
   @throws[IOException]
-  override def uncompress(is: InputStream, os: OutputStream): Unit = passThrough(is, os)
+  override def decompress(input: Array[Byte]): Array[Byte] = input
 }
 
 class ZlibCompressor(val properties: Map[String, CompressionSetting]) extends Compressor {
@@ -98,17 +107,23 @@ class ZlibCompressor(val properties: Map[String, CompressionSetting]) extends Co
   override def getId = "zlib"
 
   @throws[IOException]
-  override def compress(is: InputStream, os: OutputStream): Unit = {
+  override def compress(input: Array[Byte]): Array[Byte] = {
+    val is = new ByteArrayInputStream(input)
+    val os = new ByteArrayOutputStream()
     val dos = new DeflaterOutputStream(os, new Deflater(level))
     try passThrough(is, dos)
     finally if (dos != null) dos.close()
+    os.toByteArray
   }
 
   @throws[IOException]
-  override def uncompress(is: InputStream, os: OutputStream): Unit = {
+  override def decompress(input: Array[Byte]): Array[Byte] = {
+    val is = new ByteArrayInputStream(input)
+    val os = new ByteArrayOutputStream()
     val iis = new InflaterInputStream(is, new Inflater)
     try passThrough(iis, os)
     finally if (iis != null) iis.close()
+    os.toByteArray
   }
 }
 
@@ -131,19 +146,26 @@ class GzipCompressor(val properties: Map[String, CompressionSetting]) extends Co
   override def getId = "gzip"
 
   @throws[IOException]
-  override def compress(is: InputStream, os: OutputStream): Unit = {
+  override def compress(input: Array[Byte]): Array[Byte] = {
+    val is = new ByteArrayInputStream(input)
+    val os = new ByteArrayOutputStream()
+
     val parameters = new GzipParameters
     parameters.setCompressionLevel(level)
     val dos = new GzipCompressorOutputStream(os, parameters)
     try passThrough(is, dos)
     finally if (dos != null) dos.close()
+    os.toByteArray
   }
 
   @throws[IOException]
-  override def uncompress(is: InputStream, os: OutputStream): Unit = {
+  override def decompress(input: Array[Byte]): Array[Byte] = {
+    val is = new ByteArrayInputStream(input)
+    val os = new ByteArrayOutputStream()
     val iis = new GzipCompressorInputStream(is, true)
     try passThrough(iis, os)
     finally if (iis != null) iis.close()
+    os.toByteArray
   }
 }
 
@@ -222,7 +244,9 @@ class BloscCompressor(val properties: Map[String, CompressionSetting]) extends C
     "compressor=" + getId + "/cname=" + cname + "/clevel=" + clevel.toString + "/blocksize=" + blocksize + "/shuffle=" + shuffle
 
   @throws[IOException]
-  override def compress(is: InputStream, os: OutputStream): Unit = {
+  override def compress(input: Array[Byte]): Array[Byte] = {
+    val is = new ByteArrayInputStream(input)
+
     val baos = new ByteArrayOutputStream
     passThrough(is, baos)
     val inputBytes = baos.toByteArray
@@ -233,11 +257,13 @@ class BloscCompressor(val properties: Map[String, CompressionSetting]) extends C
     JBlosc.compressCtx(clevel, shuffle, 1, inputBuffer, inputSize, outBuffer, outputSize, cname, blocksize, 1)
     val bs = cbufferSizes(outBuffer)
     val compressedChunk = util.Arrays.copyOfRange(outBuffer.array, 0, bs.getCbytes.toInt)
-    os.write(compressedChunk)
+    compressedChunk
   }
 
   @throws[IOException]
-  override def uncompress(is: InputStream, os: OutputStream): Unit = {
+  override def decompress(input: Array[Byte]): Array[Byte] = {
+    val is = new ByteArrayInputStream(input)
+
     val di = new DataInputStream(is)
     val header = new Array[Byte](JBlosc.OVERHEAD)
     di.readFully(header)
@@ -248,7 +274,8 @@ class BloscCompressor(val properties: Map[String, CompressionSetting]) extends C
     di.readFully(inBytes, header.length, compressedSize - header.length)
     val outBuffer = ByteBuffer.allocate(uncompressedSize)
     JBlosc.decompressCtx(ByteBuffer.wrap(inBytes), outBuffer, outBuffer.limit, 1)
-    os.write(outBuffer.array)
+
+    outBuffer.array
   }
 
   private def cbufferSizes(cbuffer: ByteBuffer) = {
@@ -268,16 +295,37 @@ class JpegCompressor() extends Compressor {
   override def toString: String = getId
 
   @throws[IOException]
-  override def compress(is: InputStream, os: OutputStream): Unit = ???
+  override def compress(input: Array[Byte]): Array[Byte] = ???
 
   @throws[IOException]
-  override def uncompress(is: InputStream, os: OutputStream): Unit = {
+  override def decompress(input: Array[Byte]): Array[Byte] = {
+    val is = new ByteArrayInputStream(input)
     val iis: ImageInputStream = createImageInputStream(is)
     val bi: BufferedImage = ImageIO.read(iis: ImageInputStream)
     val raster = bi.getRaster
     val dbb: DataBufferByte = raster.getDataBuffer.asInstanceOf[DataBufferByte]
     val width = raster.getWidth
     val data = dbb.getData.grouped(width).toList
-    os.write(data.flatten.toArray)
+    data.flatten.toArray
   }
 }
+
+class CompressedSegmentationCompressor(dataType: PrecomputedDataType, volumeSize: Array[Int], blockSize: Vec3Int)
+    extends Compressor {
+  override def getId: String = "compressedsegmentation"
+
+  override def toString: String = s"compressor=$getId/dataType=${dataType.toString}"
+
+  override def decompress(input: Array[Byte]): Array[Byte] =
+    dataType match {
+      case PrecomputedDataType.uint32 =>
+        CompressedSegmentation32.decompress(input, volumeSize, blockSize)
+      case PrecomputedDataType.uint64 =>
+        CompressedSegmentation64.decompress(input, volumeSize, blockSize)
+      case _ =>
+        throw new UnsupportedOperationException(
+          "Can not use compressed segmentation for datatypes other than u32, u64.")
+    }
+
+  override def compress(input: Array[Byte]): Array[Byte] = ???
+}

webknossos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/n5/N5ChunkReader.scala

@@ -4,9 +4,7 @@ import com.scalableminds.webknossos.datastore.datareaders.{ChunkReader, ChunkTyp
 import com.scalableminds.webknossos.datastore.datavault.VaultPath
 import com.typesafe.scalalogging.LazyLogging
 
-import java.io.{ByteArrayInputStream, ByteArrayOutputStream}
 import scala.collection.immutable.NumericRange
-import scala.util.Using
 
 object N5ChunkReader {
   def create(vaultPath: VaultPath, header: DatasetHeader): ChunkReader =
@@ -26,23 +24,19 @@ class N5ChunkReader(header: DatasetHeader, vaultPath: VaultPath, typedChunkReade
 
   override protected def readChunkBytesAndShape(
       path: String,
-      range: Option[NumericRange[Long]]): Option[(Array[Byte], Option[Array[Int]])] =
-    Using.Manager { use =>
-      def processBytes(bytes: Array[Byte], expectedElementCount: Int): Array[Byte] = {
-        val is = use(new ByteArrayInputStream(bytes))
-        val os = use(new ByteArrayOutputStream())
-        header.compressorImpl.uncompress(is, os)
-        val output = os.toByteArray
-        val paddedBlock = output ++ Array.fill(header.bytesPerElement * expectedElementCount - output.length) {
-          header.fillValueNumber.byteValue()
-        }
-        paddedBlock
+      range: Option[NumericRange[Long]]): Option[(Array[Byte], Option[Array[Int]])] = {
+    def processBytes(bytes: Array[Byte], expectedElementCount: Int): Array[Byte] = {
+      val output = header.compressorImpl.decompress(bytes)
+      val paddedBlock = output ++ Array.fill(header.bytesPerElement * expectedElementCount - output.length) {
+        header.fillValueNumber.byteValue()
       }
-
-      for {
-        bytes <- (vaultPath / path).readBytes(range)
-        (blockHeader, data) = dataExtractor.readBytesAndHeader(bytes)
-        paddedChunkBytes = processBytes(data, blockHeader.blockSize.product)
-      } yield (paddedChunkBytes, Some(blockHeader.blockSize))
-    }.get
+      paddedBlock
+    }
+
+    for {
+      bytes <- (vaultPath / path).readBytes(range)
+      (blockHeader, data) = dataExtractor.readBytesAndHeader(bytes)
+      paddedChunkBytes = processBytes(data, blockHeader.blockSize.product)
+    } yield (paddedChunkBytes, Some(blockHeader.blockSize))
+  }
 }