Scan Annotation Pipeline

The SSTK provides the annotation tools used for semantic segmentation labeling of the ScanNet and Matterport3D datasets. For ScanNet, we also provide an interface for aligning 3D CAD models to the reconstructed scan.

If you use either the semantic segmentation labeler or the scan to CAD model aligner, please cite:

@inproceedings{dai2017scannet,
    title={ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes},
    author={Dai, Angela and Chang, Angel X. and Savva, Manolis and Halber, Maciej and Funkhouser, Thomas and Nie{\ss}ner, Matthias},
    booktitle = {Proc. Computer Vision and Pattern Recognition (CVPR), IEEE},
    year = {2017}
}

An overview of the architecture of the annotation pipeline for ScanNet is shown below.

The annotation pipeline use a MySQL database for storing annotations and an Apache Solr index for indexing assets.

See MySQL installation instructions for installing your own instance of MySQL and run the provided DDL scripts in scripts/db to create the annotation tables. Once you have setup a MySQL server, you should edit config.annDb in server/config/index.js to point to your MySQL instance.

For the scan model alignment annotation tool, we use solr to index and query ShapeNetCore and ShapeNetSem models. Please register at https://www.shapenet.org to get access to ShapeNet models.

Preparing your assets for annotation

Please see Preparing assets for annotation for how to prepare your assets for annotation.

Below are some details of the input required for annotating scans.

Semantic segmentation of scans (3D Reconstructions)

The input is a reconstruction scan .ply that has been pre-segmented.

If you have a ply format mesh of the scan, use the ScanNet Segmentator to create a vertex-based segmentation using Felzenswalb and Huttenlocher's Graph Based Image Segmentation algorithm on computed mesh normals.

Once you have those, you can update the Basic metadata json example to point to them (see formats[0].path and surfaces.file). The asset should be ready for semantic segmentation annotation.

You also need to update server/proj/scannet/tasks/segment_annotation/default.config.yml to point to the assets you want to annotate by updating the following fields:

scansToAnnotate: <filename of your file with list of scans that you people to annotate>
idPrefix: <your asset source>

Testing access: To test access, substitute for the modelId URL parameter with your assetId (<yourAssetName.assetId>) in this URL: http://localhost:8010/scans/segment-annotator-single?modelId=nyuv2.office_0001c

Exporting annotations: For exporting the semantic segmentation annotations with ssc/export-annotated-ply.js, the segment-annotations-manual field for the basic example specifies the following: segment-annotations-manual.files.segments for the over-segmentation and the segment-annotations-manual.files.segmentGroups for the annotated semantic segmentation.

Scan model alignment

For retrieving 3D CAD models and aligning them to your scan, you will first need to have a semantic segmentation. This can either be output from the manual semantic segmentation (above) or automatically generated segmentation. The semantic segmentation is specified in json format as described in Segment Annotation Formats.

Which semantic segmentation is used is specified as the 'segmentType' and segmentAnnotationId parameter to the scan model aligner. When the segmentAnnotationId is specified, it is taken directly from the database. The segmentation used is determined by the segmentType field. See Scan Model Aligner for more details on the scan model aligner UI.

A precomputed semantic segmentation can also be specified - this is also determined by the segmentType field (without any segmentAnnotationId specified). The precomputed semantic segmentation can be specified as two separate files (one for the initial over-segmentation, and one for the annotation) or as one combined file (see NYUv2 surfaces example).

The scan-model-alignments field for the basic example specifies the URLs for fetching list of annotated asset ids (annotatedAssetIds) and annotation ids for a given asset (annId) for exporting the scan model alignments.