Interactive Segmentation with ClickSEG

Overview • Pretrained models • How to Run • Controls • Acknowledgment

Overview

Application key points:

• Manually selected ROI
• Deploy on GPU(faster) or CPU(slower)
• Accurate predictions in most cases
• Correct prediction interactively with red and green clicks
• Select one of 10 pretrained models
• Models are class agnostic, you can segment any object from any domain

ClickSeg Interactive segmentation algorithms allow users to explicitly control the predictions using interactive input at several iterations, in contrast to common semantic and instance segmentation algorithms that can only input an image and output a segmentation mask in one pass. Such interaction makes it possible to select an object of interest and correct prediction errors.

Besides segmenting new objects, proposed method allows to correct external masks, e.g. produced by other instance or semantic segmentation models. A user can fix false negative and false positive regions with positive (green) and negative (red) clicks, respectively.

Pretrained models

CDNet: Conditional Diffusion for Interative Segmentation (ICCV2021)

config:

Input Size: 384 x 384
Previous Mask: No
Iterative Training: No

Train Dataset	Model	GrabCut	Berkeley	Pascal VOC	COCO MVal	SBD	DAVIS	DAVIS585 from zero	DAVIS585 from init
		NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%
SBD	ResNet34 (89.72 MB)	1.86/2.18	1.95/3.27	3.61/4.51	4.13/5.88	5.18/7.89	5.00/6.89	6.68/9.59	5.04/7.06
COCO+ LVIS	ResNet34 (89.72 MB)	1.40/1.52	1.47/2.06	2.74/3.30	2.51/3.88	4.30/7.04	4.27/5.56	4.86/7.37	4.21/5.92

FocalClick: Towards Practical Interactive Image Segmentation (CVPR2022)

config:

S1 version: coarse segmentator input size 128x128; refiner input size 256x256.  
S2 version: coarse segmentator input size 256x256; refiner input size 256x256.  
Previous Mask: Yes
Iterative Training: Yes

Train Dataset	Model	GrabCut	Berkeley	Pascal VOC	COCO MVal	SBD	DAVIS	DAVIS585 from zero	DAVIS585 from init
		NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%
COCO+ LVIS	SegFormerB0-S1 (14.38 MB)	1.60/1.86	2.05/3.29	3.54/4.22	3.08/4.21	4.98/7.60	5.13/7.42	6.21/9.06	2.63/3.69
COCO+ LVIS	SegFormerB0-S2 (14.38 MB)	1.40/1.66	1.59/2.27	2.97/3.52	2.65/3.59	4.56/6.86	4.04/5.49	5.01/7.22	2.21/3.08
COCO+ LVIS	SegFormerB3-S2 (174.56 MB)	1.44/1.50	1.55/1.92	2.46/2.88	2.32/3.12	3.53/5.59	3.61/4.90	4.06/5.89	2.00/2.76
Combined Datasets	SegFormerB3-S2 (174.56 MB)	1.22/1.26	1.35/1.48	2.54/2.96	2.51/3.33	3.70/5.84	2.92/4.52	3.98/5.75	1.98/2.72

Efficient Baselines using MobileNets and PPLCNets

config:

Input Size: 384x384.
Previous Mask: Yes
Iterative Training: Yes

Train Dataset	Model	GrabCut	Berkeley	Pascal VOC	COCO MVal	SBD	DAVIS	DAVIS585 from zero	DAVIS585 from init
		NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%	NoC 85/90%
COCO+ LVIS	MobileNetV2 (7.5 MB)	1.82/2.02	1.95/2.69	2.97/3.61	2.74/3.73	4.44/6.75	3.65/5.81	5.25/7.28	2.15/3.04
COCO+ LVIS	PPLCNet (11.92 MB)	1.74/1.92	1.96/2.66	2.95/3.51	2.72/3.75	4.41/6.66	4.40/5.78	5.11/7.28	2.03/2.90
Combined Datasets	MobileNetV2 (7.5 MB)	1.50/1.62	1.62/2.25	3.00/3.61	2.80/3.96	4.66/7.05	3.59/5.24	5.05/7.12	2.06/2.97
Combined Datasets	PPLCNet (11.92 MB)	1.46/1.66	1.63/1.99	2.88/3.44	2.75/3.89	4.44/6.74	3.65/5.34	5.02/6.98	1.96/2.81

Prediction preview (SegformerB3-S2):

How to Run

1. Start the application from Ecosystem.

2. Select the pretrained model and deploy it on your device by clicking Serve button.

3. You'll see Model has been successfully loaded message indicating that the application has been successfully started and you can work with it from now on.

Controls

Key	Description
Left Mouse Button	Place a positive click
Shift + Left Mouse Button	Place a negative click
Scroll Wheel	Zoom an image in and out
Right Mouse Button + Move Mouse	Move an image
Space	Finish the current object mask
Shift + H	Higlight instances with random colors
Ctrl + H	Hide all labels

— Auto add positivie point to rectangle button (ON by default for SmartTool apps)

— SmartTool selector button, switch between SmartTool apps and models

Acknowledgment

This app is based on the great work ClickSEG: A Codebase for Click-Based Interactive Segmentation github.