spconv 2 now supported; doc updated; cumm is required now

DerrickXuNu · Dec 19, 2022 · d0c64ad · d0c64ad
1 parent efd7241
commit d0c64ad
Show file tree

Hide file tree

Showing 12 changed files with 112 additions and 152 deletions.
diff --git a/README.md b/README.md
@@ -18,6 +18,7 @@ paper [OPV2V.](https://arxiv.org/abs/2109.07644)
 </p>
 
 ## News:
+- 12/16/2022: Both spconv 1.2.1 and spconv 2.x are supported! 
 - 12/04/2022: The log replay tool for OPV2V is online now! With 
 this toolbox, you can 100% replay all the events in the offline dataset and add/change any sensors/groundtruth you
 want to explore the tasks that the origin dataset do not support. Check [here](logreplay/README.md) to see more details.
@@ -32,6 +33,9 @@ want to explore the tasks that the origin dataset do not support. Check [here](l
     - [X] [Pixor](https://arxiv.org/abs/1902.06326)
     - [X] [VoxelNet](https://arxiv.org/abs/1711.06396)
     - [X] [SECOND](https://www.mdpi.com/1424-8220/18/10/3337)
+- Support multiple sparse convolution versions
+  - [X] Spconv 1.2.1
+  - [X] Spconv 2.x
 - Support  SOTA multi-agent perception models:
     - [x] [Attentive Fusion [ICRA2022]](https://arxiv.org/abs/2109.07644)
     - [x] [Cooper [ICDCS]](https://arxiv.org/abs/1905.05265)
@@ -99,23 +103,23 @@ The evaluation results  will be dumped in the model directory.
 ## Benchmark and model zoo
 ### Results on OPV2V LiDAR-track ([email protected] for no-compression/ compression)
 
-|                    | Backbone   | Fusion Strategy  | Bandwidth (Megabit), <br/> before/after compression| Default Towns    |Culver City| Download |
-|--------------------| --------   | ---------------  | ---------------                | -------------    |-----------| -------- |
-| Naive Late         | PointPillar        | Late      |    **0.024**/**0.024** |   0.781/0.781        | 0.668/0.668         |    [url](https://drive.google.com/file/d/1WTKooW6k0exLqoIE5Czqy6ptycYlgKZz/view?usp=sharing)   |
-| [Cooper](https://arxiv.org/abs/1905.05265)       | PointPillar        | Early  |   7.68/7.68   | 0.800/x         | 0.696/x       | [url](https://drive.google.com/file/d/1N1p6syxGSKD18ELgtBQoSuUzR8tX1JeE/view?usp=sharing)     | 
-| [Attentive Fusion](https://arxiv.org/abs/2109.07644)         | PointPillar        | Intermediate  | 126.8/1.98   | 0.815/0.810       | 0.735/0.731        | [url](https://drive.google.com/file/d/1u4w13SDzdGq6Irh2PHxT-qIlNXRT3z6Z/view?usp=sharing)     | 
-| [F-Cooper](https://arxiv.org/abs/1909.06459)         | PointPillar        | Intermediate  | 72.08/1.12    | 0.790/0.788     | 0.728/0.726        | [url](https://drive.google.com/file/d/1CjXu9Y2ZTzJA6Oo3hnqFhbTqBVKq3mQb/view?usp=sharing)     | 
-| [V2VNet](https://arxiv.org/abs/2008.07519)         | PointPillar        | Intermediate  | 72.08/1.12    | **0.822**/0.814     | 0.734/0.729    | [url](https://drive.google.com/file/d/14xl_gNEIHcDw-SvQyO1ioQwyzGym-tKX/view?usp=sharing)     | 
-| [FPV-RCNN](https://arxiv.org/abs/2109.11615)         | PV-RCNN        | Intermediate(2 stage)  | 0.24/0.24    | 0.820/**0.820**     | **0.763**/**0.763**    | [url](https://drive.google.com/file/d/1iOVi7holJ-Cu2P3dRv5HmOWlB5lkLukJ/view)     | 
-| Naive Late         | VoxelNet        | Late  | **0.024**/**0.024**    | 0.738/0.738          | 0.588/0.588        | [url]()    |
-| Cooper    | VoxelNet        | Early   |   7.68/7.68  | 0.758/x        | 0.677/x        | [url](https://drive.google.com/file/d/14WD7iLLyyCJJ3lApbYYdr5KOUM1ACnve/view?usp=sharing)     | 
-| Attentive Fusion        | VoxelNet        | Intermediate |   576.71/1.12   | **0.864**/**0.852**        | **0.775**/**0.746**       | [url](https://drive.google.com/file/d/16q8CfcB8dS4EVhJMvvEfn0gM2ynxZB3E/view?usp=sharing)      | 
-| Naive Late         | SECOND        | Late |  **0.024**/**0.024**    |  0.775/0.775        |0.682/0.682        | [url](https://drive.google.com/file/d/1VG_FKe1mKagPVGXH7UGHpyaM5q3cxtD8/view?usp=sharing)      |
-| Cooper    | SECOND        | Early  |   7.68/7.68   |  0.813/x       |  0.738/x     | [url](https://drive.google.com/file/d/1Z9io1VNcU-urcRW8l0ogWCTVCB53mw4N/view?usp=sharing)     | 
-| Attentive         | SECOND        | Intermediate |  63.4/0.99     |   **0.826**/**0.783**     | **0.760**/**0.760**    | [url](https://drive.google.com/file/d/1zEB8EyZ0X-WQykHFOM0pVwI8jXunRz1Z/view?usp=sharing)      | 
-| Naive Late         | PIXOR        | Late |    **0.024**/**0.024** |    0.578/0.578       |  0.360/0.360      | [url]()      |
-| Cooper    | PIXOR        | Early |   7.68/7.68    |   0.678/x      | **0.558**/x      | [url](https://drive.google.com/file/d/1ZDLjtizZCuV6D92LloEPKRIw-LqxfE1j/view?usp=sharing)     | 
-| Attentive         | PIXOR        | Intermediate  |   313.75/1.22  |  **0.687**/**0.612**      | 0.546/**0.492**       | [url]()      |
+|                    | Spconv Version| Backbone   | Fusion Strategy  | Bandwidth (Megabit), <br/> before/after compression| Default Towns    |Culver City| Download |
+|--------------------| -------|--------   | ---------------  | ---------------                | -------------    |-----------| -------- |
+| Naive Late         | 1.2.1 | PointPillar        | Late      |    **0.024**/**0.024** |   0.781/0.781        | 0.668/0.668         |    [url](https://drive.google.com/file/d/1WTKooW6k0exLqoIE5Czqy6ptycYlgKZz/view?usp=sharing)   |
+| [Cooper](https://arxiv.org/abs/1905.05265)   | 1.2.1     | PointPillar        | Early  |   7.68/7.68   | 0.800/x         | 0.696/x       | [url](https://drive.google.com/file/d/1N1p6syxGSKD18ELgtBQoSuUzR8tX1JeE/view?usp=sharing)     | 
+| [Attentive Fusion](https://arxiv.org/abs/2109.07644)  | 1.2.1        | PointPillar        | Intermediate  | 126.8/1.98   | 0.815/0.810       | 0.735/0.731        | [url](https://drive.google.com/file/d/1u4w13SDzdGq6Irh2PHxT-qIlNXRT3z6Z/view?usp=sharing)     | 
+| [F-Cooper](https://arxiv.org/abs/1909.06459)   | 1.2.1       | PointPillar        | Intermediate  | 72.08/1.12    | 0.790/0.788     | 0.728/0.726        | [url](https://drive.google.com/file/d/1CjXu9Y2ZTzJA6Oo3hnqFhbTqBVKq3mQb/view?usp=sharing)     | 
+| [V2VNet](https://arxiv.org/abs/2008.07519)  | 1.2.1        | PointPillar        | Intermediate  | 72.08/1.12    | **0.822**/0.814     | 0.734/0.729    | [url](https://drive.google.com/file/d/14xl_gNEIHcDw-SvQyO1ioQwyzGym-tKX/view?usp=sharing)     | 
+| [FPV-RCNN](https://arxiv.org/abs/2109.11615)   | 1.2.1       | PV-RCNN        | Intermediate(2 stage)  | 0.24/0.24    | 0.820/**0.820**     | **0.763**/**0.763**    | [url](https://drive.google.com/file/d/1iOVi7holJ-Cu2P3dRv5HmOWlB5lkLukJ/view)     | 
+| Naive Late    | 1.2.1      | VoxelNet        | Late  | **0.024**/**0.024**    | 0.738/0.738          | 0.588/0.588        | [url]()    |
+| Cooper   | 1.2.1  | VoxelNet        | Early   |   7.68/7.68  | 0.758/x        | 0.677/x        | [url](https://drive.google.com/file/d/14WD7iLLyyCJJ3lApbYYdr5KOUM1ACnve/view?usp=sharing)     | 
+| Attentive Fusion    | 1.2.1     | VoxelNet        | Intermediate |   576.71/1.12   | **0.864**/**0.852**        | **0.775**/**0.746**       | [url](https://drive.google.com/file/d/16q8CfcB8dS4EVhJMvvEfn0gM2ynxZB3E/view?usp=sharing)      | 
+| Naive Late    | 1.2.1      | SECOND        | Late |  **0.024**/**0.024**    |  0.775/0.775        |0.682/0.682        | [url](https://drive.google.com/file/d/1VG_FKe1mKagPVGXH7UGHpyaM5q3cxtD8/view?usp=sharing)      |
+| Cooper  | 1.2.1   | SECOND        | Early  |   7.68/7.68   |  0.813/x       |  0.738/x     | [url](https://drive.google.com/file/d/1Z9io1VNcU-urcRW8l0ogWCTVCB53mw4N/view?usp=sharing)     | 
+| Attentive     | 1.2.1     | SECOND        | Intermediate |  63.4/0.99     |   **0.826**/**0.783**     | **0.760**/**0.760**    | [url](https://drive.google.com/file/d/1zEB8EyZ0X-WQykHFOM0pVwI8jXunRz1Z/view?usp=sharing)      | 
+| Naive Late    | 1.2.1      | PIXOR        | Late |    **0.024**/**0.024** |    0.578/0.578       |  0.360/0.360      | [url]()      |
+| Cooper | 1.2.1    | PIXOR        | Early |   7.68/7.68    |   0.678/x      | **0.558**/x      | [url](https://drive.google.com/file/d/1ZDLjtizZCuV6D92LloEPKRIw-LqxfE1j/view?usp=sharing)     | 
+| Attentive   | 1.2.1       | PIXOR        | Intermediate  |   313.75/1.22  |  **0.687**/**0.612**      | 0.546/**0.492**       | [url]()      |
 
 **Note**: 
 * We suggest using **PointPillar** as the backbone when you are creating your method and try to compare with

diff --git a/docs/md_files/installation.md b/docs/md_files/installation.md
@@ -39,17 +39,24 @@ If conda install failed,  install through pip
 pip install -r requirements.txt
 ```
 
-### 2. Pytorch Installation (>=1.8)
+### 2. Pytorch Installation (>=1.8, tested on 1.8-1.12.0)
 Go to https://pytorch.org/ to install pytorch cuda version.
 
-### 3. Spconv (1.2.1 requred)
-OpenCOOD currently uses the old spconv version to generate voxel features. We will 
-upgrade to spconv 2.0 in the short future. To install spconv 1.2.1, please follow the guide in https://github.com/traveller59/spconv/tree/v1.2.1.
+### 3. Spconv (1.2.1 or 2.x)
+OpenCOOD support both spconv 1.2.1 and 2.x to generate voxel features. 
 
+To install spconv 1.2.1, please follow the guide in https://github.com/traveller59/spconv/tree/v1.2.1.
+
+To install spconv 2.x, please run the following commands (if you are using cuda 11.3):
+```python
+pip install spconv-cu113
+```
 #### Tips for installing spconv 1.2.1:
 1. make sure your cmake version >= 3.13.2
 2. CUDNN and CUDA runtime library (use `nvcc --version` to check) needs to be installed on your machine.
 
+
+
 ### 4. Bbx IOU cuda version compile
 Install bbx nms calculation cuda version
 

diff --git a/environment.yml b/environment.yml
@@ -14,3 +14,4 @@ dependencies:
     - tensorboardX
     - shapely
     - einops
+    - cumm
diff --git a/opencood/data_utils/pre_processor/sp_voxel_preprocessor.py b/opencood/data_utils/pre_processor/sp_voxel_preprocessor.py
@@ -9,7 +9,7 @@
 
 import numpy as np
 import torch
-
+from cumm import tensorview as tv
 from opencood.data_utils.pre_processor.base_preprocessor import \
     BasePreprocessor
 
@@ -18,11 +18,14 @@ class SpVoxelPreprocessor(BasePreprocessor):
     def __init__(self, preprocess_params, train):
         super(SpVoxelPreprocessor, self).__init__(preprocess_params,
                                                   train)
+        self.spconv = 1
         try:
+            # spconv v1.x
             from spconv.utils import VoxelGeneratorV2 as VoxelGenerator
         except:
-            from spconv.utils import VoxelGenerator
-
+            # spconv v2.x
+            from spconv.utils import Point2VoxelCPU3d as VoxelGenerator
+            self.spconv = 2
         self.lidar_range = self.params['cav_lidar_range']
         self.voxel_size = self.params['args']['voxel_size']
         self.max_points_per_voxel = self.params['args']['max_points_per_voxel']
@@ -37,23 +40,41 @@ def __init__(self, preprocess_params, train):
         self.grid_size = np.round(grid_size).astype(np.int64)
 
         # use sparse conv library to generate voxel
-        self.voxel_generator = VoxelGenerator(
-            voxel_size=self.voxel_size,
-            point_cloud_range=self.lidar_range,
-            max_num_points=self.max_points_per_voxel,
-            max_voxels=self.max_voxels
-        )
+        if self.spconv == 1:
+            self.voxel_generator = VoxelGenerator(
+                voxel_size=self.voxel_size,
+                point_cloud_range=self.lidar_range,
+                max_num_points=self.max_points_per_voxel,
+                max_voxels=self.max_voxels
+            )
+        else:
+            self.voxel_generator = VoxelGenerator(
+                vsize_xyz=self.voxel_size,
+                coors_range_xyz=self.lidar_range,
+                max_num_points_per_voxel=self.max_points_per_voxel,
+                num_point_features=4,
+                max_num_voxels=self.max_voxels
+            )
 
     def preprocess(self, pcd_np):
         data_dict = {}
-        voxel_output = self.voxel_generator.generate(pcd_np)
+        if self.spconv == 1:
+            voxel_output = self.voxel_generator.generate(pcd_np)
+        else:
+            pcd_tv = tv.from_numpy(pcd_np)
+            voxel_output = self.voxel_generator.point_to_voxel(pcd_tv)
         if isinstance(voxel_output, dict):
             voxels, coordinates, num_points = \
                 voxel_output['voxels'], voxel_output['coordinates'], \
                 voxel_output['num_points_per_voxel']
         else:
             voxels, coordinates, num_points = voxel_output
 
+        if self.spconv == 2:
+            voxels = voxels.numpy()
+            coordinates = coordinates.numpy()
+            num_points = num_points.numpy()
+
         data_dict['voxel_features'] = voxels
         data_dict['voxel_coords'] = coordinates
         data_dict['voxel_num_points'] = num_points

diff --git a/opencood/loss/ciassd_loss.py b/opencood/loss/ciassd_loss.py
@@ -114,7 +114,7 @@ def forward(self, output_dict, label_dict):
 
         return loss
 
-    def logging(self, epoch, batch_id, batch_len, writer):
+    def logging(self, epoch, batch_id, batch_len, writer, pbar=None):
         """
         Print out  the loss function for current iteration.
 

diff --git a/opencood/loss/fpvrcnn_loss.py b/opencood/loss/fpvrcnn_loss.py
@@ -85,7 +85,7 @@ def forward(self, output_dict, label_dict):
 
         return loss
 
-    def logging(self, epoch, batch_id, batch_len, writer):
+    def logging(self, epoch, batch_id, batch_len, writer, pbar=None):
         """
         Print out  the loss function for current iteration.
 

diff --git a/opencood/loss/pixor_loss.py b/opencood/loss/pixor_loss.py
@@ -66,7 +66,7 @@ def forward(self, output_dict, target_dict):
 
         return total_loss
 
-    def logging(self, epoch, batch_id, batch_len, writer):
+    def logging(self, epoch, batch_id, batch_len, writer, pbar=None):
         """
         Print out  the loss function for current iteration.
 
@@ -85,10 +85,16 @@ def logging(self, epoch, batch_id, batch_len, writer):
         reg_loss = self.loss_dict['reg_loss']
         cls_loss = self.loss_dict['cls_loss']
 
-        print("[epoch %d][%d/%d], || Loss: %.4f || cls Loss: %.4f"
-              " || reg Loss: %.4f" % (
-                  epoch, batch_id + 1, batch_len,
-                  total_loss.item(), cls_loss.item(), reg_loss.item()))
+        if pbar is None:
+            print("[epoch %d][%d/%d], || Loss: %.4f || Conf Loss: %.4f"
+                " || Loc Loss: %.4f" % (
+                    epoch, batch_id + 1, batch_len,
+                    total_loss.item(), cls_loss.item(), reg_loss.item()))
+        else:
+            pbar.set_description("[epoch %d][%d/%d], || Loss: %.4f || Conf Loss: %.4f"
+                  " || Loc Loss: %.4f" % (
+                      epoch, batch_id + 1, batch_len,
+                      total_loss.item(), cls_loss.item(), reg_loss.item()))
 
         writer.add_scalar('Regression_loss', reg_loss.item(),
                           epoch * batch_len + batch_id)

diff --git a/opencood/loss/point_pillar_loss.py b/opencood/loss/point_pillar_loss.py
@@ -212,7 +212,7 @@ def add_sin_difference(boxes1, boxes2, dim=6):
         return boxes1, boxes2
 
 
-    def logging(self, epoch, batch_id, batch_len, writer):
+    def logging(self, epoch, batch_id, batch_len, writer, pbar=None):
         """
         Print out  the loss function for current iteration.
 
@@ -230,13 +230,19 @@ def logging(self, epoch, batch_id, batch_len, writer):
         total_loss = self.loss_dict['total_loss']
         reg_loss = self.loss_dict['reg_loss']
         conf_loss = self.loss_dict['conf_loss']
+        if pbar is None:
+            print("[epoch %d][%d/%d], || Loss: %.4f || Conf Loss: %.4f"
+                " || Loc Loss: %.4f" % (
+                    epoch, batch_id + 1, batch_len,
+                    total_loss.item(), conf_loss.item(), reg_loss.item()))
+        else:
+            pbar.set_description("[epoch %d][%d/%d], || Loss: %.4f || Conf Loss: %.4f"
+                  " || Loc Loss: %.4f" % (
+                      epoch, batch_id + 1, batch_len,
+                      total_loss.item(), conf_loss.item(), reg_loss.item()))
 
-        print("[epoch %d][%d/%d], || Loss: %.4f || Conf Loss: %.4f"
-              " || Loc Loss: %.4f" % (
-                  epoch, batch_id + 1, batch_len,
-                  total_loss.item(), conf_loss.item(), reg_loss.item()))
 
         writer.add_scalar('Regression_loss', reg_loss.item(),
                           epoch*batch_len + batch_id)
         writer.add_scalar('Confidence_loss', conf_loss.item(),
-                          epoch*batch_len + batch_id)
+                          epoch*batch_len + batch_id)
diff --git a/opencood/loss/voxel_net_loss.py b/opencood/loss/voxel_net_loss.py
-Original file line number
+Diff line change
@@ Expand Up / @@ -14,3 +14,4 @@ dependencies: @@
         - tensorboardX
         - shapely
         - einops
+        - cumm