Low performance on the nuscenes validation set

[lhiceu]

Hello @JialianW
I test the performance of the model you provided (nuscenes.pth)on the nuscenes validation set following the readme. But I got low performance which is different from the results in the paper.

### Final results ###

Per-class results:
		AMOTA	AMOTP	RECALL	MOTAR	GT	MOTA	MOTP	MT	ML	FAF	TP	FP	FN	IDS	FRAG	TID	LGD
bicycle 	0.000	1.495	0.552	0.000	1993	0.000	0.793	27	38	683.2	864	35219	893	236	90	1.07	1.82
bus     	0.062	1.420	0.224	0.394	2112	0.088	0.811	9	87	17.5	470	285	1638	4	19	0.70	4.02
car     	0.088	0.893	0.114	0.520	58317	0.059	0.486	133	3379	55.1	6582	3158	51646	89	166	2.04	4.35
motorcy 	0.000	1.579	0.545	0.000	1977	0.000	0.851	29	24	494.5	819	22430	899	259	118	1.14	2.32
pedestr 	0.000	1.394	0.639	0.000	25423	0.000	0.873	486	198	2454.6	10887	143963	9168	5368	1778	0.60	1.73
trailer 	0.000	1.614	0.543	0.000	2425	0.000	1.084	26	29	2907.8	869	151552	1108	448	169	1.32	2.16
truck   	0.003	1.341	0.093	0.139	9650	0.013	0.939	23	509	21.2	884	761	8756	10	30	0.70	3.08

Aggregated results:
AMOTA	0.022
AMOTP	1.391
RECALL	0.387
MOTAR	0.150
GT	14556
MOTA	0.023
MOTP	0.834
MT	733
ML	4264
FAF	947.7
TP	21375
FP	357368
FN	74108
IDS	6414
FRAG	2370
TID	1.08
LGD	2.78
Eval time: 3056.8s 

I also tested the results from CenterTrack and got good performance. So I ruled out the issues of data preparation and nuScenes dataset API.
What anything else should I do to get the same performance as you provided? Thank you.

[lhiceu]

Details about opt.txt :

==> commit hash: b'0443c36\n'
==> torch version: 1.3.1
==> cudnn version: 7603
==> Cmd:
['test.py', 'tracking,ddd', '--exp_id', 'nuScenes_3Dtracking', '--dataset', 'nuscenes', '--pre_hm', '--track_thresh', '0.1', '--gpus', '0', '--inference', '--load_model', '../models/nuscenes.pth', '--clip_len', '2', '--trades']
==> Opt:
  K: 100
  add_05: False
  amodel_offset_weight: 1
  arch: dla_34
  aug_rot: 0
  backbone: dla34
  batch_size: 32
  box_nms: -1
  chunk_sizes: [32]
  clip_len: 2
  custom_dataset_ann_path: 
  custom_dataset_img_path: 
  data_dir: /media/he/Disk/TraDeS/src/lib/../../data
  dataset: nuscenes
  dataset_version: 
  debug: 0
  debug_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking/debug
  debugger_theme: white
  deform_kernel_size: 3
  demo: 
  dense_reg: 1
  dep_weight: 1
  depth_scale: 1
  dim_weight: 1
  dla_node: dcn
  down_ratio: 4
  efficient_level: 0
  embedding: False
  eval_val: False
  exp_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd
  exp_id: nuScenes_3Dtracking
  fix_res: True
  fix_short: -1
  flip: 0.5
  flip_test: False
  fp_disturb: 0
  gpus: [0]
  gpus_str: 0
  head_conv: {'hm': [256], 'reg': [256], 'wh': [256], 'dep': [256], 'rot': [256], 'dim': [256], 'amodel_offset': [256]}
  head_kernel: 3
  heads: {'hm': 10, 'reg': 2, 'wh': 2, 'dep': 1, 'rot': 8, 'dim': 3, 'amodel_offset': 2}
  hm_disturb: 0
  hm_hp_weight: 1
  hm_weight: 1
  hp_weight: 1
  hungarian: False
  ignore_loaded_cats: []
  inference: True
  input_h: 448
  input_res: 800
  input_w: 800
  keep_res: False
  kitti_split: 3dop
  load_model: ../models/nuscenes.pth
  load_results: 
  lost_disturb: 0
  lr: 0.000125
  lr_step: [60]
  ltrb: False
  ltrb_amodal: False
  ltrb_amodal_weight: 0.1
  ltrb_weight: 0.1
  map_argoverse_id: False
  master_batch_size: 32
  max_age: -1
  max_frame_dist: 3
  model_output_list: False
  msra_outchannel: 256
  nID: -1
  neck: dlaup
  new_thresh: 0.1
  nms: False
  no_color_aug: False
  no_pause: False
  no_pre_img: False
  no_repeat: True
  non_block_test: False
  not_cuda_benchmark: False
  not_idaup: False
  not_max_crop: False
  not_prefetch_test: False
  not_rand_crop: False
  not_set_cuda_env: False
  not_show_bbox: False
  not_show_number: False
  num_classes: 10
  num_epochs: 70
  num_head_conv: 1
  num_iters: -1
  num_layers: 101
  num_stacks: 1
  num_workers: 4
  nuscenes_att: False
  nuscenes_att_weight: 1
  off_weight: 1
  optim: adam
  out_thresh: 0.1
  output_h: 112
  output_res: 200
  output_w: 200
  overlap_thresh: -1
  pad: 31
  pre_hm: True
  pre_img: True
  pre_thresh: 0.1
  print_iter: 0
  prior_bias: -4.6
  public_det: False
  qualitative: False
  reg_loss: l1
  reset_hm: False
  resize_video: False
  resume: False
  reuse_hm: False
  root_dir: /media/he/Disk/TraDeS/src/lib/../..
  rot_weight: 1
  rotate: 0
  same_aug_pre: False
  save_all: False
  save_dir: /media/he/Disk/TraDeS/src/lib/../../exp/tracking,ddd/nuScenes_3Dtracking
  save_framerate: 30
  save_img_suffix: 
  save_imgs: []
  save_point: [90]
  save_results: False
  save_video: False
  scale: 0
  seed: 317
  seg: False
  seg_feat_channel: 8
  shift: 0
  show_track_color: True
  skip_first: -1
  tango_color: False
  task: tracking,ddd
  test: False
  test_dataset: nuscenes
  test_focal_length: -1
  test_scales: [1.0]
  track_thresh: 0.1
  tracking: True
  tracking_weight: 1
  trades: True
  trainval: False
  transpose_video: False
  use_kpt_center: False
  use_loaded_results: False
  val_intervals: 10000
  velocity: False
  velocity_weight: 1
  video_h: 512
  video_w: 512
  vis_gt_bev: 
  vis_thresh: 0.3
  weights: {'hm': 1, 'reg': 1, 'wh': 0.1, 'dep': 1, 'rot': 1, 'dim': 1, 'amodel_offset': 1, 'cost_volume': 1.0}
  wh_weight: 0.1
  window_size: 7
  zero_pre_hm: False
  zero_tracking: False