Too long training time

[surheaven]

I train on single NVIDIA RTX 3090 GPU. The data is only 270 frames, 1400 epochs run for five days. But github and the supplemental files say it only takes 48h, why is that? I see in the code that the maximum epoch of train is 8000. Is it necessary to run so many epochs.

[MoyGcc]

What about your iters per second? It's about 6 iter/s on my local PC and around 3.5 iter/s on my institutional server. In both cases, 48 hours should be enough for a decent result.

[surheaven]

Epoch 2449: 100%|██████▉| 259/260 [02:15<00:00, 1.91it/s, loss=0.0283, v_num=iox1, rgb_loss=0.0277, eikonal_loss=0.00417, bce_loss=0.0273, opacity_sparse_loss=0.000,
Epoch 2868: 7%| | 19/259 [00:11<02:26, 1.63it/s, loss=0.0215, v_num=iox1, rgb_loss=0.0211, eikonal_loss=0.00268, bce_loss=0.0202, opacity_sparse_loss=0.000, in_shape_loss=0.0126]Epoch 2973: 84%|▊| 218/259 [01:57<00:22, 1.86it/s, loss=0.0263, v_num=iox1, rgb_loss=0.0258, eikonal_loss=0.00245, bce_loss=0.0346, opacity_sparse_loss=0.000, in_shape_loss=0.0011

[hakanErgin]

I am testing the preprocessed demo on my work laptop with quadro RTX 3000 6gb. had to change pixel_per_batch to 256

Extremely slow, but I think I may have a misconfiguration or something.