No Scalar Data Found

[I-CANT-CODE]

I am having issues with TensorBoard.

I am basically training the network 10 times in a for loop, each iteration training on a different version of the same data set, like such:

for j in range(1,10):

    folder_name = path to folder [j] + "/*.png"
    checkpoint_path = "checkpoints_"+str(j)
    
   data_provider = image_util.ImageDataProvider(folder_name,data_suffix=".png",mask_suffix = "_LABEL.png")
    data_provider.n_class = 3

    net[j] = unet.Unet(layers=3,cost_kwargs = {"class_weights":[.4,.3,.3]}, features_root=128, channels=1, n_class=3)

    trainer[j] = unet.Trainer(net[j],optimizer = "adam", opt_kwargs = dict(learning_rate=.00001))

    path[j] = trainer[j].train(data_provider, output_path=checkpoint_path, training_iters = 32, epochs=2)

but for some reason in TensorBoard only half of these sessions' scalar data shows up on the graphs. Why is this?

Edit: so the data does eventually show up in TensorBoard but it takes around 20 minutes AFTER it finishes training each iteration to finally show up in the graph. Any Idea why?

[I-CANT-CODE]

OK, issue was resolved by using a time.wait(5) at the end of the loop. Maybe some strange non-blocking thread that needed time to run before moving on to the next training sesh

[I-CANT-CODE]

Ok issue was not resolved, TensorBoard only works sometimes after all loops are finished, still shows up empty during training for some reason.