fcnxs example validation accuracy does not change

[sinaj]

Hello,
I am trying to use the example code for image segmentation. I simply changed the train.lst and val.lst to replace my images, but when I run it, although the train accuracy changes, the validation accuracy does not change at all (it is always 0.730370). Is there any potential situation that it could happen?

Here is the log:

INFO:root:Start training with cpu(0)
INFO:root:Saved checkpoint to "model_pascal/FCN32s_VGG16-0001.params"
INFO:root:--->Epoch[0] Train-accuracy=0.500000
INFO:root: in eval process...
INFO:root:batch[4] Validation-accuracy=0.730370
INFO:root:Saved checkpoint to "model_pascal/FCN32s_VGG16-0002.params"
INFO:root:--->Epoch[1] Train-accuracy=0.500000
INFO:root: in eval process...
INFO:root:batch[4] Validation-accuracy=0.730370
INFO:root:Saved checkpoint to "model_pascal/FCN32s_VGG16-0003.params"
INFO:root:--->Epoch[2] Train-accuracy=0.750000
INFO:root: in eval process...
INFO:root:batch[4] Validation-accuracy=0.730370
INFO:root:Saved checkpoint to "model_pascal/FCN32s_VGG16-0004.params"
INFO:root:--->Epoch[3] Train-accuracy=1.000000
INFO:root: in eval process...
INFO:root:batch[4] Validation-accuracy=0.730370
INFO:root:Saved checkpoint to "model_pascal/FCN32s_VGG16-0005.params"

[xiaowei-hu]

during training, your labels may be all equaled to zeros, because 0.730370 is the background's ratio

[tornadomeet]

@sinaj ,you may check you data, i think it may be something wrong there, especially the label.

[liangfu]

@sinaj any update? as I was having exactly the same problem, even after checking the label.

[Hjy20255]

@sinaj i have same error,did you solve it

[sinaj]

@Hjy20255 No, could not solve it.

[liangfu]

I was having similar problem when porting fcn8s from vgg16 to resnet, changing use_global_stats from True to False in symbol_fcnxs.py file solved the problem for me. see my local repository https://github.com/liangfu/mx-fcn .

[Hjy20255]

[

]

[Hjy20255]

sorry,i can not find use_global_stats in symbol_fcnxs.py
and so i add use_global_stats=False in symbol_fcnxs.py
but val acc all the same value!
my experiment ：
i train my own dataset using fcn—xs
for label images： The real color diagram becomes the index diagram
for example（0，0，0）--->0 ,USE the index diagram it can run successfly. BUT USE real color diagram meet errors.

[liangfu]

Try debug the output of each layer by adding following code before executing forward pass (in this case, the code exists in solver.py file)

import time
def stat_helper(name, array):
  """wrapper for executor callback"""
  import ctypes
  from mxnet.ndarray import NDArray
  from mxnet.base import NDArrayHandle, py_str
  array = ctypes.cast(array, NDArrayHandle)
  array = NDArray(array, writable=False).asnumpy()
  # array.wait_to_read()
  print (name, array.shape, np.mean(array), np.std(array), ('%.1fms' % (float(time.time()-stat_helper.start_time)*1000)))
  # print (name, array.shape, ('%.1fms' % (float(time.time()-stat_helper.start_time)*1000)))
  stat_helper.start_time=time.time()
  stat_helper.start_time=float(time.time())
  self.executor.set_monitor_callback(stat_helper)

And there should be no numerical overflow (displayed as inf for example) in any layer.