The loss becomes negative

[FiReTiTi]

Hi,
I jut ran a CNN built with Keras on a big training set, and I has weird loss values at each epoch (see below):
66496/511502 [==>...........................] - ETA: 63s - loss: 8.2800
66528/511502 [==>...........................] - ETA: 63s - loss: -204433556137039776.0000

345664/511502 [===================>..........] - ETA: 23s - loss: 8.3174
345696/511502 [===================>..........] - ETA: 23s - loss: -39342531075525840.0000

214080/511502 [===========>..................] - ETA: 41s - loss: 8.3406
214112/511502 [===========>..................] - ETA: 41s - loss: -63520753730220536.0000

How is that possible? The loss becomes suddenly to big and the value gets bigger than the double encoding?
Is there a way to avoid it?

Regards,

[the-moliver]

Do you really think that's enough information for anyone to be able to answer your question?

[fchollet]

The loss is just a scalar that you are trying to minimize. It's not supposed to be positive! For instance a cosine proximity loss will usually be negative (trying to make proximity as high as possible by minimizing a negative scalar).

[FiReTiTi]

Hi, thank you for your answers.
The last layer is a dense layer with a sigmoid, so the value should not be negative:
model.add(Dense(1)) model.add(Activation('sigmoid'))
What really surprise me is that from one batch to the next, there is such a fall.
@the-moliver: which information would you need?

[ymcui]

What is your training objective, binary_crossentropy or others?
And if you are using GPU to train your network, the datatype should be in float32 (theano restriction).

[FiReTiTi]

Hi, thank you for your help.
Yes, the training objective is binary_crossentropy.
And yes, all my data are already float32, I made sure of that.

[ymcui]

@FiReTiTi
I don't think binary_crossentropy could return the negative values.
In Theano backend,

crossentropy(t,o) = -(t*log(o) + (1 - t)*log(1 - o)).

t and o are all in range of [0,1], making the whole equation non-negative.
your output log more seems like an overflow.
Maybe you can check on your data ? or you can extract a proportion of data to test if the model still gives the negative loss.

66496/511502 [==>...........................] - ETA: 63s - loss: 8.2800
66528/511502 [==>...........................] - ETA: 63s - loss: -204433556137039776.0000

[FiReTiTi]

If the loss cannot be negative, then does it mean it goes over the encoding limits and then it loops back into the negative values?

[FiReTiTi]

I am still working on the same data, and here is an other weird thing:
Epoch 140/3000
5s - loss: 0.5968 - val_loss: 0.4191
Epoch 141/3000
5s - loss: 0.5974 - val_loss: 0.4556
Epoch 142/3000
5s - loss: 0.5979 - val_loss: 0.4382
Epoch 143/3000
5s - loss: 6.0467 - val_loss: 11.1324
Epoch 144/3000
5s - loss: 7.7176 - val_loss: 11.1324
Epoch 145/3000
5s - loss: 7.7176 - val_loss: 11.1324
Epoch 146/3000
5s - loss: 7.7176 - val_loss: 11.1324
And nothing changes during the next 2850 epochs, perfectly identical.

[philipperemy]

@FiReTiTi please give more information about your model if you want help on that.

In your last case, your optimiser is likely to be stuck in a local minima. That could explain why it remains identical during all your next iterations.

[FiReTiTi]

Here is the model:
model = Sequential()
model.add(Convolution2D(8, 7, 7, border_mode='valid', input_shape=(1, 31, 31), activation='tanh'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(16, 5, 5, border_mode='valid', activation='tanh'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(32, 3, 3, border_mode='valid', activation='relu'))
model.add(Convolution2D(65, 1, 1, border_mode='valid', activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(23))
model.add(Activation('tanh'))
model.add(Dropout(0.1))
model.add(Dense(11))
model.add(Activation('sigmoid'))
model.add(Dropout(0.1))
model.add(Dense(1))
model.add(Activation('sigmoid'))
optimizer = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='binary_crossentropy', optimizer=optimizer)
model.fit(dataset, labels, batch_size=batch_size, nb_epoch=nb_epoch, shuffle=True, validation_split=0.1, verbose=2)

The dataset contains 70 000 images of size 31x31.
What I don't understand is why there is this sudden jump of the loss and then the loss is stuck.

[sunshineatnoon]

@FiReTiTi May I ask how did you solve the negative loss problem? I ran into the same problem, my loss is a customed loss with a bunch of mse, so it shouldn't be negative either. It looks like this:

Epoch 1/15
   32/33102 [..............................] - ETA: 15552s - loss: -88028794.750
   48/33102 [..............................] - ETA: 11176s - loss: -1419246161.8
   64/33102 [..............................] - ETA: 8987s - loss: -13590295485.3
   80/33102 [..............................] - ETA: 7674s - loss: -107586018455.
   96/33102 [..............................] - ETA: 6797s - loss: -661847078867.
  112/33102 [..............................] - ETA: 6172s - loss: -3960883097561
  128/33102 [..............................] - ETA: 5702s - loss: -3047999712303
  144/33102 [..............................] - ETA: 5337s - loss: -2318531227797
  320/33102 [..............................] - ETA: 3720s - loss: -1825597231654244712448.0000

[FiReTiTi]

I haven't, it still happen time to time :-(
Use smaller NNs seems to reduce the phenomenon.

[sunshineatnoon]

@FiReTiTi Thanks for your reply. I found my problem, I use a custom loss and accidentally put y_pred and y_true in the wrong order when passing them to my loss function, so maybe it's not the same reason in your case.

[philipperemy]

@sunshineatnoon so you were using a non symmetric loss function like the cross entropy?