Dropout

[patriciaaa82]

Hi,

In the "dropout from scratch" chapter, there is no significative difference between adding or not adding dropout. See metrics below:

with drop_prob=0.5:
Epoch 0. Loss: 0.7281168998689048, Train_acc 0.84805, Test_acc 0.8542
Epoch 1. Loss: 0.3915857726824017, Train_acc 0.92135, Test_acc 0.9232
Epoch 2. Loss: 0.28589152397083284, Train_acc 0.94455, Test_acc 0.9453
Epoch 3. Loss: 0.24532910775307348, Train_acc 0.95731664, Test_acc 0.9544
Epoch 4. Loss: 0.2158778892831953, Train_acc 0.96416664, Test_acc 0.9577
Epoch 5. Loss: 0.18741829898958554, Train_acc 0.9690667, Test_acc 0.9629
Epoch 6. Loss: 0.17892182123475542, Train_acc 0.97436666, Test_acc 0.9669
Epoch 7. Loss: 0.16173455391374822, Train_acc 0.97606665, Test_acc 0.9697
Epoch 8. Loss: 0.14435731281497297, Train_acc 0.97805, Test_acc 0.9715
Epoch 9. Loss: 0.13987530898537387, Train_acc 0.9802333, Test_acc 0.9728

with drop_prob=0:
Epoch 0. Loss: 0.5414872900664346, Train_acc 0.8639, Test_acc 0.8664
Epoch 1. Loss: 0.2935891257485201, Train_acc 0.91835, Test_acc 0.9187
Epoch 2. Loss: 0.1943884894438662, Train_acc 0.94588333, Test_acc 0.9443
Epoch 3. Loss: 0.16208293540151902, Train_acc 0.9608, Test_acc 0.9575
Epoch 4. Loss: 0.12486255719026844, Train_acc 0.96991664, Test_acc 0.965
Epoch 5. Loss: 0.1082405275668826, Train_acc 0.97546667, Test_acc 0.9675
Epoch 6. Loss: 0.08227119813514386, Train_acc 0.97891665, Test_acc 0.9709
Epoch 7. Loss: 0.07985300555672155, Train_acc 0.98141664, Test_acc 0.9731
Epoch 8. Loss: 0.06968508473642689, Train_acc 0.98436666, Test_acc 0.9736
Epoch 9. Loss: 0.05624080957929232, Train_acc 0.98793334, Test_acc 0.9774

It would be nice to see an improvement when using dropout wrt. not using it. Or perhaps I'm missing something?
Best,
Patricia

[theoneandonlywoj]

This is just a simple example. In real projects dropout is important.