Verification on Training Details #8
Description
Hi,
I am trying to reproduce the CIFAR-10 experiment with wide resnet. However I was only able to achieve 2.96% Test Error with SAM and 3.5% Test Error with SGD. Since I am using only 4 GPUs for training, I understand that the performance of SAM could be different. However in case of SGD, if all hyperparameters are same, I believe the performance should be similar.
I tried my best to replicate the hyperparameters mentioned in the paper. However some parameters were not so clearly stated. So it would be grateful, if you could verify my training details. Here is the hyperparameters I used for training:
learning rate : 0.1
epoch : 200 (and 400 in case of SGD)
batch size : 256
scheduler : cosine
weight decay : 0.0005
rho : 0.05
beta : 0.9
Nesterov : True