Compare some image classification models.

git clone --recurse-submodules https://github.com/enhuiz/image-classification.git

python -m src.train yaml=config/cifar100/resnet18.yml

Where config/cifar100/resnet18.yml can be any configuration you want to run.

bash ./scripts/plot.sh

python -m src.summary

• It appears that the ResNet model severely overfits on the CIFAR100 dataset. Models with lower training accuracy have larger test accuracy.
• The Mix-style method [1] helps alleviate this overfitting clearly.

• Why does the loss suddenly drop periodically (after each epoch) for ResNet? Is it because of memorization?

• The use of relative positional encoding [2] has been shown to improve model convergence and enhance testing performance when the model is trained for a specific number of steps.
• Implementing a diagonal mask [3] results in a slight improvement in performance.
• The learnable temperature for attention [3] does not improve performance in this particular setting, although it does result in a decrease in temperature. Specifically, for a log version implementation, the temperature is 0.9±0.0407, and for the vanilla version, the temperature is 0.892±0.0457.

[1] https://github.com/KaiyangZhou/mixstyle-release

[2] https://arxiv.org/pdf/1901.02860.pdf

[3] https://arxiv.org/pdf/2112.13492.pdf