Adaptive gradients and momentum in neural-network training

A comparison of adaptive-gradient and momentum methods across networks on MNIST and CIFAR-10, examining loss and accuracy trajectories.

Report date

9 October 2025

Original authors

Pang (Jeff) Liu

Report language

English · 22 pages

Main observations

Experiments cover MLP, CNN and VGG13 models, examining AdaGrad, RMSProp, Adam and alternative momentum settings.

Scope and limitations

Comparisons depend on the original architectures, learning rates and training budgets; they do not establish a universal optimizer ranking.