Muon and a Selective Survey on Steepest Descent in Riemannian and Non-Riemannian Manifolds
Muon from first principles, what makes it different from other optimizers, and why it works so well.
Muon
Napkin Math on Non-Euclidean Trust Region Optimization
A possible reason why Muon converges faster & does better at higher learning rates than Adam.
Steepest Descent Under Schatten-p Norms
Why Muon still work despite not perfectly semi-orthogonalizing the gradients.
Squeezing 1-2% Efficiency Gains Out of Muon by Optimizing the Newton-Schulz Coefficients
Simply switching to Muon can already get you 2x efficiency gains. But you can squeeze out an extra 1-2% by optimizing the Newton-Schulz coefficients.
CASPR Without Accumulation is Muon
The CASPR optimizer, a variant of Shampoo, reduces to Muon when we remove the accumulation on the preconditioners.
Deep Learning Optimizers as Steepest Descent in Normed Spaces
Instead of asking, ‘Which optimizer should I use?’ ask, ‘In which space do my features live in?’