Fast, Numerically Stable, and Auto-Differentiable Spectral Clipping via Newton-Schulz Iteration
A small step towards hardware-architecture-optimizer codesign in deep learning.
Machine Learning
Muon and a Selective Survey on Steepest Descent in Riemannian and Non-Riemannian Manifolds
Muon from first principles, what makes it different from other optimizers, and why it works so well.
Napkin Math on Non-Euclidean Trust Region Optimization
A possible reason why Muon converges faster & does better at higher learning rates than Adam.
Blocked Matrix Formulation of Linear Attention Mechanisms
The blocked matrix formulation of linear attention mechanisms, multi-step online gradient descent at inference time, and chunk-wise parallelism.
Steepest Descent Under Schatten-p Norms
Why Muon still work despite not perfectly semi-orthogonalizing the gradients.
Squeezing 1-2% Efficiency Gains Out of Muon by Optimizing the Newton-Schulz Coefficients
Simply switching to Muon can already get you 2x efficiency gains. But you can squeeze out an extra 1-2% by optimizing the Newton-Schulz coefficients.
CASPR Without Accumulation is Muon
The CASPR optimizer, a variant of Shampoo, reduces to Muon when we remove the accumulation on the preconditioners.
GRPO's Main Flaw
GRPO may not be the best choice for training reasoning models. Here’s why.
(Linear) Attention as Test-Time Regression
A unifying framework for linear attention mechanisms as test-time regression and how to parallelize training and inference.
Deep Learning Optimizers as Steepest Descent in Normed Spaces
Instead of asking, ‘Which optimizer should I use?’ ask, ‘In which space do my features live in?’