Critical Batch Size for Steepest Descent Under Arbitrary Norms
First-order optimization under arbitrary norms with Nesterov momentum (and weight decay) yields a universal critical batch size formula.
Ponder
Franz Louis Cesista’s Newsletter. Artificial intelligence, math, and logistics, from the inside–among other topics. Highly relevant for nerds and machine learning engineers, useful for those working in tech.
Rethinking Maximal Update Parametrization: Steepest Descent on Finsler-Structured (Matrix) Geometries via Dual Ascent
To guarantee fast and robust model training, we can recast the optimization problem as steepest descent on Finsler-structured geometries. Here we show how to compute the optimal updates via dual ascent.
Rethinking Maximal Update Parametrization: Steepest Descent on the Spectral Ball
Novel optimizers for maximally updating both the weights and activations of neural networks while keeping weight norms under control. To get there, we needed to invent an efficient, GPU/TPU-friendly method for eigenvalue clipping and solve the Steepest Descent problem on the Positive Semidefinite Cone, Convex Spectrahedron, and finally on the Spectral Ball.
Steepest Descent on Finsler-Structured (Matrix) Manifolds
Fast and robust model training.
Heuristic Solutions for Steepest Descent on the Stiefel Manifold
What would Muon look like if we constrained the weights to be semi-orthogonal?
Sensitivity and Sharpness of n-Simplicial Attention
Towards a maximal update parameterization of n-simplicial attention
Adam with Aggressive Gradient Clipping ≈ Smoothed SignSGD/NormSGD
Why does Adam with aggressive gradient value/norm clipping have sparse updates and do well with higher learning rates? Here we show that it is essentially equivalent to a smoothed version of SignSGD/NormSGD.
Fast, Numerically Stable, and Auto-Differentiable Spectral Clipping via Newton-Schulz Iteration
A small step towards hardware-architecture-optimizer codesign in deep learning.
Muon and a Selective Survey on Steepest Descent in Riemannian and Non-Riemannian Manifolds
Muon from first principles, what makes it different from other optimizers, and why it works so well.
Napkin Math on Non-Euclidean Trust Region Optimization
A possible reason why Muon converges faster & does better at higher learning rates than Adam.
Blocked Matrix Formulation of Linear Attention Mechanisms
The blocked matrix formulation of linear attention mechanisms, multi-step online gradient descent at inference time, and chunk-wise parallelism.
Steepest Descent Under Schatten-p Norms
Why Muon still work despite not perfectly semi-orthogonalizing the gradients.
Squeezing 1-2% Efficiency Gains Out of Muon by Optimizing the Newton-Schulz Coefficients
Simply switching to Muon can already get you 2x efficiency gains. But you can squeeze out an extra 1-2% by optimizing the Newton-Schulz coefficients.
CASPR Without Accumulation is Muon
The CASPR optimizer, a variant of Shampoo, reduces to Muon when we remove the accumulation on the preconditioners.
GRPO's Main Flaw
GRPO may not be the best choice for training reasoning models. Here’s why.
(Linear) Attention as Test-Time Regression
A unifying framework for linear attention mechanisms as test-time regression and how to parallelize training and inference.
Deep Learning Optimizers as Steepest Descent in Normed Spaces
Instead of asking, ‘Which optimizer should I use?’ ask, ‘In which space do my features live in?’
ChatGPT May Have Developed Seasonal Depression
Could ChatGPT’s shorter responses be an indication of something more bizarre going on?
The Human Mind May Be Universal
Years of experience in building artificial minds led me to believe that these AIs may end up seeming more ‘human’ than we currently imagine them to be.
Vaccine Search as a Computational Problem
A thought dump on mRNA vaccines and the future of computational biology