Schedule

Tuesday, 11th

8:45 - 10:00	Check-in at Meta Lobby
9:00 - 10:10	Breakfast Optional breakfast provided by Meta
10:15 - 10:30	Opening Remarks	David Kanter MLCommons
10:30 - 11:15	Intro Talk: The AlgoPerf Benchmark Motivation, Rules, and Results The goal of the AlgoPerf: Training Algorithms competition is to evaluate practical speed-ups in neural network training achieved solely by improving the underlying training algorithms. In the external tuning ruleset, submissions must provide workload-agnostic hyperparameter search spaces, while in the self-tuning ruleset they must be completely hyperparameter-free. In both rulesets, submissions are compared on time-to-result across multiple deep learning workloads, training on fixed hardware. This talk presents the inaugural AlgoPerf competition’s results, which drew 18 diverse submissions from 10 teams. Our investigation reveals several key findings: (1) The winning submission in the external tuning ruleset, using Distributed Shampoo, demonstrates the effectiveness of non-diagonal preconditioning over popular methods like Adam, even when compared on wall-clock runtime. (2) The winning submission in the self-tuning ruleset, based on the Schedule Free AdamW algorithm, demonstrates a new level of effectiveness for completely hyperparameter-free training algorithms. (3) The top-scoring submissions were surprisingly robust to workload changes. We also discuss the engineering challenges encountered in ensuring a fair comparison between different training algorithms. These results highlight both the significant progress so far, and the considerable room for further improvements.	Frank Schneider University of Tübingen
11:15 - 12:00	Spotlight Talk I: External Tuning Track Winner Scaling Beyond Diagonal Preconditioners for Training Neural Networks At-Scale Over the past decade, diagonal scaling-based adaptive subgradient methods like AdaGrad and Adam(W) have been widely adopted for neural network training due to their simplicity, linear memory requirements, and robustness to hyperparameter tuning. However, full-matrix preconditioned adaptive subgradient methods, which offer superior performance, are often hindered by their quadratic memory and cubic computational complexity. Recent advances in approximate Kronecker factorization-based methods, including K-FAC and Shampoo, offer a promising alternative, achieving faster convergence than diagonal approximation-based methods while remaining computationally tractable. This talk provides an overview of the Shampoo algorithm and presents our distributed PyTorch implementation, highlighting key heuristics, performance optimizations, and systems/scalability developments that enable efficient training of million- and billion-parameter models. We will also share key insights and learnings from our AlgoPerf PyTorch Shampoo submission, including generalizable configurations and future research directions.	Anna Cai & Michael Shi Meta AI
12:00 - 13:30	Lunch Break Hosted by Meta
13:30 - 14:15	Spotlight Talk II: Self-Tuning Track Winner Schedules & Schedule-Free Learning I will introduce an alternative view of learning rate schedules, where they are considered as a technique for ensuring optimal convergence rates for the last iterate of an optimization procedure. This view leads to highly predictive theory of optimal learning rate schedules, explaining learning rate warmup and annealing procedures used in practice. Going beyond this, I will show how this viewpoint suggests Schedule-Free approaches, where learning rate schedules are replaced by iterate averaging schemes, which yield a number of benefits: no need to specify the stopping time in advance, smoother loss curves and often better eval metrics. This approach is ideally suited to the AlgoPerf competition’s time-to-target setup.	Aaron Defazio Meta AI
14:15 - 15:00	Lightning Talks AlgoPerf Submissions & their Follow-Ups Niccolò Ajroldi: Weight Averaging Techniques on AlgoPerf David Tweedle: Applying Randomized Singular Value Decomposition During Training Sourabh Medapati: Lessons from competing in AlgoPerf v0.5	Niccolò Ajroldi David Tweedle Sourabh Medapati
15:00 - 15:15	Coffee Break
15:15 - 16:15	Roundtable Discussion A moderated open audience discussion on "The Future of Training Algorithms"	Frank Schneider University of Tübingen
16:15 - 17:00	Networking

Wednesday, 12th

8:45 - 10:00	Check-in at Meta Lobby
9:00 - 10:10	Breakfast Optional breakfast provided by Meta
10:15 - 11:00	Invited Talk I How does Gradient Descent Work? Optimization is the engine of deep learning, yet the theory of optimization has had little impact on the practice of deep learning. Why? In this talk, we will first show that traditional theories of optimization fail to explain the convergence of the simplest optimization algorithm — deterministic gradient descent — in deep learning. Whereas traditional theories assert that gradient descent converges because the curvature of the loss landscape is “a priori” small, in reality gradient descent converges because it actively avoids high-curvature regions of the loss landscape. Understanding this behavior requires Taylor expanding to third order, which is one order higher than normally used in optimization theory. While the “fine-grained” dynamics of gradient descent involve chaotic oscillations that are difficult to analyze, we will demonstrate that the “time-averaged” dynamics are, fortunately, much more tractable. We will show that our time-averaged analysis yields highly accurate quantitative predictions in a variety of deep learning settings. Since gradient descent is the simplest optimization algorithm, we hope that this analysis can set the stage for a mathematical theory of optimization in deep learning. This talk is based partly on works by Alex Damian and Eshaan Nichani.	Jeremy Cohen Flatiron Institute
11:00 - 11:45	Invited Talk II What is the best O(n) Hessian query? Ill conditioned loss landscapes are ubiquitous in machine learning, and they slow down optimization. Preconditioning the gradient to make the loss more isotropic is a natural solution, but is challenging for extremely large problems, as direct access to the problem Hessian is prohibitively expensive. We present two fresh approaches to preconditioning using tools from randomized numerical linear algebra and online convex optimization for efficient access to Hessian information, motivated by the question: what is the most useful information we can query from the problem Hessian using O(n) memory and compute?	Madeleine Udell Stanford University
11:45 - 13:30	Lunch Break Hosted by Meta
13:30 - 14:15	Invited Talk III Stochastic-Gradient-based Algorithms for Nonconvex Constrained Optimization and Learning This talk focuses on recent work by my research group and collaborators on the design and analysis of stochastic-gradient-based algorithms for solving nonconvex constrained optimization problems. Our primary motivation is to offer better alternatives to regularization methods (also known as penalty or soft-constraint methods) for inducing structure in a solution or incorporating prior knowledge in a learning problem. Instead, our algorithms are designed to solve constrained problems directly. (Such methods are sometimes referred to as hard-constraint methods.) Even though soft-constraint and hard-constraint methods can be seen to be equivalent, in some sense, using classical penalty function theory, their performance in practice is radically different. Therefore, our aim has been to design such methods such that they are computationally efficient and have rigorous convergence and worst-case complexity guarantees. Our numerical experiments with physics-informed learning and fair supervised learning problems have been encouraging, but we expect that our proposed strategies could also be successful beyond the settings we have considered so far. The talk will also touch upon possible connections between our approach and reduced-space methods for training and/or fine-tuning machine learning models.	Frank E. Curtis Lehigh University
14:15 - 15:15	Roundtable Discussion A moderated open audience discussion on "Training Algorithms in Production"	Michael Rabbat Meta AI
15:15 - 15:30	Coffee Break
15:30 - 16:30	Panel Discussion The Future of AlgoPerf	George Dahl Google DeepMind
16:30 - 17:00	Closing Remarks	Priya Kasimbeg Google DeepMind