Welcome to the Data Science Colloquium of the ENS.
This colloquium is organized around data sciences in a broad sense with the goal of bringing together researchers with diverse backgrounds (including for instance mathematics, computer science, physics, chemistry and neuroscience) but a common interest in dealing with large scale or high dimensional data.
The colloquium is followed by an open buffet around which participants can meet and discuss collaborations.
These seminars are made possible by the support of the CFM-ENS Chair “Modèles et Sciences des Données”.
You can check the list of the next seminars below and the list of past seminars.
Videos of some of the past seminars are available online.
The colloquium is organized by:
04 Avril 2024, 11h00-12h00 (Paris time), room Amphi Jaures (29 Rue d'Ulm).
Lénaïc Chizat (EPFL)
Title: A Formula for Feature Learning in Large Neural Networks
Abstract: Deep learning succeeds by doing hierarchical feature learning, but tuning hyperparameters such as initialization scales, learning rates, etc., only give indirect control over this behavior. This calls for theoretical tools to predict, measure and control feature learning. In this talk, we will first review various theoretical advances (signal propagation, infinite width dynamics, etc) that have led to a better understanding of the subtle impact of hyperparameters and architectural choices on the training dynamics. We will then introduce a formula which, in any architecture, quantifies feature learning in terms of more tractable quantities: statistics of the forward and backward passes, and a notion of alignment between the feature updates and the backward pass which captures an important aspect of the nature of feature learning. This formula suggests normalization rules for the forward and backward passes and for the layer-wise learning rates. To illustrate these ideas, I will discuss the feature learning behavior of ReLU MLPs and ResNets in the infinite width and depth limit.