Webpage of Edouard Oyallon

I am EUnlike the common French name 'Édouard',
there is no accent on the 'E'.douard Oyallon, a CNRS researcher in the team MLIA of Sorbonne University. Prior to it, you might have met me at Flatiron Institute (CCM), Ecole Polytechnique (DepMap), CentraleSupélec (Opis), INRIA Lille (SequeL, with Michal Valko), Ecole Normale Supérieure (DATA, with Stéphane Mallat) or even at ENS Cachan, campus de Ker Lann. My research interests used to lie in the foundations of machine learning techniques, examining both their applied and theoretical aspects, including the symmetries of deep neural networks. Currently, I am developing algorithms for large-scale distributed and decentralized training, utilizing asynchronous methods and increased parallelism to accelerate the training of larger neural networks.

I’m looking to connect with experienced students and collaborators who are actively working on optimizing and accelerating large-scale LLM training (up to exascale). If you want to collaborate, please reach out at edouard[d.o.t]oyallon[a.t]cnrs[d.o.t]fr!

Interested in the theory of deep learning? Check my lecture notes.
Interested in training LLMs in practice? Check my class webpage.
Interested in joining an academic community around AI + HPC? Join the Working Group HPC4Learning.

My Google Scholar.
My GitHub.
My CV.

Students/Postdoctoral researchers, past and present:

2026-now: Hugo Fruchet, intern (ENS Saclay).
2025-now: Pietro Cagnasso, intern (Politecnico di Torino) > PhD student, co-supervised with Eugene Belilvosky.
2025-now: Danyal Ahmed, intern (University of Cambridge) > PhD student.
2021-2025: Adel Nabli, intern (CentraleSupélec/Université de Montréal) > PhD student, co-supervised with Eugene Belilovsky.
2021-2024: Léo Grinsztajn, intern (Ecole Polytechnique) > PhD student, co-supervised with Gaël Varoquaux.
2021-2024: Louis Fournier, intern (Ecole Polytechnique) > PhD student.
2022-2024: Stéphane Rivaud (SonyCSL), Postdoc.
2023: Thomas Pumir (Princeton University), Postdoc.
2020-2021: Louis Leconte, intern (ENS Paris-Saclay), co-supervised with Aymeric Dieuleveut and Eric Moulines.
2021: Jakob Maier, intern (Institut Polytechnique de Paris).
2021: Jiang Ruiyao, intern (Ecole Polytechnique).

On-going Grants/projects

2023-2029, Partner of PEPR SHARP project.
2026-2030, Partner of DAIMOS project.

Academic service:

2026-now, Organizer of the CNRS Research working group HPC for Learning.
2019-now, One of the core developer and maintainer of Kymatio, http://www.kymat.io. Wavelet Scattering Transforms in Python with GPU acceleration.
2022, Co-organiser of the GdR ISIS Recent Advances in Graph Machine Learning.
2019-2022, Member of the MALIA group at the Société Française de Statistique (SFdS).
2021, Organizer of the Learning and Deep Learning session at Journées MAS 2021.
2021-2022, Co-organizer of the AAAI 2021 Workshop: Learning Network Architecture During Training.
2020-2021, Co-organizer with SFdS, Okwin and AccentureLab of the Federated Learning workshop.
2019, Co-organizer of the ICLR 2019 Workshop: Learning with Limited Labeled Data: Representation Learning for Weak Supervision and Beyond.
2017, Co-organizer of the NeurIPS 2017 Workshop: Learning with Limited Labeled Data: Weak Supervision and Beyond.

Recent preprints/technical reports:

Bakong K., Massoulié L., Oyallon E. and Scaman K. - Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation, preprint.
Serrano L., Han J., Oyallon E., Ho S. and Morel R. - Test-time Generalization for Physics through Neural Operator Splitting, preprint. https://arxiv.org/abs/2602.00884
Janson P., Oyallon E. and Belilovsky E. - Stabilizing Native Low-Rank LLM Pretraining, preprint.
Singh V., Khalid Z., Oyallon E. and Belilovsky E. - Model Parallelism With Subnetwork Data Parallelism, preprint. https://arxiv.org/abs/2507.09029

List of publications:

Thérien B., Joseph C.-É., Knyazev B., Oyallon E., Rish I., and Belilovsky E. - LO: Compute-Efficient Meta-Generalization of Learned Optimizers, ICLR 2026. https://arxiv.org/abs/2406.00153
Nabli A. and Oyallon E. - Decentralized Asynchronous Optimization with DADAO allows Decoupling and Acceleration, JMLR 2025. https://hal.science/hal-05270499
Nabli A., Fournier L., Erbacher P., Serrano L., Belilovsky E., and Oyallon E. - ACCO: Accumulate while you Communicate, Hiding Communications in Distributed LLM Training, NeurIPS 2025. https://hal.science/hal-04592562
Morel R., Han J., Oyallon E. - DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction, ICML 2025. https://arxiv.org/abs/2504.19496
Rivaud S., Fournier L., Pumir T., Belilovsky E., Eickenberg M., and Oyallon E. - PETRA: Parallel End-to-end Training with Reversible Architectures, ICLR 2025. (Spotlight) https://openreview.net/forum?id=0fhzSFsGUT
Fournier L. and Oyallon E. - Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks, NeurIPS OPT2024. https://arxiv.org/abs/2403.08837
Fournier L., Nabli A., Aminbeidokhti M., Pedersoli M., Belilovsky E., and Oyallon E. - WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average, NeurIPS OPT2024. https://hal.science/hal-04588075
Oyallon E. - Contributions to Local, Asynchronous and Decentralized Learning, and to Geometric Deep Learning, HDR manuscript, 2023. https://hal.science/tel-04334118, slides
Grinsztajn L., Kim M.J., Oyallon E. and Varoquaux G. - Modeling string entries for tabular data prediction: do we need big large language models?, NeurIPS TRL 2023. https://openreview.net/forum?id=WXNmnmpRBJ
Nabli A., Belilovsky E. and Oyallon E. - A²CiD²: Accelerating Asynchronous Communication in Decentralized Deep Learning, NeurIPS 2023. https://hal.science/hal-04124318
Legate G., Bernier N., Caccia L., Oyallon E., Belilovsky E., Guiding The Last Layer in Federated Learning with Pre-Trained Models, NeurIPS 2023. https://arxiv.org/abs/2306.03937
Tenison I., Sreeramadas S. A., Mugunthan V., Oyallon E., Rish I., Belilovsky E. - Gradient Masked Averaging for Federated Learning, TMLR 2023. https://arxiv.org/abs/2201.11986
Fournier L., Patel A., Eickenberg M., Oyallon E. and Belilovsky E. - Preventing Dimensional Collapse in Contrastive Local Learning with Subsampling, ICML LLW 2023. https://hal.science/hal-04156218
Fournier L., Rivaud, S., Belilovsky E., Eickenberg M. and Oyallon E. - Can Forward Gradient Match Backpropagation?, ICML 2023. https://hal.science/hal-04119829
Nabli A. and Oyallon E. - DADAO: Decoupled Accelerated Decentralized Asynchronous Optimization, ICML 2023. https://hal.science/hal-03737694
Grinsztajn L., Oyallon E. and Varoquaux G. - Why do tree-based models still outperform deep learning on tabular data?, NeurIPS 2022. https://hal.science/hal-03723551/
Sergeant-Perthuis G., Maier J., Bruna J. and Oyallon E. - On Non-Linear operators for Geometric Deep Learning, NeurIPS 2022. https://hal.science/hal-03711864/
Laousy O., Chassagnon G., Oyallon E., Paragios N., Revel M.-P., Vakalopoulou M - Deep Reinforcement Learning for L3 Slice Localization in Sarcopenia Assessment, International Workshop on Machine Learning in Medical Imaging 2021. https://arxiv.org/abs/2107.12800
Grinsztajn N., Preux P. and Oyallon E. - Low-rank projections of GCNs Laplacian, ICLR 2021 Workshop GTRL. https://openreview.net/forum?id=DnlDTxsQ8Uz
Thiry L., Arbel M., Belilovsky E. and Oyallon E. - The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods, ICLR 2021. https://openreview.net/forum?id=aYuZO9DIdnn
Oyallon E. - Interferometric Graph Transform: a Deep Unsupervised Graph Representation, ICML 2020. https://arxiv.org/abs/2006.05722
Belilovsky E., Eickenberg M. and Oyallon E. - Decoupled Greedy Learning of CNNs, ICML 2020. https://arxiv.org/abs/1901.08164
Andreux M., Angles T., Exarchakis G., Leonarduzzi R., Rochette G., Thiry L., Zarka J., Mallat S., Andén J., Belilovsky E., Bruna J., Lostanlen V., Matthew J. Hirn, Oyallon E., Zhang S., Cella C., Eickenberg, M. - Kymatio: Scattering Transforms in Python, JMLR software 2020. https://arxiv.org/abs/1812.11214
Chizat L., Oyallon, E. and Bach F. - On Lazy Training in Differentiable Programming, NeurIPS 2019. https://arxiv.org/abs/1812.07956
Belilovsky E., Eickenberg M. and Oyallon E. - Greedy Layerwise Learning Can Scale to ImageNet, ICML 2019. https://arxiv.org/abs/1812.11446, slides
Belilovsky E., Eickenberg M., Oyallon E. - Do Deep Convolutional Network Layers Need to be Trained End-to-End?, NeurIPS CRACT workshop 2018.
Oyallon E., Belilovsky E., Zagoruyko S., and Valko M. - Compressing the Input for CNNs with the First-Order scattering Transform, ECCV 2018. https://arxiv.org/abs/1809.10200, poster
Scieur D., Oyallon E., d'Aspremont, A. and Bach, F. - Nonlinear Acceleration of CNNs, ICLR workshop 2018. https://openreview.net/forum?id=HkNpF_kDM, long version https://arxiv.org/abs/1805.09639
Oyallon E., Zagoruyko S., Huang G., Komodakis N., Lacoste-Julien S., Blaschko M., and Belilovsky E. - Scattering Networks for Hybrid Representation Learning, TPAMI 2018. https://arxiv.org/abs/1809.06367
Jacobsen J.-H., Smeulders A.W.M. and Oyallon E. - i-RevNet: Deep Invertible Networks, ICLR 2018. https://openreview.net/forum?id=HJsjkMb0Z
Oyallon E. - Analyzing and Introducing Structures in Deep Convolutional Neural Networks, PhD thesis, 2017. https://hal.archives-ouvertes.fr/tel-02353134v1, slides, intro to my thesis (french)
Oyallon E., Belilovsky E., and Zagoruyko S. - Scaling the Scattering Transform: Deep Hybrid Networks, ICCV 2017. https://arxiv.org/abs/1703.08961, poster
Jacobsen J.-H., Oyallon E., Mallat, S. and Smeulders, A.W.M. - Multiscale Hierarchical Convolutional Networks, ICML PADL 2017. https://arxiv.org/abs/1703.04140, poster
Oyallon E. - Building a Regular Decision Boundary with Deep Networks, CVPR 2017. https://arxiv.org/abs/1703.01775, poster
Oyallon E. and Mallat S. - Deep Roto-translation Scattering for Object Classification, CVPR 2015. http://arxiv.org/abs/1412.8659, poster
Oyallon E. and Rabin J. - An Analysis of the SURF Method, IPOL 2015. http://www.ipol.im/pub/art/2015/69/
Oyallon E., Mallat S. and Sifre L. - Generic Deep Networks with wavelet Scattering, ICLR 2014 workshop. http://arxiv.org/abs/1312.5940, poster

Teaching/trainings:

2026, Lecturer, Training and Deploying Large-Scale Models, MVA, webpage of the class
Generic notes of my Deep Learning teaching can be found here. Feedbacks to improve them are welcome!
2023, Lecturer, Advanced topics in Deep Learning, at IPP, webpage of the class
2022, Lecturer, Advanced topics in Deep Learning, at IPP
2021, Lecturer, Deep Learning, at HiParis2021!'s summer school, slides | Lecturer, Advanced topics in Deep Learning, at IPP | Guest lecturer, Deep Learning, at Concordia University | Lecturer, Deep Learning, at CIRM - notebooks are here, videos part 1/part 2, writing notes.
2019, Lecturer, Reinforcement Learning, CentraleSupélec | Lecturer, Deep Learning in Practice, MVA/CentraleSupélec, with Guillaume Charpiat, webpage of the class.
2018, Tutorial at Ateliers Statistiques de la Société Française de Statistique | Teaching assistant, Deep Learning, MVA.
2017, Corporate Seminar Series with Sébastien Loustau.
2014-2017, Teaching assistant, ENSAE, fundamental probability classes and calculus.