Knowledge

This page contains papers and related content linked to the loss landscape challenge. If you have a paper or article that you would like to share, email ideami@ideami.com.

• Why Flatness Correlates With Generalization For Deep Neural Networks
  huofeng Zhang, Isaac Reid, Guillermo Valle Pérez, Ard Louis
  Submitted on 10 March, 2021;
  arXiv:2103.06219

• Quantum Earth Mover’s Distance: A New Approach to Learning Quantum Data
  Bobak Toussi Kiani, Giacomo De Palma, Milad Marvian, Zi-Wen Liu, Seth Lloyd
  Submitted on 08 Jan, 2021;
  arXiv: 2101.03037

• Towards a Better Global Loss Landscape of GANs
  Ruoyu Sun, Tiantian Fang, Alex Schwing
  Submitted on 10 Nov, 2020;
  arXiv: 2011.04926

• Approximation and convergence of GANs training: an SDE approach
  Haoyang Cao, Xin Guo
  Submitted on 2 Jun, 2020;
  arXiv: 2006.02047

• Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
  Pu Zhao, Pin-Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, Xue Lin
  Submitted on 30 Apr, 2020;
  arXiv: 2005.00060

• Revisiting Loss Landscape for Adversarial Robustness
  Dongxian Wu, Yisen Wang, Shu-tao Xia
  Submitted on 13 Apr, 2020;
  arXiv: 2004.05884

• Understanding Global Loss Landscape of One-hidden-layer ReLU Neural Networks
  Bo Liu
  Submitted on 12 Feb, 2020;
  arXiv: 2002.04763

• Deep Ensembles: A Loss Landscape Perspective
  Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan
  Submitted on 5 Dic, 2019;
  arXiv: 1912.02757

• Emergent properties of the local geometry of neural loss landscapes
  Authors: Stanislav Fort, Surya Ganguli
  Submitted on 14 Oct, 2019;
  arXiv:1910.05929
  

• Loss Landscape Sightseeing with Multi-Point Optimization
  Authors: Ivan Skorokhodov, Mikhail Burtsev
  Submitted on 9 Oct, 2019;
  arXiv:1910.03867
  

• How noise affects the Hessian spectrum in overparameterized neural networks
  Authors: Mingwei Wei, David J Schwab
  Submitted on 1 Oct, 2019;
  arXiv:1910.00195

 

• Visualizing and Understanding the Effectiveness of BERT
  Authors: Yaru Hao, Li Dong, Furu Wei, Ke Xu
  Submitted on 15 August, 2019;
  arXiv:1908.05620
  

 

• The Difficulty of Training Sparse Neural Networks
  Authors: Utku Evci, Fabian Pedregosa, Aidan Gomez, Erich Elsen
  Submitted on 25 June, 2019;
  arXiv:1906.10732

 

• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
  Authors: Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, Rong Ge
  Submitted on 14 June, 2019;
  arXiv:1906.06247

 

• Large Scale Structure of Neural Network Loss Landscapes
  Stanislav Fort, Stanislaw Jastrzebski
  Submitted on 11 Jun 2019
  arXiv:1906.04724

 

• Understanding Generalization through Visualizations
  W. Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K. Terry, Furong Huang, Tom Goldstein
  Submitted 7 June 2019
  arXiv:1906.03291
  https://arxiv.org/abs/1906.03291

 

• The Effect of Network Depth on the Optimization Landscape
  Behrooz Ghorbani, Ying Xiao, Shankar Krishnan
  Submitted 28 May 2019
  Link

 

• Visualizing Loss Landscape of Deep Neural Networks…..but can we Trust them?
  Jae Duk Seo
  May 5, 2019
  Link

 

• Negative eigenvalues of the Hessian in deep neural networks
  Guillaume Alain, Nicolas Le Roux, Pierre-Antoine Manzagol
  Submitted on 6 Feb 2019
  arXiv:1902.02366

 

• Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
  Charles H. Martin, Michael W. Mahoney
  Submitted on 2 Oct 2018
  arXiv:1810.01075

 

• On the loss landscape of a class of deep neural networks with no bad local valleys
  Quynh Nguyen, Mahesh Chandra Mukkamala, Matthias Hein
  Submitted on 27 Sep 2018
  arXiv:1809.10749

 

• The Goldilocks zone: Towards better understanding of neural network loss landscapes
  Stanislav Fort, Adam Scherlis
  Submitted on 6 Jul 2018
  arXiv:1807.02581
  

 

• The loss landscape of overparameterized neural networks
  Y Cooper
  Submitted on 26 Apr 2018
  arXiv:1804.10200

 

• Measuring the Intrinsic Dimension of Objective Landscapes
  Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski
  Submitted on 24 Apr 2018
  arXiv:1804.08838
  

 

• A Mean Field View of the Landscape of Two-Layers Neural Networks
  Song Mei, Andrea Montanari, Phan-Minh Nguyen
  Submitted on 18 Apr 2018
  arXiv:1804.06561
  

 

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
  Jonathan Frankle, Michael Carbin
  Submitted on 9 Mar 2018
  arXiv:1803.03635

 

• Essentially No Barriers in Neural Network Energy Landscape
  Felix Draxler, Kambis Veschgini, Manfred Salmhofer, Fred A. Hamprecht
  Submitted on 2 Mar 2018
  arXiv:1803.00885

 

• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
  Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson
  Neural Information Processing Systems 2018
  Submitted on 27 Feb 2018
  arXiv:1802.10026

 

• Visualizing the Loss Landscape of Neural Nets (A favourite of mine)  *********
  Li, Hao and Xu, Zheng and Taylor, Gavin and Studer, Christoph and Goldstein, Tom
  Neural Information Processing Systems 2018
  Submitted on 28 Dec 2017
  arXiv:1712.09913

 

• Sharp Minima Can Generalize For Deep Nets
  Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio
  Submitted on 28 March 2017
  arXiv:1703.04933