Loss

The Lands

This gallery showcases a few examples of the pieces produced by the LL project. The creations below belong to Phase 1. Phase 2 is currently in the initial stages. Writings that go deep into these and other visualizations are being prepared.

LR COASTER visualizes a learning rate stress test during the training of a convnet. We ride along the minimizer while exploring its nearby surroundings. I use extreme changes in the learning rate to illustrate how the morphology and dynamics of the loss landscape change in response to the changes in the learning rate. The resolution (300K loss values calculated per frame) allows us to explore the change in morphology. More details and related analysis about this and other visualizations will be published in the future.

SENTINEL visualizes the optimization process of a convnet during training mode, moving from a high loss value through the creation of an edge horizon to the final convexity and minimum. We ride along the minimizer while exploring its nearby surroundings. More details and related analysis about this and other visualizations will be published in the future.

WALTZ-RES visualizes the difference in morphology and dynamics between two resnet-25 networks, one with skip connections and one without. In this fragment of the visualization, we can see the first 2 and a half epochs of the training process. We ride along the minimizer while exploring its nearby surroundings. More details and related analysis about this and other visualizations will be published in the future.

EDGE HORIZON visualizes a loss landscape in extreme resolution, using 1 million loss points captured during the training of a convnet. The morphology of the landscape during the training phase is influenced by the parameters of the network. More details and related analysis about this and other visualizations will be published in the future.

GOBLIN takes us on a journey from above the edge horizon of the loss landscape of a convnet, during its training process, through the edge horizon (laterally) and to the perspective from below its dynamic convexity. We ride along the minimizer while exploring its nearby surroundings. More details and related analysis about this and other visualizations will be published in the future.

DOWN UNDER goes deep below the loss landscape of the training process of a convnet (while training mode is active), giving us a perspective from below, as the minimizer’s dynamics transform the nearby surroundings during its journey towards its final destination. We ride along the minimizer while exploring its nearby surroundings. More details and related analysis about this and other visualizations will be published in the future.

GENTLY follows the gentle change in the surroundings of the minimizer as we follow its gradual descent. We ride along the minimizer while exploring its nearby surroundings. More details and related analysis about this and other visualizations will be published in the future.

Preparation phase

The gallery above will be expanded with more creations and associated writings over time. Before I began creating my own landscapes, there was a preparation phase in which I worked with existing data from other sources. An example of that phase is the landscape right below (which uses data from the excellent paper “Visualizing the Loss Landscape of Neural Nets” by Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer and Tom Goldstein. I also produced simulations like the last video of the gallery, which is a simulation that was created before the Loss Landscape project began. All the Loss Landscape videos use real data and real networks except the very last one of this page, which was also the very first loss landscape video I created.

LL is led by Javier Ideami, A.I researcher, multidisciplinary creative director, engineer and entrepreneur. Contact Ideami on ideami@ideami.com

xyz

Let’s Get Started

Going deep