In-Depth
2021 PyTorch Developer Day Conference Recap
The goal of the event was to bring together leading researchers and engineers who use the PyTorch deep neural network code library.
- By Pure AI Editors
- 01/04/2022
The 2021 PyTorch Developer Day (PTDD) conference was held Dec. 1-2, 2021. The goal of the event was to bring together leading researchers and engineers who use the PyTorch deep neural network code library. PyTorch (Facebook/Microsoft) and TensorFlow (Google) are the two dominant deep neural libraries used in industry and academia. The 2021 PTDD event was virtual rather than in-person, due to the covid pandemic.
The history of PyTorch is rather complicated, but, briefly, a C++ tensor library for deep neural networks, named Torch, was released in 2007. Because working with C++ is tedious and error-prone, a Python language wrapper library -- PyTorch -- was released in 2017. Most of the PyTorch development work was done by Facebook Research. In 2020, Microsoft took on the responsibility of maintaining PyTorch builds for the Windows operating system.
PTDD is the premier technical conference for PyTorch. Invitations to attend were limited. And unlike many machine learning research conferences that accept hundreds of papers, PTDD accepted only about 20 full talks and 50 smaller poster-style presentations.
The core 50 PTDD poster sessions were divided into six categories:
- Tools and Core Library
- Model Optimization and Interpretability
- Extensions and Training
- Performance and Deployment
- Audio, Image, Video
- Healthcare and Responsible AI
Brief Summaries of Five Representative Presentations
"Flow Forecast: A Deep Learning for Time Series Forecasting, Classification, and Anomaly Detection Framework," by I. Godfried and A. Polishko. Flow Forecast features unit/integration tests, model visualizations and native cloud provider integration. Flow Forecast allows researchers to experiment with new model architectures and benchmark their results. Flow Forecast supports a broad variety of deep time series models such as LSTMs and Transformers.
"SaLinA: Easy Programming of Sequential Decision Learning and Reinforcement Learning Models in PyTorch," by L. Denoyer, A. de la Fuente, S. Duong, JB Gaya, PA Kamienny and D. Thompson. SaLinA is a lightweight library that extends PyTorch modules for the development of sequential decision models. SaLinA can be used for Reinforcement Learning (including model-based with differentiable environments, multi-agent RL), and also in a supervised/unsupervised learning settings for instance for NLP, Computer Vision, etc.
[Click on image for larger view.] Figure 1: PyTorch Developer Day Virtual Event Application Interface
"PyTorch for R," by D. Falbel. The PyTorch for the R language project was released in 2020. The project allows R users to use PyTorch functionality to benefit from PyTorch's speed, power and flexibility. The presentation summarized project progress and recent developments, and described plans for the future.
"Video Transformer Network," by D. Neimark, O. Bar, M. Zohar and D. Asselmann. Video Transformer Network (VIN) is a transformer-based framework for video recognition. VIN is inspired by recent developments in vision transformers. Unlike standard video action recognition approaches that rely on 3D ConvNets, VIN classifies actions by attending to the entire video sequence information.
"A Framework for Bayesian Neural Networks," by S. Karimi, B. Gokkaya, A. Flower, E. Emamjomeh-Zadeh, A. Templeton, I. Kabul and E. Meijer. A Bayesian neural network (BNN) has weights and biases that are probability distributions instead of single fixed values. The BNN framework provides tools to build BNN models, estimate the uncertainty of their predictions, and transform existing models into their BNN counterparts.
A Closer Look at One Poster Presentation
"Variational Autoencoder Reconstruction Probability Anomaly Detection," by J. McCaffrey, R. Loynd, B. Xia and A. Minnich. Anomaly detection using deep neural autoencoder reconstruction error is a well-known and effective technique. Reconstruction error anomaly detection compares X and reconstructed X, and when they differ greatly, X is likely anomalous in some way.
[Click on image for larger view.] Figure 2: A Poster Presentation from the 2021 PyTorch Developer Day Conference
Recent research has explored an evolution of the autoencoder reconstruction error technique, called variational autoencoder (VAE) reconstruction probability. Briefly, a source data item X generates an internal (u1, s1) mean and standard deviation (equivalently, variance or log-variance) which define a multivariate probability distribution P.
The P(u1, s1) distribution determines a Q(u2, s2) distribution which is sampled to generate a reconstructed X. The VAE reconstruction probability technique determines how likely it is that the source item X came from the Q(u2, s2) distribution, and if the likelihood is small, X is tagged as anomalous. Experiments on synthetic datasets suggest that the autoencoder reconstruction error and VAE reconstruction probability techniques identify different types of anomalous data items.
Brief Summaries of Three Longer Presentations
"Scientific Computing in PyTorch," by M. Ruberry, Software Engineer, Meta AI. A review of how PyTorch has improved its support for scientific computing over the past year, with a focus on support for complex numbers, spectral operations, linear algebra and statistical physics.
"State of PyTorch 2021," by D. Dzhulgakov, Software Engineer, Meta AI. A review of who builds PyTorch, new and upcoming improvements to the framework and ways to get involved in PyTorch library development.
"TorchAudio," by M. Hira, Software Engineer, Meta AI. A review of what TorchAudio is, a discussion of recent releases and a preview of upcoming features.
Attendee Comments
The Pure AI editors spoke with Dr. James McCaffrey, who attended and presented at the 2021 PyTorch Developer Day conference. McCaffrey commented, "The PyTorch Developer Day conference was a breath of fresh air compared to many of the virtual conferences I've participated in since the beginning of the pandemic." He explained, "Almost two years into the pandemic, all of my colleagues that I've spoken to have reached virtual-event mental saturation. But the PyTorch Developer Day event avoided the Zoom-Fatigue effect by holding a small, rather elite and focused event. The conference organizers did a very good job of filtering attendees and presentations."
Another 2021 PTDD attendee, C. Lee, noted, "The PyTorch Developer Day conference clearly focused on practical and useful topics for the PyTorch library, rather than theoretical or research topics." Lee added, "I liked the pragmatic emphasis, and the size of the event allowed me to review every presentation."