News
Google Open Sources Library for Training Large-Scale Neural Network Models
This week Google announced that it has open-sourced GPipe, a library that can be used to train deep neural networks (DNNs) used in large-scale machine learning projects, like language processing, speech recognition and image recognition.
Google noted in its announcement that current state-of-the-art image models are facing limitations imposed by GPU memory, even with the many advances in that area.
With GPipe and "pipeline parallelism," the company discussed in this paper how it uses GPipe in a distributed manner to "overcome this limitation," thanks to the library's use of synchronous stochastic gradient descent, combined with the parallel architecture and use of multiple sequential layers.
"Importantly, GPipe allows researchers to easily deploy more accelerators to train larger models and to scale the performance without tuning hyperparameters," the company commented.
"To demonstrate the effectiveness of GPipe, we trained an AmoebaNet-B with 557 million model parameters and input image size of 480 x 480 on Google Cloud TPUv2s," it continued.
"This model performed well on multiple popular datasets, including pushing the single-crop ImageNet accuracy to 84.3%, the CIFAR-10 accuracy to 99%, and the CIFAR-100 accuracy to 91.3%."
In contrast, Google points out that the winner of the 2017 ImageNet challenge, Squeeze-and-Excitation Networks, achieved 82.7 percent top-1 accuracy with 145.8 million parameters.
Much more detail about GPipe can be found in the official announcement here.
According to Google, the core GPipe library is open-sourced under the Lingvo framework.
About the Author
Becky Nagel is the vice president of Web & Digital Strategy for 1105's Converge360 Group, where she oversees the front-end Web team and deals with all aspects of digital projects at the company, including launching and running the group's popular virtual summit and Coffee talk series . She an experienced tech journalist (20 years), and before her current position, was the editorial director of the group's sites. A few years ago she gave a talk at a leading technical publishers conference about how changes in Web browser technology would impact online advertising for publishers. Follow her on twitter @beckynagel.