News

Open Source Project from Amazon Cloud Optimizes Machine Learning Models for the Edge

• By David Ramel
• 02/06/2019

A new open source project from the Amazon cloud platform, Amazon Web Services (AWS), optimizes machine learning models to run better in the cloud and on resource-constrained network edge devices.

Called Neo-AI, the new open source offering is based on Neo functionality introduced last fall as part of the Amazon SageMaker service, used to train machine learning models once and then deploy them to various locations.

At the time, AWS described SageMaker Neo as a "deep learning model compiler [that] lets customers train models once, and run them anywhere with up to 2X improvement in performance."

AWS said the new offering helps developers and users tune machine learning models to optimize them for the many different different platforms, hardware and software configurations they may have to work with. This was described as a time-consuming, manual process further complicated by the lack of compute and storage resources typically found on edge devices to which models are often deployed.

"Neo-AI eliminates the time and effort needed to tune machine learning models for deployment on multiple platforms by automatically optimizing TensorFlow, MXNet, PyTorch, ONNX, and XGBoost models to perform at up to twice the speed of the original model with no loss in accuracy," AWS said in a Jan. 23 blog post.

"Additionally, it converts models into an efficient common format to eliminate software compatibility problems," AWS said. "On the target platform, a compact runtime uses a small fraction of the resources that a framework would typically consume. By making optimization easier, Neo-AI allows sophisticated models to run on resource-constrained devices, where they can unlock innovation in areas such as autonomous vehicles, home security and anomaly detection."

Currently Neo-AI supports platforms from Intel, NVIDIA, and ARM, while support for Xilinx, Cadence and Qualcomm is said to be coming soon. All of those organizations are expected to help steer the project with community contributions. One main component of the project, a runtime, is reportedly currently deployed on devices from many vendors including ADLINK, Lenovo, Leopard Imaging, Panasonic and others.

The source code for Neo-AI is available on GitHub.