In-Depth
Foundational Models Might Revolutionize Time Series Regression Problems
Researchers are looking at adapting LLM techniques that have revolutionized natural language to see if they can be applied to time series regression problems and provide a gigantic leap in capability.
- By Pure AI Editors
- 04/08/2024
Among all the ways advanced machine learning techniques are being applied to solve real-world problems is a potential revamp of time series regression. The goal of a time series regression problem is to predict a single numeric value that will occur in the future. For example, an airline company must be able to predict the number of passengers that will want to fly a particular route over the next few months. A bad prediction can cost a company millions of dollars, either in lost revenue, or in wasted resources.
The Current Time Series Regression Landscape
Mathematicians have studied time series regression for decades using classical statistics techniques such as ARIMA (auto-regressive integrated mean average). Over the past 10 years or so, data scientists have studied time series regression problems using machine learning techniques, such as LSTM (long short-term memory) systems and standard neural network systems. But it's probably fair to say that there haven't been any major breakthroughs in decades.
[Click on image for larger view.] Figure 1: Standard Techniques Such as Neural Networks Can Be Used for Time Series Problems
Researchers are now looking at adapting the large language model (LLM) techniques that have revolutionized natural language to see if they can be applied to time series regression problems and provide a gigantic leap in capability. Such modern techniques are vastly different from traditional time series regression techniques.
Foundational Models
Large language models, such as GPT-4 which forms the basis of the ChatGPT chatbot, start by training a so-called foundational model using a huge amount of language data (such as the entire text of Wikipedia). An LLM foundational model is analogous to a high school student who has a strong grasp of the English language and basic facts, and is now ready to go to college to learn specific knowledge. Put another way, an LLM does not start from scratch for each conversation.
Traditional time series regression techniques start from scratch and create and train a new model for each specific dataset. Short-range forecasts are often accurate but long-range forecasts are often not much more than guesses.
[Click on image for larger view.] Figure 2: Standard Techniques Create a Model from Scratch for Each Specific Problem
Researchers at Google (A. Das, W. Kong, R. Sen, Y. Zhou) recently published a paper titled "A Decoder-Only Foundation Model for Time-Series Forecasting." The paper describes a system called TimesFM.
The TimesFM base model was trained using a very large set of time series data, containing over 100 billion real world time points. The TimesFM model has 200 million parameters, which is large, but not as large as current LLMs such as GPT-3, which has 175 billion parameters.
The zero-shot performance (out-of-the box, with no fine-tuning) of the TimesFM model achieves close to the state-of-the-art results on a wide range of previously unseen datasets. The TimesFM model was trained using both synthetic data (using statistical models and physics simulations) and real-world data (such as pageviews from Wikipedia).
Wrapping Up
The Pure AI editors spoke with Dr. James McCaffrey, who works at Microsoft Research, and has studied time series regression for decades. McCaffrey commented, "There have been a few previous attempts at creating foundational models for time series data, but none were entirely successful, probably because they didn't have enough training data."
He added, "Foundational models for time series regression have the potential to completely revolutionize the field. I am cautiously optimistic that time series regression foundational models such as TimesFM model could provide a powerful new tool for time series forecasts."