From determining how many spare parts to order and planning staff schedules to predicting sales and anticipating customer demand, accurate forecasts help keep costs low and service levels high. In this article, we explore five exciting trends that are shaping the future of forecasting.
1. Probabilistic forecasting for demand and planning
The future is full of uncertainties, with multiple possible outcomes for any forecast. Traditional forecasting often provides a single expected value or most likely outcome, which is essentially a best guess over potential scenarios. However, this approach doesn’t capture the full range of possibilities and their probabilities. Enter probabilistic forecasting, which offers a spectrum of potential outcomes, each with its associated probability.
Imagine a manufacturing company needing to keep a stock of critical parts. A traditional forecast might give a single number, leading to a 50% chance of having a stock shortage. Probabilistic forecasting, on the other hand, provides a range of outcomes with confidence levels, such as a 90% chance of avoiding a shortage. This approach can significantly improve decision-making and resource allocation.
There are various techniques to generate probabilistic predictions, some producing full distributions and some focusing on confidence intervals. Conformal prediction is an exciting recent approach, which is especially useful with machine learning models. Check out our code implementation for a practical example. Evaluating these forecasts requires careful selection of metrics that truly reflect your needs.
2. Hierarchical forecasting for complex operations
As companies grow, so does the complexity of their operations and the number of forecasts required. For instance, a retailer might need forecasts for tens of thousands of products across hundreds of distribution centers and thousands of stores. Typically, they might want both store-level demand forecasts as well as distribution-center-level and even country or product group level forecasts. Hierarchical forecasting involves generating forecasts at multiple organizational levels, leveraging information across levels, and ultimately ensuring that these forecasts are coherent.
Techniques like top-down or bottom-up forecasting help manage this complexity. A standard modern approach is hierarchical reconciliation, where forecasts are made at all levels and then adjusted to match totals across the hierarchy. This method often outperforms forecasting only at the lower levels in hierarchies.
3. ML, Neural Networks, AutoML and Foundation Models
Traditional forecasting models relied on statistical methods. They were created by writing down a model for the expected process, fine-tuning this model based on the data we observe, and then extending it into the future to make predictions. However, with the explosion of data complexity, forecasters are turning to machine learning (ML) for more powerful methods that rely more on data and less on assumptions about the underlying process. These models learn from all available data at once, instead of one series at a time like statistical models. They can also easily integrate additional data such as weather data, promotions, holidays, or even economic factors for more accurate and nuanced predictions.
Starting with gradient-boosted trees like LightGBM, forecasters now often use neural networks, such as N-BEATS, for large datasets. Going further, AutoML techniques automatically search through a wide range of potential models and select the best ones. Often multiple models are combined, known as ensembling, to provide even better results. More recently foundation models, large networks which have already been trained for a wide variety of tasks, have also been developed. They potentially allow for accurate forecasts with minimal data and effort. However, these still need to be perfected.
4. Explainable forecasting to build trust
While advanced techniques such as machine learning can yield highly accurate forecasts, these forecasts often struggle with adoption within companies. Stakeholders may hesitate to trust them due to the black-box nature of the algorithms used. For example, if a forecasting algorithm predicts low demand in a period with an active promotion, a planner might simply ignore it and add additional demand anyway. The goal of explainable forecasting is to help users understand why algorithms produce the forecasts they do.
The most straightforward approach is to select models that are inherently simple to explain, so-called interpretable models. However, these are often not expressive enough for complex tasks. When using ML models, explanations are generally only approximate. Common techniques include surrogate models and Shapley values, but these can sometimes fail to clarify the situation. Often good design beats complex algorithms here. Transparently showing stakeholders which features the algorithm uses and how the forecast looks under different scenarios can build trust and understanding.
5. MLOps and automation for effective forecasting processes
The sheer volume of forecasts required today makes manual tuning impractical. Robust tools are needed to incorporate new data, clean it, generate forecasts, identify potential failures for further inspection and track metrics over time. All of this needs to be automated, integrated with the organization's data infrastructure and run reliably and at scale. This often requires a combination of forecasting knowledge and software engineering experience. The task of operationalizing forecasting (or any machine learning task) is often referred to as MLOps.
MLOps is a huge field requiring experience both with machine learning algorithms and with software and data engineering. To get started with MLOps, we invite you to explore our content on MLOps or take our MLOps Maturity Scan.
Conclusion
There are a lot of exciting developments in the world of forecasting, such as probabilistic, hierarchical, neural and explainable forecasting. Although it might be tempting to immediately switch to a fully automated Probabilistic Hierarchical Neural Network for all your forecasts, a solid understanding of your data and processes remains central to successful forecasting. Simple, well-executed methods can often rival complex ones when supported by carefully curated data. Ultimately, the value of a forecast is measured by its effectiveness in driving business decisions, so advanced techniques should be used with this goal in mind.
Are you not sure which forecasting models would be best suited for your business? Our experts are happy to help you identify the best forecasting strategy!
