When Should Businesses Trust AI Predictions?

Every forecast in business eventually becomes a decision. An airline predicts demand, then decides whether to sell a seat today or save it for a higher-paying traveler tomorrow. A retailer forecasts product demand, then decides what to stock, display, or recommend. An advertising platform predicts future opportunities, then decides how quickly to spend a limited budget.

We often focus on making the prediction more accurate. But there is another question that matters just as much: What should a decision-maker do when they do not know how accurate the prediction is? This is the problem my coauthors and I study in online resource allocation: how to benefit from predictions without having to assume they are right.

A Forecast Is Not a Decision

Online resource allocation describes a large class of business problems in which requests arrive one at a time and resources are limited. Airline seats, product inventories, advertising budgets, and delivery capacity can all fit this framework. The difficulty is that using a resource now may mean losing a better opportunity later. If we can forecast what is coming, we can reserve resources for the future. In our research, we represent a prediction through “shadow prices”: estimates of how valuable it is to preserve each unit of a scarce resource. A high shadow price says, in effect, “this resource may be more valuable later.” But forecasts are not facts. Even a strong forecasting method can perform differently across products, locations, seasons, or time periods. A strategy that follows a good forecast can outperform a conservative strategy. The same strategy can fail badly when the forecast is wrong.

Two Seats and a Difficult Choice

Consider this example: Imagine an airline has only two adjacent seats left on a flight. A traveler has just arrived and wants to buy one of them for $100. The airline has to decide immediately: sell the seat now, or keep both seats available in case a more valuable booking arrives later. If the next traveler also wants just one seat for $100, selling to the first traveler was the right choice. But suppose the next request comes from two people traveling together who are willing to pay $250, but only if they can get both adjacent seats. Selling the first seat means losing that booking entirely. When the first traveler arrives, the airline does not yet know which type of demand will come next. Now suppose the airline has a forecast suggesting that a higher-value booking is likely to arrive later. If that forecast is accurate, holding the two seats is the better decision. But if the forecast is wrong and no such booking appears, the airline may end up with an empty seat it could have sold earlier. The challenge, then, is not simply having a prediction. It is deciding how much to trust that prediction when its accuracy is uncertain.

What If the World Changes?

There is another complication: the environment generating demand may itself be uncertain. Many decision models assume that customer arrivals are reasonably stable. In other settings, demand can shift because of a competitor, a viral trend, a holiday, a supply disruption, or a sudden market change. Our research treats these as points on a spectrum from relatively stationary to highly nonstationary. The theory shows a fundamental tradeoff: if you do not know whether the environment is stable and do not know whether a prediction is accurate, no algorithm can always get the best possible outcome in every scenario.

Knowing When to Switch

Our algorithm begins by behaving as though the environment is reasonably stable, using the prediction to guide decisions while controlling how quickly resources are consumed. At regular points, it checks whether observed performance is still consistent with that assumption. If the evidence suggests that the environment has become too nonstationary, the algorithm switches to a more conservative strategy designed for difficult conditions. That fallback can still use the prediction when it helps, but it does not depend on the prediction being accurate. The algorithm does not need to know in advance how good the forecast is or what kind of arrival process it will face. It learns how cautious it needs to be while decisions are being made. This suggests a broader principle for AI in business: do not treat a prediction as a one-time yes-or-no bet. Build decision systems that can use a forecast, monitor whether the world still supports it, and change course when it does not.

How should the decision process use a model whose reliability may change over time? In inventory management, pricing, advertising, capacity planning, and other operational settings, the answer may be neither “follow the prediction” nor “ignore the prediction.”

What Happens With Real Retail Data?

We tested the approach using two years of online transaction data from clothing retailer H&M. The full dataset contained more than 100,000 products; for the large-scale experiment, we focused on 5,000 high-transaction products, representing about 13.7 million transactions. We generated forecasts using three common methods: Prophet, which identifies trends and recurring seasonal patterns; ARIMA, which uses patterns in past demand to predict what comes next; and Exponential Smoothing, which emphasizes more recent observations when forecasting future demand. The point was not to identify one forecasting champion. It was to see whether the decision algorithm could remain useful as forecast quality changed. Across the three methods, the algorithm moved toward the better of two extremes: following the prediction and relying on a robust online strategy. On average, it recovered 68 percent of the normalized gap between those benchmarks with Prophet forecasts, 58 percent with ARIMA, and 53 percent with Exponential Smoothing. The goal is to benefit when a forecast is strong without paying an excessive price when it is weak.

Build AI Systems With an Escape Hatch

Businesses sometimes frame AI adoption as a choice between trusting the model and trusting the old process. That framing misses the more useful question: How should the decision process use a model whose reliability may change over time? In inventory management, pricing, advertising, capacity planning, and other operational settings, the answer may be neither “follow the prediction” nor “ignore the prediction.” It may be to use the prediction when the evidence supports it, while preserving a robust fallback when it does not. As AI forecasting becomes more powerful, this distinction will matter more, not less. Better predictions create more value only when organizations have decision systems capable of using them well. The future of AI in operations is not about choosing between prediction and robustness. It is about designing systems that know how to use both.

Further Reading

Lin An, Andrew A. Li, Benjamin Moseley, and Gabriel Visotsky, “Best of Many in Both Worlds: Online Resource Allocation with Predictions under Unknown Arrival Model.” https://arxiv.org/abs/2402.13530