Building a Forecasting System (from Scratch)

Introduction

Building a Forecasting System (from Scratch)

Introduction

Let’s address the elephant in the room: there is a significant gap in forecasting between research and its practical application in industry.

There, I said it.

On one hand, practitioners misinterpret fundamental concepts, such as proper model evaluation or the identification of underlying dynamics. Others, often charmed or even enchanted by a hyped method or model, blindly implement it and pray to some ancient ML God for it to work. On the other hand, researchers often over-engineer models that struggle in production environments. We have a ton of new methods that make unrealistic assumptions about their applicability. In short, researchers tend to miss (or forget) the practical constraints that define success in the field.

This article is (yet another) attempt to bridge this gap.

It is neither a formal academic paper nor a technical whitepaper. Its purpose is to document the design and engineering of an end-to-end Demand Forecasting System (prototype), drawing on knowledge accumulated by the research community and our own practical experience in the field.

A bit of context: in our latest project, we partnered with small-to-medium enterprises (SMEs) in the ready-to-eat meal sector — ranging from school caterers and bistros to B2B suppliers, each with varying degrees of digital maturity. Our primary goal was to improve their entire demand forecasting pipelines, specifically addressing the critical trade-off between minimizing food waste and maintaining high operational efficiency.

By replacing intuition-based planning with a data-driven framework, we achieved significant, quantifiable improvements:

  • Accuracy: Forecasting accuracy improved by margins ranging from 13% to over 133% across five distinct partners.
  • Sustainability: Simulation results indicate a potential 67% reduction in food waste per meal for specific case studies.
  • Efficiency: Cost approximations point to a reduction in operational costs between 4% and 20%.
  • Adoption: Four partners actively tested and used our deployed system to drive their weekly production decisions.

To achieve these results across multiple distinct companies within a limited timeframe, we could not rely on tailored, one-off solutions. Instead, we developed a generalizable, robust, and scalable approach broken down into distinct, testable modules. Our methodology is grounded in modern MLOps principles and aligned with academic forecasting literature.

We focus on 5 core pillars:

  1. Scoping the business problem,
  2. Data collection and curation,
  3. Efficient and effective modeling,
  4. Turning predictions to actionable decisions.
  5. Deploying a scalable, cost-friendly cloud-based solution

To sprinkle some extra hype onto our system, we enhanced our forecasting architecture with targeted agentic modules. Instead of complex, open-ended multi-agent systems, we introduced discrete, specialized agents responsible for verifiable tasks — such as extracting features from unstructured recipe text or composing weekly reports — ensuring the pipeline remains robust and deterministic.

Figure 1: The End-to-End Forecasting Framework: A high-level view of the five core modules developed to transform raw SME data into actionable production decisions.

In this post, we detail our approach from start to finish, sharing technical insights on how to take meaningful academic literature and translate it into a robust, useful forecasting system.


1. Scoping.

A common misconception when designing a forecasting solution is that the whole process is model-centric. Assuming that the algorithm sits at the core of the system is wrong. In production, the model is merely a utility function within a larger decision-support framework. A forecast is only valuable if it contains the specific signal required to optimize that decision.

Scoping is the critical process where you acquire a holistic understanding of the business problem in order to architect a solution. In our case, the goal was to produce a forecast that acts as a functional input to a downstream decision engine. The nature of the decision dictates the necessary forecasting properties, such as the cross-sectional level, forecasting horizon, and granularity. More details can be found in this article [1], this book [2] or this YouTube playlist [3]. If the model output does not align with the requirements of the operation, the prediction loses its utility regardless of its accuracy.

To gain this in-depth understanding of the unique business practices and operational limitations of our partners, we utilized a formal interview process supported by a technical questionnaire. The primary goal was not just to understand single use cases, but to frame the distinct problem for each company and identify common patterns across the sector. This allowed us to design a generalizable forecasting solution that was robust enough to scale, yet flexible enough to incorporate case-specific characteristics without requiring bespoke engineering for every deployment.

We identified two distinct downstream business decisions:

  • Daily Meal Preparation: A short-term, newsvendor-type problem where forecasts for the next 5–7 days are required to plan production quantities and organize kitchen shifts.
  • Ingredient Reordering: A problem involving a longer forecast horizon (often 14 to 21 days) to manage inventory for raw materials with longer lead times.

To satisfy both constraints, we designed a pipeline that generates forecasts on a daily resolution for up to three weeks into the future, updated on a weekly basis. This approach allows the short-term planner to view the immediate daily trajectory for kitchen prep, while the procurement team can consider the decomposed daily predictions (more on that later) over the specific lead-time windows.

A critical engineering decision during this phase was determining the level of hierarchy at which to model demand. While inventory decisions are made at the ingredient level (e.g., kg of potatoes), the demand signal — driven by taste, seasonality, and menu fatigue — exists at the meal level (e.g., Steak with Fries). Forecasting aggregated ingredients directly dilutes the clear signal from individual meals, making it impossible to detect the impact of key underlying demand drivers.

Therefore, we structured the system to forecast demand at the meal-client level. These forecasts are passed through a translation layer using a Bill of Materials (recipe matrix) to deterministically calculate the required raw materials.

Figure 2: The Translation Layer. Decoupling the demand signal from inventory management: Forecasting is performed on the end-product (meal) to capture consumer behavior, then mapped via recipes to raw ingredients for replenishment.

This decomposition allows us to evaluate the system based on downstream decision quality rather than relying entirely on forecasting error measures. The latter, despite being useful performance proxies, often correlate poorly with actual business performance (refer to [4] and [5] for more) Instead, we track KPIs that explicitly matter to our partners:

  • Availability: Of both finished meals and the raw ingredients required for preparation.
  • Efficiency: Inventory costs.
  • Sustainability: Minimization of food waste.

The last crucial scoping task is the identification of demand predictors. We must determine which quantifiable parameters influence demand and how we can access and track these signals systematically. The requirement to move from abstract influence to concrete input leads us directly to our next module:


2. Building Robust Data Pipelines

Scoping provides the theoretical map of demand drivers, effectively identifying what should be tracked. However, in applied machine learning, there is often a divergence between identifying a predictor and successfully engineering the infrastructure to monitor it.

For anyone who has deployed models in production, it is evident that the majority of the development lifecycle is not spent tuning hyperparameters, but rather constructing the underlying data pipelines. This involves connecting fragmented data lakes to structured data warehouses, designing robust ETL (Extract, Transform, Load) processes, and managing feature stores that ensure training-serving skew is minimized.

The challenge for SMEs in 2025 is rarely the absence of data. We are way past that. Companies now possess years of transactional history, stock logs, and customer records. The barrier is accessibility and structure. Their collected data often resides in what can be best described as “data swamps” — fragmented legacy systems, ad-hoc Excel files with inconsistent formatting, or free-text fields used as makeshift databases. The engineering task, therefore, is to impose strict schema validation on this entropy to ensure reproducibility at inference time.

To systematize this process, we adopted a covariate classification framework I first saw in a presentation from a Google practitioner at ISF 2023 (unfortunately, I could not find a public reference to link). This framework categorizes features into four distinct levels of accessibility and complexity:

  • Level 1: Univariate Features. Lagged values and rolling statistics derived strictly from the target time series.
  • Level 2: Local Exogenous Features. Structured, readily available external data such as price lists, holiday calendars, and promotion schedules.
  • Level 3: Internal Unstructured Features. Information that exists within the organization but is not model-ready, such as raw text in menu descriptions or operator notes.
  • Level 4: External Global Features. Macroeconomic indicators or competitor actions outside the company’s direct influence. While theoretically valuable, integrating Level 4 data often proves prohibitive for SMEs due to the high costs of external data streams and the complexity of maintaining reliable API integrations for signals that may offer only marginal predictive gains.

Our scoping phase revealed that for the ready-to-eat meal sector, the highest predictive value — and the highest technical debt — resided in Level 3: the unstructured meal descriptions. A raw text string like “Kip met friet en appelmoes” (Chicken with fries and applesauce) contains critical signals regarding protein type and starch content that drive demand cannibalization. However, most models (like the one we discuss later) require structured, categorical inputs rather than raw text to perform optimally.

To bridge this gap, we leveraged Large Language Models (LLMs) — not as forecasters, but as specialized data transformation agents. We adhere to a philosophy of “Agentic Automation” for discrete, verifiable tasks over open multi-agent frameworks. The agent’s role is strictly defined: it ingests heterogeneous recipe information — whether from legacy logs or new entries — and maps it deterministically to a set of predefined categories.

These categories were not established arbitrarily by the model but were defined empirically and in collaboration with domain experts during the scoping phase to capture specific demand variability. Consequently, the agent extracts these concepts into fixed columns such as ProteinGroup (“POULTRY”, “PORK”) and StarchGroup (“FRIES”, “MASH”). This approach effectively converts unstructured noise into high-signal categorical features that explain phenomena like menu fatigue and cannibalization.

Figure 3: The “Agentic” Data Transformation Pipeline. Using specialized LLM agents to deterministically extract categorical demand drivers (e.g., Protein Group, Starch Type) from unstructured, noisy recipe descriptions and save them a Feature Store database.

The result of this phase is a robust, client-specific pipeline that ingests raw, messy operational data and outputs a structured tabular format that explicitly includes the identified demand drivers per case. See the impact of our framework in the figure below:

Figure 4: The Value of Unstructured Data. Percentage of error reduction (over a Seasonal Naive baseline) comparing our ML model with and without the extra covariates extracted by the LLM agent.

We argue that the usage of LLMs will unlock new potential in external covariate construction, introducing a whole new dimension for building more accurate forecasting models.


3. Value-Driven Modelling

Building forecasting models is a well-researched topic — perhaps “too well-researched.” We have a zoo of models and model comparisons that are often non-applicable in production settings due to reliability or consistency issues. Yet, instead of trying to understand what drives performance gains and how to make models more robust, we often take our screwdriver and, through trial and error, attempt to engineer another model to extract an extra 0.01% on a picked metric.

Something is genuinely wrong here. But that is a discussion for another article.

The reality is that from this giant zoo of models, only a handful are used in production, and these are rarely under the spotlight. In my eyes, this is a key reason why academia and practice are currently misaligned. (Note: Here I exclude case-specific models designed by large labs for singular use cases — examples include among others DeepETA for Uber [6] and the Price-Elasticity Transformer from Zalando [7]).

In general, the core principle in model selection for applied forecasting is finding the equilibrium between model capacity, data characteristics (length and width), the quality and quantity of covariates, and the signal-to-noise ratio. It is a dance between overfitting, underfitting, model capacity, and time series relatedness. The safest practice is to start with lower-complexity models and incrementally increase capacity until you reach your first added-value milestone — agreed upon during scoping — or hit diminishing returns. Sometimes, a 0.1% accuracy gain might not be worth the development time better spent on other parts of the forecasting pipeline. Other times, this 0.1% is extremely important.

More on Forecasting Value Added in [8].

For this project, we opted for global Gradient Boosted Decision Tree (GBDT) models, specifically LightGBM, for a number of reasons:

  • Performance Consistency: GBDTs consistently top applied forecasting competitions, with substantial evidence from practitioners regarding their efficacy (e.g., VN1, VN2, M5 competition).
  • Global Modeling: The data availability and complexity across our partners were sufficient to justify a global model over many individual battle-tested local models. Please do not forget to test your favourite global model with benchmarks like (seasonal) Naive, (seasonal) Window Average, ETS and Foundation Models that exceel in setting a good baseline for your optimized model.
  • Off-the-shelf Performance: We needed a model with strong off-the-shelf performance that required minimal, and preferably automated, model structure selection. More on that later.
  • Covariate Handling: Our rich set of covariates, discussed in the previous section, is much easier to incorporate into GBDTs, which have demonstrated dominance in tabular data applications. There are both theoretical (some related work here: [9]) and practical findings (see here [10]) on why GBDTs do better on high-dimensional feature spaces.

Yes, I see you, NN enthusiast. It does better even if you add your favorite embeddings (and introduce this extra layer of complexity/uncertainty). Actually, the GBDTs vs. Neural Networks debate for forecasting is a fantastic one with many factors to consider, but we will leave that for another day.

I will skip the granular details of how we fit our LightGBM. A journal article covering that methodology is coming soon. Spoiler alert: auto-fitting AutoML methods (like AutoGluon here [11]) are underperforming in this domain. We argue that this is a key reason why the performance of GBDTs is underreported in most articles. Fitting a GBDT requires expertise and an in-depth understanding of the underlying dynamics of the task — specifically, building/selecting features/parameters that efficiently explain variability — and the correct order in which to optimize them. Approaches like Grid Search on vast potential feature and hyperparameter spaces are often mathematically doomed.

To that end, we have developed a framework (which captures around 70% of the total LGBM forecasting potential) using a recipe of pre-defined optimization steps rather than blind search. However, we believe squeezing that extra 30% of accuracy depends on the modeler’s expertise to understand the task, how the model works and build proper search configurations.

Figure 5: Our SSOF Optimization Trajectory. Step-by-step error reduction during the automated calibration process. This structured approach captures ~70% of the model’s potential performance before manual expert tuning. Detailed explaination of each step will be provided in a future journal publication.

Before reaching the results section, I want to briefly discuss the pipeline engineering around the model. We believe that this is usually overlooked despite being a fundamental part of building a robust Forecasting System.

Our system is focused on four engineering pillars:

  • Intelligent Gap Filling: Real-world data doesn’t just have missing values. It has structural gaps (holidays, closures, shortages). Naively filling these with zeros or NaNs might teach your model some unwanted behavior (e.g., “no data = no demand”). Context-aware gap filling is essential.
  • Lagged-Window Engineering: In demand forecasting services, many items experience intermittent behavior. Including items that sold twice in six months generates lag features dominated by zeros, creating training noise. We built an adaptive filtering system that smartly drops series with insufficient recent activity and filters out training rows where lag features provide zero gradient signal.
  • Covariate Alignment & The Feature Store: A common failure point in production forecasting is the misalignment between training and inference data, particularly with “Item-Time” covariates like daily menus. In a training context, this data is historical fact. We can easily do cross-validation and split the table accordingly. At inference, it is a future schedule that must be retrieved and mapped correctly. This requires a sophisticated data fusion pipeline. We implemented a Feature Store architecture that acts as the single source of truth for these predictors. It ingests disparate data streams — historical sales, future menu calendars, and static item metadata — and fuses them using composite keys (item_id and timeframe). This engineering ensures that features are handled without future leakage during training and are deterministically mapped from future schedules during inference.
Figure 6: The Data Fusion Engine. A detailed architecture of the ETL pipelines and Feature Store, ensuring that time-based, item-based, and dynamic covariates are correctly aligned for both training and inference.
  • Smart Ensembling: Those familiar with forecasting know the benefit of ensembling (see my article here [12] and an in-depth review here [13]) . In production, however, where time series have no clear patterns (trailing zeros due to out-of-stock, spikes due to promotions, etc.), naively ensembling using the mean or the median can result in out-of-context predictions. We implemented a heuristic-based ensembling combining our LGBM with Seasonal Models (Seasonal Naive and Seasonal Window Average). If the models agree within a certain threshold, we ensemble. If they diverge significantly (e.g., the LGBM has detected a complex pattern due to a promotion), we use a heuristic to pick the correct forecasting model.

To conclude the section, we present some results of our experiments. The point of our article is not to showcase the performance of the model but to detail how to build a proper ecosystem around your picked model.

Table 1: Comparative Model Performance (RMSSE). Evaluation of the Global LightGBM model against statistical baselines and foundation models across five distinct partner case studies.

4. From Predictions to Decisions

As mentioned, the aim of a forecast is to support a decision-making task. It is an input to an optimization function aimed at minimizing a specific high-level business utility function (usually costs). To formalize this, we are looking for the optimal action a* at time t that minimizes an expected loss function:

where:

  • A is the set of actions.
  • L is the utility function (usualy costs)
  • y_t+h is the prediction for time t+h
  • S_t environmental variables (stock-in-hand, stock in transit, etc)

Our system is designed to support two types of decisions: the short-term preparation of meals (Daily Meal Prep) and the medium-term procurement of raw materials (Ingredient Reordering). Furthermore, different partners have different decision-making constraints. Some require a 99% service level (due to agreements with retailers), while others are fine with much lower service levels. The system must be designed with enough flexibility to incorporate such heterogeneity.

Building an optimization engine requires translating forecasting into a risk management problem. A point forecast is just a non-actionable illusion of certainty [2]. To support robust decision-making, we need to quantify the distribution of probable outcomes. To that end, we have developed a non-parametric method that relies on local bootstrapping inference that adapts to the unique volatility of each item. An academic publication discussing this in depth is on its way. Meanwhile, you can look at the slides of my latest presentation at the International Symposium on Forecasting (ISF) in Beijing, China, where I detail an outdated version of our approach, available here: [Link]

On the optimization side, daily meal preparation resembles the classic newsvendor problem. The kitchen must prepare enough meals to meet the daily demand while also minimizing leftovers, since prepared meals have zero residual value the next day. Mathematically, the optimal production quantity is defined by the “Critical Ratio”: q = C_u / (C_u + C_o), where C_o is the cost of overproduction (waste) and C_u is the cost of underproduction (lost sales and reputational damage). However, in practice, accurately quantifying the cost of a lost sale (C_u) — essentially the price of customer disappointment — is notoriously difficult for SMEs.

To simulate the impact of our approach, we relied on a set of heuristics (that include the unique hard constraints set by every client) to quantify inventory costs (we used a similar methodology to [14]). Given these inputs, for every item, the system automatically suggests the corresponding quantile to the user, who is responsible for the final decision. Figure 7 shows the impact ouf our proposed engine in terms of costs for the different partners.

Figure 7: Economic Optimization Results. Average inventory costs by partner and targeted service level. The simulation demonstrates how the probabilistic framework reduces holding costs while ensuring availability.

The second optimization task involves reordering decisions. While demand signals exist at the meal level, inventory is managed at the ingredient level. To address this, we use the deterministic translation layer described in Figure 2. The complexity here lies in the aggregation of uncertainty. A single ingredient, such as potatoes, supports multiple end products. Summing the safety stocks of individual meals would result in bloated inventory (risk pooling). Instead, we aggregate the probabilistic distributions of the dependent meals first, and then calculate the reorder point for the ingredient based on its specific lead time and the target service level.

To operationalize these optimization principles, we engineered a flexible simulation framework. During scoping, we established the reordering constraints (hot/cold ingredients, batches, etc.) and lead times for the different partners. The architecture of the discrete-event simulation framework accepts the probabilistic forecast distributions and the current inventory states as inputs, and iteratively steps through time to simulate the trajectory of inventory positions under these constraints. This allows us to evaluate the efficacy of different ordering policies (actions a_t) against historical demand patterns without exposing our partners to operational risk.

Crucially, the system is agnostic to the specific business goals of each partner. Unique Replenishment Policies are given as configuration objects. This allows us to run identical simulations for a hospital requiring a 99% service level and a budget-conscious caterer targeting 85%, simply by injecting different cost parameters and lead-time constraints into the simulation environment. Based on the simulated trajectories, the system generates recommended ingredient reorder quantities for domain experts, who validate or adjust the output against uncaptured operational nuances.

Empirical evaluation of this framework demonstrates its efficacy, yielding estimated cost savings ranging from 9% to 22%. In addition to optimizing the financial trade-offs identified during scoping, our approach directly addresses the environmental objectives, showing a potential reduction in food waste of up to 52% while adhering to strict service level constraints.

Figure 7: Environmental Impact. Average food waste per meal (kg) across partners. The model significantly outperforms standard practices, particularly at higher service coverage levels.

Our goal is to release a modified version of the simulation engine as a Python package pretty soon. Given its flexibility, we believe it is going to be useful for the community in various scenarios.


5. Deployment

Taking a forecasting model from a Proof of Concept (POC) to production is a task that is usually underestimated. The aim of this section is to bring these two worlds closer together by describing our system architecture and the constraints imposed when taking a forecasting model to production.

Our system has the following requirements:

  • Multi-tenancy: We needed a single platform capable of serving seven distinct partners, each with isolated data schemas and strict privacy requirements.
  • Cost Sensitivity: Unlike technology giants, our partners are SMEs operating with thin margins. They cannot sustain the overhead of servers or compute clusters running 24/7 for a task that occurs once a week.
  • Batch Nature: Forecasting is not a real-time event like fraud detection. It is a heavy, scheduled process that requires significant compute power for a few minutes, followed by days of inactivity.

To align our infrastructure with these realities, we engineered a Serverless Event-Driven Architecture described below:

Figure 8: The Event-Driven Serverless Architecture. A comprehensive view of the deployment stack, separating the Control Module (Vercel), Compute Module (AWS Lambda/Step Functions), and State Module (S3/Supabase) to minimize costs for SMEs.

Our system is composed of 3 core modules:

  1. The Control Module (Vercel): This is the user’s entry point. We built an intuitive dashboard hosted on Vercel that handles authentication and user interaction. Partners log in here to upload their files and review results before their decision making.
  2. The Compute Module (AWS Microservices — Lambdas):
  • Microservice A (The Data Pipeline): A Lambda containing partner-specific preprocessing pipelines converting raw data into a structured format.
  • Microservice B (The Forecaster): A Lambda containing the Fitting-Forecasting-Post Forecasting (Inference) logic of our LGBM module.
  • Microservice C (The Optimizer): A Lambda containing the simulation engine.

3. The State Module (AWS S3 & Supabase)

  • AWS S3 (The Data Lake & Working Memory): S3 serves a dual purpose. First, it is the Data Lake where clients upload raw historical data (CSVs). Second, it acts as the high-speed “glue” for passing heavy intermediate DataFrames (Parquet files) between functions.
  • Supabase (The Long-Term Memory): Stores metadata, job status, and the final structured decisions that the dashboard queries.

The workflow is described bellow:

Phase 1: Ingestion (The Data Lake) The process begins in the Vercel frontend. The user uploads their raw weekly sales data (CSVs or Excel) to a dedicated S3 bucket (s3://raw-data/{client_id}/). Once confirmed, the frontend triggers a Step Function to initiate Phase 2.

Phase 2: The Data Pipeline (Microservice A) The first Lambda spins up to standardize the input. It loads the raw batch from the S3 Data Lake and runs the partner-specific preprocessing logic defined during the Scoping phase.

Phase 2.5: The Update Step (Feature Store & Monitoring) Before forecasting, the system must update its state. This intermediate step takes the cleaned data from Phase 2 and performs three critical actions:

  1. Append to History: The new sales data is appended to the master training DataFrame (the “Full History”).
  2. Update Covariates: The Feature Store (in Supabase) is updated with the newly extracted categorical features (eg menus, holidays).
  3. Ground Truth Evaluation: The system matches the actual sales from this new batch against the predictions made last week. These “True Values” are written to the database, instantly updating the monitoring plots on the frontend so users can visualize model performance.

Phase 3: The Forecaster (Microservice B) This microservice handles the heavy lifting by refitting the Global LightGBM model from scratch (more on that later). Then it post-processes forecasts, generating the full forecast distribution.

Phase 4: The Optimizer (Microservice C) A Step Function triggers the optimization engine, passing the S3 path of the forecast. This service instantly loads the forecast file — restoring the full probability distribution without significant database latency — and runs the optimization simulations. The final actinable results are writtin to Supabase and are immediately queryable by the user’s dashboard.

Phase 5: The Agentic Layer (Interaction) To bridge the gap between raw data and actionable strategy, we integrated a specialized agentic layer as the final step in the pipeline. This layer serves two distinct functions: proactive reporting and reactive decision support.

  • Automated Reporting Agent Once the new forecast is persisted in Supabase, a dedicated reporting agent triggers to contextualize the results. Instead of presenting users only with a static table of numbers, the agent analyzes the new batch against historical performance and inventory patterns to generate a tailored weekly summary. This includes performance analysis, demand drivers weekly influence (specific meal on the menu) and any extra query each partner is interested in.
  • Interactive Chat Agent: In addition, we include a chatting agent (with context awereness) where the users can query their data using natural language to validate the model’s suggestions before finalizing orders. Example queries: “What was the demand for ‘Vol-au-vent’ the last time it was on the menu?” or “How many kg of carrots did we order this time last month?”.

Important Disclaimer: This system was designed as a functional prototype to allow SMEs to investigate the value of a forecasting system. It is not a commercial SaaS product. While robust, we acknowledge that certain architectural decisions may require further optimization for larger-scale production environments.


6. Discussion.

Before concluding, I want to address two aspects of production forecasting that are often overlooked or misunderstood. The first is the frequency of model updates (refitting), and the second is the psychological hurdle of building trust in AI predictions.

Regarding the refitting frequency of Global Operational Forecasting models, the literature is surprisingly sparse. Two notable papers [15] on global models and [16] on local models systematically investigated this issue. Both reached a similar conclusion: less frequent refitting yields massive reductions in computational costs while retaining, or even marginally improving, forecast stability.

We would like to challenge these notions. The “prohibitive cost” argument relies on specific scoping assumptions that do not necessarily hold in modern production environments. The estimated millions in savings assume a legacy approach: either fitting billions of disjointed local models (where Cost = N_{series} * Model_{cost}) or continuously retraining massive architectures that require heavy GPU (or CPU) utilization.

In our serverless architecture, the cost of refitting a global LightGBM model on a weekly batch of data (typical operation for many SMEs) using AWS Lambdas is negligible. Thus, the trade-off shifts entirely to accuracy. Our experiments consistently showed that weekly refitting never degraded performance compared to static models, and most of the time, it improved it. Consequently, we argue that for weekly batch inference (using a production-based model like LGBM) in the SME sector, the “when to refit” question is trivial: you should always refit.

Figure 9: The Case for Weekly Refitting. Analysis of model performance decay over time. The “Performance Decay” trajectory illustrates the necessity of frequent retraining to adapt to shifting data distributions.

The real engineering bottleneck — and where the true computational cost lies — is not in Refitting (updating the trees with new data) but in Recalibration (feature engineering/selection & hyperparameter optimization).

While our SSOF framework described in Section 3 consistently unlocks about 70% of the model’s potential performance (with the remaining 30% requiring expert manual engineering), it is computationally intensive. Running this full recalibration loop every week is wasteful. This leads us to what we believe is the actual frontier for this kind of research: Drift-Triggered Recalibration. Instead of recalibrating on a fixed schedule (e.g., every week, which is costly, vs. every month, which risks using a model on drifted data), the system should only trigger a full structural update when it detects significant Data Drift — a fundamental shift in the underlying distribution of the demand signal. We are currently investigating this trade-off and expect to share our research findings in the near future.

The final barrier to putting a forecasting model into production is not technological — it is psychological. In the SME sector, particularly in high-stakes environments, decision-making has historically relied on “gut feeling” and domain expertise. A kitchen manager knows intuitively the effect of the different external drivers on a meal’s demand. When a “Black Box” model outputs a number without context, it creates friction. If we view the forecasting system purely as a code repository, we fail. The system must be designed to communicate with decision-makers, not just databases.

This friction is a phenomenon known as Algorithm Aversion (see [17] for a comprehensive review) . In our experience, trust in an automated system is incredibly fragile. A human planner might over-produce by 20% for years and call it a “safety buffer,” but if the AI over-produces by 10% once, it is labeled “broken.” Users tend to penalize algorithms more heavily for errors than they do their own judgment. In the early stages of our deployment, we observed that users were looking for reasons to reject the model to regain a sense of control. The turning point where trust is lost usually happens at the first significant variance between the model’s output and the user’s intuition, unless the system provides the mechanisms to explain or validate that variance.

To deal with this, we implemented a “Shadow Mode” protocol. Instead of deploying the model and asking the planner to follow its predictions, we architected a parallel validation phase. For several weeks, the system generated predictions in the background, logging them alongside the planner’s manual orders and the ground-truth actuals. We then visualized and monitored these signals on our dashboard. When a planner consistently sees improved predictions from an ML model, some of this skepticism is mitigated.

Ultimately, we found that trust is built through specific “Aha moments” — instances where the model correctly predicts complex non-linear patterns that human intuition misses. For example, in one case study, the model automatically lowered forecasts for schools when specific proteins (e.g., pork) appeared on the menu due to demographic sensitivities. Seeing the algorithm capture this nuance without manual intervention proved its value more than any metric could.

By combining this “Shadow Mode” validation with regular workshops that decomposed our pipeline, we successfully moved our partners to a hybrid decision-making culture, where the algorithm handles the baseline complexity, and the human expert fine-tunes the decisions.


Conclusion

The objective of this post was to look beyond the forecasting model itself and discuss the architecture of a complete, end-to-end forecasting system. Transitioning from a theoretical Proof of Concept to a production environment requires a holistic approach that treats the model as just one component within a larger decision-support framework.

Our approach is decomposed into five distinct, correlated, and iterative steps. We began by scoping the specific business constraints to ensure our solution aligned perfectly with the partner’s operational decisions. This guided our data collection, where we built robust pipelines to transform and fuse messy, unstructured logs — using LLMs to parse menu descriptions — into clean, actionable predictors. For modeling, we opted for the never-disappointing global GBDT. Crucially, we developed a probabilistic inference method that allowed us to convert these predictions into dec

isions. We tackled two optimization tasks: one using a Newsvendor solution and the second by designing a flexible simulation engine. Finally, we architected and deployed a scalable, serverless, cloud-based system that automates this entire chain, using “shadow mode” validation to bridge the gap between algorithmic output and human trust.

We believe that the gap between academic forecasting research and industrial application remains wide, but it is bridgeable. To close this, our view is that nowadays the spotlight is entirely on new model architectures, overlooking all the other crucial components required to extract value from a forecast. We do not claim that our solution is perfect. It is just an attempt to discuss some of the elements that, in our experience, are important in helping companies take advantage of the rich forecasting literature.

We encourage the community and other applied researchers to share more of their experiences — both good and bad — so we can collectively build a more robust blueprint for the future of applied forecasting.

Thanks for reading.


References

[1] Ma, Shaohui, Stephan Kolassa, and R. Fildes. “Retail forecasting: research and practice.” Lancaster University Management School. Lancaster, UK (2018).

[2] Kolassa, Stephan, Bahman Rostami-Tabar, and Enno Siemsen. Demand forecasting for executives and professionals. Chapman and Hall/CRC, 2023.

[3] Lancaster CMAF. (n.d.). Business Forecasting Principles [YouTube playlist]. YouTube. Retrieved January 27, 2026, from https://www.youtube.com/playlist?

[4] Kourentzes, Nikolaos, Ivan Svetunkov, and Juan R. Trapero. “Connecting forecasting and inventory performance: a complex task.” Available at SSRN 3878176 (2021).

[5] Kourentzes, Nikolaos, Juan R. Trapero, and Devon K. Barrow. “Optimising forecasting models for inventory planning.” International Journal of Production Economics 225 (2020): 107597.

[6] Wu, Fan, and Lixia Wu. “DeepETA: a spatial-temporal sequential neural network model for estimating time of arrival in package delivery system.” Proceedings of the AAAI conference on artificial intelligence. Vol. 33. №01. 2019.

[7] Kunz, Manuel, et al. “Deep Learning based Forecasting: a case study from the online fashion industry.” Forecasting with artificial intelligence: theory and applications. Cham: Springer Nature Switzerland, 2023. 279–311.

[8] Fildes, Robert, Paul Goodwin, and Shari De Baets. “Forecast value added in demand planning.” International Journal of Forecasting 41.2 (2025): 649–669.

[9] Ki, Dohyeong, and Adityanand Guntuboyina. “What Functions Does XGBoost Learn?.” arXiv preprint arXiv:2601.05444 (2026).

[10] McElfresh, Duncan, et al. “When do neural nets outperform boosted trees on tabular data?.” Advances in Neural Information Processing Systems 36 (2023): 76336–76369.

[11] Shchur, Oleksandr, et al. “AutoGluon–TimeSeries: AutoML for probabilistic time series forecasting.” International Conference on Automated Machine Learning. PMLR, 2023.

[12] Theodosiou, F. (2022, December 8). Forecasting with multiple temporal aggregations. Part 1: Uncertainty and forecast combinations. Medium. https://medium.com/@filotastheodosiou/forecasting-with-multiple-temporal-aggregations-part-1-uncertainty-and-forecast-combinations-fdcc1f0646b9

[13] Wang, Xiaoqian, et al. “Forecast combinations: An over 50-year review.” International Journal of Forecasting 39.4 (2023): 1518–1547.

[14] Wang, Shengjie, Yanfei Kang, and Fotios Petropoulos. “Combining probabilistic forecasts of intermittent demand.” European Journal of Operational Research 315.3 (2024): 1038–1048.

[15] Zanotti, M. (2025). On the retraining frequency of global models in retail demand forecasting. Machine Learning with Applications, 22, Article 100769. https://doi.org/10.1016/j.mlwa.2025.100769

[16] Spiliotis, Evangelos, and Fotios Petropoulos. “On the update frequency of univariate forecasting models.” European Journal of Operational Research 314.1 (2024): 111–121.

[17] Jussupow, Ekaterina, Izak Benbasat, and Armin Heinzl. “Why are we averse towards algorithms? A comprehensive literature review on algorithm aversion.” (2020).