Compare forecasts and policies¶

A good comparison changes one thing and keeps everything else fixed: the demand, the opening stock, the lead time, the costs, the scoring window. run_comparison does the bookkeeping. This recipe compares two ordering rules across three SKUs and picks the cheaper one.

1. Scenario¶

Three teas with different demand rates, twelve weeks of daily Poisson demand, and opening stock worth five days of sales:

import numpy as np
import pandas as pd

from stockcast.core import InventoryStateDataFrame, SimulationEngine
from stockcast.policies import OrderUpToPolicy, ReorderPointPolicy

opening = pd.Timestamp("2026-01-05")
rates = {"tea_250g": 6.0, "coffee_1kg": 3.0, "matcha_50g": 1.5}
skus = list(rates)
L, R, n = 2, 4, 84

rng = np.random.default_rng(5)
dates = pd.date_range(opening + pd.Timedelta(days=1), periods=n, freq="D")
demand = pd.DataFrame([
    {"unique_id": sku, "period": p, "date": d, "y": float(rng.poisson(rate))}
    for sku, rate in rates.items() for p, d in enumerate(dates)
])

inventory = InventoryStateDataFrame.from_observed(
    pd.DataFrame({"unique_id": skus, "on_hand": [5 * r for r in rates.values()]}),
    opening_date=opening,
)

2. Two policies from the same forecast¶

Both policies use the same forecast (Poisson demand at each SKU's rate); they differ in how they use it. \((R, S)\) reviews every 4 days and covers \(L + R = 6\) days. \((s, S)\) checks every day, so its reorder point covers \(L + 1 = 3\) days, and orders up to \(s\) plus a week of demand:

paths = np.random.default_rng(0)


def window_quantile(rate, horizon, probability=0.95):
    return float(np.quantile(paths.poisson(rate, size=(10_000, horizon)).sum(axis=1),
                             probability))


rs_targets = pd.DataFrame({
    "unique_id": skus,
    "S": [window_quantile(r, L + R) for r in rates.values()],
    "end": opening + pd.Timedelta(days=L + R),
})
order_up_to = OrderUpToPolicy(lead_time=L, review_period=R, freq="D", service_level=0.95,
                              allow_backorders=False, date_column="end").fit(
    rs_targets, target_column="S", forecast_origin=opening,
)

ss_targets = pd.DataFrame({
    "unique_id": skus,
    "s": [window_quantile(r, L + 1) for r in rates.values()],
    "end": opening + pd.Timedelta(days=L + 1),
})
ss_targets["S"] = ss_targets["s"] + [7 * r for r in rates.values()]
reorder_point = ReorderPointPolicy(lead_time=L, review_period=1, freq="D",
                                   policy_type="sS", service_level=0.95,
                                   allow_backorders=False, date_column="end").fit(
    ss_targets, forecast_origin=opening,
    reorder_point_column="s", order_up_to_column="S",
)

3. Run¶

Two weeks of warm-up let both policies settle before scoring starts:

comparison = SimulationEngine().run_comparison(
    policies={"(R, S) every 4 days": order_up_to, "(s, S) daily check": reorder_point},
    demand_source=demand, inventory=inventory, warmup_periods=14, random_seed=5,
)

4. Score¶

from stockcast.evaluation import (
    InventoryEvaluator, avg_on_hand, fill_rate, order_event_count, total_cost,
)

costs = {
    "holding_cost_per_unit_period": 0.02,
    "shortage_cost_per_unit": 2.00,
    "order_cost_per_sku_line": 5.00,
    "order_cost_per_unit": 0.0,
    "cost_components": ["holding", "shortage", "ordering"],
}


def score(groupby):
    return pd.concat({
        label: InventoryEvaluator().fit(comparison[label], window="scoring").evaluate(
            [fill_rate, avg_on_hand, order_event_count, total_cost],
            groupby=groupby, context=costs)
        for label in comparison
    }).droplevel(1)


score([]).round(3)

                     fill_rate  avg_on_hand  order_event_count  total_cost
(R, S) every 4 days        1.0       13.314                 17      310.92
(s, S) daily check         1.0       19.238                 26      220.80

score(["unique_id"])[["unique_id", "fill_rate", "total_cost"]].round(3)

                      unique_id  fill_rate  total_cost
(R, S) every 4 days    tea_250g        1.0      114.30
(R, S) every 4 days  coffee_1kg        1.0      101.80
(R, S) every 4 days  matcha_50g        1.0       94.82
(s, S) daily check     tea_250g        1.0       87.92
(s, S) daily check   coffee_1kg        1.0       69.98
(s, S) daily check   matcha_50g        1.0       62.90

Results¶

Both rules served every sale. The difference is in how they spend money:

	Order lines	Ordering cost	Holding cost	Total
\((R, S)\) every 4 days	51	255.00	55.92	310.92
\((s, S)\) daily check	28	140.00	80.80	220.80

\((R, S)\) orders every SKU at every review, 51 order lines at €5 each. \((s, S)\) orders a SKU only when it runs low, and then orders a week's worth, so it places about half as many lines and carries a little more stock. With a €5 fee per line, that trade wins for every SKU. Lower the fee, or raise the cost of holding stock, and the answer can flip; rerun score with new rates to find out. (The numbers come from sku_order_line_count, ordering_cost, and holding_cost, evaluated the same way.)

Variations¶

Two forecasts, one policy. Fit the same policy class on targets from two forecasting models and compare them the same way. That measures forecasts by what they do to stock and cost, not only by accuracy.
Rolling refits. Pass policy_schedules=[...] with one snapshot dictionary per policy (Refresh targets as forecasts roll).
Lost sales against backorders. Fit one policy with allow_backorders=False and one with True, and create the inventory without allow_backorders; each branch then runs in its policy's mode.
Many demand paths. Loop over several seeds and average the metrics. Several paths give a distribution of outcomes.
Plots. plot_summary_comparison(comparison) and plot_comparison_dashboard(comparison) draw the comparison (Plots).

Notebook 06 walks through forecast and policy comparisons on five SKUs with smooth forecasts.