Step 7 of 9

Evaluate a run¶

InventoryEvaluator reads the event table and computes metrics over the window and grouping you choose.

Setup: the tea shop from steps 1 to 5

import numpy as np
import pandas as pd

from stockcast.core import InventoryStateDataFrame, SimulationEngine
from stockcast.policies import OrderUpToPolicy
from stockcast.utils import DemandGenerator

sku = "tea_250g"
opening_date = pd.Timestamp("2026-01-05")
lead_time, review_period = 2, 4
horizon = lead_time + review_period

demand = DemandGenerator(
    [sku], first_date=opening_date + pd.Timedelta(days=1), freq="D",
    random_seed=3, negative_demand_handling="clip_zero",
).seasonal(n_periods=56, base=6.0, amplitude=2.0, season_length=7, std=2.0)
demand["y"] = demand["y"].round()

inventory = InventoryStateDataFrame.from_observed(
    pd.DataFrame({"unique_id": [sku], "date": [opening_date], "on_hand": [30.0]}),
    allow_backorders=False,   # some pages step this state by hand
)

paths = np.random.default_rng(42).poisson(6.0, size=(10_000, horizon))


def tea_policy(probability):
    """Order-up-to policy whose target is the given quantile of 6-day demand."""
    target = pd.DataFrame({
        "unique_id": [sku],
        "target": [np.quantile(paths.sum(axis=1), probability)],
        "date": [opening_date + pd.Timedelta(days=horizon)],   # the last day it covers
    })
    return OrderUpToPolicy(
        lead_time=lead_time, review_period=review_period, freq="D",
        service_level=probability, allow_backorders=False,
    ).fit(target, target_column="target")


policy = tea_policy(0.95)
run_settings = dict(random_seed=3)
result = SimulationEngine().run(
    policy=policy, demand_source=demand, inventory=inventory, **run_settings,
)

Fit and evaluate¶

The evaluator follows the same fit / apply rhythm as policies. fit selects the event table rows; evaluate computes the metrics:

from stockcast.evaluation import (
    InventoryEvaluator, avg_on_hand, cycle_service_level, fill_rate, order_event_count,
)

evaluator = InventoryEvaluator().fit(result)   # the scoring window
evaluator.evaluate(metrics=[fill_rate, avg_on_hand, order_event_count])

   fill_rate  avg_on_hand  order_event_count
0        1.0    18.607143                 14

Two choices shape the answer:

window: which part of the run you are describing. fit uses the scoring window, like result.summary(); pass window="all", "warmup", or "settlement" for another part.
groupby: the grain of the answer. By default everything is pooled into one row; groupby=["unique_id"] gives one row per SKU, and any event table column works.

Service metrics¶

"Service level" means different things to different people, so Stockcast gives each meaning its own metric.

Fill rate: the share of demand served straight from the shelf.

\[ \text{fill rate} = \frac{\sum_t \text{fulfilled}_t}{\sum_t \text{demand}_t} \]

Demand-period service level: the share of periods with demand in which nothing was short.

\[ \frac{\#\{t : \text{demand}_t > 0,\ \text{shortage}_t = 0\}}{\#\{t : \text{demand}_t > 0\}} \]

Cycle service level: the share of replenishment cycles (from the arrival of one order to the arrival of the next) with no shortage at all. An order that arrives in several deliveries counts once. Because the first and last cycles of a run are cut short, you choose whether to include them:

evaluator.evaluate(
    metrics=[cycle_service_level],
    groupby=[],
    context={"include_partial_cycles": False},
)

   cycle_service_level
0                  1.0

A target probability of 0.95 is a statement about the forecast. These metrics measure what the whole system delivered. They are related, and they are not the same number. Comparing the two is one of the main uses of a simulation.

Costs¶

Cost metrics multiply event table quantities by rates you supply in context:

Metric	Formula	Rate key(s)
`holding_cost`	\(\sum_t h \cdot \text{on hand}_{\text{end},t}\)	`holding_cost_per_unit_period`
`shortage_cost`	\(\sum_t p \cdot \text{shortage}_t\)	`shortage_cost_per_unit`
`ordering_cost`	\(\sum_t (K \cdot \text{lines}_t + c_o \cdot q_t)\)	`order_cost_per_sku_line`, `order_cost_per_unit`
`purchase_cost`	\(\sum_t c \cdot q_t\)	`purchase_cost_per_unit`
`total_cost`	sum of the components you list	`cost_components` + their rates

from stockcast.evaluation import holding_cost, shortage_cost, total_cost

costs = {
    "holding_cost_per_unit_period": 0.02,   # €0.02 per pack per day on the shelf
    "shortage_cost_per_unit": 2.00,         # lost margin per pack not sold
    "order_cost_per_sku_line": 5.00,        # delivery fee per order
    "order_cost_per_unit": 0.0,
    "cost_components": ["holding", "shortage", "ordering"],
}
evaluator.evaluate(
    metrics=[holding_cost, shortage_cost, total_cost],
    groupby=[],
    context=costs,
)

   holding_cost  shortage_cost  total_cost
0         20.84            0.0       90.84

You set every rate, including zeros. Unspecified costs are not filled in, so a total contains only the components you list.

When you only need the total, give the rates to TotalCost. Each rate you pass adds its component:

from stockcast.evaluation import TotalCost

evaluator.evaluate([TotalCost(holding=0.02, shortage=2.00,
                              order_per_line=5.00, order_per_unit=0.0)])

   total_cost
0       90.84

Custom metrics¶

A metric is any function that takes the event table slice and the context:

def units_left_at_the_end(events, context=None):
    return events.sort_values("period")["ending_on_hand"].iloc[-1]

evaluator.evaluate(metrics=[fill_rate, units_left_at_the_end], groupby=["unique_id"])

  unique_id  fill_rate  units_left_at_the_end
0  tea_250g        1.0                   25.0

Summary¶

InventoryEvaluator().fit(result).evaluate(metrics), with optional window, groupby, and context.
Fill rate, demand-period service level, and cycle service level answer three different questions.
Costs use rates you supply; custom metrics are plain functions.

See also: Evaluation and metrics · Put costs on a run · Notebook 06

Next: Compare scenarios