9. Choosing the Best Forecasting KPI

Teaching rewrite — the Metrics step. Original wording; examples recomputed.

Introduction

Lesson 8 showed four metrics crowning four different forecasts. This lesson stress-tests them on two hard cases — outliers and intermittent demand — and lands on a practical recommendation: track MAE and bias together, optionally combined into one score.

Metric	Formula
Bias	average(error)
MAE	average(|error|)
MAE%	sum(|error|) / sum(demand)
MAPE	average(|error| / demand)
RMSE	√average(error²)

---

9.1 Extreme demand (outliers)

Metric	Reaction to one huge demand spike
RMSE	Over-reacts — the squared error of the spike can dominate the whole score
MAPE	Barely reacts — the spike’s error is divided by the huge demand, and a too-low forecast’s error is capped at 100%
MAE	Reacts proportionately — affected, but not dominated

Worked example: one outlier

Demand 100, 100, 100, 400; forecast 100 every period → errors 0, 0, 0, −300.

Metric	Value
MAE	300 / 4 = 75 → MAE% = 300 / 700 = 42.9%
RMSE	√(90,000 / 4) = 150 → RMSE% = 150 / 175 = 85.7%
MAPE	(0 + 0 + 0 + 75%) / 4 = 18.75%

One spike doubles RMSE% relative to MAE%, while MAPE makes the miss look small.

---

9.2 Intermittent demand

Many items sell only occasionally. Example: demand over 12 periods is mostly zero, with sales of 9, 6, 12, 9 every third period (average 3 per period).

Forecast	Bias%	MAE%	RMSE%	MAPE	Score (MAE% + |Bias%|)
A: 3 every period	0%	133.3%	147.2%	undefined	133.3%
B: 0 every period	−100%	100%	178.0%	undefined	200%

• MAE prefers B — forecasting zero forever — because the median of intermittent demand is 0. A forecast of zero is useless to a planner.
• MAPE can’t be computed at all (division by zero).
• RMSE and bias prefer A, the useful forecast.
• The combined score also prefers A.

---

Checkpoint: hard cases

Quiz

Checkpoint — sections 9.1 and 9.2

1 / 4

Which metric is most distorted by a single extreme demand spike?

Pick an answer to see if you are right.

---

9.3 The best compromise: MAE + bias

Metric	Simple	Unbiased	Error weighting	Outlier sensitivity
MAE	✅	❌ (aims at median)	Uniform	Moderate ✅
MAPE	✅	❌ (pushes down)	Over-weights low-demand periods 💥	Too low ❌
RMSE	❌	✅	Over-weights big errors	Too high ❌
MAE + Bias	✅	✅	Uniform	Moderate ✅

No single metric is perfect, so combine two:

Score = MAE + |Bias| Score% = MAE% + |Bias%|

The bias term pulls MAE’s median-seeking (downward) tendency back toward the mean.

Use the absolute bias

Forget the absolute value and you reward under-forecasting (negative bias would lower the score).

• Use the score to compare forecasts quantitatively.
• With management, show MAE and bias separately — easier to understand than a combined percentage.
• Track it for every relevant lag (lesson 6); analysts look per lag, management may prefer a single scorecard.

---

9.4 Case study — part 2

Forecast	Bias%	MAE%	MAPE	RMSE%	Score%
1 (380)	−0.5%	68.4%	352.9%	94.4%	68.9%
2 (260)	−31.9%	65.7%	239.1%	99.6%	97.6%
3 (200)	−47.6%	67.6%	187.1%	105.7%	115.2%

Forecast 1 wins clearly: 68.4 + 0.5 = 68.9%.

Forecasts 2 and 3 carry huge negative bias (−32% and −48%). Any consistent bias beyond about ±5% should be treated as suspicious — something is wrong with the model or the process. If safety stocks are set in days of cover, under-forecasting also shrinks the buffers, compounding shortages.

How to say it to management: “Forecast 1 is the best balance of accuracy and bias, and both matter for running the supply chain.”

---

Checkpoint: the score

Quiz

Checkpoint — sections 9.3 and 9.4

1 / 4

MAE% 40%, Bias% −8%. Score%?

Pick an answer to see if you are right.

---

Rank: choosing metrics

Quiz

Metric selection

1 / 2

Rank these metric set-ups from MOST to LEAST recommended for a planning process.

OrderDrag the handle to rearrange — top should be first.

1. ⋮⋮1RMSE and bias for a single product
2. ⋮⋮2MAPE alone
3. ⋮⋮3MAE alone
4. ⋮⋮4MAE and bias (or their combined score), per lag

Drag items into the correct order (or use ↑↓), then check.

---

Check: choosing the KPI

Quiz

Choosing the best forecasting KPI

1 / 5

Which pair is recommended for assessing forecast quality?

Pick an answer to see if you are right.

---

Takeaways

Remember

• RMSE over-reacts to outliers; MAPE under-reacts and breaks on zeros; MAE alone can favour a zero forecast for intermittent items
• Use MAE + bias — or Score = MAE + |Bias| — as the default
• Treat consistent |bias| > ~5% as a red flag
• Show MAE and bias separately to management; compute per lag
• Avoid MAPE

Further Reading

• 8. Forecasting metrics
• Next: 10. What is a good forecast error?
• Background: N. Vandeput, Demand Forecasting Best Practices (Manning) — used as a study outline