Skip to main content

8. Forecasting Metrics: Bias, MAE, MAPE, RMSE

Teaching rewrite — the Metrics step of the framework (Part 2: measuring forecasting quality). Original wording; all numbers recomputed.

Introduction​

You can only improve what you measure. But the metric you choose shapes the forecasts people produce: one metric quietly rewards under-forecasting, another over-reacts to outliers, another ignores your most valuable products. There is no single perfect indicator — each avoids some traps and falls into others.

This lesson covers:

  • Accuracy vs bias — two different questions
  • How to compute Bias, MAE, MAPE, and RMSE, plus their percentage versions
  • What each one rewards or punishes
  • A case study: three forecasts, four metrics, four different winners
Measure against demand

Compare forecasts with demand, not constrained sales (lessons 2–3). Otherwise a forecast that predicted a shortage looks perfect.

8.1 Accuracy and bias​

ConceptQuestion it answers
AccuracyHow big are the errors? (spread between forecast and actual, ignoring direction)
BiasWhich direction do errors lean? Are forecasts on average too high or too low?

Think of a dartboard: darts tightly grouped but off to one side are accurate-ish but biased; darts scattered evenly around the bullseye are unbiased but inaccurate. You want tight and centred.

The two can conflict:

  • Unbiased but inaccurate — swings wildly around demand → noisy signal to production.
  • Accurate but biased — always a bit low → steady lost sales. (Consistent bias often comes from intentional manual edits; see the review-process lessons.)

8.2 Error and bias​

Error for period t:

et = ft − dt (forecast minus demand)

Positive = over-forecast; negative = under-forecast.

Bias = average error = Σet / n

Bias% = Σet / Σdt (total error ÷ total demand over the same periods)

Worked example (used throughout this lesson)​

PeriodDemandForecastErrorAbs. errorAPESquared error
Jan1015+5550%25
Feb512+77140%49
Mar85−3337.5%9
Apr108−2220%4
Total3340+71787
  • Bias = 7 / 4 = 1.75 units
  • Bias% = 7 / 33 = 21.2% (over-forecasting)

Why scale? A bias of 43 units is excellent if average demand is 10,000, terrible if it is 20.

Attention points
  • Compute bias over many periods (or many products), not one product in one period.
  • Divide error and demand over the same time window — a classic spreadsheet mistake is dividing by demand from a longer range.
  • Bias alone can hide big errors: +50 and −50 average to zero.

How much bias is normal? Statistical and ML engines often end slightly negative across portfolios: they see old products declining but can’t foresee new launches. Human processes, by contrast, often drift positive under management pressure.

8.3 Mean Absolute Error (MAE)​

MAE = Σ|et| / n MAE% = Σ|et| / Σdt

From the table: MAE = 17 / 4 = 4.25 units; MAE% = 17 / 33 = 51.5%.

Compute MAE% from the totals at the end — not period by period (that would be MAPE).

ProsCons
Simple to compute and explainMinimising MAE pulls the forecast toward the median of demand, which is usually below the mean for skewed demand → tends to under-forecast
Easy to express as a %

Median vs mean: values 1, 1, 2, 3, 4, 4, 5, 6, 6, 100 have median 4 but mean 13.2.

8.4 Mean Absolute Percentage Error (MAPE)​

MAPE = (1/n) Σ |et| / dt — each period divided by its own demand

From the table: (50% + 140% + 37.5% + 20%) / 4 = 61.9%.

Why it is flawed:

  • Skewed weighting — an error in a low-demand period counts far more than the same error in a high-demand period (Feb’s 7-unit error = 140%; Jan’s 5-unit error = 50%).
  • Asymmetric cap — forecasting 0 can never exceed a 100% error, but over-forecasting has no ceiling → minimising MAPE pushes forecasts down.
  • Undefined whenever demand is zero.
Naming confusion

Many people compute MAE% (total abs error ÷ total demand) and call it “MAPE”. Always state the formula when comparing numbers.

Checkpoint: bias, MAE, MAPE​

Quiz

Checkpoint — sections 8.1 to 8.4

1 / 4

Demand 20, 30; forecast 25, 20. What is the bias?

Pick an answer to see if you are right.

8.5 Root Mean Square Error (RMSE)​

RMSE = √( Σet² / n ) RMSE% = RMSE / average demand

Read the name backwards as a recipe: error → square → mean → root.

From the table: √(87 / 4) = √21.75 = 4.66 units; average demand = 33/4 = 8.25 → RMSE% = 56.5%.

ProsCons
Penalises big errors most — matching supply chain reality (large misses cause stock-outs and dead stock; small ones barely matter)Harder to explain
Linked to safety-stock formulasOver-reacts to outliers
Minimising it tends to give an unbiased forecast (it aims at the mean)Doesn’t scale well across portfolios (lesson 11)

8.6 Case study — three flat forecasts​

A product has 21 periods of demand (total 8,019, average ≈ 382, median 254). Demand is volatile: 170, 254, 360, 221, 351, 1,034, 632, 142, 421, 200, 405, 904, 8, 52, 528, 60, 97, 1,482, 70, 233, 395.

Three (deliberately simple) flat forecasts:

ForecastValue every periodBias%MAE%MAPERMSE%
1380−0.5%68.4%352.9%94.4%
2260−31.9%65.7%239.1%99.6%
3200−47.6%67.6%187.1%105.7%

Look at the bold “winners”:

  • Bias and RMSE pick Forecast 1 (≈ the mean).
  • MAE picks Forecast 2 (closest to the median, 254).
  • MAPE picks Forecast 3 — the lowest forecast — exactly the downward pull described above.

Which would you choose for your supply chain? Lesson 9 settles it.

Checkpoint: RMSE and the case study​

Quiz

Checkpoint — sections 8.5 and 8.6

1 / 4

Errors over 4 periods: 3, −1, 4, −2. RMSE?

Pick an answer to see if you are right.

Rank: metric mechanics​

Quiz

Metric mechanics

1 / 2

Rank the RMSE recipe steps.

OrderDrag the handle to rearrange — top should be first.

  1. 1Average the squares
  2. 2Take the square root
  3. 3Square each error
  4. 4Compute each error

Drag items into the correct order (or use ↑↓), then check.

Check: forecasting metrics​

Quiz

Forecasting metrics

1 / 6

Accuracy measures ____; bias measures ____.

Pick an answer to see if you are right.

Takeaways​

Remember
  • Track accuracy (size) and bias (direction) — they can conflict
  • Error = forecast − demand; Bias% = total error ÷ total demand
  • MAE is simple but aims at the median → slight under-forecast
  • MAPE over-weights low-demand periods, rewards under-forecasting, breaks on zeros — avoid it
  • RMSE punishes big errors, aims at the mean, but over-reacts to outliers
  • Different metrics can crown different forecasts

Further Reading​