Skip to main content

10. What Is a Good Forecast Error?

Teaching rewrite — the Metrics step: judging accuracy against a fair benchmark. Original wording; examples recomputed.

Introduction​

Your tool reports an MAE of 36% and a bias of +1.6%. Good or bad? Common answers don’t hold up:

Common answerProblem
“Better than last year”What if last year was exceptional (e.g. a pandemic year)?
“Better than our target”Chicken-and-egg: how was the target set?
“Better than some other forecast”Which one?
“It looks right — follows trend and seasonality”Not standardised; doesn’t scale to thousands of items
“Industry benchmark”Not comparable (lesson 12)
“Demand COV says it’s hard to forecast”Misleading — see 10.2

Achievable accuracy depends on the data: national monthly phone sales are far easier than one model at one store on one day; collaborative manufacturers can be very accurate, fashion struggles. You can’t know a “good” error in advance — you must compare against a simple benchmark on the same data.

This lesson covers:

  • Naïve, moving-average, and seasonal benchmarks
  • Fair benchmarking rules (same lag, out-of-sample, ongoing)
  • Why the coefficient of variation (COV) is a poor measure of forecastability

10.1 Benchmarks​

A benchmark is a simple, free forecasting method. If your model or process can’t beat it, it isn’t adding value — and if the benchmark wins, you have just found your new model.

Naïve forecast​

Always predicts the last observed value. Easy to beat — so beating it proves little. Be wary of vendors who celebrate beating the naïve forecast.

Moving average​

Average of the last n periods; 3–6 periods usually works best. A much fairer contender.

Worked example: choosing a benchmark​

Demand history: 100, 120, 90, 110, 130, 95, | 105, 125, 100, 115, 120, 90 (forecast the last 6 periods, one period ahead).

BenchmarkMAE%Bias%
Naïve (MA1)16.0%+0.8%
MA39.7%+1.5%
MA69.3%+0.4%

Use the best one (MA6 here) — otherwise you compare yourself with an artificially weak benchmark.

Seasonal benchmarks​

For seasonal demand, use the same period in previous cycles:

  • Seasonal naïve — last cycle’s same period (e.g. last Monday for next Monday).
  • Seasonal moving average — average of several cycles (e.g. the last three Mondays).

For a product with a strong weekday pattern, a seasonal moving average can cut MAE roughly in half compared to a plain moving average that mixes Mondays with Saturdays.

Rules for fair benchmarking​

  1. Try several moving averages and pick the best.
  2. Same lag — a forecast made three months ahead must be compared with a benchmark that only had data up to three months before.
  3. Out-of-sample — don’t compare on the period used to choose or tune your model. A model picked as the best of 15 on 2025 data will naturally beat the benchmark on 2025; the real test is 2026. Otherwise you are measuring overfitting.
  4. Keep tracking — benchmarking is continuous, not a one-off (lesson 12).

Checkpoint: benchmarks​

Quiz

Checkpoint — section 10.1

1 / 4

Why is the naïve forecast a weak benchmark?

Pick an answer to see if you are right.

10.2 Why COV is not a forecastability measure​

Coefficient of variation: COV = σ / μ (standard deviation ÷ mean).

Many tools use COV to label items “easy” or “hard” to forecast. Problems:

ProblemExplanation
Data leakageCOV compares each period with the same period’s overall mean — information a forecaster never has. Comparing 2025 demand with the 2025 average looks better than any genuine forecast made from 2024 data
Trend inflates COVA steadily growing product has high variation around its mean, yet a moving average follows it easily
Seasonality inflates COVA highly seasonal product has high COV, yet a seasonal benchmark forecasts it well

Only for flat demand (no trend, no seasonality) does COV roughly track forecastability — and even then it flatters itself through leakage. Deseasonalising or detrending first doesn’t fix it: that also uses hindsight and isn’t a simple benchmark.

Worked example: trend fools COV​

Demand grows 10 units per period: 10, 20, 30, 40, 50, 60.

  • Mean = 35; standard deviation ≈ 17.1 → COV ≈ 49% (“hard to forecast”).
  • A simple trend-following forecast (last value + last change) predicts each period exactly from period 3 on → error 0.

High COV, perfectly forecastable.

Better: judge forecastability by the error of a good benchmark on the item.

Checkpoint: COV​

Quiz

Checkpoint — section 10.2

1 / 4

COV is computed as…

Pick an answer to see if you are right.

Rank: benchmarking practice​

Quiz

Benchmarking practice

1 / 2

Rank the steps to judge whether a forecast is “good”.

OrderDrag the handle to rearrange — top should be first.

  1. 1Compare scores; if the benchmark wins, adopt it or fix the model.
  2. 2Pick the best benchmark on the evaluation period.
  3. 3Repeat every cycle.
  4. 4Compute several simple benchmarks (MA3–MA6, seasonal MA) at the same lag.
  5. 5Compute your forecast’s MAE and bias on the same out-of-sample period.

Drag items into the correct order (or use ↑↓), then check.

Check: good forecast error​

Quiz

What is a good forecast error?

1 / 5

What is the best way to know if a 36% MAE is good?

Pick an answer to see if you are right.

Takeaways​

Remember
  • You can’t know a good error in advance — benchmark it
  • Naïve is too weak; use the best of several moving averages (3–6) or seasonal benchmarks
  • Compare at the same lag, out-of-sample, and continuously
  • COV leaks future information and is inflated by trend and seasonality — don’t use it as forecastability
  • Judge forecastability by benchmark error

Further Reading​