Skip to main content

11. Measuring Forecasting Accuracy on a Product Portfolio

Teaching rewrite — the Metrics step, scaled from one item to thousands. Original wording; numbers recomputed.

Introduction​

Lessons 8–10 judged one time series. Planners manage hundreds or thousands. Adding up errors across products mixes apples and pears: a 500-unit miss on nails and a 2-unit miss on anvils are not equally important. Many companies retreat to reporting accuracy per category to reduce the mix — but there is a better way: value-weighted KPIs.

This lesson covers:

  • Why unweighted totals over-emphasise high-volume items
  • Why averaging percentages across products fails
  • Value-weighted bias, MAE, and RMSE
  • Why RMSE doesn’t scale to portfolios
  • Why you shouldn’t penalise over- and under-forecasts differently

11.1 Unweighted metrics on a portfolio​

ProductDemandForecastErrorAbs. errorSquared error
Anvil1210−224
Hammer100150+50502,500
Nail1,5001,000−500500250,000
Total1,6121,160−452552
  • Bias% = −452 / 1,612 = −28.0%
  • MAE% = 552 / 1,612 = 34.2%

Both are driven almost entirely by nails — the cheapest item — because they sell in the largest volume.

Averaging percentages doesn’t help​

ProductError %Absolute % error
Anvil−16.7%16.7%
Hammer+50.0%50.0%
Nail−33.3%33.3%
Average0.0%33.3%

The average error % says zero bias — clearly false (we under-forecast by 452 units overall). Averaging percentages also treats every product as equally important and breaks on zero demand. Averaging percentages across products and periods is (almost) always a bad idea.

11.2 Value-weighted KPIs​

Give each product a weight w reflecting its importance — e.g. unit cost (to manage invested capital), margin (to manage profit), price, or a strategic weight.

Weighted error: ew = w × (f − d) Weighted demand: dw = w × d

KPIAbsolutePercentage
BiasΣew / nΣew / Σdw
MAEΣ|ew| / nΣ|ew| / Σdw
RMSE√(Σew² / n)RMSE / average dw

Errors become money: a 5-unit error on a 2item=2 item = 10.

Worked example: weighting by unit cost​

ProductDemandForecastUnit costWeighted demandWeighted errorWeighted abs. error
Anvil1210€50€600−€100€100
Hammer100150€20€2,000+€1,000€1,000
Nail1,5001,000€0.10€150−€50€50
Total€2,750+€850€1,150
  • Weighted Bias% = 850 / 2,750 = +30.9% (over-invested, mainly in hammers)
  • Weighted MAE% = 1,150 / 2,750 = 41.8%

The story flips: the hammer is now the main offender, not the nail. Even the sign of the bias changed — in units we were short; in money we were over-stocked.

Checkpoint: weighting​

Quiz

Checkpoint — sections 11.1 and 11.2

1 / 4

Why do unweighted portfolio metrics mislead?

Pick an answer to see if you are right.

11.3 RMSE doesn’t scale​

Squaring errors across very different items lets a few large items dominate. In the example:

  • Unweighted RMSE ≈ √((4 + 2,500 + 250,000) / 3) ≈ 290 units — almost all from nails.
  • Weighted RMSE ≈ √((10,000 + 1,000,000 + 2,500) / 3) ≈ €581 — almost all from hammers.

RMSE can be useful for one item; for portfolios, use weighted MAE + weighted bias.

11.4 Don’t penalise over- and under-forecasts differently​

It is tempting to weight positive and negative errors differently because the costs differ: one unit too many costs holding or spoilage; one unit too few costs a sale and goodwill. But a metric that punishes under-forecasting more will produce biased forecasts. Other teams will notice, lose trust, and start making their own forecasts — or second-guessing yours (“planners inflate, so I’ll cut”).

Handle cost asymmetry where it belongs: in service-level targets and safety stocks set by inventory management — not in the demand forecast.

Checkpoint: RMSE and asymmetry​

Quiz

Checkpoint — sections 11.3 and 11.4

1 / 4

Why avoid RMSE for a whole portfolio?

Pick an answer to see if you are right.

Rank: portfolio metrics​

Quiz

Portfolio metric choices

1 / 2

Rank these portfolio metrics from MOST to LEAST recommended.

OrderDrag the handle to rearrange — top should be first.

  1. 1Unweighted MAE% across all items
  2. 2Value-weighted MAE% and Bias%
  3. 3Average of item-level MAPEs
  4. 4Unweighted MAE% and Bias% per category

Drag items into the correct order (or use ↑↓), then check.

Check: portfolio accuracy​

Quiz

Measuring accuracy on a product portfolio

1 / 4

What does a value-weighted KPI do?

Pick an answer to see if you are right.

Takeaways​

Remember
  • Unweighted totals are dominated by high-volume items; averaged percentages hide bias and ignore importance
  • Use value-weighted MAE and bias (weights: cost, margin, price, or strategic)
  • Weighting can change both who the offender is and the sign of the bias
  • RMSE doesn’t scale to portfolios
  • Don’t encode cost asymmetry in the forecast metric — use service levels and safety stock

Further Reading​