How to Check a Digital Twin’s Accuracy Claim
Six Questions to Ask Before You Trust a Line Simulation

Every line simulation vendor says its model is accurate. Very few say accurate against what, over what period, or how it was checked. This guide gives you the questions that turn an accuracy claim into something you can test, whoever the vendor is, us included.

Why the claim needs checking

A model can be made to match almost any single number. Tune enough parameters against last month’s OEE and the model will reproduce last month’s OEE, whether or not it understands the line. That match tells you the model can be fitted. It doesn’t tell you the model can predict, and prediction is the only reason to buy it. You’ll use it to ask what happens if you add a buffer, speed up a filler or fix a failure mode, and none of those futures has been measured yet.

The questions below separate a model that has been fitted from one that has been validated.

1. Accurate against what?

The only reference that counts is the plant’s own measured performance: OEE, or good output against scheduled time, taken from the line’s own records. A match against rated speeds, a vendor benchmark, another model or a demonstration line says nothing about yours.

Ask: What measured figure was the model compared with, and where did that figure come from?

2. Over what period?

Lines lose most of their output to stops, and the stops that matter most are often the rare, long ones. A one-shift or one-week comparison can miss them entirely. The comparison period should be long enough to include the line’s normal mix of products, changeovers and long repairs. Months is a reasonable floor, and a year covers seasonal mix.

The simulated period should match the measured one too. A one-year simulation compared with a year of plant data is a like-for-like test, and a single short run compared with an annual average is not.

Ask: How long was the comparison period, and did the simulated run cover the same span?

3. Was it tested out of sample?

This is the question that matters most. If the failure and repair data were fitted on the same weeks the model was scored on, a good match is partly guaranteed. The stronger test fits the model on one period and scores it on another the model has never seen. Failing that, the model should be frozen, with no more tuning, before the scoring data is looked at.

Ask: Which period was used to build and fit the model, and which period was used to score it? Was anything adjusted after the scoring data was seen?

4. Does it match part by part, not just in total?

A line-level OEE can match while the parts underneath are wrong. One machine’s stops can be overstated and another’s understated, and the errors cancel. That model will still give the wrong answer the first time you ask what fixing one of those machines is worth, which is exactly the question you bought it for.

A validated model should also reproduce the parts:

• Each machine’s availability, and the stop counts and durations for each failure mode.
• Starved and blocked time for each machine, which shows whether the model’s buffers and speeds interact the way the line’s do. Finding the bottleneck explains why these two figures matter.
• Output by product or product category, because a line that runs several formats rarely runs them all equally well.

Per-failure-mode detail is also what makes the model’s gains believable. The published paper below puts it plainly: with one averaged failure mode per machine, the gains and losses of different fixes would be “less accurate and essentially identical to each other.” Micro-stops vs breakdowns shows why.

Ask: Show me the comparison for each machine and each product category, not just the line total.

5. What does “within N” mean?

“Within 2%” can mean several different things, and the difference is large.

• Points or percent. At 60% OEE, two percentage points is 58% to 62%. Two percent relative is 58.8% to 61.2%. Ask which one is meant.
• One run or many. A simulation with random stops gives a different answer on every run. A fair claim compares the average over many replications with the measured figure and reports the spread. A single lucky run proves little.
• Line total or every category. “Within N points for the plant” is a weaker claim than “within N points for each product category.”
• Before or after tuning. See question 3.

A precise claim reads something like this: the average of the replications of a one-year run differs from a year of measured OEE by less than N percentage points, for each product category, with the model built from a period other than the one it was scored on. The closer a vendor’s claim comes to that sentence, the more it’s worth.

Ask: Points or percent, how many replications, and at what level of detail?

6. Who checked it, and can you see it?

A claim made in a sales deck is weaker than one made in a peer-reviewed paper. A model checked by the vendor is weaker than one checked by someone who doesn’t sell it. The best evidence is a published method, an independent validator and data you can inspect.

Ask: Is the validation published? Who did it? Can I see the comparison, and better still, run the same test on my own line?

The checklist

Our published validation

Here is how ReliaSim answers the same questions.

Fischel and Lange modeled a real multi-line food plant and published the work at the 2020 Winter Simulation Conference. Their model is a discrete rate and reliability model with up to twenty failure modes on each of more than twenty unit operations. Before anyone used it for a decision, a one-year simulation had to land within 1% of the plant’s actual OEE for each product category, and it did.

That published model was then rebuilt in ReliaSim and independently validated by Tom Lange, a co-author of the paper. The ReliaSim model came within 1% of the plant’s measured OEE, and within 1% of the original model as well. Here “within 1%” means the simulated OEE differs from the measured OEE by less than one percentage point.

• Against what: the plant’s measured OEE.
• Over what period: a one-year simulation compared with a year of plant data.
• Part by part: per failure mode in the model, and scored for each product category.
• Who checked it: published at a peer-reviewed conference, and the ReliaSim rebuild validated by one of the paper’s authors.

The full write-up is the published OEE validation case study, and ReliaSim vs ExtendSim covers the rebuild and the run times. The same build, validate, then predict order is the ReliaSim method, and it applies to every model we build.