An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

Decisions and Modelsintermediate

Designing a KPI that works

Q: A support team is measured on average time to first response. Which companion measure best protects against the obvious gaming?

the share of tickets resolved without the customer having to make contact again. The cheapest way to cut first-response time is a fast, empty acknowledgement, which leaves the customer to come back. A measure of issues genuinely settled on the first exchange moves the wrong way exactly when that gaming happens. Ticket volume and staffing describe workload rather than quality, and last quarter figure is only a benchmark for the same flawed measure.

Q: Two teams report conversion rates of 2.5 and 3.4 per cent for the same month and the same site. What should be inspected first?

the written definition each team used, covering numerator, denominator, filters and time window. The arithmetic, the refresh timing and a regional or order-type exclusion could each produce a gap this size, so none of them stands out as the place to start. The written definitions do, because the numerator, the denominator, the filters and the time window sit in one place and a side-by-side read takes a single pass. Sessions against unique visitors is the usual culprit, since people return and sessions therefore exceed visitors. Definitions will not settle everything on their own. If the two read the same, an arithmetic slip, a stale table or an undocumented exclusion is still live, and each needs its own check. If neither team has written its definition down, that is already the finding.

A key performance indicator is a number chosen to steer behaviour, which is what separates it from an ordinary statistic. A workable KPI has an unambiguous definition covering numerator, denominator and time window, an owner who can move it, and a link to a decision someone will actually make. It also needs a guardrail, a second measure that catches the damage the first one invites. The failure mode is predictable. Once a measure carries consequences, people optimise the measure, and any gap between the measure and the goal becomes the thing the organisation pays for.

Why it matters

A call centre told to cut average call time will cut it. Some of that comes from working smarter and some from hanging up on people, and the number itself cannot tell the difference. That is why a measure carrying a bonus needs a companion measure that would move the wrong way if people gamed it. Choose the pair together, never one on its own.

Before you read on — recall

A support team is measured on average time to first response. Which companion measure best protects against the obvious gaming?

Formulas

A metric tree

R = V \times c \times \bar{A}

Revenue

R

decomposes into visitors

V

, conversion rate

c

and average order value

\bar{A}

. With 200,000 visitors, a conversion rate of two and a half per cent and an average order of A$80, revenue is A$400,000. Lifting conversion alone to two point eight per cent takes it to A$448,000. The decomposition matters because each factor has a different owner and a different lever, so it tells you who can act.

Why the denominator is a decision

c = \frac{\text{orders in the window}}{\text{unique visitors in the window}}

Swap unique visitors for sessions and the same business reports a different conversion rate, usually a lower one, because a single person may visit several times. Neither definition is wrong. What is wrong is leaving the choice unwritten, because two teams then report different numbers for the same month and the meeting turns into an argument about arithmetic.

Worked examples

Scenario

A logistics company makes on-time delivery percentage the headline KPI for depot managers, with a quarterly bonus attached.

Solution

Within a quarter the number improves. Some of the gain is real. Some comes from quoting longer delivery windows, which makes on-time easier to hit while customers wait longer, and some from recording a delivery as complete when the van reaches the suburb rather than the door. Neither is fraud, and both are rational responses to the measure as written. The repair is to define the promise date as the one quoted to the customer at order time, then pair the KPI with average quoted lead time and with delivery complaints per thousand parcels.

Scenario

A university service desk reports 40 numbers on a monthly dashboard. Leadership says the dashboard is not useful.

Solution

Forty numbers is a reporting habit rather than a set of indicators. Ask which decisions the leadership group actually makes each month, then keep only measures that could change one of them. Usually that leaves three or four, each with a named owner, a written definition and a threshold that triggers a conversation. Everything else moves to a reference page for people who need the detail. A dashboard where everything is important tells a reader nothing about where to look first.

Common mistakes

✗A KPI is simply an important number. It is a number with consequences attached, and consequences change behaviour. Choosing one without asking how it could be met dishonestly leaves the gaming to chance.
✗More indicators give a fuller picture. Past a handful, attention splinters and nothing gets acted on. A small set with clear owners beats a long list nobody can prioritise.
✗A KPI can only be gamed by dishonest people. Most gaming is ordinary and rational. Staff meet the measure they are judged on, and the shortfall appears in whatever the measure failed to capture.
✗Once defined, a KPI should never change, for the sake of comparability. Definitions do need to be stable enough to compare periods, but a measure tied to a strategy that has moved on steers the organisation towards last year goal. Review it deliberately and record the change.

Revision bullets

•A KPI carries consequences, which is why it changes behaviour
•Write down numerator, denominator, time window and filters
•Every KPI needs an owner who can genuinely move it
•Pair each KPI with a guardrail that catches the obvious gaming
•Decompose a headline metric into factors with different owners
•Keep the set small enough that people know where to look first

Quick check

A support team is measured on average time to first response. Which companion measure best protects against the obvious gaming?

Two teams report conversion rates of 2.5 and 3.4 per cent for the same month and the same site. What should be inspected first?

Connected topics

Three Analytics Types Analytics in Practice Measures & Dimensions Quality Dimensions What Is a Decision A/B Testing Dashboards Data Storytelling Truncated Axes Degree Centrality Social Analytics Business Model Mapping Analytics Leverage Filter Bubbles

More in Decisions and Models

Decision Process Models & Abstraction Decision Support Supervised vs Unsup.

Sources

Ridgway (1956)
Ridgway, V. F. "Dysfunctional Consequences of Performance Measurements." Administrative Science Quarterly, 1(2), 1956.
Early account of how single, composite and multiple performance measures each distort behaviour once people are judged by them.
Campbell (1979)
Campbell, D. T. "Assessing the impact of planned social change." Evaluation and Program Planning, 2(1), 1979.
Argues that the more a quantitative indicator is used for decision making, the more it will be corrupted and the more it will distort what it was meant to monitor.
Strathern (1997)
Strathern, M. "Improving ratings: audit in the British University system." European Review, 5(3), 1997.
Source of the widely quoted formulation that a measure ceases to be a good measure once it becomes a target.
Kaplan & Norton (1992)
Kaplan, R. S., & Norton, D. P. "The Balanced Scorecard: Measures That Drive Performance." Harvard Business Review, 70(1), 1992.
Argues that a single financial indicator is too narrow to steer an organisation, and proposes a small balanced set of measures instead.

How to cite this page

Dr. Phil's Quant Lab. (2026). Designing a KPI that works. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-kpi-design