An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.
Where analytics has leverage
Leverage is how much total value a better decision produces. It is not the same as how interesting the problem is, how much data exists, or how sophisticated the method would be. Three quantities multiply: how often the decision is made, how much value is at stake each time, and what share of that stake a better decision can realistically recover. A fourth condition gates the lot, which is whether anyone is able to act. The shape of the answer follows from the shape of the leverage. Many small decisions call for an automated model, few large ones call for analysis and decision support.
Why it matters
Two jobs can be worth the same money and need completely different solutions. Saving four dollars on a call that happens a million times a year is worth as much as saving four million dollars once. The first has to be automatic because no person can be in the loop a million times. The second should never be automatic, because it happens once and someone will have to defend it. Work out the frequency before you work out the technique.
Decision A is made 800,000 times a year with A$4 at stake each time. Decision B is made once a year with A$5,000,000 at stake. A model can recover about 4 per cent of the stake in either case. Which one should be built as an automated model rather than a one-off analysis?
Formulas
Worked examples
A utility is choosing between two analytics projects. Routing inbound calls: 1,200,000 calls a year, about A$6 of handling cost at stake per call. Negotiating the annual bulk supply contract: one decision a year, A$8,000,000 at stake. Both look recoverable at roughly the same rate.
Take call routing at a 5 per cent recovery. V equals 1,200,000 multiplied by A$6 multiplied by 0.05, which is A$360,000 a year. Take the contract at a 3 per cent recovery. V equals 1 multiplied by A$8,000,000 multiplied by 0.03, which is A$240,000 a year. Both are worth doing, and they need opposite solutions. Routing must be automated and monitored, since no analyst can sit inside 1.2 million calls. The contract needs a careful one-off analysis, a written argument and a human signature, because it happens once and someone must defend it.
A retention model is improved from 0.81 to 0.86 on a standard accuracy measure. The operational rule contacts the top 200 accounts each month. Comparing the two ranked lists, 193 of the 200 accounts are the same.
Count the change in actions, not the change in the metric. Seven slots on the list differ, so fourteen accounts change contact status, seven newly contacted and seven dropped. Everything the improvement can be worth is confined to those fourteen. Whether that is trivial or substantial is not yet known, because it turns on what those accounts are worth, on how much being contacted actually changes their behaviour, and on what a contact costs. Most of the accuracy gain landed in a region of the ranking the decision rule never reads. The productive move is to price those fourteen swaps, then consider widening what the decision is allowed to vary.
Common mistakes
- ✗The biggest data source is where the leverage is. Leverage attaches to decisions, not to volume. A large event log connected to no decision has a leverage of exactly zero, and a small, well-owned table attached to a frequent decision can be worth a great deal.
- ✗A more accurate model always creates more value. Value only appears when the action changes. If the improvement occurs away from the threshold the decision rule actually uses, behaviour is unchanged and the gain is worth nothing operationally.
- ✗One-off strategic decisions are not really analytics work. A single decision with a large stake can outrank a million small ones. It calls for analysis and decision support rather than an automated model, which is a difference in form rather than a reason to decline the work.
- ✗High leverage means the project should be built. Recoverable value also requires the ability to act. If the decision is fixed by contract, blocked by an approval cycle or politically closed, the calculated value is not available and the honest recommendation is to fix that first.
Revision bullets
- •Annual value equals frequency times stake times recoverable share
- •Recoverable share is capped by current decision quality and by irreducible randomness
- •High frequency and low stakes points to automation and monitoring
- •Low frequency and high stakes points to decision support with a human decision
- •Count the actions that change, not the metric; a gain that changes no action is worth nothing
- •Ability to act is a gate, not a bonus
Quick check
Decision A is made 800,000 times a year with A$4 at stake each time. Decision B is made once a year with A$5,000,000 at stake. A model can recover about 4 per cent of the stake in either case. Which one should be built as an automated model rather than a one-off analysis?
A model improves from 0.81 to 0.86 accuracy, but the operational rule contacts the same top 200 accounts either way, and 193 of them are unchanged. What is the value of the improvement?
Connected topics
More in How Analytics Gets Built
Sources
- Howard (1966)Howard, R. A. "Information Value Theory." IEEE Transactions on Systems Science and Cybernetics, 2(1), 22-26, 1966.The formal statement that information has value only through the decision it changes, which is the underlying principle here.
- Raiffa (1968)Raiffa, H. Decision Analysis: Introductory Lectures on Choices under Uncertainty. Addison-Wesley, 1968.Introductory treatment of valuing information against the decision it informs rather than against its own accuracy.
- Davenport & Harris (2007)Davenport, T. H., & Harris, J. G. Competing on Analytics: The New Science of Winning. Harvard Business School Press, 2007.Argues for concentrating analytics effort on repeated, distinctive decisions rather than spreading it across available data.