An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.
Personalisation and the privacy trade-off
Personalisation improves as a model learns more about a person, so the thing that makes it useful is the thing that makes it intrusive. That is the personalisation-privacy paradox: people report valuing privacy highly and then trade it for a discount or a small convenience. Two mechanisms make the trade-off worse than it looks on the surface. Inference means an organisation can derive an attribute it never collected, from ordinary purchases and clicks. And context matters more than secrecy, because a fact can be perfectly true, freely given, and still wrongly used when it flows somewhere the person never expected it to go.
Why it matters
A good waiter remembering that you take your coffee black is charming. A waiter who remembers your last eleven visits, who you came with, and the fact you switched to decaf in March is not charming. Nothing about the information changed. What changed is how much of it there is and how far it has travelled from the moment you offered it.
A supermarket chain says its recommender cannot discriminate on health status because health is not in the dataset. Why is that claim weak?
Worked examples
A large retailer wants to reach customers early in a major life event, before competitors do, using nothing but loyalty card data.
The mechanic is an ordinary supervised model. Take customers whose life event is already known from a registry or a gift registry sign-up and label them positive. Label everyone else negative. Score each product line by how much more often it appears in positive baskets than in the rest. Unscented lotion, large handbags, cotton balls and certain supplements move together and produce a customer-level score. The retailer never asked a question and never bought a health record. It inferred one from grocery lines. Duhigg's 2012 account of this practice is widely retold and the specific anecdote in it has been questioned, but the method is elementary and any loyalty programme with a few million baskets can reproduce it.
A health insurer offers members a premium discount if they share the feed from their fitness tracker.
Step count is not secret, and members shared it willingly with the tracker in the first place. The problem is the flow. Data offered to a fitness app for self-tracking now reaches a party whose job is to price risk, and a low step count can raise a premium. Contextual integrity says the test is not whether a fact is private but whether the flow matches the norms of the context it came from. The design fixes are a hard barrier between the feed and pricing, or a programme that is genuinely optional in the sense that declining costs a member nothing.
Common mistakes
- ✗People who click through consent banners clearly do not care about privacy. Stated preferences and clicking behaviour diverge sharply, and the divergence is largely produced by the design of the choice: the default, the friction, and a benefit that is immediate against a cost that is vague and later.
- ✗If we never collect sensitive attributes we cannot act on them. Sensitive attributes are inferable from mundane behaviour, so a model can act on health, financial distress or a life event without any of those ever appearing in a field. Absence from the schema is not absence from the system.
- ✗Anonymising the data solves the problem. Removing names rarely removes identifiability, because combinations of ordinary attributes are close to unique. Re-identification usually means joining the release to another dataset rather than breaking anything.
- ✗More personalisation always sells more. Personalisation that is visibly built on information the customer did not knowingly hand over can reduce trust and response, so relevance and intrusiveness trade off against each other rather than rising together.
Revision bullets
- •The paradox: stated privacy preference is high, revealed trade-off is cheap
- •Inference: sensitive attributes derived from ordinary behaviour, never collected
- •Contextual integrity: judge the flow, not whether the fact is secret
- •Defaults and friction do most of the work in any consent decision
- •Removing names is not anonymity when attribute combinations are near unique
Quick check
A supermarket chain says its recommender cannot discriminate on health status because health is not in the dataset. Why is that claim weak?
Two designs collect exactly the same data. Design A opts users in by default with a buried setting to leave. Design B asks once, clearly, with no penalty for declining. A regulator is likely to treat them differently mainly because
Connected topics
More in Ethics and Governance
Sources
- Awad & Krishnan (2006)Awad, N. F., & Krishnan, M. S. "The Personalization Privacy Paradox: An Empirical Evaluation of Information Transparency and the Willingness to be Profiled Online for Personalization." MIS Quarterly, 30(1), 13-28, 2006.The canonical information-systems statement of the paradox and of the role information transparency plays in it.
- Acquisti, A., Brandimarte, L., & Loewenstein, G. "Privacy and human behavior in the age of information." Science, 347(6221), 509-514, 2015.Reviews why privacy preferences are uncertain, context-dependent and highly sensitive to how a choice is presented.
- Nissenbaum (2010)Nissenbaum, H. Privacy in Context: Technology, Policy, and the Integrity of Social Life. Stanford University Press, 2010.Contextual integrity: the appropriateness of an information flow is judged against the norms of the context the information was given in.
- Duhigg, C. "How Companies Learn Your Secrets." The New York Times Magazine, 16 February 2012.Popular account of inferring a life event from ordinary purchase data. The underlying method is straightforward even where the anecdote itself is disputed.