An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

Text and Social Dataintermediate

Web mining

Q: A funnel runs at 12 per cent view to cart, 40 per cent cart to checkout and 65 per cent checkout to purchase. A manager argues the team must attack the 12 per cent step because it is the worst rate. What is the most accurate response?

The steps multiply, so an equal relative gain at any step adds the same number of sales; pick the step where a gain is achievable. Overall conversion is the product of the step rates, so lifting any one of them by ten per cent relative raises the total by ten per cent relative, whichever step it is. The low first step is still informative, since it hints that the mismatch sits in price, availability or traffic quality, but the case for working there is achievability rather than arithmetic.

Q: A retailer wants to know which of two product page layouts causes more purchases. Usage logs show layout B has the higher conversion rate among visitors who saw it. What is the problem?

Visitors were not randomly assigned, so the gap may reflect who saw each layout rather than the layout itself. Observational log data records what happened, not what would have happened under the other layout. If layout B went to returning visitors, to mobile users or to traffic from one campaign, the audience differs along with the layout. Random assignment is what separates the two, which is why layout questions belong to controlled experimentation rather than log analysis.

Web mining applies analytics to the web itself, and it splits three ways. Content mining treats page text, images and product listings as a corpus to extract from. Structure mining treats hyperlinks as a graph and ranks pages by who links to them, which is where PageRank sits. Usage mining treats server logs and clickstreams as behaviour, reconstructing sessions, paths and funnels out of raw request records. The three answer different questions. Content tells you what is out there, structure tells you what the web treats as authoritative, and usage tells you what your own visitors actually did.

Why it matters

Three ways to study a shopping centre. Read every sign and price tag, which is content. Map which shops the walkways connect and which corner everybody has to pass, which is structure. Follow the footprints on the floor to see where people entered, wandered and gave up, which is usage. Same building, three completely different studies, and confusing one for another is the usual mistake.

Before you read on — recall

A funnel runs at 12 per cent view to cart, 40 per cent cart to checkout and 65 per cent checkout to purchase. A manager argues the team must attack the 12 per cent step because it is the worst rate. What is the most accurate response?

Formulas

PageRank

PR(p) = \frac{1 - d}{N} + d \sum_{q \to p} \frac{PR(q)}{L(q)}

Each page passes its score along its outgoing links, split evenly, and

d

(typically 0.85) is the chance a surfer keeps clicking rather than jumping somewhere at random.

L(q)

is the number of links leaving page

q

. A link from a highly ranked page with few outgoing links is worth far more than a link from a crowded page nobody visits.

Funnel step rates and overall conversion

r_{k} = \frac{v_{k+1}}{v_{k}}, \qquad r_{\text{overall}} = \prod_{k} r_{k}

Ten thousand product views leading to 1,200 carts, 480 started checkouts and 312 purchases give step rates of 12, 40 and 65 per cent and an overall rate of 3.12 per cent. Because the steps multiply, a ten per cent relative gain at any single step lifts the total by the same ten per cent relative.

Worked examples

Scenario

Work PageRank by hand on a three-page site so the mechanics are visible. Page A links to B and C. Page B links to C. Page C links back to A. Use a damping factor of 0.85.

Solution

Start every page at one third. In the first pass A receives all of C's score and lands on one third again. B receives half of A's score, giving 0.19. C receives half of A plus all of B, giving 0.48. Iterate until the values stop moving and they settle near 0.39 for A, 0.21 for B and 0.40 for C. C ranks highest because two pages point at it. A finishes a close second on a single inbound link, because that link comes from the strongest page on the site. That is the whole idea: a link is a vote, weighted by the voter and divided among everyone the voter points at.

Scenario

An online retailer declares that its checkout page is broken, because only 3 per cent of product viewers buy anything.

Solution

Split the funnel first. Of 10,000 product views, 1,200 reach a cart, 480 start checkout and 312 complete, giving step rates of 12, 40 and 65 per cent. Since the steps multiply, a ten per cent relative improvement at any one of them adds the same 31 sales, so "attack the worst rate" is not automatically the right instruction. What the split does give you is a diagnosis. A checkout converting 65 per cent of the people who start it is not demonstrably broken, but calling it healthy needs a comparison with earlier periods on the same site or a sector benchmark, whereas 88 per cent of viewers leaving without adding anything points at price, delivery cost, stock availability or the product page. The numbers say where to look, and business context says where a gain is actually achievable.

Common mistakes

✗A page view equals a person. Logs record requests, not people. One visitor generates many requests, shared devices and office networks blur identity, and crawlers and monitoring bots produce traffic that never converts. Sessions have to be rebuilt with explicit rules, commonly by cutting after a period of inactivity, and bots have to be filtered before any rate is calculated.
✗Search traffic reveals what a population is doing. It reveals what some people typed, on one search service, through an interface that itself shapes what gets typed. The Google Flu Trends failure is the standing case: a model built on search terms tracked influenza well for a period, then drifted badly, partly because the platform kept changing underneath it.
✗If it is on the web it can be collected and reused. Site terms, copyright, personal data law and technical measures such as robots directives all constrain collection, and the rules vary by jurisdiction and by what you do with the material afterwards. Treat permission as a design question rather than an afterthought.
✗Structure mining and usage mining answer the same question. Structure asks who the web points at, a statement about authority across sites you do not control. Usage asks what your visitors did, a statement about behaviour on a site you do control. Neither substitutes for the other.

Revision bullets

•Three branches: content (what pages say), structure (who links to whom), usage (what visitors did)
•PageRank: a link is a vote, weighted by the voter and split across its outgoing links
•The damping factor models a surfer who sometimes jumps rather than clicks
•Usage mining rebuilds sessions from raw requests, then paths and funnels
•Step rates multiply, so an equal relative gain at any step is worth the same
•Requests are not people; filter bots and define sessions before reporting any rate

Quick check

A retailer wants to know which of two product page layouts causes more purchases. Usage logs show layout B has the higher conversion rate among visitors who saw it. What is the problem?

Connected topics

A/B Testing What SNA Is Social Analytics Social Data Sources Text Mining Pipeline Personalisation

More in Text and Social Data

NLP Sentiment Analysis

Sources

Kosala & Blockeel (2000)
Kosala, R., & Blockeel, H. "Web mining research: A survey." ACM SIGKDD Explorations Newsletter, 2(1), 1-15, 2000.
Source of the content, structure and usage division used throughout this node.
Brin & Page (1998)
Brin, S., & Page, L. "The anatomy of a large-scale hypertextual Web search engine." Computer Networks and ISDN Systems, 30(1-7), 107-117, 1998.
Original description of PageRank and the random surfer model behind the damping factor.
Lazer et al. (2014)
Lazer, D., Kennedy, R., King, G., & Vespignani, A. "The Parable of Google Flu: Traps in Big Data Analysis." Science, 343(6176), 1203-1205, 2014.
The reference case on why a model built on web signals drifts when the platform beneath it changes.

How to cite this page

Dr. Phil's Quant Lab. (2026). Web mining. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-web-mining