An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

Text and Social Dataintermediate

The text mining pipeline

Q: In a collection of 4,000 support tickets, the word "password" appears in 3,960 of them. What happens to its tf-idf weight, and what does that imply?

The weight is near zero, so the word does almost nothing to separate one ticket from another. idf is the log of 4,000 divided by 3,960, which is about 0.004, so even a ticket using the word ten times ends with a negligible weight. The word can be central to the business and still useless for telling tickets apart, and that distinction is exactly what tf-idf enforces.

Q: Two documents produce a cosine similarity of 0.85 on a tf-idf matrix. Which conclusion is safe?

They draw on a strongly overlapping set of distinctive terms. Cosine on a tf-idf matrix measures overlap in weighted vocabulary. High similarity means the two documents use the same distinctive words. It says nothing about stance, since a glowing review and a scathing one about the same product share most of their vocabulary, and nothing about length, which the normalisation deliberately removes.

A text mining pipeline turns documents into a table a model can read. Collect, clean, tokenise and normalise, then build a term-document matrix whose rows are terms, whose columns are documents, and whose cells hold counts. That matrix is almost entirely zeros, because any single document uses a tiny slice of the vocabulary. Raw counts favour common words, so the cells are usually reweighted by tf-idf, which rewards a term for being frequent inside a document and rare across the collection. Clustering, classification, topic discovery and similarity search can all run on the reweighted matrix, but tf-idf is one representation among several, alongside raw counts and learned embeddings.

Why it matters

Imagine ruling up a giant spreadsheet. Every distinct word gets a row, every document gets a column, and you tally. The tally on its own is not informative, because "the" wins every column. The trick is to score a word higher when it is busy in one document and quiet everywhere else. That is what makes "refund" stand out in a complaint while "the" disappears from every column at once.

Before you read on — recall

In a collection of 4,000 support tickets, the word "password" appears in 3,960 of them. What happens to its tf-idf weight, and what does that imply?

Formulas

Term frequency and document frequency

\mathrm{tf}_{t,d} = \text{count of term } t \text{ in document } d, \qquad \mathrm{df}_{t} = \text{number of documents containing } t

Two different counts doing two different jobs. Term frequency is local to one document. Document frequency is a property of the whole collection, and it is the one that tells you whether a word is distinctive.

Inverse document frequency

\mathrm{idf}_{t} = \log_{10}\!\left(\frac{N}{\mathrm{df}_{t}}\right)

With

N = 3

documents, a term appearing in two of them scores

\log_{10}(1.5) = 0.176

and a term appearing in only one scores

\log_{10}(3) = 0.477

. A term present in every document scores exactly zero, which is how tf-idf quietly removes words like "the" without any stop-word list.

tf-idf weight

w_{t,d} = \mathrm{tf}_{t,d} \times \mathrm{idf}_{t}

In the worked example below, "battery" appears twice in the first review and scores

w = 2 \times 0.176 = 0.352

, while "died" appears once and scores

w = 1 \times 0.477 = 0.477

. The rarer word wins despite being half as frequent, which is exactly the intent.

Cosine similarity between two documents

\cos(d_{1}, d_{2}) = \frac{\sum_{t} w_{t,d_{1}} w_{t,d_{2}}}{\sqrt{\sum_{t} w_{t,d_{1}}^{2}} \; \sqrt{\sum_{t} w_{t,d_{2}}^{2}}}

Compares direction rather than size, so a long document and a short one can still match. On the raw counts below, reviews one and two score 0.40 while reviews one and three score 0.25, even though one and three share the complaint word "poor".

Worked examples

Scenario

Build the term-document matrix by hand for three short reviews so the mechanics are visible. Review one: "The battery died fast. Battery life is poor." Review two: "Great battery life, great screen." Review three: "The screen is poor."

Solution

Lowercase, split on whitespace and punctuation, and drop the stop words "the" and "is". Review one leaves battery twice plus died, fast, life and poor. Review two leaves great twice plus battery, life and screen. Review three leaves screen and poor. The matrix has seven rows and three columns. The battery row reads 2, 1, 0. The life row reads 1, 1, 0. The poor row reads 1, 0, 1. The screen row reads 0, 1, 1. The great row reads 0, 2, 0. The died row and the fast row each read 1, 0, 0. Eleven of twenty-one cells are non-zero on three tiny reviews, and the emptiness only grows with real data.

Scenario

Using those same three reviews, decide which two are most similar, then check whether the answer matches what a person would say.

Solution

Treat each column as a vector over the seven terms. Reviews one and two share battery and life, giving a dot product of 3 and a cosine of about 0.40. Reviews one and three share only poor, giving a dot product of 1 and a cosine of exactly 0.25. The arithmetic says one is closer to two. A person reading them says one and three belong together, because both are complaints while two is praise. Bag-of-words similarity measures shared vocabulary, not shared stance, which is precisely why sentiment needs its own model instead of falling out of the matrix.

Common mistakes

✗The term-document matrix preserves the meaning of a document. It preserves which words appeared and how often. Word order, syntax and negation are gone, so "service was good, food was not" and "food was good, service was not" produce identical columns.
✗Removing stop words and applying tf-idf do the same job. Stop-word removal is a hard delete decided in advance from a fixed list. Tf-idf is a soft weighting computed from the collection itself, so it will down-weight a word that happens to be ubiquitous in your corpus even when no standard list contains it.
✗A sparse matrix is a sign of poor data quality. Sparsity is the normal state of text. A vocabulary of fifty thousand terms against documents of two hundred words gives well over 99 per cent zeros, which is why text tooling stores only the non-zero cells.
✗The pipeline is a fixed recipe you run once. Vocabulary drifts, new products and new slang arrive, and a matrix built on last year's vocabulary quietly stops matching this year's documents. The cleaning and weighting steps need the same monitoring as the model sitting on top of them.

Revision bullets

•Stages: collect, clean, tokenise and normalise, build the matrix, weight, analyse, act
•Term-document matrix: terms as rows, documents as columns, counts in cells, mostly zeros
•idf is the log of N over document frequency; a word in every document scores zero
•tf-idf = local frequency multiplied by global rarity
•Cosine similarity compares direction, so length does not decide the match
•Bag of words discards order and negation, so stance needs a separate model

Quick check

In a collection of 4,000 support tickets, the word "password" appears in 3,960 of them. What happens to its tf-idf weight, and what does that imply?

Two documents produce a cosine similarity of 0.85 on a tf-idf matrix. Which conclusion is safe?

Connected topics

Forms of Data Data Cleansing NLP Sentiment Analysis Web Mining CRISP-DM

More in Text and Social Data

Social Analytics Social Data Sources

Sources

Manning, Raghavan & Schutze (2008)
Manning, C. D., Raghavan, P., & Schutze, H. Introduction to Information Retrieval. Cambridge University Press, 2008.
Free online text covering the term-document matrix, tf-idf weighting and the vector space model.
Salton & Buckley (1988)
Salton, G., & Buckley, C. "Term-weighting approaches in automatic text retrieval." Information Processing & Management, 24(5), 513-523, 1988.
The comparative study that established tf-idf style weighting as the working default.
Blei, Ng & Jordan (2003)
Blei, D. M., Ng, A. Y., & Jordan, M. I. "Latent Dirichlet Allocation." Journal of Machine Learning Research, 3, 993-1022, 2003.
The topic model most often run on top of a term-document matrix when the themes are unknown in advance.

How to cite this page

Dr. Phil's Quant Lab. (2026). The text mining pipeline. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-text-mining-pipeline