An independent study reference written by Dr Phuc V. Nguyen. It is not official subject material — for assessment requirements always follow your subject outline and vUWS.

Text and Social Dataintermediate

Natural language processing

Q: A team applies a standard stop-word list before training a complaint classifier, and accuracy on negative reviews falls sharply. What is the most likely cause?

The list removed negation words such as "not" and "never", flattening opposite meanings onto the same representation. Generic stop-word lists were built for search, where common function words add little. In sentiment and complaint work the negation carries the signal. Strip "not" and the phrase "not helpful" becomes indistinguishable from "helpful". The remedy is a task-specific list, not abandoning the step.

Q: Two support tickets are represented as embeddings. Ticket A runs to 300 words, ticket B to 20, both about a billing error, and their cosine similarity is high. What does that tell you?

They point in a similar direction in the vector space, which is why the length gap does not dominate the score. Cosine similarity divides by the length of each vector, so it measures the angle rather than the magnitude. That is precisely why it suits text, where a long document and a short one can be about the same thing. High similarity means shared context and vocabulary patterns, not identical wording.

Natural language processing is the set of techniques that let a computer treat human language as data. The work is layered. Tokenisation cuts a string into units. Normalisation collapses surface variants such as case, plurals and tense. Tagging labels each token with its part of speech. Named entity recognition marks people, organisations and places. Representation converts the result into numbers, first as plain word counts and now usually as embeddings, dense vectors positioned so that words used in similar contexts sit close together. Every layer involves a judgement, and those judgements propagate into whatever model sits on top.

Why it matters

Language is easy for people because we arrive carrying context. A computer sees only characters. Teaching it to read means stripping away surface variation, then giving each word a position in a space where neighbours mean similar things. Once "invoice" and "bill" sit near each other and far from "battery", a model can group support tickets it has never seen before, without anyone writing a rule about invoices.

Before you read on — recall

A team applies a standard stop-word list before training a complaint classifier, and accuracy on negative reviews falls sharply. What is the most likely cause?

Formulas

Bigram probability, the simplest model of what follows what

P(w_{n} \mid w_{n-1}) = \frac{C(w_{n-1}, w_{n})}{C(w_{n-1})}

Count the pairs, then divide by how often the first word appeared alone. In a corpus of support messages where "cannot" occurs 4,000 times and the pair "cannot log" occurs 900 times,

P(\text{log} \mid \text{cannot}) = 900/4{,}000 = 0.225

. Crude, but this is how a machine gets its first sense of context.

Cosine similarity between two word vectors

\cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert}

Take toy two-dimensional vectors: invoice at

\mathbf{u} = (0.9,\, 0.1)

, bill at

\mathbf{v} = (0.8,\, 0.2)

and battery at

\mathbf{w} = (0.1,\, 0.95)

. Then invoice and bill score 0.99 while invoice and battery score 0.21. The measure compares direction rather than length, which is why a long text and a short one on the same topic still score alike.

Worked examples

Scenario

A telecommunications provider receives roughly 40,000 free-text support messages a month and wants them routed automatically to the right team.

Solution

Run the layers in order and inspect each one. Tokenisation must keep account numbers and product codes intact rather than splitting them at hyphens. Normalisation needs a choice between stemming, which chops endings by rule, and lemmatisation, which maps a word to its dictionary form. Stemming is fast and blunt, reducing both "universal" and "university" to the same truncated token. Lemmatisation keeps them apart and turns "better" into "good", which a sentiment layer downstream depends on. Entity recognition then picks out product names so a message about a specific handset reaches hardware. Only after all of that does the classifier see numbers, and most accuracy problems in systems like this trace back to a preprocessing decision rather than to the model.

Scenario

A model trained on English customer reviews is applied in a Vietnamese market where posts routinely mix Vietnamese, English product names and emoji inside one sentence.

Solution

Almost every layer breaks. Vietnamese writes many compounds as separate syllables, so English-style whitespace tokenisation produces fragments carrying no meaning on their own. Diacritics are frequently dropped in casual writing, which multiplies the surface forms of a single word. Emoji carry sentiment the text does not state. The fix is not a stronger classifier but a language-appropriate tokeniser, a normalisation step that standardises diacritics, and training data drawn from the same register as the target text. Language technology does not travel across languages by default.

Common mistakes

✗Natural language processing now means a large language model. Large models are one option and often an expensive one. Tokenising, counting and a simple classifier still solve a great many routing, tagging and search problems faster, more cheaply, and with an audit trail a business can inspect.
✗Preprocessing is a mechanical step with no consequences. Removing stop words deletes "not", which reverses meaning. Lowercasing merges the company Apple with the fruit. Every cleaning decision is a modelling decision and belongs in the documentation with its reason.
✗A system that handles English will handle other languages. Tokenisation, word order, morphology and even what counts as a word differ by language, and a model trained on one language holds no knowledge of another. Multilingual work needs language-specific components and language-specific evaluation.
✗Word embeddings capture meaning objectively. They capture which words appear in similar contexts in the training text. That is a statistical fact about a corpus, so stereotypes present in the corpus are reproduced in the geometry, which is why embedding-based systems need bias testing before they are deployed.

Revision bullets

•Layers: tokenise, normalise, tag, recognise entities, represent as numbers
•Stemming chops by rule and over-merges; lemmatisation maps to the dictionary form
•Bag of words ignores order; embeddings place similar-context words close together
•Cosine similarity compares direction, so document length does not dominate
•Removing stop words can delete negation and reverse the meaning
•Nothing transfers across languages for free

Quick check

A team applies a standard stop-word list before training a complaint classifier, and accuracy on negative reviews falls sharply. What is the most likely cause?

Two support tickets are represented as embeddings. Ticket A runs to 300 words, ticket B to 20, both about a billing error, and their cosine similarity is high. What does that tell you?

Connected topics

Forms of Data Supervised vs Unsup.Sentiment Analysis Text Mining Pipeline Algorithmic Bias Agentic AI

More in Text and Social Data

Social Analytics Social Data Sources Web Mining

Sources

Jurafsky & Martin, SLP (3rd ed. draft)
Jurafsky, D., & Martin, J. H. Speech and Language Processing, 3rd edition draft. Stanford University.
Freely available reference covering tokenisation, tagging, entity recognition, n-gram models and vector semantics.
Mikolov et al. (2013)
Mikolov, T., Chen, K., Corrado, G., & Dean, J. "Efficient Estimation of Word Representations in Vector Space." arXiv:1301.3781, 2013.
Introduced the word2vec embeddings that made distributional word vectors practical at scale.
Devlin et al. (2019)
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." NAACL-HLT, 2019. arXiv:1810.04805.
A foundational pretrained contextual model, extended and adapted by much of the text classification work that followed.
Bolukbasi et al. (2016)
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings." NeurIPS, 2016. arXiv:1607.06520.
Demonstrates that embeddings reproduce the social regularities of their training corpus.

How to cite this page

Dr. Phil's Quant Lab. (2026). Natural language processing. Derivatives Atlas. https://phucnguyenvan.com/concept/ba-nlp