← →/space · click bg to advance · F fullscreen
DSAIT4090 · Lexical processing & language modelling
Statistical Language Models
Assigning probabilities to text — and measuring it, with nothing more than counting.
A. Anand · study deck
Motivation
Why language modelling?
Almost everything people know is written down. For a computer to use that knowledge — to answer, translate, summarise — it first has to model how language works.
The linguist's view
Describe language with rules written by hand — grammar, syntax, parts of speech.
The computational view
Describe language with probability — learn from how people actually use words, and let fluency and meaning fall out of the statistics.
The shift
No grammar is written by hand. The model learns one thing from data — what usually comes next — and that turns out to be enough to read, write and reason over text.
The core move
A sentence, by probability
“Understanding” becomes one concrete number: give every sentence a probability. Sensible, fluent sentences score high; nonsense scores low.
- ✓ high P “the cat sat on the mat” — fluent and sensible.
- ✗ low P “mat the on cat sat the” — the same words, scrambled: not how we speak.
- ✗ low P “colourless green ideas sleep furiously” — perfectly grammatical, yet nobody says it.
Not the same as grammar
That last sentence is where we part ways with the linguist: a grammar calls it well-formed, but the language model still finds it unlikely. We model what is probable, not merely what is allowed.
The object
What is a language model?
Definition
A language model assigns a probability to a piece of text.
Equivalently, it predicts the next word: given a context, it outputs a probability for every word in the vocabulary.
- Machine translation: P(“I went to the lecture”) >> P(“I flew to the lecture”)
- Speech recognition: P(“I saw a van”) >> P(“eyes awe an”)
- Classification / autocomplete / generation — all rest on scoring word sequences.
Question
The model reads “new telescope spots water on a distant ___”.
What should it output for the blank?
Not one word — think about what form the answer takes.
Think — then press → for the answer.
Answer
A distribution over the whole vocabulary
new telescope spots water on a distant ___
planet
0.62
star
0.15
moon
0.12
galaxy
0.07
ocean
0.04
illustrative numbers — the distribution sums to 1
A language model predicts what comes next — a probability for every word in the vocabulary.
Definitions
A language model, precisely
Language model
A probability distribution over token sequences — two equivalent views:
the joint probability of a whole sequence, factored into next-token predictions.
Example — the two views on one sentence
Each factor is a next-word probability; multiply them to get the whole-sentence probability.
The question for the rest of the lecture
Everything now reduces to one problem: how do we estimate these probabilities — the P(wₜ | context) — from data?
Before we count — what do we count?
What is a token?
To estimate P(wₜ | context) by counting, we first have to decide what a “word” — a token — actually is. We cut the text into tokens, and count those.
- For this lecture a token is just a word (split on spaces and punctuation). It keeps the counting intuitive.
- Words are a poor unit, though: the vocabulary never ends (new names, typos → unseen), and many languages don't split on spaces.
- Real models use sub-word tokens (byte-level BPE) — we come back to how at the end of this deck (and the full story is Deck 3). For now: think “words”.
Question
Given a large corpus of text, how can we estimate these probabilities?
We need them to make predictions — and all we have is counts of what people actually wrote.
Think — then press → for the answer.
A little vocabulary first
Three terms, in pictures
corpus — the text we learn from
<s> new telescope spots water </s>
<s> telescope spots a planet </s>
<s> new telescope spots a planet </s>
<s> water on a distant planet </s>
n-gram — a short run of tokens
unigram spots · bigram spotsa · trigram spotsaplanet
parameter — one probability per n-gram
— how likely a is right after spots
Estimating it just means reading it off the corpus: spots a appears 2×, spots 3× →
We build a bigram model: one parameter P(word | previous word) for each pair, read straight from these counts.
The simplest useful model
Training = counting
All three n-grams follow the same count-and-normalise recipe — we work through the bigram here.
Maximum-likelihood estimate (bigram)
P(wᵢ | wᵢ₋₁) =
count(wᵢ₋₁ wᵢ)
count(wᵢ₋₁)
- Corpus: four short sentences built from the same handful of words.
- <s> and </s> mark sentence start/end — so the model also learns how sentences begin and finish.
- No network, no gradients — just counts and a division.
Question
A bigram's parameters are the probabilities P(word | previous word). Given this corpus, how do we find one — say P(a | spots)?
training corpus
<s> new telescope spots water </s>
<s> telescope spots a planet </s>
<s> new telescope spots a planet </s>
<s> water on a distant planet </s>
Think — then press → for the answer.
Answer
A bigram model is just counting
training corpus
<s> new telescope spots water </s>
<s> telescope spots a planet </s>
<s> new telescope spots a planet </s>
<s> water on a distant planet </s>
spots occurs 3× → a twice, water once
Count the pair spots a (2), divide by how often spots occurs (3): P(a | spots) = ⅔ — exact, from the corpus on the left.
Recap
Counts in, distribution out
training corpus
<s> new telescope spots water </s>
<s> telescope spots a planet </s>
<s> new telescope spots a planet </s>
<s> water on a distant planet </s>
P(· | <s>)
new ½ · telescope ¼ · water ¼
P(· | spots)
a ⅔ · water ⅓
P(· | a)
planet ⅔ · distant ⅓
The idea
Each row is a conditional distribution that sums to 1. The parameters are these normalised counts — training by maximum likelihood is just count & divide.
From words to sentences
Scoring a whole sentence: the chain rule
We can estimate one next-word probability by counting. To score a whole sentence, multiply them — that is the chain rule, and it is exact.
P(w₁,…,wₙ) = P(w₁)·P(w₂|w₁)·P(w₃|w₁,w₂)···P(wₙ|w₁,…,wₙ₋₁)
Exact — no approximation yet
The chain rule just rewrites the joint probability as a product of next-word factors; nothing is lost. The trouble is only in estimating those factors.
The catch
But the full history can't be counted
The last factors condition on a long history — a seven-word context almost never appears even once. So how much history can we actually afford?
The fix for sparsity
Shorten the history
If the full history can’t be counted, condition on only the last few words instead.
This is the Markov assumption — the single idea that makes n-gram models possible.
| unigram | P(w₅) | no history · counts very reliable |
| bigram | P(w₅|w₄) | one word back |
| trigram | P(w₅|w₃,w₄) | two words back |
| full | P(w₅|w₁,w₂,w₃,w₄) | no assumption · hopelessly sparse |
Question
The exact long history never appears in any corpus. So how can we estimate the next word at all?
And if we throw history away, what do we lose?
Think — then press → for the answer.
Answer
Markov assumption & the n-gram trade-off
newtelescopespotswateronadistantplanet
unigram
no history · counts reliable
bigram
one word back
trigram
two words back
full
hopelessly sparse
How much history is enough? More context means richer predictions but sparser counts.
Recap
A trade-off, not a free lunch
The Markov assumption
P(wᵢ | w₁…wᵢ₋₁) ≈ P(wᵢ | wᵢ₋ₙ₊₁…wᵢ₋₁) — keep only the last n−1 words.
Too short (bigram)
“the telescope we bought last year finally spotted a ___” — the bigram sees only “a”. It has forgotten telescope, so planet and pancake look equally likely. Reliable counts, weak predictions.
Too long (6-gram)
“spots water on a distant ___” as a 6-gram — that exact history almost never recurs, so its count is 0 and it predicts nothing. Rich context, but unusable.
The real test
Scoring a sentence never seen in training
Our running example — new telescope spots water on a distant planet — is not in the corpus.
A model that only memorised sentences would be stuck here.
But a bigram model only needs pairs, and it can walk the sentence one bigram at a time,
from <s> to </s>, multiplying the factors.
Question
The example sentence isn’t in the corpus. Can a bigram model still assign it a probability — and what is it?
Nine factors (eight words + the end token). What do they multiply to?
Think — then press → for the answer.
Answer
Where every factor comes from
<s>newtelescopespotswateronadistantplanet</s>
Each factor is a count ratio C(pair)/C(first word). A 1 means that word was always followed by the next in this corpus; only four factors fall below 1.
Recap & a warning
An unseen sentence scores — unless a pair is missing
The result
Every bigram in the sentence appears in the corpus, so each factor is > 0 and P(sentence) = 1/36 — generalising from pieces it has seen, not memorising whole sentences.
The alternate world — one unseen pair
Score instead “<s> new planet …”. The pair new planet never occurs, so with plain counting — and the whole product collapses to P = 0, however reasonable the rest.
One missing bigram kills the sentence. That is the zero problem — and the reason we will need smoothing.
Question
The same table of counts can score a sentence. Can it also write one?
If so, with what procedure — and will it only ever repeat the corpus?
Think — then press → for the answer.
Answer
Generating text by sampling
<s>new…
next-word distribution · one random sample
To generate: look up the next-word distribution, sample one word, append, repeat.
Recap
Score and generate — same counts
Idea
From <s>: look up the next-word distribution, sample one word, append it, repeat until </s>.
Output — “new telescope spots a distant planet” — never appeared in the corpus, yet every adjacent pair in it did. Novelty by chaining seen bigrams.
Temperature rescales that distribution before sampling: T = 1 is the model's own probabilities (what A1 uses); lower T is greedier, higher T more random. More in Deck 8 · Decoding.
Practical issue 1
Tiny probabilities
A sentence probability is a product of many numbers below 1 — it gets vanishingly small fast.
- 1000-word document, each word ≈ 0.1 → P ≈ 10⁻¹⁰⁰⁰.
- Smallest 64-bit float ≈ 10⁻³⁰⁸. Anything smaller rounds to exactly 0.
Question
Multiply a thousand small probabilities on a real computer. What happens — and how do we avoid it?
Think — then press → for the answer.
Answer
Underflow & log-probabilities
a 1,000-word document, each word at p = 0.1:
Fix: add log-probabilities instead of multiplying probabilities.
Recap
Work in log space
log ∏ᵢ p(wᵢ|wᵢ₋₁) = ∑ᵢ log p(wᵢ|wᵢ₋₁)
Fix
The log of a product is the sum of the logs — so add log-probabilities instead of multiplying probabilities. The running sum never underflows, and log is monotonic so comparisons are unchanged.
For our sentence: log(1/36) = −3.58.
Practical issue 2
Zeros & sparsity
Most word pairs are never seen, even in a huge corpus. An unseen bigram has count 0 → probability 0.
And because we multiply, a single zero makes the whole sentence impossible —
e.g. new planet spots water: the pair new planet never occurred, so P=0 however reasonable the rest.
Question
A single unseen pair sends the whole sentence probability to zero. How do we stop that — without lying about what we saw?
Think — then press → for the answer.
Answer
Add-one smoothing & backoff
Add-one smoothing
Pretend every pair was seen once more: +1 on top, +V below — still sums to 1.
P(planet | new): 0 → 1/11
Backoff
Unseen bigram? Fall back to the unigram. Less context, but never zero.
P(planet) = 3/22
unseen ≠ impossible — steal a little mass for what you didn't see
Two ways to banish the zero — because unseen is not the same as impossible.
Recap
Never exactly zero
Two fixes · one assumption
Add-one smoothing: pretend every pair was seen once more — add 1 on top, add V=9 below, so it still sums to 1. P(planet|new): 0 → 1/11.
Add-k (general): add a fraction k instead of 1 — ; add-one is just k=1. Pick k on a held-out dev split. (this is the smoother you implement in A1.)
Backoff: unseen bigram → fall back to the unigram. P(planet) = 3/22. Less context, but not zero. Smarter smoothers like Kneser–Ney redistribute mass by how many contexts a word appears in — the classic n-gram baseline (optional in A1).
Underlying assumption: unseen ≠ impossible. Steal a little mass for everything you didn’t see.
Evaluation
Is the model any good?
- Held-out rule: train on one set of text, measure on text the model has never seen — otherwise it can just memorise.
- Here: the four corpus sentences train; the example sentence is held out to test.
Perplexity
The sum is the negative log-likelihood (NLL) of the text; perplexity is just exp(NLL / N) — the average per token.
Question
Two models — a unigram and a bigram — each score the same held-out sentence. Which is better, and how do we put one number on it?
Each assigns the sentence a probability; lower or higher perplexity — which way is good?
Think — then press → for the answer.
Answer
Perplexity: bigram vs unigram
Same held-out sentence new telescope spots water on a distant planet (N = 9), scored by two models on the same corpus.
unigram — P(wᵢ), no context
| new | 2/22 | a | 3/22 |
| telescope | 3/22 | distant | 1/22 |
| spots | 3/22 | planet | 3/22 |
| water | 2/22 | </s> | 4/22 |
| on | 1/22 | | |
bigram — P(wᵢ|wᵢ₋₁), one word of context
| new | 1/2 | a | 1 |
| telescope | 1 | distant | 1/3 |
| spots | 1 | planet | 1 |
| water | 1/3 | </s> | 1 |
| on | 1/2 | | |
Context makes each factor larger → higher P(W) → lower perplexity. The bigram wins.
Recap
One number: lower is better
Perplexity
= inverse probability of the test set, normalised by length = exp(average negative log-prob per token). Lower is better.
Bigram: = 1.49 (as unsure as choosing between ~1.5 words). Unigram, ignoring context: 9.92. Context really helps.
Measured over tokens (e.g. BPE) — so comparable only between models with the same tokenizer.
Perplexity today
Perplexity hasn't gone away
- It is still the core intrinsic metric for pretraining LLMs — tracked throughout training on held-out web text.
- Benchmarks reported in perplexity: Penn Treebank, WikiText-2 / 103, The Pile, LAMBADA, 1B Word.
| Model | Perplexity ≈ | Test set · tokenizer |
| 5-gram (Kneser–Ney) | ~141 | PTB · words |
| LSTM (2016) | ~48 | WikiText-103 · words |
| Transformer-XL (2019) | ~18 | WikiText-103 · words |
| GPT-2 1.5B (2019) | ~17.5 | WikiText-103 · words, zero-shot |
| LLaMA-2 7B (2023) | ~5.5 | WikiText-2 · own BPE |
| Frontier 70B+ | ~3 | web text · own BPE |
Read it carefully
Lower is better — but these are not directly comparable: different test sets and, above all, different tokenizers. The drop from ~18 to ~3 is mostly sub-word vs word counting, not a 6× better model.
Instruction-tuned / chat models are not really comparable to pure language models either: tuning trades raw next-token likelihood for following instructions, so their perplexity on plain text can even look worse.
Comparing across tokenizers
Perplexity per byte
Two models with different tokenizers can't be ranked by per-token perplexity — the denominator N counts different things. Normalise instead by something both share: the UTF-8 bytes of the text.
Per token vs per byte
Why per byte is the fair comparison
Same NLL (negative log-likelihood — the summed −log p over the text), divided by a count that is identical for every model — the bytes of the text as written. A coarse tokenizer that spends fewer, bigger tokens no longer looks artificially better.
The two are linked exactly by bytes-per-token: . (in log₂ this is bits per byte, the standard cross-model report — and exactly what A1 asks you to compute.)
Answer
Recap & the bridge to cross-entropy
- A language model gives P(next word | history)
- Chain rule: P(sentence) is a product of next-word probabilities
- Markov assumption: keep only the last n−1 words
- n-gram models are counts; sample from them to generate
- Use log-probs; smooth or back off for unseen pairs
- Perplexity on held-out text: lower is better
Next: softmax & cross-entropy — the loss that trains a neural network.
Recap
Statistical language models
- Chain rule → sentence probability; Markov → countable n-grams.
- Logs fix underflow; smoothing & backoff fix zeros.
- Perplexity on held-out text measures the result.
The bridge
Look inside perplexity: the average negative log-probability per token has a name — it is the cross-entropy. So perplexity = e^(cross-entropy).
Next: cross-entropy & softmax — the loss that trains a neural network.
Back to tokens
Words, or tokens?
We have been talking about words — and quietly kept saying tokens. Are they the same thing?
- No — or rather, it depends. A token is whatever unit we choose to cut text into; a word is just one choice of unit, and usually a poor one.
- Tokens are the lifeblood of every large language model — the atoms it reads, counts and predicts. Choosing them well is half the game.
Two questions decide everything that follows
1 · How do we cut text into tokens? 2 · How big should the vocabulary be — and why does that change the numbers we just computed?
Why not just words?
A word vocabulary never ends
word-level vocabulary
exoplanets→[UNK]
any word unseen in training collapses to one unknown token — its meaning is lost
The problemNew names, typos, rare words — a word vocabulary can't contain them all, so each becomes [UNK]. Characters avoid that but make sequences very long (our headline: 45 character-tokens vs 8 words).
The middle ground
Subwords: pick the vocabulary
| characters | subwords | words |
| vocabulary size | ~100s | 30k–200k | millions |
| tokens · our headline | 45 | 8 | 8 |
| tokens · exoplanets | 10 | 3 | [UNK] |
exoplanets →␣exoplanets
Frequent words stay whole; rare words split into known pieces — short sequences, never [UNK]. You choose the vocabulary size.
The algorithm
Byte-Pair Encoding (BPE)
Start from single characters, then repeat one step until the vocabulary is the size you want:
- Count every adjacent pair of symbols across the corpus (weighted by word frequency).
- Merge the most frequent pair into one new token; add it to the vocabulary.
- Repeat. Each merge adds exactly one token — so vocabulary size = how many merges you do.
e r → erer _ → er_n e → nene w → newl o → lo
(9)(9)(8)(8)(7)
No unknowns, ever
Byte-level BPE
a new word never seen in training: lower
lower→lower_
replay the learned merges in order → known pieces, no [UNK]
start from the 256 raw bytes, not characters
706C616E6574
any text in any script is a byte sequence — nothing falls outside the vocabulary
Why A1 starts at 257Byte-level is lossless — no [UNK], no dropped whitespace. The base vocabulary is the 256 bytes (+ one end marker) = the 257 point in your sweep; every merge on top makes tokens longer and the sequence shorter.
Tying it to A1
Vocabulary size is a dial
Same text, different vocabulary → a different number of tokens. That is why your sweep's per-token perplexity moves even though the text never changes — and why the fair comparison is per byte.
Bigger vocabulary helps
Fewer, longer tokens → shorter sequences; each token carries more context, fewer factors to multiply.
Bigger vocabulary hurts
More token types → each bigram is seen fewer times → sparser counts, shakier estimates (worse without smoothing).
A1 · findings Q3You sweep 257 / 1k / 8k and explain, with your own numbers, what a larger vocabulary buys and what it costs — judged on per-byte perplexity, the only axis that stays fair across sizes.
The service behind A1 — and the rest of the course
TULIP — TU Delft's LLM platform
- What it is: an OpenAI-compatible API to selected, locally hosted language models — a pilot service from REIT and ICT for research, education and institutional use.
- We are piloting it for this course: you will use TULIP in the assignments and the projects, starting with A1.
- How to use it: connect over eduVPN with your personal key; calls look just like the OpenAI API. Each of you has a token budget — the kit tracks it and stops before you run out.
- Specifics live in each assignment: endpoints, model names and limits are spelled out there (A1 uses model chat).
Your turn · A1 — the task
Assignment 1 — what you build
In one line: build an n-gram language model — specifically a bigram with add-k smoothing — compare it to a modern LM (TULIP) using perplexity, and reason about your results and your experience.
- 1 · Tokenizer (provided). A byte-level BPE tokenizer (vocab 8000) trained on the kit's text — lossless, no [UNK].
- 2 · Your n-gram model. A bigram over those tokens with add-k smoothing — you write fit and prob.
- 3 · Score with TULIP. Turn TULIP's per-token log-probs into a document NLL (you write doc_nll).
- 4 · Perplexity, two ways. On your held-out set, per token and per byte — per byte is the fair comparison. Repeated over a vocab sweep 257 / 1k / 8k.
- 5 · Generate & compare. One continuation per prompt from your n-gram and from TULIP (temperature 1.0).
One command
python a1.py --student-number <yours> — about 2–3 minutes on a laptop, no GPU. Your student number fixes your personal held-out set.
Your turn · A1 — findings & grading
Two questions about your own numbers
- Q1. Rank 257-vocab n-gram, 8k n-gram and TULIP by per-token then per-byte PPL — why do the rankings disagree, and which is the fair comparison?
- Q2. Compare your 8k n-gram and TULIP per byte — is it what you expected from a bigram vs a large neural model? Give two reasons, each backed by your own evidence (the 10 most surprising tokens, or how the samples read).
How it's graded — 10 points
Correctness (automatic) · 5 — your numbers are recomputed on your own held-out set (n-gram 2 · TULIP scoring 2 · sweep & counts 1); no hidden reference, nothing compared to a classmate. Understanding (TA) · 5 — findings Q1–Q2 (3) + reflection (2); answers that use your numbers score high.
Your turn · A1 — how to submit
Submitting your work
One zip to Brightspace · A1_<studentnumber>.zip
a1_results.json · tokenizer.json · sweep/ · samples.txt · findings.md (Q1–Q2) · reflection.md + AI-use · code/
- Build it with python make_zip.py <studentnumber> — it runs the format checker; fix every error before you upload.
- Do not include the dataset, your .env file, or the TULIP cache. Use the same student number you passed to load_lm.
- TULIP setup first: eduVPN + your key in .env, then python tulip_check.py — do this early, not on deadline night. Runs stop at a 400k-token budget.