← →/space · click bg to advance · F fullscreen
DSAIT4090 · Lexical processing & language modelling

Statistical Language Models

Assigning probabilities to text — and measuring it, with nothing more than counting.
A. Anand · study deck
Motivation

Why language modelling?

Almost everything people know is written down. For a computer to use that knowledge — to answer, translate, summarise — it first has to model how language works.

The linguist's view

Describe language with rules written by hand — grammar, syntax, parts of speech.

The computational view

Describe language with probability — learn from how people actually use words, and let fluency and meaning fall out of the statistics.

The shift

No grammar is written by hand. The model learns one thing from data — what usually comes next — and that turns out to be enough to read, write and reason over text.

The core move

A sentence, by probability

“Understanding” becomes one concrete number: give every sentence a probability. Sensible, fluent sentences score high; nonsense scores low.
Not the same as grammar

That last sentence is where we part ways with the linguist: a grammar calls it well-formed, but the language model still finds it unlikely. We model what is probable, not merely what is allowed.

The object

What is a language model?

Definition A language model assigns a probability to a piece of text. Equivalently, it predicts the next word: given a context, it outputs a probability for every word in the vocabulary.
Question
The model reads “new telescope spots water on a distant ___”.
What should it output for the blank?
Not one word — think about what form the answer takes.
Think — then press → for the answer.
Answer

A distribution over the whole vocabulary

new telescope spots water on a distant  ___ 
planet
0.62
star
0.15
moon
0.12
galaxy
0.07
ocean
0.04
illustrative numbers — the distribution sums to 1
A language model predicts what comes next — a probability for every word in the vocabulary.
Definitions

A language model, precisely

Language model A probability distribution over token sequences — two equivalent views: the joint probability of a whole sequence, factored into next-token predictions.
Example — the two views on one sentence

Each factor is a next-word probability; multiply them to get the whole-sentence probability.

The question for the rest of the lecture

Everything now reduces to one problem: how do we estimate these probabilities — the P(wₜ | context) — from data?

Before we count — what do we count?

What is a token?

To estimate P(wₜ | context) by counting, we first have to decide what a “word” — a token — actually is. We cut the text into tokens, and count those.
Question
Given a large corpus of text, how can we estimate these probabilities?
We need them to make predictions — and all we have is counts of what people actually wrote.
Think — then press → for the answer.
A little vocabulary first

Three terms, in pictures

corpus — the text we learn from
<s> new telescope spots water </s>
<s> telescope spots a planet </s>
<s> new telescope spots a planet </s>
<s> water on a distant planet </s>
n-gram — a short run of tokens
unigram spots  ·  bigram spotsa  ·  trigram spotsaplanet
parameter — one probability per n-gram
— how likely a is right after spots
Estimating it just means reading it off the corpus: spots a appears 2×, spots 3×  → 
We build a bigram model: one parameter P(word | previous word) for each pair, read straight from these counts.
The simplest useful model

Training = counting

All three n-grams follow the same count-and-normalise recipe — we work through the bigram here.
Maximum-likelihood estimate (bigram) P(wᵢ | wᵢ₋₁) = count(wᵢ₋₁ wᵢ)
count(wᵢ₋₁)
Question
A bigram's parameters are the probabilities P(word | previous word). Given this corpus, how do we find one — say P(a | spots)?
training corpus
<s> new telescope spots water </s>
<s> telescope spots a planet </s>
<s> new telescope spots a planet </s>
<s> water on a distant planet </s>
Think — then press → for the answer.
Answer

A bigram model is just counting

training corpus
<s> new telescope spots water </s>
<s> telescope spots a planet </s>
<s> new telescope spots a planet </s>
<s> water on a distant planet </s>
spots occurs 3× → a twice, water once
Count the pair spots a (2), divide by how often spots occurs (3): P(a | spots) = ⅔ — exact, from the corpus on the left.
Recap

Counts in, distribution out

training corpus
<s> new telescope spots water </s>
<s> telescope spots a planet </s>
<s> new telescope spots a planet </s>
<s> water on a distant planet </s>
P(· | <s>)
new ½ · telescope ¼ · water ¼
P(· | spots)
a ⅔ · water ⅓
P(· | a)
planet ⅔ · distant ⅓
The idea

Each row is a conditional distribution that sums to 1. The parameters are these normalised counts — training by maximum likelihood is just count & divide.

From words to sentences

Scoring a whole sentence: the chain rule

We can estimate one next-word probability by counting. To score a whole sentence, multiply them — that is the chain rule, and it is exact.
P(w₁,…,wₙ) = P(w₁)·P(w₂|w₁)·P(w₃|w₁,w₂)···P(wₙ|w₁,…,wₙ₋₁)
Exact — no approximation yet

The chain rule just rewrites the joint probability as a product of next-word factors; nothing is lost. The trouble is only in estimating those factors.

The catch

But the full history can't be counted

The last factors condition on a long history — a seven-word context almost never appears even once. So how much history can we actually afford?
The fix for sparsity

Shorten the history

If the full history can’t be counted, condition on only the last few words instead. This is the Markov assumption — the single idea that makes n-gram models possible.
unigramP(w₅)no history · counts very reliable
bigramP(w₅|w₄)one word back
trigramP(w₅|w₃,w₄)two words back
fullP(w₅|w₁,w₂,w₃,w₄)no assumption · hopelessly sparse
Question
The exact long history never appears in any corpus. So how can we estimate the next word at all?
And if we throw history away, what do we lose?
Think — then press → for the answer.
Answer

Markov assumption & the n-gram trade-off

newtelescopespotswateronadistantplanet
unigram
no history · counts reliable
bigram
one word back
trigram
two words back
full
hopelessly sparse
How much history is enough? More context means richer predictions but sparser counts.
Recap

A trade-off, not a free lunch

The Markov assumption

P(wᵢ | w₁…wᵢ₋₁) ≈ P(wᵢ | wᵢ₋ₙ₊₁…wᵢ₋₁) — keep only the last n−1 words.

Too short (bigram)

“the telescope we bought last year finally spotted a ___” — the bigram sees only “a”. It has forgotten telescope, so planet and pancake look equally likely. Reliable counts, weak predictions.

Too long (6-gram)

“spots water on a distant ___” as a 6-gram — that exact history almost never recurs, so its count is 0 and it predicts nothing. Rich context, but unusable.

The real test

Scoring a sentence never seen in training

Our running example — new telescope spots water on a distant planet — is not in the corpus. A model that only memorised sentences would be stuck here.
But a bigram model only needs pairs, and it can walk the sentence one bigram at a time, from <s> to </s>, multiplying the factors.
Question
The example sentence isn’t in the corpus. Can a bigram model still assign it a probability — and what is it?
Nine factors (eight words + the end token). What do they multiply to?
Think — then press → for the answer.
Answer

Where every factor comes from

<s>newtelescopespotswateronadistantplanet</s>
Each factor is a count ratio C(pair)/C(first word). A 1 means that word was always followed by the next in this corpus; only four factors fall below 1.
Recap & a warning

An unseen sentence scores — unless a pair is missing

The result

Every bigram in the sentence appears in the corpus, so each factor is > 0 and P(sentence) = 1/36 — generalising from pieces it has seen, not memorising whole sentences.

The alternate world — one unseen pair

Score instead “<s> new planet …”. The pair new planet never occurs, so with plain counting — and the whole product collapses to P = 0, however reasonable the rest.

One missing bigram kills the sentence. That is the zero problem — and the reason we will need smoothing.

Question
The same table of counts can score a sentence. Can it also write one?
If so, with what procedure — and will it only ever repeat the corpus?
Think — then press → for the answer.
Answer

Generating text by sampling

<s>new…
next-word distribution  ·  one random sample
new
½
telescope
¼
water
¼
To generate: look up the next-word distribution, sample one word, append, repeat.
Recap

Score and generate — same counts

Idea

From <s>: look up the next-word distribution, sample one word, append it, repeat until </s>.

Output — “new telescope spots a distant planet” — never appeared in the corpus, yet every adjacent pair in it did. Novelty by chaining seen bigrams.

Temperature rescales that distribution before sampling: T = 1 is the model's own probabilities (what A1 uses); lower T is greedier, higher T more random. More in Deck 8 · Decoding.

Practical issue 1

Tiny probabilities

A sentence probability is a product of many numbers below 1 — it gets vanishingly small fast.
Question
Multiply a thousand small probabilities on a real computer. What happens — and how do we avoid it?
Think — then press → for the answer.
Answer

Underflow & log-probabilities

a 1,000-word document, each word at p = 0.1:
Fix: add log-probabilities instead of multiplying probabilities.
Recap

Work in log space

log ∏ᵢ p(wᵢ|wᵢ₋₁) = ∑ᵢ log p(wᵢ|wᵢ₋₁)
Fix

The log of a product is the sum of the logs — so add log-probabilities instead of multiplying probabilities. The running sum never underflows, and log is monotonic so comparisons are unchanged.

For our sentence: log(1/36) = −3.58.

Practical issue 2

Zeros & sparsity

Most word pairs are never seen, even in a huge corpus. An unseen bigram has count 0 → probability 0.
And because we multiply, a single zero makes the whole sentence impossible — e.g. new planet spots water: the pair new planet never occurred, so P=0 however reasonable the rest.
Question
A single unseen pair sends the whole sentence probability to zero. How do we stop that — without lying about what we saw?
Think — then press → for the answer.
Answer

Add-one smoothing & backoff

Add-one smoothing

Pretend every pair was seen once more: +1 on top, +V below — still sums to 1.

P(planet | new): 0 → 1/11

Backoff

Unseen bigram? Fall back to the unigram. Less context, but never zero.

P(planet) = 3/22

unseen ≠ impossible — steal a little mass for what you didn't see
Two ways to banish the zero — because unseen is not the same as impossible.
Recap

Never exactly zero

Two fixes · one assumption

Add-one smoothing: pretend every pair was seen once more — add 1 on top, add V=9 below, so it still sums to 1. P(planet|new): 0 → 1/11.

Add-k (general): add a fraction k instead of 1 — ; add-one is just k=1. Pick k on a held-out dev split. (this is the smoother you implement in A1.)

Backoff: unseen bigram → fall back to the unigram. P(planet) = 3/22. Less context, but not zero. Smarter smoothers like Kneser–Ney redistribute mass by how many contexts a word appears in — the classic n-gram baseline (optional in A1).

Underlying assumption: unseen ≠ impossible. Steal a little mass for everything you didn’t see.

Evaluation

Is the model any good?

Perplexity
The sum is the negative log-likelihood (NLL) of the text; perplexity is just exp(NLL / N) — the average per token.
Question
Two models — a unigram and a bigram — each score the same held-out sentence. Which is better, and how do we put one number on it?
Each assigns the sentence a probability; lower or higher perplexity — which way is good?
Think — then press → for the answer.
Answer

Perplexity: bigram vs unigram

Same held-out sentence new telescope spots water on a distant planet (N = 9), scored by two models on the same corpus.
unigram — P(wᵢ), no context
new2/22a3/22
telescope3/22distant1/22
spots3/22planet3/22
water2/22</s>4/22
on1/22
9.92
bigram — P(wᵢ|wᵢ₋₁), one word of context
new1/2a1
telescope1distant1/3
spots1planet1
water1/3</s>1
on1/2
1.49
Context makes each factor larger → higher P(W) → lower perplexity. The bigram wins.
Recap

One number: lower is better

Perplexity

= inverse probability of the test set, normalised by length = exp(average negative log-prob per token). Lower is better.

Bigram: = 1.49 (as unsure as choosing between ~1.5 words). Unigram, ignoring context: 9.92. Context really helps.

Measured over tokens (e.g. BPE) — so comparable only between models with the same tokenizer.

Perplexity today

Perplexity hasn't gone away

ModelPerplexity ≈Test set · tokenizer
5-gram (Kneser–Ney)~141PTB · words
LSTM (2016)~48WikiText-103 · words
Transformer-XL (2019)~18WikiText-103 · words
GPT-2 1.5B (2019)~17.5WikiText-103 · words, zero-shot
LLaMA-2 7B (2023)~5.5WikiText-2 · own BPE
Frontier 70B+~3web text · own BPE
Read it carefully

Lower is better — but these are not directly comparable: different test sets and, above all, different tokenizers. The drop from ~18 to ~3 is mostly sub-word vs word counting, not a 6× better model.

Instruction-tuned / chat models are not really comparable to pure language models either: tuning trades raw next-token likelihood for following instructions, so their perplexity on plain text can even look worse.

Comparing across tokenizers

Perplexity per byte

Two models with different tokenizers can't be ranked by per-token perplexity — the denominator N counts different things. Normalise instead by something both share: the UTF-8 bytes of the text.
Per token vs per byte
Why per byte is the fair comparison

Same NLL (negative log-likelihood — the summed −log p over the text), divided by a count that is identical for every model — the bytes of the text as written. A coarse tokenizer that spends fewer, bigger tokens no longer looks artificially better.

The two are linked exactly by bytes-per-token: . (in log₂ this is bits per byte, the standard cross-model report — and exactly what A1 asks you to compute.)

Answer

Recap & the bridge to cross-entropy

Next: softmax & cross-entropy — the loss that trains a neural network.
Recap

Statistical language models

The bridge

Look inside perplexity: the average negative log-probability per token has a name — it is the cross-entropy. So perplexity = e^(cross-entropy).

Next: cross-entropy & softmax — the loss that trains a neural network.
Back to tokens

Words, or tokens?

We have been talking about words — and quietly kept saying tokens. Are they the same thing?
Two questions decide everything that follows

1 · How do we cut text into tokens?    2 · How big should the vocabulary be — and why does that change the numbers we just computed?

Why not just words?

A word vocabulary never ends

word-level vocabulary
exoplanets→[UNK]
any word unseen in training collapses to one unknown token — its meaning is lost
The problem

New names, typos, rare words — a word vocabulary can't contain them all, so each becomes [UNK]. Characters avoid that but make sequences very long (our headline: 45 character-tokens vs 8 words).

The middle ground

Subwords: pick the vocabulary

characterssubwordswords
vocabulary size~100s30k–200kmillions
tokens · our headline4588
tokens · exoplanets103[UNK]
exoplanets →␣exoplanets
Frequent words stay whole; rare words split into known pieces — short sequences, never [UNK]. You choose the vocabulary size.
The algorithm

Byte-Pair Encoding (BPE)

Start from single characters, then repeat one step until the vocabulary is the size you want:
e r → erer _ → er_n e → nene w → newl o → lo (9)(9)(8)(8)(7)
No unknowns, ever

Byte-level BPE

a new word never seen in training: lower
lower→lower_
replay the learned merges in order → known pieces, no [UNK]
start from the 256 raw bytes, not characters
706C616E6574
any text in any script is a byte sequence — nothing falls outside the vocabulary
Why A1 starts at 257

Byte-level is lossless — no [UNK], no dropped whitespace. The base vocabulary is the 256 bytes (+ one end marker) = the 257 point in your sweep; every merge on top makes tokens longer and the sequence shorter.

Tying it to A1

Vocabulary size is a dial

Same text, different vocabulary → a different number of tokens. That is why your sweep's per-token perplexity moves even though the text never changes — and why the fair comparison is per byte.

Bigger vocabulary helps

Fewer, longer tokens → shorter sequences; each token carries more context, fewer factors to multiply.

Bigger vocabulary hurts

More token types → each bigram is seen fewer times → sparser counts, shakier estimates (worse without smoothing).

A1 · findings Q3

You sweep 257 / 1k / 8k and explain, with your own numbers, what a larger vocabulary buys and what it costs — judged on per-byte perplexity, the only axis that stays fair across sizes.

The service behind A1 — and the rest of the course

TULIP — TU Delft's LLM platform

TULIP — TU Delft LLM Infrastructure Platform
Your turn · A1 — the task

Assignment 1 — what you build

In one line: build an n-gram language model — specifically a bigram with add-k smoothing — compare it to a modern LM (TULIP) using perplexity, and reason about your results and your experience.
One command

python a1.py --student-number <yours> — about 2–3 minutes on a laptop, no GPU. Your student number fixes your personal held-out set.

Your turn · A1 — findings & grading

Two questions about your own numbers

How it's graded — 10 points

Correctness (automatic) · 5 — your numbers are recomputed on your own held-out set (n-gram 2 · TULIP scoring 2 · sweep & counts 1); no hidden reference, nothing compared to a classmate. Understanding (TA) · 5 — findings Q1–Q2 (3) + reflection (2); answers that use your numbers score high.

Your turn · A1 — how to submit

Submitting your work

One zip to Brightspace · A1_<studentnumber>.zip a1_results.json · tokenizer.json · sweep/ · samples.txt · findings.md (Q1–Q2) · reflection.md + AI-use · code/
Dates

Released 19 Nov · due 26 Nov, 23:59.  Open the A1 brief →