Course 8 of 13 · Generative AI / LLMs

Gen AI Foundational Models for NLP & Language Understanding

How text becomes numbers — one-hot to embeddings to Word2Vec to seq2seq, and the bottleneck that produced attention

10 hours · 2 modules · PyTorch, torchtext, Gensim

How text becomes numbers, and how language models worked before transformers. Do not skip this as history — embeddings are live infrastructure (every vector database in Course 12–13 stores them), and the encoder–decoder framing here is what the transformer replaced piece by piece.

Module 1 — Fundamentals of Language Understanding

One-hot encoding

Each word is a vector of vocabulary length: 1 in its own position, 0 everywhere else.

vocab = [cat, dog, run]
cat = [1,0,0]   dog = [0,1,0]   run = [0,0,1]

Three fatal problems:

1. Dimensionality — 50,000-word vocabulary means 50,000-dimensional vectors
2. Sparsity — one non-zero value per vector; almost all storage is zeros
3. No semantics — every pair is equidistant. cat and dog are exactly as similar as cat

and run. Cosine similarity between any two distinct one-hot vectors is 0.

That third point is the motivation for everything else in this module.

Bag of Words

Represent a document by word counts, ignoring order.

"the cat sat on the mat" → {the:2, cat:1, sat:1, on:1, mat:1}

Order is destroyed: "dog bites man" and "man bites dog" are identical. Still a strong baseline for topic classification.

TF-IDF improves on raw counts by down-weighting words that appear everywhere:

TF-IDF(t,d) = TF(t,d) × log(N / DF(t))

Terms frequent in this document but rare across the corpus score highest. "the" gets crushed; "mitochondria" survives.

Embeddings

A dense, low-dimensional, learned vector per token — typically 100–1024 dimensions.

The properties that matter:

• Dense — every dimension carries information
• Semantic — similar meanings land near each other in the space
• Compositional — the famous king − man + woman ≈ queen vector arithmetic
• Learned — from co-occurrence in real text, not hand-designed

Underlying principle (the distributional hypothesis): words that appear in similar contexts have similar meanings. Everything from Word2Vec to BERT to modern embedding APIs is an elaboration of that one idea.

emb = nn.Embedding(num_embeddings=vocab_size, embedding_dim=100)
vecs = emb(torch.tensor([5, 23, 99]))    # (3, 100)

nn.Embedding is a lookup table that happens to be trainable — mathematically a one-hot vector times a weight matrix, implemented as an index lookup because that's vastly faster.

EmbeddingBag

Averages (or sums) the embeddings of a variable-length sequence in one operation — no padding required, faster and more memory-efficient than embed-then-mean.

emb = nn.EmbeddingBag(vocab_size, embed_dim, mode='mean')
out = emb(text_flat, offsets)     # offsets marks where each document starts

The offsets tensor is how it handles ragged batches: concatenate all documents into one flat tensor and record start positions. Standard for fast text classification (the AG_NEWS classifier lab uses exactly this).

Similarity

cosine_similarity(a,b) = (a·b) / (‖a‖·‖b‖)

Range −1 to 1. Cosine, not Euclidean, because direction encodes meaning and magnitude mostly encodes frequency. This is the metric every vector database in Course 13 uses.

Module 2 — Word2Vec and Sequence-to-Sequence Models

Word2Vec

Learn embeddings by training a shallow network on a fake task; throw away the network and keep the weight matrix. The embeddings are the by-product, and that's the clever part.

Two architectures:

CBOW (Continuous Bag of Words) — predict the target word from its surrounding context. [the, sat, on, the] → cat Faster; better on frequent words; smooths over the data.

Skip-gram — predict the context words from the target. cat → [the, sat, on, the] Slower; better on rare words and small corpora. Generally the better choice for quality.

Mnemonic: CBOW = context→word, Skip-gram = word→context. Guaranteed quiz item.

Window size controls what "similar" means: small windows (2) capture syntactic similarity (interchangeable parts of speech), large windows (10) capture topical similarity.

Negative sampling — the softmax over a 50k vocabulary is prohibitively expensive at every step, so instead: take the true context pair plus k random "negative" pairs and train a binary classifier. Turns one 50,000-way softmax into k+1 binary decisions. This optimisation is what made Word2Vec trainable at scale.

from gensim.models import Word2Vec

model = Word2Vec(sentences, vector_size=100, window=5, min_count=5,
                 sg=1,        # 1 = skip-gram, 0 = CBOW
                 negative=5, workers=4)
model.wv.most_similar('king')
model.wv.most_similar(positive=['king','woman'], negative=['man'])

Related methods to recognise by name: GloVe (factorises a global co-occurrence matrix rather than using local windows) and FastText (embeds character n-grams, so it can build vectors for words it has never seen — strong on morphologically rich languages and typos).

The fundamental limitation: Word2Vec gives one static vector per word. "bank" has a single embedding covering riverbank and financial institution. Contextual embeddings (ELMo, BERT, and everything after) fix this by producing a different vector depending on the sentence — which is the argument for the next course.

N-gram language models

Predict the next word from the previous n−1 words, using counts:

P(wₙ | wₙ₋₂, wₙ₋₁) = count(wₙ₋₂, wₙ₋₁, wₙ) / count(wₙ₋₂, wₙ₋₁)

The Markov assumption: only the last n−1 words matter. Wrong, but tractable.

Problems: unseen n-grams get probability 0 (fixed with smoothing — Laplace/add-one, Kneser-Ney); the table grows exponentially with n; no generalisation between similar words (having seen "black cat" tells the model nothing about "dark cat").

Neural n-gram LM fixes generalisation: embed the context words, concatenate, run through an MLP, softmax over the vocabulary. Because similar words have similar embeddings, the model generalises across them automatically.

class NGramLM(nn.Module):
    def __init__(self, vocab_size, embed_dim, context_size, hidden):
        super().__init__()
        self.emb = nn.Embedding(vocab_size, embed_dim)
        self.fc1 = nn.Linear(context_size * embed_dim, hidden)
        self.fc2 = nn.Linear(hidden, vocab_size)
    def forward(self, x):                      # x: (batch, context_size)
        e = self.emb(x).view(x.size(0), -1)    # concatenate context embeddings
        return self.fc2(torch.relu(self.fc1(e)))   # raw logits

Still fixed-length context. RNNs remove that constraint; transformers remove it and the sequential bottleneck.

Evaluating a language model — perplexity

Perplexity = exp(cross-entropy loss)

Interpretation: the effective number of equally-likely choices the model is deciding between at each step. Lower is better. Perplexity 10 means the model is about as uncertain as if choosing uniformly among 10 words.

Only comparable between models sharing the same tokenizer and test set — a point people routinely get wrong when comparing published numbers.

Sequence-to-sequence models

Map an input sequence to an output sequence of different length: translation, summarisation, dialogue.

INPUT → [Encoder RNN] → context vector → [Decoder RNN] → OUTPUT

Encoder consumes the input and compresses it into a fixed-size context (its final hidden state). Decoder generates the output token by token, conditioned on that context and its own previous outputs.

class Encoder(nn.Module):
    def __init__(self, vocab, emb, hid):
        super().__init__()
        self.embedding = nn.Embedding(vocab, emb)
        self.rnn = nn.GRU(emb, hid, batch_first=True)
    def forward(self, x):
        outputs, hidden = self.rnn(self.embedding(x))
        return outputs, hidden                     # hidden = the context vector

class Decoder(nn.Module):
    def __init__(self, vocab, emb, hid):
        super().__init__()
        self.embedding = nn.Embedding(vocab, emb)
        self.rnn = nn.GRU(emb, hid, batch_first=True)
        self.fc  = nn.Linear(hid, vocab)
    def forward(self, x, hidden):
        out, hidden = self.rnn(self.embedding(x), hidden)
        return self.fc(out), hidden

Teacher forcing — during training, feed the decoder the true previous token rather than its own prediction. Trains much faster and more stably. Downside: exposure bias — at inference the model must consume its own (possibly wrong) outputs, a situation it never experienced in training, so errors compound. Mitigation: scheduled sampling, mixing true and predicted tokens with a decaying probability.

The bottleneck problem — the key insight of the module: the entire input sequence must be squeezed into one fixed-size vector. For a 50-word sentence that vector is hopelessly overloaded, and translation quality degrades sharply with input length.

The fix is attention. Instead of one context vector, the decoder computes a weighted combination of all encoder hidden states at every output step, weighting whichever input positions are relevant right now.

score(sᵢ, hⱼ) → softmax → αᵢⱼ → context_i = Σⱼ αᵢⱼ · hⱼ

Then the 2017 realisation: if attention is doing the real work, drop the recurrence entirely. That is Course 9.

Decoding strategies — how you turn logits into text:

• Greedy — take the argmax each step. Fast, repetitive, locally myopic.
• Beam search — keep the k best partial sequences. Better for translation; can be bland.
• Sampling with temperature — divide logits by T before softmax. T<1 sharpens (conservative),

T>1 flattens (creative), T→0 is greedy.

• Top-k — sample from the k most likely tokens.
• Top-p / nucleus — sample from the smallest set whose cumulative probability exceeds p.

Adapts the candidate pool to the model's confidence; the modern default.

You'll meet these again as API parameters in Courses 12–13, so learn them properly here.

Course 8 in one page

• One-hot: huge, sparse, semantically useless — every pair equidistant
• Embeddings: dense, learned, semantic; nn.Embedding is a trainable lookup table
• Compare with cosine similarity
• CBOW = context→word; Skip-gram = word→context (better on rare words)
• Negative sampling replaces a full-vocabulary softmax with k+1 binary decisions
• Word2Vec/GloVe are static — one vector per word regardless of context; BERT fixes this
• FastText uses character n-grams → handles unseen words and typos
• N-gram LMs: Markov assumption, zero-probability problem, no generalisation; neural LMs

generalise via embeddings

• Perplexity = exp(cross-entropy), lower is better, only comparable within a tokenizer
• Seq2seq = encoder → context vector → decoder; teacher forcing speeds training, causes exposure bias
• The fixed context vector is the bottleneck; attention removes it; removing recurrence gives you

the transformer

• Decoding: greedy / beam / temperature / top-k / top-p