Tokens and Embeddings

Adapted from: Jay Alammar & Maarten Grootendorst, Hands-On Large Language Models (O'Reilly, 2024), Chapter 2 — rewritten and condensed for teaching, with code examples adapted from the original.

Tokens and embeddings are two of the most central ideas behind how large language models actually work. A model never sees raw text — it sees a sequence of small chunks called tokens, and to do any computation on them at all, it first has to turn those tokens into numeric vectors called embeddings.

This chapter walks through both halves of that pipeline: how tokenizers break text into tokens, and how those tokens become embeddings — first as the static, per-token vectors a model starts from, then as the richer contextualized vectors a trained model produces. From there we zoom out from single tokens to whole sentences and documents (the basis of things like semantic search), and then sideways, to see how the exact same embedding idea — originally popularized by word2vec — powers recommendation systems that have nothing to do with language at all.

---

LLM Tokenization

If you've used a chat-style LLM playground, you may have noticed the response doesn't appear all at once — it streams in piece by piece, one token at a time. That's not just an output quirk: tokens are also how the model reads its input. Before a prompt ever reaches the model, it passes through a tokenizer that chops it into pieces — sometimes whole words, sometimes fragments of words, sometimes punctuation.

Downloading and running a real tokenizer

Here's what that actually looks like in code, loading a small instruction-tuned model and its paired tokenizer:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct",
    device_map="cuda",
    torch_dtype="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

With both loaded, generation is a three-step dance: tokenize the prompt into IDs, hand those IDs to the model, then decode whatever IDs come back:

prompt = "Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|>"

# Tokenize the input prompt
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")

# Generate the text
generation_output = model.generate(
    input_ids=input_ids,
    max_new_tokens=20,
)

# Print the output
print(tokenizer.decode(generation_output[0]))

Notice the model itself never receives the text prompt — it receives input_ids, a tensor of plain integers. Printing input_ids makes this obvious:

tensor([[    1, 14350,   385,  4876, 27746,  5281,   304, 19235,   363,   278,
         25305,   293, 16423,   292,   286,   728,   481, 29889, 12027,  7420,
           920,   372,  9559, 29889, 32001]], device='cuda:0')

Each number is the unique ID of a specific token, looked up in a table the tokenizer maintains internally — its vocabulary. To see what each ID actually stands for, decode them one at a time:

for id in input_ids[0]:
    print(tokenizer.decode(id))

<s>
Write
an
email
apolog
izing
to
Sarah
for
the
trag
ic
garden
ing
m
ish
ap
.
Exp
lain
how
it
happened
.
<|assistant|>

A few things worth noticing:

• <s> (ID 1) is a special token marking the start of the text — it isn't "real" text at all.
• Whole words show up as single tokens: Write, an, email.
• Longer or less common words get split into fragments: apolog + izing, trag + ic.
• Punctuation gets its own token.

There's a subtlety about spaces here: the tokenizer doesn't emit a separate token for the space character. Instead, fragment tokens like izing and ic carry a hidden marker meaning "I'm glued directly onto the token before me, no space." A token without that marker is assumed to have a space in front of it. That's the tokenizer's way of encoding whitespace without wasting a whole token on it.

Generation works the same way in reverse: the model emits token IDs one at a time (e.g. 3323 → 'Sub', then 622 → 'ject', together forming "Subject"), and the tokenizer's decode turns those back into text — you can even hand it a list of IDs at once:

print(tokenizer.decode(3323))
print(tokenizer.decode(622))
print(tokenizer.decode([3323, 622]))
print(tokenizer.decode(29901))

Sub
ject
Subject
:

Check: how tokenization works

Quiz

How LLM tokenization works

1 / 2

What does a language model actually receive as input — the raw prompt text, or something else?

Pick an answer to see if you are right.

🧪 A tiny illustration of "token IDs → text" round-tripping

---

How Does the Tokenizer Break Down Text?

Three design decisions, made well before you ever call .generate(), determine exactly how a tokenizer will chop up a given piece of text:

1. The tokenization method — chosen by the model's creators. Byte pair encoding (BPE) is widely used by GPT-family models; WordPiece is used by BERT. Both aim to find an efficient set of tokens to represent a text dataset, just via different algorithms.
2. Tokenizer parameters — vocabulary size, which special tokens to include, and so on (covered in more depth below).
3. The training dataset — even with identical method and parameters, a tokenizer trained on English prose will differ from one trained on source code or a multilingual corpus.

Tokenizers aren't only used on the way in, either — the exact same tokenizer is used on the way out, translating each generated token ID back into readable text.

Word vs. Subword vs. Character vs. Byte Tokens

The tokenization scheme in the walkthrough above — mixing whole words and word fragments — is called subword tokenization, and it's by far the most common approach today. But it's not the only option:

Scheme	How it works	Strength	Weakness
Word tokens	Each whole word is one token	Simple, was standard pre-LLM (word2vec era)	Can't handle new/unseen words; wastes vocabulary on near-duplicates like apology, apologize, apologetic
Subword tokens	Mix of whole words and fragments	Handles new words by falling back to known fragments; more expressive vocabulary (one apolog root + reusable suffixes like -y, -ize, -etic)	Slightly more complex to decode (fragment-gluing rules)
Character tokens	Every character is its own token	Never truly "unknown" — can always fall back to raw letters	Makes modeling harder (spelling out "p-l-a-y" instead of one token for "play"); wastes context length
Byte tokens	Break text into the raw bytes behind Unicode characters	Fully general, "tokenization-free"; strong in multilingual settings (see CANINE and ByT5)	Even more tokens needed per word than character-level

Subword tokenization strikes a practical balance: for a model with a 1,024-token context window, subword tokenization typically fits roughly 3x more actual text than character-level tokenization would, since subword tokens average about three characters each.

Byte-level fallback isn't the same as a byte-level tokenizer

Some subword tokenizers — GPT-2 and RoBERTa among them — include individual bytes in their vocabulary purely as a fallback for characters they can't otherwise represent. That's different from a true byte-level ("tokenization-free") tokenizer, which uses bytes to represent everything, not just edge cases.

Check: tokenization schemes

Quiz

Word, subword, character, and byte tokens

1 / 3

Why did word-level tokenization fall out of favor for modern NLP models?

Pick an answer to see if you are right.

🧪 A toy BPE-style merge step

Real BPE training repeatedly merges the most frequent adjacent pair of symbols in a corpus. Here's a stripped-down illustration of a couple of merge rounds:

---

Comparing Trained LLM Tokenizers

The best way to see how these design choices play out is to run the same text through several real, trained tokenizers and compare. A good stress-test string includes capitalization, non-English characters, emoji, code-like syntax, whitespace, and numbers:

text = """
English and CAPITALIZATION
🎵🐦
show_tokens False None elif == >= else: two tabs:"		" Three tabs: "			"
12.0*50=600
"""

Here's a condensed tour of what happens, from older to newer tokenizers:

Tokenizer	Method	Vocab size	Notable behavior on the test string
BERT (uncased, 2018)	WordPiece	30,522	Lowercases everything; drops newlines; emoji/Chinese become [UNK]; wraps input in [CLS] ... [SEP]
BERT (cased, 2018)	WordPiece	28,996	Same as above but preserves case — "CAPITALIZATION" explodes into 8 fragment tokens (CA, ##PI, ##TA, ##L, ##I, ##Z, ##AT, ##ION)
GPT-2 (2019)	BPE	50,257	Preserves newlines and case; represents 4 spaces as 3 tokens (indentation is expensive); emoji become several undecodable-looking tokens that still round-trip correctly
Flan-T5 (2022)	SentencePiece (BPE/unigram)	32,100	No newline or whitespace-run tokens at all (bad for code); emoji/Chinese both collapse to <unk>
GPT-4 (2023)	BPE	~100,000+	A single token for runs of whitespace up to 83 characters long; elif gets its own token; represents most words in fewer tokens overall (code-aware)
StarCoder2 (2024)	BPE	49,152	Code-focused: whitespace runs are one token; each digit is its own token (600 → 6, 0, 0) to help the model reason about numbers; adds <filename>/<reponame> tokens for multi-file code context
Galactica	BPE	50,000	Science-focused: special tokens for citations ([START_REF] / [END_REF]), step-by-step reasoning (<work>), and more
Phi-3 / Llama 2	BPE	32,000	Reuses the Llama 2 tokenizer, plus chat-role tokens: `<

A few patterns worth internalizing from this table:

• Whitespace handling matters a lot for code. A tokenizer that needs 3–4 tokens to represent one indentation level makes it harder for a model to track indentation depth reliably — collapsing a whitespace run into a single token is a deliberate, code-aware design choice (GPT-4, StarCoder2, Galactica all do this).
• Digit splitting is a deliberate choice, too. StarCoder2 assigns each digit its own token specifically so numbers are represented consistently — compare that to GPT-2, where 870 is one token but 871 splits into two (8 + 71), an inconsistency that plausibly makes arithmetic harder to learn.
• Special tokens encode a model's intended use case. Fill-in-the-middle tokens (<fim_prefix>, etc.) exist because those models are trained to complete code given both what comes before and after the cursor. Chat-role tokens exist because those models are trained for multi-turn conversation. A tokenizer's special-token list is a pretty direct readout of what the model was built to do.

Check: comparing real tokenizers

Quiz

Comparing trained tokenizers

1 / 3

Why do code-focused tokenizers like GPT-4's and StarCoder2's represent a run of whitespace (e.g. 4 spaces) as a single token, instead of one token per space?

Pick an answer to see if you are right.

---

Tokenizer Properties, Recap

Pulling the three design axes together:

• Tokenization method — the algorithm (BPE, WordPiece, SentencePiece/unigram, ...) that decides how to build an efficient vocabulary from a dataset.
• Tokenizer parameters — choices like vocabulary size (30K and 50K used to be typical; 100K+ is increasingly common), which special tokens to include (beginning/end-of-text, padding, unknown, classification, masking, or domain-specific ones like Galactica's <work>), and how to handle capitalization.
• The training dataset's domain — the same method and parameters, trained on a different dataset, produce a meaningfully different tokenizer. A tokenizer trained on prose will tokenize indented Python code far less efficiently than one deliberately trained on a code-heavy dataset.

---

Token Embeddings

Tokenization solves half the problem: representing language as a sequence of discrete units. The other half is finding a good numeric representation for each of those units — one a model can actually compute with. That numeric representation is the embedding.

A coherent model isn't automatically a factual one

A model trained on enough tokens starts to capture real patterns in language — grammar, style, even facts, if its training data (like Wikipedia) contained a lot of them. But fluency and factual reliability are different things. Around early 2023, some LLMs were informally dubbed "Google killers" before users discovered that generation alone isn't a reliable substitute for search — which is a big part of why retrieval-augmented generation (RAG), combining search with generation, became so popular.

A language model holds an embedding for every token in its tokenizer's vocabulary

Once a tokenizer is trained, it's paired permanently with a specific language model — you can't swap in a different tokenizer for an already-trained model without retraining. Internally, the model holds one embedding vector per vocabulary entry, stored together as an embedding matrix. When you download a pretrained model, this matrix is part of what you're downloading.

Before training begins, these vectors start out randomly initialized, exactly like the rest of the model's weights. Training is what shapes them into vectors that actually encode useful meaning.

From static to contextualized embeddings

The embedding matrix gives every token a single, fixed ("static") vector, regardless of context — the word "bank" gets the same starting vector whether it's next to "river" or "account." But a trained language model can do better: as tokens pass through the model's layers, their vectors are updated based on surrounding context, producing contextualized embeddings that actually distinguish "river bank" from "savings bank."

These contextualized vectors are the backbone of tasks like named-entity recognition and extractive summarization, and — perhaps surprisingly — the same underlying idea also powers AI image generation systems like DALL·E, Midjourney, and Stable Diffusion.

Here's what producing contextualized embeddings looks like in code, using DeBERTa v3, a small and efficient model well-suited to token embeddings:

from transformers import AutoModel, AutoTokenizer

# Load a tokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base")
# Load a language model
model = AutoModel.from_pretrained("microsoft/deberta-v3-xsmall")

# Tokenize the sentence
tokens = tokenizer('Hello world', return_tensors='pt')
# Process the tokens
output = model(**tokens)[0]

Checking the shape of the result:

output.shape
# torch.Size([1, 4, 384])

Reading that shape right to left: 384 is the size of each embedding vector, 4 is the number of tokens, and the leading 1 is the batch dimension (used when processing several inputs together — irrelevant here, since we only passed one sentence).

Why four tokens for two words? Because the tokenizer wraps the input with its own special tokens:

for token in tokens['input_ids'][0]:
    print(tokenizer.decode(token))

[CLS]
Hello
world
[SEP]

Each of those four tokens gets its own 384-dimensional contextualized vector — the raw output a language model actually produces, and the foundation everything downstream (classification heads, similarity search, and more) is built on.

Check: token embeddings

Quiz

Static vs. contextualized token embeddings

1 / 3

What is stored in a language model's "embedding matrix"?

Pick an answer to see if you are right.

🧪 Static vs. "contextualized" similarity, illustrated with toy vectors

---

Text Embeddings (for Sentences and Whole Documents)

Plenty of applications don't want a vector per token — they want a single vector representing a whole sentence, paragraph, or document. This is exactly what dedicated text embedding models produce: feed in a chunk of text, get back one vector that captures its overall meaning.

One simple way to get there is to average all of a model's token embeddings together. But the best-performing text embedding models are trained specifically for this task, using a package like sentence-transformers:

from sentence_transformers import SentenceTransformer

# Load model
model = SentenceTransformer("sentence-transformers/all-mpnet-base-v2")

# Convert text to text embeddings
vector = model.encode("Best movie ever!")

vector.shape
# (768,)

That's it — one 768-dimensional vector representing the entire sentence. The number of dimensions is a property of the specific embedding model, not something you choose per-call. These sentence/document vectors are the foundation for applications like semantic search, topic modeling, clustering, and retrieval-augmented generation — comparing two pieces of text becomes as simple as comparing two vectors.

Check: text embeddings

Quiz

Sentence and document embeddings

1 / 2

What is the simplest way to turn a sequence of token embeddings into one embedding for the whole sentence?

Pick an answer to see if you are right.

---

Word Embeddings Beyond LLMs

Embeddings turn out to be useful for far more than text. Assigning a meaningful vector to "an object" is a pattern that shows up in recommendation systems, robotics, and beyond — and the technique that made this idea famous, well before modern LLMs, was word2vec.

Using pretrained word embeddings

You can download pretrained word vectors (word2vec, GloVe, and others) with the gensim library:

import gensim.downloader as api

# Download embeddings (66MB, glove, trained on wikipedia, vector size: 50)
model = api.load("glove-wiki-gigaword-50")

Once loaded, you can explore the embedding space directly — for instance, finding the nearest neighbors of "king":

model.most_similar([model['king']], topn=11)

[('king', 1.0000001192092896),
 ('prince', 0.8236179351806641),
 ('queen', 0.7839043),
 ('ii', 0.7746230363845825),
 ('emperor', 0.7736247777938843),
 ('son', 0.766719400882721),
 ('uncle', 0.7627150416374207),
 ('kingdom', 0.7542161345481873),
 ('throne', 0.7539914846420288),
 ('brother', 0.7492411136627197),
 ('ruler', 0.7434253692626953)]

Notably, this vector space wasn't produced by an LLM at all — it comes from word2vec, a much simpler and older method. Yet "king" landing near "queen," "prince," and "throne" shows the embedding space genuinely captured meaningful relationships between words.

The word2vec algorithm: skip-gram and negative sampling

Just like an LLM, word2vec learns from examples generated out of ordinary text. Take a sentence — say, "Thou shalt not make a machine in the likeness of a human mind." Slide a window across it (window size 2 means "look two words to each side"), and every pass of that window generates training examples pairing the center word with each of its neighbors.

The model being trained is deliberately simple: given two words, predict 1 if they tend to appear near each other, 0 if they don't. This idea — the sliding window producing (center word, neighbor word) pairs — is called skip-gram.

There's a catch, though: if every training example is labeled "1" (they're all neighbor pairs pulled from real sentences), the model can trivially "win" by always predicting 1. To prevent that, the training set is padded with negative examples: pairs of words that were not found near each other, generated simply by picking random words from the vocabulary and labeling them 0. This works surprisingly well without needing to be clever about which random words to pick — the underlying idea is called noise-contrastive estimation. Together, skip-gram (generating positive pairs) and this random negative-sampling trick are the two core ideas behind word2vec — and behind a lot of other embedding techniques far beyond it.

Concretely, training proceeds like this:

1. Build a vocabulary from the corpus, make the usual tokenization decisions (capitalization, punctuation, vocabulary size).
2. Randomly initialize an embedding vector for every vocabulary entry — a matrix of shape vocab_size × embedding_dimensions.
3. Train a small neural network on millions (or billions) of (word A, word B, label) examples, where the network takes the two words' current embeddings and predicts whether they're neighbors.
4. Every time a prediction is wrong, the training process nudges the relevant embeddings so the model does a little better next time.

By the end, the embeddings for every word in the vocabulary have been shaped by millions of "are these two neighbors?" comparisons — which is exactly why semantically related words (like "king" and "throne") end up near each other in the resulting vector space.

This idea generalizes far beyond words

"Take two vectors and predict whether they're related" is one of the most reusable ideas in machine learning — it's the same underlying pattern behind training language models for sentence embeddings and retrieval, and it's also central to bridging entirely different modalities (an image and a caption, predicted as "matching" or not), which is foundational to how AI image generation systems work.

Check: word2vec

Quiz

word2vec: skip-gram and negative sampling

1 / 3

In word2vec's skip-gram approach, what does the sliding window over a sentence produce?

Pick an answer to see if you are right.

🧪 Generating skip-gram training pairs from a toy sentence

---

Embeddings for Recommendation Systems

Because word2vec only cares about "sequences of tokens with neighbor relationships," it doesn't actually need those tokens to be words at all. Swap "song" in for "word" and "playlist" in for "sentence," and the exact same algorithm learns to embed songs based on which ones tend to show up together in human-curated playlists.

Using a dataset of playlists collected from hundreds of US radio stations (compiled by Shuo Chen at Cornell), here's what the resulting recommendations look like in practice. Asking for songs similar to Michael Jackson's "Billie Jean":

Title	Artist
Kiss	Prince & The Revolution
Wanna Be Startin' Somethin'	Michael Jackson
The Way You Make Me Feel	Michael Jackson
Holiday	Madonna
Don't Stop 'Til You Get Enough	Michael Jackson

That's a genuinely sensible neighborhood — other Michael Jackson tracks, plus artists from the same era and genre. Trying 2Pac's "California Love" instead surfaces a similarly on-target list of 90s rap and hip-hop tracks (Nas, Notorious B.I.G., Snoop Dogg, and others) — the embedding space organizes itself by genre and era without ever being told what "genre" or "era" mean.

Training the song embedding model

The training code is nearly identical to training word2vec on text — only the "sentences" are playlists and the "words" are song IDs:

import pandas as pd
from urllib import request

# Get the playlist dataset file
data = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/train.txt')

# Parse the playlist dataset file. Skip the first two lines as
# they only contain metadata
lines = data.read().decode("utf-8").split('\n')[2:]

# Remove playlists with only one song
playlists = [s.rstrip().split() for s in lines if len(s.split()) > 1]

# Load song metadata
songs_file = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/song_hash.txt')
songs_file = songs_file.read().decode("utf-8").split('\n')
songs = [s.rstrip().split('\t') for s in songs_file]

songs_df = pd.DataFrame(data=songs, columns=['id', 'title', 'artist'])
songs_df = songs_df.set_index('id')

Each entry in playlists is just a list of song IDs — exactly analogous to a tokenized sentence:

print('Playlist #1:\n', playlists[0], '\n')
print('Playlist #2:\n', playlists[1])

Playlist #1:
 ['0', '1', '2', '3', '4', '5', ..., '43']

Playlist #2:
 ['78', '79', '80', '3', '62', ..., '210']

Training reuses gensim's Word2Vec implementation directly — no song-specific code required:

from gensim.models import Word2Vec

# Train our Word2Vec model
model = Word2Vec(
    playlists, vector_size=32, window=20, negative=50,
    min_count=1, workers=4,
)

A minute or two later, every song has a trained embedding, queryable the same way as any word2vec vocabulary:

song_id = 2172
model.wv.most_similar(positive=str(song_id))

[('2976', 0.9977465271949768),
 ('3167', 0.9977430701255798),
 ('3094', 0.9975950717926025),
 ('2640', 0.9966474175453186),
 ('2849', 0.9963167905807495)]

Song 2172 turns out to be Metallica's "Fade To Black" — and its five nearest neighbors (Van Halen, Dio, Guns N' Roses, Judas Priest) are all comfortably within the same heavy metal / hard rock neighborhood, despite the model never being told what a "genre" is. It only ever learned from co-occurrence in playlists.

Check: embeddings for recommendations

Quiz

Embeddings for recommendation systems

1 / 2

What is the key insight that lets word2vec be reused for song recommendations instead of just words?

Pick an answer to see if you are right.

---

Chapter Summary

Key takeaways

1. Tokenizers are the first and last step of every LLM call. Text goes in as a string, comes out as input_ids; generated input_ids get decoded back into text on the way out.
2. Subword tokenization is the dominant scheme today, balancing the vocabulary-expressiveness problems of word-level tokenization against the context-length cost of character- or byte-level tokenization.
3. Three design choices shape any tokenizer: the algorithm (BPE, WordPiece, SentencePiece/unigram), the parameters (vocabulary size, special tokens, capitalization handling), and the training dataset's domain (general text vs. code vs. multilingual vs. scientific).
4. Comparing real tokenizers (BERT → GPT-2 → Flan-T5 → GPT-4 → StarCoder2 → Galactica → Phi-3) shows tokenizer design tracking a model's intended use directly — code models get cheap whitespace and per-digit numbers; chat models get role tokens; science models get citation tokens.
5. A model's embedding matrix gives every vocabulary token a starting vector; as tokens flow through the model's layers, they become contextualized embeddings that actually reflect surrounding meaning — the basis for tasks like named-entity recognition and summarization.
6. Text embedding models compress a whole sentence or document into one fixed-size vector, powering semantic search, clustering, and RAG.
7. word2vec predates modern contextualized embeddings but introduced two ideas — skip-gram (generating neighbor pairs via a sliding window) and negative sampling (contrasting real neighbor pairs against random ones) — that generalize far beyond text, including to non-language domains like song recommendation, where "playlist" stands in for "sentence" and "song" stands in for "word."

The next step is looking at what happens inside the model between tokenization and output — how transformer-based LLMs actually process a sequence of token embeddings to generate text, one token at a time.

Exercises

The first two exercises run on the real gensim library — actual Word2Vec training, not a mock — since training on a small in-memory corpus needs no GPU, no internet access, and no pretrained model download. The rest are pure-Python implementations of the underlying math.

Activity: Train a real Word2Vec model

Two toy "playlists" (lists of word-tokens) are provided, selected by an input line (0 or 1). Train a Word2Vec model on the given sentences with vector_size=16, window=2, min_count=1, workers=1, seed=42, and print the size of its resulting vocabulary.