Text Classification

Adapted from: Jay Alammar & Maarten Grootendorst, Hands-On Large Language Models (O'Reilly, 2024), Chapter 4 — rewritten and condensed for teaching, with code examples adapted from the original.

Text classification — assigning a label to a piece of text — is everywhere: sentiment analysis, intent detection, entity extraction, language identification. It's also a great lens for comparing how representation models (like BERT) and generative models (like GPT) actually differ in practice, since both can be pointed at the exact same task, just through very different mechanics.

This lesson works through the same running example — classifying movie review sentiment as positive or negative — using four progressively different techniques: a task-specific representation model, an embedding model paired with a lightweight classifier, zero-shot classification with no labeled data at all, and two generative models (one open-source, one closed-source API).

Always compare against a real baseline

Before reaching for an LLM, it's worth benchmarking against a classic, unglamorous baseline: TF-IDF features feeding a logistic regression classifier. It's fast, cheap, and a genuinely strong baseline — if an LLM-based approach can't clearly beat it, that's worth knowing.

---

The Dataset: Movie Review Sentiment

The examples in this lesson use the rotten_tomatoes dataset from the Hugging Face Hub — 5,331 positive and 5,331 negative short movie reviews, split into train/validation/test sets:

from datasets import load_dataset

# Load our data
data = load_dataset("rotten_tomatoes")
data

DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 8530
    })
    validation: Dataset({
        features: ['text', 'label'],
        num_rows: 1066
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 1066
    })
})

Every technique in this lesson trains (if it trains at all) on the train split and reports results on test; validation is there if you need it for hyperparameter tuning without peeking at test data. Each review is labeled 1 (positive) or 0 (negative) — this is binary sentiment classification.

Check: the classification setup

Quiz

Dataset and evaluation splits

1 / 2

Why does this workflow evaluate every technique on the test split rather than the train split?

Pick an answer to see if you are right.

---

Text Classification with Representation Models

Classifying with a pretrained representation model comes in two flavors:

1. A task-specific model — a representation model (like BERT) that's already been finetuned by someone else for exactly the task you need, e.g. sentiment analysis.
2. An embedding model — a representation model finetuned to produce general-purpose embeddings, which you then feed into your own lightweight classifier (or use directly for zero-shot classification, covered below).

Both start life as the same kind of foundation model; the difference is purely in what they were finetuned to output. In this lesson both stay frozen — no training of the big model happens here, only (in one case) a small classifier on top of it.

Model selection

With tens of thousands of classification models and thousands of embedding models available, picking one isn't trivial. A few things matter beyond raw benchmark performance: language compatibility, underlying architecture, model size, and — easy to underestimate — inference speed in production.

On architecture: encoder-only models like BERT remain a strong, efficient choice for task-specific work — they're typically far smaller than comparable generative models while still excelling at focused tasks. Several BERT variants have emerged over the years, each with different tradeoffs:

Model family	What it changed
BERT	The original encoder-only baseline
RoBERTa	A more robustly optimized BERT pretraining recipe
DistilBERT	A distilled, smaller/faster/cheaper version of BERT
ALBERT	Parameter-sharing tricks for a "lite" BERT
DeBERTa	Disentangled attention for improved representations

Solid starting points/baselines include the base BERT, RoBERTa, DistilBERT, and DeBERTa models, bert-tiny, and ALBERT base v2. For this lesson's task-specific example, we'll use Twitter-RoBERTa-base for Sentiment Analysis — a RoBERTa model finetuned on tweets, not movie reviews, which makes it a good test of how well a model generalizes outside its original training domain.

For selecting an embedding model, the MTEB leaderboard (Massive Text Embedding Benchmark) is the standard starting point — it benchmarks both open and closed embedding models across many tasks. This lesson uses sentence-transformers/all-mpnet-base-v2, a small but strong performer.

Check: choosing representation models

Quiz

Model selection for representation-based classification

1 / 2

Why might an encoder-only model like BERT be preferred over a generative model for a task-specific classification job?

Pick an answer to see if you are right.

---

Using a task-specific model

Loading the task-specific model is a one-liner with Hugging Face's pipeline:

from transformers import pipeline

# Path to our HF model
model_path = "cardiffnlp/twitter-roberta-base-sentiment-latest"

# Load model into pipeline
pipe = pipeline(
    model=model_path,
    tokenizer=model_path,
    return_all_scores=True,
    device="cuda:0",
)

Notice the tokenizer loads alongside the model (it would load automatically either way — passing it explicitly just makes visible what's happening under the hood, per Lesson 2). One of subword tokenization's real practical payoffs shows up here: even a word this tokenizer never saw during training can still get a reasonable representation, built by combining familiar subword fragments.

Running inference over the whole test split:

import numpy as np
from tqdm import tqdm
from transformers.pipelines.pt_utils import KeyDataset

# Run inference
y_pred = []
for output in tqdm(pipe(KeyDataset(data["test"], "text")), total=len(data["test"])):
    negative_score = output[0]["score"]
    positive_score = output[2]["score"]
    assignment = np.argmax([negative_score, positive_score])
    y_pred.append(assignment)

Reading a classification report

Evaluating predictions means comparing them against the true labels. A small helper keeps this consistent across every technique in this lesson:

from sklearn.metrics import classification_report

def evaluate_performance(y_true, y_pred):
    """Create and print the classification report"""
    performance = classification_report(
        y_true, y_pred,
        target_names=["Negative Review", "Positive Review"],
    )
    print(performance)

evaluate_performance(data["test"]["label"], y_pred)

To understand what comes out of that function, it helps to first understand the confusion matrix — the four ways a binary prediction can land, depending on whether it's correct (True/False) and which class it predicted (Positive/Negative):

	Predicted Positive	Predicted Negative
Actually Positive	True Positive (TP)	False Negative (FN)
Actually Negative	False Positive (FP)	True Negative (TN)

From those four counts, four standard metrics follow:

• Precision — of everything the model called positive, how much actually was? TP / (TP + FP). High precision means few false alarms.
• Recall — of everything that actually was positive, how much did the model catch? TP / (TP + FN). High recall means few misses.
• Accuracy — the fraction of all predictions (both classes) that were correct.
• F1 score — the harmonic mean of precision and recall, balancing both into one number.

Here's the actual classification report for the task-specific model:

	precision	recall	f1-score	support
Negative Review	0.76	0.88	0.81	533
Positive Review	0.86	0.72	0.78	533
accuracy			0.80	1066
macro avg	0.81	0.80	0.80	1066
weighted avg	0.81	0.80	0.80	1066

Throughout this lesson, "the F1 score" means the weighted average row's F1 column — 0.80 here — chosen so every class is weighted fairly regardless of how many examples it has. An 0.80 F1 from a model that was never trained on movie reviews at all is a genuinely strong result.

Check: confusion matrix and metrics

Quiz

Confusion matrices and evaluation metrics

1 / 2

A spam filter has high recall but low precision on the "spam" class. What does that mean in practice?

Pick an answer to see if you are right.

🧪 Computing precision, recall, and F1 from a confusion matrix

---

Classification with embeddings (supervised)

What if there's no ready-made task-specific model for your exact problem? Rather than finetuning a representation model yourself (expensive, and covered elsewhere), you can separate the problem into two cheaper steps:

1. Feature extraction — run text through a frozen, general-purpose embedding model to get a vector per document.
2. Classification — train a lightweight classifier (like logistic regression) directly on those vectors.

The embedding model never needs to be finetuned — all the trainable work happens in a small, fast classifier, which can even train on a CPU.

from sentence_transformers import SentenceTransformer

# Load model
model = SentenceTransformer("sentence-transformers/all-mpnet-base-v2")

# Convert text to embeddings
train_embeddings = model.encode(data["train"]["text"], show_progress_bar=True)
test_embeddings = model.encode(data["test"]["text"], show_progress_bar=True)

train_embeddings.shape
# (8530, 768)

8,530 training documents, each represented by a 768-dimensional vector — exactly the embedding shape you'd expect from this model (see Lesson 2). Training the classifier is now just ordinary supervised learning on fixed-length numeric features:

from sklearn.linear_model import LogisticRegression

# Train a logistic regression on our train embeddings
clf = LogisticRegression(random_state=42)
clf.fit(train_embeddings, data["train"]["label"])

# Predict previously unseen instances
y_pred = clf.predict(test_embeddings)
evaluate_performance(data["test"]["label"], y_pred)

	precision	recall	f1-score	support
Negative Review	0.85	0.86	0.85	533
Positive Review	0.86	0.85	0.85	533
accuracy			0.85	1066
macro avg	0.85	0.85	0.85	1066
weighted avg	0.85	0.85	0.85	1066

F1 climbs to 0.85 — better than the off-domain task-specific model, while only ever training a small logistic regression. If GPU access is a constraint, this same approach works with embeddings from an external API (Cohere and OpenAI both offer one) instead of running sentence-transformers locally, making the whole pipeline CPU-only.

Zero-shot classification: no labeled data at all

Labeled data is expensive to collect. Before investing in it, it's worth checking whether a task is even feasible without any labels — using zero-shot classification. You still need to know what your labels mean, just not any examples of them.

The trick: describe each label in plain language — "This is a negative movie review" — and embed that description exactly like you'd embed a document. Now both documents and labels live in the same embedding space, and you can measure how close each document sits to each label description using cosine similarity (the cosine of the angle between two vectors — closer to 1 means more similar, regardless of vector magnitude).

# Create embeddings for our labels
label_embeddings = model.encode(["A negative review", "A positive review"])

from sklearn.metrics.pairwise import cosine_similarity

# Find the best matching label for each document
sim_matrix = cosine_similarity(test_embeddings, label_embeddings)
y_pred = np.argmax(sim_matrix, axis=1)

evaluate_performance(data["test"]["label"], y_pred)

	precision	recall	f1-score	support
Negative Review	0.78	0.77	0.78	533
Positive Review	0.77	0.79	0.78	533
accuracy			0.78	1066
macro avg	0.78	0.78	0.78	1066
weighted avg	0.78	0.78	0.78	1066

An F1 of 0.78 with zero labeled training examples — using nothing but two short label descriptions — is a genuinely strong result, and a good demonstration of just how much general-purpose utility lives inside a good embedding model.

Label wording is a lever you can pull

"A negative review" versus "A very negative movie review" will produce different embeddings, and often different results — the more specific description ties the label more tightly to the actual domain (movies) and pushes toward the extremes the two classes represent. It's worth experimenting with label phrasing the same way you'd experiment with a prompt.

Check: embeddings for supervised and zero-shot classification

Quiz

Embedding-based classification approaches

1 / 3

What is the key benefit of the two-step "embed, then classify" approach over finetuning a whole representation model?

Pick an answer to see if you are right.

🧪 Zero-shot classification by cosine similarity, with toy embeddings

---

Text Classification with Generative Models

Everything so far has used a representation model that outputs a numeric class score directly. Generative models — GPT-family models included — work completely differently: they take in a sequence of tokens and produce another sequence of tokens (sequence-to-sequence), not a class label. Handed a raw movie review with no other context, a generative model has no built-in notion of "classify this" at all.

To get useful classification behavior out of a generative model, you have to tell it what to do — via the instruction or prompt you give it. Iterating on that prompt to reliably get the output format and behavior you want is called prompt engineering.

Using Flan-T5 (encoder-decoder)

Most of this book's examples split neatly into encoder-only (BERT-style) or decoder-only (GPT-style) models. But the original transformer architecture was actually encoder-decoder — and models in that lineage, like T5 (Text-to-Text Transfer Transformer), are also generative, sequence-to-sequence models.

T5 pretrains with a twist on masked language modeling: instead of masking single tokens, it masks whole spans of consecutive tokens and trains the model to reconstruct them. The bigger idea, though, is in how T5 gets finetuned: every downstream task — translation, summarization, classification, whatever — gets reframed as a text-in, text-out problem and trained together, all as one unified format. Flan-T5 pushes this further, finetuning across well over a thousand tasks phrased as natural-language instructions, much closer to how modern instruction-following chat models are trained.

Loading it for classification uses the text2text-generation pipeline (the standard task for encoder-decoder models):

# Load our model
pipe = pipeline(
    "text2text-generation",
    model="google/flan-t5-small",
    device="cuda:0",
)

Since this model has no built-in notion of "sentiment classification," every input needs an explicit instruction prefix:

# Prepare our data
prompt = "Is the following sentence positive or negative? "
data = data.map(lambda example: {"t5": prompt + example['text']})

# Run inference
y_pred = []
for output in tqdm(pipe(KeyDataset(data["test"], "t5")), total=len(data["test"])):
    text = output[0]["generated_text"]
    y_pred.append(0 if text == "negative" else 1)

Because the output is generated text, not a class index, it has to be mapped back into 0/1 by hand before evaluation:

evaluate_performance(data["test"]["label"], y_pred)

	precision	recall	f1-score	support
Negative Review	0.83	0.85	0.84	533
Positive Review	0.85	0.83	0.84	533
accuracy			0.84	1066
macro avg	0.84	0.84	0.84	1066
weighted avg	0.84	0.84	0.84	1066

An F1 of 0.84, from the smallest Flan-T5 variant, with a one-line instruction — a solid first look at what generative models bring to classification.

Using ChatGPT (closed-source API)

The training story behind ChatGPT connects directly back to Lesson 1's post-training coverage: OpenAI first collected instruction data (human-written examples of good responses) for initial finetuning, then collected preference data — multiple candidate outputs, manually ranked best to worst — used to further align the model toward outputs people actually prefer. Preference data captures something instruction data alone can't: the nuance of what makes one good answer better than another good answer, not just "good vs. bad."

Using a closed-source model means calling an API instead of loading weights locally:

import openai

# Create client
client = openai.OpenAI(api_key="YOUR_KEY_HERE")

def chatgpt_generation(prompt, document, model="gpt-3.5-turbo-0125"):
    """Generate an output based on a prompt and an input document."""
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": prompt.replace("[DOCUMENT]", document)},
    ]
    chat_completion = client.chat.completions.create(
        messages=messages,
        model=model,
        temperature=0,
    )
    return chat_completion.choices[0].message.content

# Define a prompt template as a base
prompt = """Predict whether the following document is a positive or negative movie review:

[DOCUMENT]

If it is positive return 1 and if it is negative return 0. Do not give any other answers.
"""

document = "unpretentious, charming, quirky, original"
chatgpt_generation(prompt, document)

Running this across the full test set and converting the string outputs to integers, exactly as with Flan-T5, gives an F1 score of 0.91 — clearly the strongest result of any technique in this lesson.

Two real caveats with closed-source API evaluation

1. Cost and rate limits. External APIs charge per request and cap how many requests you can send per minute — running an evaluation like this cost a few cents, but that adds up fast at scale. Hitting a rate limit mid-run is normal; the standard fix is exponential backoff — retry after a short sleep, and double that sleep each time you get rate-limited again, up to some maximum number of attempts.
2. Unknown training data. Since OpenAI doesn't disclose exactly what ChatGPT was trained on, there's no way to rule out this exact benchmark dataset having leaked into training somewhere — which would inflate the apparent score. This is a genuine limitation of benchmarking closed models, not just this dataset.

Check: generative models for classification

Quiz

Generative models for classification

1 / 3

Why does classifying with a generative model require a carefully written prompt, when a task-specific representation model does not?

Pick an answer to see if you are right.

🧪 Exponential backoff, simulated

---

Chapter Summary

Key takeaways

1. Task-specific representation models (a finetuned BERT-family model) can be used directly for classification, and even generalize reasonably well outside their original training domain.
2. Embedding + lightweight classifier separates expensive feature extraction (a frozen embedding model) from cheap training (e.g. logistic regression), often outperforming an off-domain task-specific model while training almost nothing.
3. Zero-shot classification with embeddings needs no labeled data at all — just a plain-language description per label, compared to document embeddings via cosine similarity — and can still reach surprisingly strong results.
4. Generative models (Flan-T5, ChatGPT) treat classification as a text-generation problem, guided entirely by the prompt — requiring the output to be parsed back into class labels, but capable of strong performance with no task-specific finetuning at all.
5. Evaluation always comes back to precision, recall, and F1, derived from the confusion matrix — with the weighted-average F1 used throughout as the single headline number.
6. Closed-source API-based classification can perform very well, but comes with real practical costs (usage fees, rate limits handled via exponential backoff) and evaluation caveats (undisclosed training data makes benchmark comparisons less trustworthy).

Every technique in this lesson assumed labels were meaningful and available (or at least describable). The next step is the harder, more open-ended case: text data with no labels at all — where clustering and topic modeling take over to find structure nobody defined in advance.

Exercises

These exercises use the real scikit-learn library — actual LogisticRegression, cosine_similarity, and the evaluation metrics — running on tiny, hand-picked embeddings instead of the full rotten_tomatoes dataset (which would need downloading real pretrained embedding models and isn't something the sandbox running these exercises can do). The classifier and the metrics are 100% real; only the input data is small enough to type out.

Activity: Train a real logistic regression classifier

A case number (0 or 1) selects a toy training set of 2D embeddings and labels, plus a single query embedding. Train sklearn.linear_model.LogisticRegression on the training data and print its predicted label for the query point.