Multimodal Large Language Models
Adapted from: Jay Alammar & Maarten Grootendorst, Hands-On Large Language Models (O'Reilly, 2024), Chapter 9 — rewritten and condensed for teaching, with code examples adapted from the original.
Every model in the previous lessons has lived entirely inside one modality: text in, text out. But language rarely lives in a vacuum — tone of voice, facial expression, and (for a machine) an accompanying image all carry meaning a pure text model can't touch. A model able to work across more than one type of data — text and images, in this lesson's case — is called multimodal. Note that "multimodal" doesn't require symmetry: a model can be built to accept an image without being able to generate one, or vice versa.
This lesson builds multimodality up in two stages: first, how images get converted into the same kind of numerical representation a Transformer already knows how to consume; second, how that representation gets wired into embedding models (CLIP) and text-generation models (BLIP-2) to produce genuinely multimodal systems.
Transformers for Vision
Transformers built their reputation on text — classification, clustering, search, generation. The Vision Transformer (ViT) asks a direct question: can the same architecture that works so well on text work on images too? It turns out yes, and ViT now outperforms the convolutional neural networks (CNNs) that used to be the default choice for image recognition.
From tokens to patches
A text Transformer's encoder can't do anything until the input is tokenized — chopped into discrete, ID-able units. Images have no equivalent vocabulary: a raw pixel carries almost no information on its own, and unlike a word, a specific patch of pixels will essentially never recur identically in another image, so there's no fixed list of "pixel-patch IDs" to assign.
ViT's answer is to tokenize images anyway — just not into words. It slices the image into a grid of fixed-size square patches (the original paper uses 16×16 patches — hence its title, "An Image Is Worth 16x16 Words"), flattens each patch, and passes it through a linear projection to produce an embedding. From that point on, each patch embedding is treated exactly like a token embedding: fed into the same encoder architecture, with no special handling for the fact that it originated from a picture rather than a word.
A single pixel is nearly meaningless on its own — it's only when pixels are grouped that recognizable structure (an edge, a texture, a shape) emerges. Patches are the smallest unit large enough to carry that structure, which is why ViT groups pixels into patches rather than trying to tokenize pixel-by-pixel.
This is the detail that matters most for everything else in this lesson: once patch embeddings enter the encoder, there is no architectural difference between processing an image and processing text. That shared representation is exactly what makes it possible to build models that reason about images and text side by side — starting with embedding models.
Check: Vision Transformers
Vision Transformers and image patching
1 / 2Why can't ViT tokenize an image the same way a text Transformer tokenizes a sentence?
Pick an answer to see if you are right.
🧪 Simulating image patchification
Multimodal Embedding Models
Text embedding models compress a document's meaning into a vector, useful for search, classification, and clustering, as earlier lessons covered. Multimodal embedding models extend that same idea across modalities: text and images are embedded into the same vector space, so a caption and the image it describes end up sitting close together — and, crucially, so does a caption compared against images it doesn't describe, just farther away.
That shared space is what unlocks cross-modal comparisons: searching for images using a text query ("pictures of a puppy"), or the reverse — finding which document best matches a given image.
The most widely used model for this is CLIP (Contrastive Language-Image Pre-training).
CLIP: connecting text and images
CLIP produces embeddings for both images and text in one shared space, which makes several tasks possible:
| Use case | How CLIP enables it |
|---|---|
| Zero-shot classification | Compare an image's embedding against the embeddings of candidate class descriptions — whichever is closest wins, with no task-specific training |
| Clustering | Cluster images and keywords together to see which keywords describe which image groups |
| Search | Find the images (or text) most similar to a text (or image) query, across huge collections |
| Generation | Multimodal embeddings can drive image-generation systems like Stable Diffusion |
How CLIP learns a shared space
CLIP starts from a large dataset of image/caption pairs. Training proceeds in three repeated steps:
- Embed both modalities. A text encoder embeds each caption; an image encoder embeds each image.
- Score similarity. Cosine similarity is computed between every image embedding and every text embedding in the batch — the same cosine similarity used for text-only embeddings in earlier lessons.
- Update toward the target. The encoders are updated to push matched image/caption pairs' embeddings closer together, and push mismatched pairs' embeddings apart.
This training strategy is called contrastive learning — the model doesn't just learn what makes things similar, it learns what makes them different, since every batch contains both correct pairs and (implicitly) incorrect ones to contrast against. Early in training, a cat photo and the caption "a picture of a cat" will have essentially random similarity; as training proceeds, that similarity climbs while the same photo's similarity to unrelated captions falls.
A model only shown matching pairs could theoretically embed everything into the same point in space and appear to "succeed" — matched pairs would trivially be similar. Contrastive learning avoids this by also penalizing high similarity between mismatched pairs within the same batch, forcing the model to spread dissimilar things apart, not just pull similar things together.
Check: CLIP concepts
CLIP and multimodal embeddings
1 / 3What does it mean for CLIP's image and text embeddings to occupy "the same vector space"?
Pick an answer to see if you are right.
🧪 Simulating contrastive learning's similarity shift
OpenCLIP in Practice
OpenCLIP is an open-source implementation of CLIP. Using it comes down to two preprocessing steps — one for text, one for images — before both feed into the same model.
from urllib.request import urlopen
from PIL import Image
# Load an AI-generated image of a puppy playing in the snow
puppy_path = "https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/puppy.png"
image = Image.open(urlopen(puppy_path)).convert("RGB")
caption = "a puppy playing in the snow"
Loading CLIP means loading three components: a text tokenizer, an image processor, and the model that turns both into embeddings.
from transformers import CLIPTokenizerFast, CLIPProcessor, CLIPModel
model_id = "openai/clip-vit-base-patch32"
# Load a tokenizer to preprocess the text
clip_tokenizer = CLIPTokenizerFast.from_pretrained(model_id)
# Load a processor to preprocess the images
clip_processor = CLIPProcessor.from_pretrained(model_id)
# Main model for generating text and image embeddings
model = CLIPModel.from_pretrained(model_id)
Tokenizing and embedding the caption
# Tokenize our input
inputs = clip_tokenizer(caption, return_tensors="pt")
inputs
{'input_ids': tensor([[49406, 320, 6829, 1629, 530, 518, 2583, 49407]]),
'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]])}
# Convert our input back to tokens
clip_tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
['<|startoftext|>', 'a</w>', 'puppy</w>', 'playing</w>', 'in</w>', 'the</w>', 'snow</w>', '<|endoftext|>']
Note the explicit start/end markers — CLIP needs a clear boundary around the text so it isn't confused with an accompanying image embedding. (There's no [CLS] token here, because in CLIP that role is effectively played by the image embedding itself.)
# Create a text embedding
text_embedding = model.get_text_features(**inputs)
text_embedding.shape
torch.Size([1, 512])
Preprocessing and embedding the image
Images need reshaping before the model can use them — CLIP expects a fixed input size regardless of the original image's dimensions:
# Preprocess image
processed_image = clip_processor(
text=None, images=image, return_tensors="pt"
)["pixel_values"]
processed_image.shape
torch.Size([1, 3, 224, 224])
The original 512×512 image is resized down to 224×224 — worth keeping in mind if you're feeding in unusually wide or tall images, since they'll be squeezed into a square and can distort.
# Create the image embedding
image_embedding = model.get_image_features(processed_image)
image_embedding.shape
torch.Size([1, 512])
The image embedding has the exact same shape as the text embedding — 512 values — which is precisely what makes comparing them meaningful.
Comparing the embeddings
# Normalize the embeddings
text_embedding /= text_embedding.norm(dim=-1, keepdim=True)
image_embedding /= image_embedding.norm(dim=-1, keepdim=True)
# Calculate their similarity
text_embedding = text_embedding.detach().cpu().numpy()
image_embedding = image_embedding.detach().cpu().numpy()
score = np.dot(text_embedding, image_embedding.T)
score
array([[0.33149648]], dtype=float32)
A similarity score of 0.33 is meaningless in isolation — there's no fixed "high" or "low" without a comparison point. Computing the same score across a small grid of unrelated images and captions shows this pair scoring clearly higher than any mismatched pair, confirming the model correctly recognized the caption as matching this image and not the others.
A shortcut with sentence-transformers
sentence-transformers wraps CLIP-family models behind the same simple .encode() interface used for text-only embedding models in earlier lessons:
from sentence_transformers import SentenceTransformer, util
# Load SBERT-compatible CLIP model
model = SentenceTransformer("clip-ViT-B-32")
# Encode the images
image_embeddings = model.encode(images)
# Encode the captions
text_embeddings = model.encode(captions)
# Compute cosine similarities
sim_matrix = util.cos_sim(image_embeddings, text_embeddings)
Check: using OpenCLIP
OpenCLIP in practice
1 / 2Why is it significant that the text and image embeddings both come out as shape [1, 512]?
Pick an answer to see if you are right.
Making Text Generation Models Multimodal
Embedding models solve half the problem — comparing across modalities. But models like Llama 2 or ChatGPT are trained purely on text: they can't natively look at an image and describe it, answer a question about it, or reason about its contents at all.
Building a text-generation model that understands images from scratch would need billions of image/text/image-caption examples and enormous compute — not a realistic starting point. BLIP-2 (Bootstrapping Language-Image Pre-training) takes a far cheaper path: instead of training a new multimodal model from the ground up, it builds a small bridge between a frozen, pretrained vision encoder and a frozen, pretrained LLM — training only the bridge itself.
The Querying Transformer (Q-Former)
That bridge is called the Querying Transformer (Q-Former). It has two internal modules that share attention layers:
- An Image Transformer, which interacts with the frozen ViT to extract visual features
- A Text Transformer, which interacts with the frozen LLM
Because the two large models (the ViT and the LLM) stay frozen throughout, BLIP-2 only has to train this comparatively small connecting piece — reusing all the capability already baked into two pretrained models rather than re-deriving it.
Training in two stages
Stage 1 trains the Q-Former to represent vision and language in a way that's mutually comparable, using image/caption pairs — much like CLIP's training data. The frozen ViT extracts image features, which feed the Q-Former's Image Transformer; the caption feeds the Q-Former's Text Transformer. Three objectives are optimized jointly:
| Objective | What it trains the model to do |
|---|---|
| Image-text contrastive learning | Pull matched image/caption embeddings together, push mismatched ones apart (the same idea as CLIP's training) |
| Image-text matching | Classify whether a given image/text pair is a genuine match or not |
| Image-grounded text generation | Generate text conditioned on information extracted from the image |
Stage 2 takes the representations learned in stage 1 and turns them into something the LLM can actually consume: a soft visual prompt. The Q-Former's learned embeddings pass through a linear projection layer so their shape matches what the LLM expects, and the result is fed into the LLM exactly the way a normal text prompt's embeddings would be. From the LLM's perspective, it isn't "seeing an image" in any special sense — it's just receiving prompt-shaped input that happens to encode visual information.
BLIP-2 doesn't teach the LLM to see. It teaches a small bridge to translate "what the image contains" into the same kind of embedding the LLM already knows how to read as a prompt.
Since BLIP-2, other visual LLMs — LLaVA, Idefics 2, and others — have taken the same basic approach: connect a pretrained CLIP-like vision encoder to a pretrained LLM, and train only the connecting projection so visual features land in the LLM's embedding space.
Check: BLIP-2 and the Q-Former
BLIP-2 and the Querying Transformer
1 / 2What is the key efficiency insight behind BLIP-2's design?
Pick an answer to see if you are right.
Preprocessing Multimodal Inputs
Loading BLIP-2 looks like loading any other Hugging Face model, plus a processor that understands both images and text:
from transformers import AutoProcessor, Blip2ForConditionalGeneration
import torch
# Load processor and main model
blip_processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained(
"Salesforce/blip2-opt-2.7b",
torch_dtype=torch.float16
)
# Send the model to GPU to speed up inference
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.vision_model and model.language_model expose the ViT and the LLM being bridged, respectively — a direct look at the two frozen components the Q-Former connects.
Preprocessing images
# Load image of a supercar
car_path = "https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-Language-Models/main/chapter09/images/car.png"
image = Image.open(urlopen(car_path)).convert("RGB")
# Preprocess the image
inputs = blip_processor(image, return_tensors="pt").to(device, torch.float16)
inputs["pixel_values"].shape
torch.Size([1, 3, 224, 224])
Same lesson as with CLIP: whatever the original image's aspect ratio, it gets squeezed into a fixed square — worth watching for with unusually wide or tall source images.
Preprocessing text
BLIP-2 tokenizes text with a GPT2Tokenizer:
# Preprocess the text
text = "Her vocalization was remarkably melodic"
token_ids = blip_processor(image, text=text, return_tensors="pt")
token_ids = token_ids.to(device, torch.float16)["input_ids"][0]
# Convert input ids back to tokens
tokens = blip_processor.tokenizer.convert_ids_to_tokens(token_ids)
tokens
['</s>', 'Her', 'Ġvocal', 'ization', 'Ġwas', 'Ġremarkably', 'Ġmel', 'odic']
The Ġ characters are an artifact of how GPT-2's tokenizer represents whitespace internally — swap them for underscores for a clearer read:
# Replace the space token with an underscore
tokens = [token.replace("Ġ", "_") for token in tokens]
tokens
['</s>', 'Her', '_vocal', 'ization', '_was', '_remarkably', '_mel', 'odic']
The underscore marks the start of a new word, which is what lets a downstream reader tell that vocal + ization are two token-pieces of the same word, while _was starts a fresh one.
Use Case 1: Image Captioning
The most direct application: convert an image into pixel values, let BLIP-2 turn that into a soft visual prompt, and let the language model generate a caption from it.
# Load an AI-generated image of a supercar
image = Image.open(urlopen(car_path)).convert("RGB")
# Convert an image into inputs and preprocess it
inputs = blip_processor(image, return_tensors="pt").to(device, torch.float16)
# Generate image ids to be passed to the decoder (LLM)
generated_ids = model.generate(**inputs, max_new_tokens=20)
# Generate text from the image ids
generated_text = blip_processor.batch_decode(
generated_ids, skip_special_tokens=True
)
generated_text = generated_text[0].strip()
generated_text
an orange supercar driving on the road at sunset
A solid caption. It's also worth testing the limits of a general-purpose model like this: it was trained on broadly public data, so anything highly domain-specific — a particular fictional character, a niche visual style — is more likely to trip it up.
For a more open-ended test, feeding it an inkblot from the classic Rorschach test produces:
a black and white ink drawing of a bat
A perfectly reasonable caption for the shape — and a good reminder that "captioning" and "correctly interpreting intent" aren't the same task; the model is describing pixels, not performing psychoanalysis.
Use Case 2: Multimodal Chat-Based Prompting
Captioning is a one-way trip: image in, caption out. Visual question answering goes further — the model receives both an image and a question about it simultaneously, and has to reason about both at once.
# Load an AI-generated image of a supercar
image = Image.open(urlopen(car_path)).convert("RGB")
# Visual question answering
prompt = "Question: Write down what you see in this picture. Answer:"
# Process both the image and the prompt
inputs = blip_processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)
# Generate text
generated_ids = model.generate(**inputs, max_new_tokens=30)
generated_text = blip_processor.batch_decode(
generated_ids, skip_special_tokens=True
)
generated_text = generated_text[0].strip()
generated_text
A sports car driving on the road at sunset
That's essentially a captioning task in disguise. The more interesting extension is chat-like follow-up: feed the previous question and the model's own answer back in as context, then ask something new.
# Chat-like prompting
prompt = "Question: Write down what you see in this picture. Answer: A sports car driving on the road at sunset. Question: What would it cost me to drive that car? Answer:"
# Generate output
inputs = blip_processor(image, text=prompt, return_tensors="pt").to(device, torch.float16)
generated_ids = model.generate(**inputs, max_new_tokens=30)
generated_text = blip_processor.batch_decode(
generated_ids, skip_special_tokens=True
)
generated_text = generated_text[0].strip()
generated_text
$1,000,000
Oddly specific, but it demonstrates the pattern: manually accumulating prior questions and answers into the prompt (the same conversation-buffer idea from the memory lesson, just applied to a vision-language model) is enough to produce chat-like behavior, even though BLIP-2 has no dedicated conversational training. The same pattern generalizes into a full interactive chatbot — an input widget appends each new question to a running memory list, rebuilds the prompt from that history, and displays the running back-and-forth — the image stays fixed throughout, but the conversation about it can keep evolving.
Check: multimodal use cases
Image captioning and visual question answering
1 / 2What is the essential difference between image captioning and visual question answering, as used with BLIP-2?
Pick an answer to see if you are right.
Chapter Summary
- Vision Transformers (ViT) make images consumable by a standard Transformer encoder by splitting them into fixed-size patches, linearly embedding each patch, and feeding those embeddings in exactly the way token embeddings would be fed for text — no architectural difference from that point onward.
- CLIP trains a shared vector space for text and images using contrastive learning: pulling matched image/caption embeddings together and pushing mismatched pairs apart, which enables zero-shot classification, cross-modal search, clustering, and generation.
- Comparing multimodal embeddings works exactly like comparing text-only embeddings — cosine similarity — but the resulting scores are only meaningful relative to other pairs, not in isolation.
- BLIP-2 avoids training a multimodal model from scratch by freezing a pretrained ViT and a pretrained LLM, and training only a small connecting Q-Former to translate visual features into a "soft visual prompt" the LLM can read like ordinary text input.
- The Q-Former is trained on three joint objectives — image-text contrastive learning, image-text matching, and image-grounded text generation — that together shape visual representations the frozen LLM can actually use.
- Multimodal text generation supports both image captioning (image in, description out) and visual question answering (image plus a specific question), and manually re-supplying prior turns as context gives it chat-like behavior despite having no built-in conversational memory.
Everything up to this point has used models exactly as they were released — prompted cleverly, chained together, made multimodal, but never actually retrained. The next part of this course turns to training and fine-tuning language models directly, starting with how to build and fine-tune a text embedding model from the ground up.