Memory Architectures

Adapted from: Jason Gregory, Game Engine Architecture, 3rd ed. (CRC Press, 2018), Chapter 3, Section 3.5 — rewritten and condensed for teaching.

The simple mental model of memory — "one giant flat array of bytes, all equally fast to touch" — is a convenient fiction. Real hardware carves memory into regions with wildly different purposes and performance characteristics: some addresses are actually joypad registers, some pages might not exist in physical RAM at all until you touch them, and the RAM you do have gets slower and slower the further it sits from the CPU core. Understanding why memory is architected this way — and how the engine can work with it rather than against it — is one of the highest-leverage things a performance-minded engine programmer can know, because it explains why a "correct" piece of code can run 50× slower than a differently-organized version doing the exact same work.

---

Memory Mapping

An n-bit address bus gives the CPU a theoretical address space of 2ⁿ bytes. Any single physical memory chip is always addressed as one contiguous block, so a computer's full address space is carved up into contiguous segments, each assigned — or mapped — to a particular physical device. On the Apple II, for instance, addresses 0xC100–0xFFFF were wired to ROM chips holding the firmware, while 0x0000–0xBFFF went to RAM.

Modern address buses are enormous relative to installed memory — a 64-bit bus can theoretically address 16 EiB, far beyond anything actually populated — so large swaths of the address space are simply left unmapped.

Memory-Mapped I/O

Not every address range needs to point at memory at all. Memory-mapped I/O maps a range of addresses to a peripheral device — a joypad, a network card — instead of RAM. Special circuitry watches for reads/writes in that range and redirects them to the device, so from the CPU's point of view, talking to hardware looks exactly like reading or writing ordinary memory. The Apple II mapped 0xC000–0xC0FF this way, letting a program control bank-switched RAM or read a game-controller pin just by touching an address.

The alternative is port-mapped I/O, where the CPU talks to devices through dedicated port registers instead of regular addresses — the approach Arduino microcontrollers use to expose direct control over their digital pins.

Video RAM

A display device reads a fixed range of memory to determine what to draw — pixel colors for raster displays, ASCII glyphs for old character-based terminals. That range is video RAM (VRAM).

On early machines (Apple II, early IBM PCs) and on consoles with unified memory (PS4, Xbox One), VRAM is just regular memory the CPU and GPU both touch directly. On a typical PC, though, the GPU lives on a separate card with its own dedicated VRAM, reachable only over an expansion bus (PCI, AGP, PCIe). That physical separation — and the cost of shuttling data across the bus — is one of the biggest reasons rendering engines and graphics APIs are as complex as they are.

Case Study: The Apple II Memory Map

The Apple II's 16-bit address bus gave it a 64 KiB address space, split across ROM, RAM, memory-mapped I/O, and two flavors of VRAM:

0xC100 - 0xFFFF  ROM (Firmware)
0xC000 - 0xC0FF  Memory-Mapped I/O
0x6000 - 0xBFFF  General-purpose RAM
0x4000 - 0x5FFF  High-res video RAM (page 2)
0x2000 - 0x3FFF  High-res video RAM (page 1)
0x0C00 - 0x1FFF  General-purpose RAM
0x0800 - 0x0BFF  Text/lo-res video RAM (page 2)
0x0400 - 0x07FF  Text/lo-res video RAM (page 1)
0x0200 - 0x03FF  General-purpose and reserved RAM
0x0100 - 0x01FF  Program stack
0x0000 - 0x00FF  Zero page (mostly reserved for DOS)

Every one of these addresses corresponded directly to a physical chip on the motherboard. Today's operating systems add a layer of indirection on top of this — which is exactly what virtual memory is for.

Check: memory mapping and I/O

Quiz

Memory mapping, MMIO, and video RAM

1 / 3

What does it mean for an address range to be "mapped" to a device?

Pick an answer to see if you are right.

---

Virtual Memory

Modern CPUs and OSes remap every address a program uses through a lookup table before it ever reaches a physical chip. The address the program works with is a virtual address; the bit pattern that actually goes out over the address bus is a physical address. A virtual address might resolve to a real RAM cell, to a block of data sitting on disk, or to nothing at all.

This indirection buys two big things: programs can address more memory than is physically installed (the overflow lives on disk), and each program gets its own private view of memory, unable to stomp on another process's — or the OS's — memory.

Virtual Memory Pages

The whole addressable space is conceptually divided into equal, power-of-two-sized chunks called pages — commonly 4 KiB or 8 KiB. A 32-bit address space with a 4 KiB page size splits into 1,048,576 pages, numbered 0x0 through 0xFFFFF:

From address	To address	Page index
0x00000000	0x00000FFF	Page 0x0
0x00001000	0x00001FFF	Page 0x1
0x00002000	0x00002FFF	Page 0x2
…	…	…
0xFFFFF000	0xFFFFFFFF	Page 0xFFFFF

Virtual-to-physical remapping always happens at page granularity — you can't remap half a page.

Virtual-to-Physical Address Translation

When the CPU touches memory, it splits the virtual address into a page index and a byte offset within that page. For a 4 KiB page, the offset is just the low 12 bits, and the page index is everything above that. For example, 0x1A7C6310 splits into offset 0x310 and page index 0x1A7C6.

The memory management unit (MMU) looks that page index up in a page table (maintained by the OS, stored in RAM) mapping virtual page indices to physical ones. If page 0x1A7C6 happens to map to physical page 0x73BB9, the MMU shifts that physical index into place and ORs it with the original offset — producing the final physical address 0x73BB9310, which is what actually goes out over the bus.

Activity: split a virtual address by hand

Assuming an 8 KiB page size (13-bit offset), split the virtual address 0x2000A1F4 into its page index and byte offset. (Answer: offset = low 13 bits = 0x01F4; page index = the rest = 0x10005.)

Handling Page Faults

If the page table says a page isn't backed by physical RAM — never allocated, or previously swapped to disk — the MMU raises an interrupt called a page fault, and the OS steps in:

• Never-allocated page: the OS typically kills the program and produces a core dump.
• Swapped-out page: the OS suspends the program, reads the page back from the swap file into physical RAM, redoes the address translation, and resumes execution — invisibly to the program, which never knows whether a page was resident or had to be pulled in from disk.

The OS tries to swap out only the least-recently-used pages, to avoid a program thrashing — repeatedly paging the same data in and out because too little physical RAM is available for its working set.

The Translation Lookaside Buffer (TLB)

Because pages are small relative to total addressable memory, a full page table can be huge — scanning it on every single memory access would be far too slow. The translation lookaside buffer (TLB) is a small, fast cache living inside the MMU (right on the CPU die) that holds the virtual-to-physical mappings for the most recently used pages. It works exactly like the memory cache hierarchies discussed below, just specialized for caching page-table entries instead of general data.

Key insight

Virtual memory is itself built on the same idea as everything else in this lesson: put a small, fast structure (the TLB) in front of a large, slow one (the full page table) and exploit the fact that real programs reuse a small set of pages over and over.

Check: virtual memory, paging, and the TLB

Quiz

Pages, the MMU, page faults, and the TLB

1 / 3

What does the memory management unit (MMU) actually do on every memory access?

Pick an answer to see if you are right.

---

Memory Architectures for Latency Reduction

Memory access latency is the time between the CPU requesting data and actually receiving it. Three things drive it:

1. the underlying cell technology — SRAM is much lower-latency than DRAM, but needs more transistors per bit, so it costs more (in dollars and in die area);
2. the number of read/write ports — a single-ported cell only permits one access at a time, while multi-ported RAM allows simultaneous accesses at the cost of more transistors per bit; and
3. the physical distance between the memory cell and the CPU core, since signals — even traveling near light-speed — still take measurable time, and every switching circuit along the way adds more.

The Memory Gap

In the earliest computers, executing an instruction and fetching from main memory took roughly comparable time — on the Intel 8086, both were around 2–4 cycles. Since then, CPU throughput has improved far faster than memory latency: a modern Core i7 can execute a register instruction in 1–10 cycles, but a main-RAM access can take around 500 cycles. This widening gulf is called the memory gap, and it's the single biggest reason performance-sensitive code spends so much effort on data layout.

Three strategies address it:

1. reduce average latency by putting small, fast memory closer to the core (register files, caches — covered below);
2. hide latency by giving the CPU other useful work to do while a memory request is in flight (out-of-order execution, prefetching, multithreading); or
3. minimize accesses by arranging a program's data so the CPU touches main memory as rarely as possible for the work it's doing.

This lesson focuses on strategy 1; strategies 2 and 3 come up again once we get to CPU pipelines and data-oriented design.

Register Files

A CPU's register file is the most extreme latency-reduction design there is: multi-ported SRAM, sitting immediately next to the ALU that uses it, with dedicated read and write ports so operations can happen in parallel. Registers skip virtual address translation, the cache hierarchy, cache coherence, and the address/data buses entirely — the ALU reaches them almost directly. That's why they're so fast, and also why register RAM is disproportionately expensive relative to general-purpose RAM: it's justified because registers are the single most frequently accessed memory in the whole machine, and there are only ever a handful of them.

Check: the memory gap and register files

Quiz

Memory access latency and the memory gap

1 / 3

Which three factors primarily determine a memory device's access latency?

Pick an answer to see if you are right.

---

Memory Cache Hierarchies

A cache hierarchy is the primary tool for taming the memory gap in real hardware. A small, very fast level 1 (L1) cache sits right on the CPU die, with latency close to the register file. A larger, slower L2 cache sits a bit further out (often shared across cores), and some systems add L3 or even L4. These caches automatically keep copies of frequently-touched data so accesses to slow main RAM stay rare.

• Cache hit — the requested data is already in the cache; served in tens of cycles.
• Cache miss — it isn't; the CPU has to reach into main RAM, at a cost of hundreds of cycles.

Cache Lines

Caching exploits two patterns real programs exhibit, known together as locality of reference:

1. Spatial locality — touching address N makes it likely you'll soon touch N+1, N+2, etc. (iterating an array is the textbook example).
2. Temporal locality — touching address N makes it likely you'll touch that same address again soon (read-modify-write on a variable).

To exploit this, caches move data in contiguous cache lines, not individual bytes. Reading one member of a struct triggers a fetch of a whole surrounding block — so if the struct's other members happen to live in that same block, they're now cheap to read too, purely as a side effect.

Mapping Cache Lines to Main RAM Addresses

The cache's address space maps onto main RAM's in a repeating pattern. Worked example: a 32 KiB cache with 128-byte lines holds 256 lines (256 × 128 = 32,768 B). Against 256 MiB of main RAM, that's (256×1024)/32 = 8192 — so main RAM is 8192× the size of the cache, meaning every single cache line corresponds to 8192 different line-sized chunks scattered across main RAM. Given any main-RAM address, you find its cache slot by taking that address modulo the cache size.

Addressing the Cache

The cache can only be addressed in whole lines. For a cache with 2ⁿ-byte lines, the low n bits of a main-RAM address are the offset within the line — strip those off to get a line-sized address, then split what's left into:

• the line index — which slot in the cache this could occupy, and
• the tag — which cache-sized block of main RAM the data actually came from (since many blocks compete for the same line index).

The full read sequence: split the address into offset/index/tag → look up the tag stored at that line index → if it matches the requested tag, hit (use the offset to grab the byte from the cached line) → if it doesn't match, miss (load the whole line from main RAM, store the new tag, then proceed as a hit).

Set Associativity and Replacement Policy

The scheme above — one main-RAM block maps to exactly one cache line — is a direct-mapped cache. Its weakness: two unrelated memory blocks that happen to map to the same line can end up repeatedly evicting each other (thrashing), even while the rest of the cache sits idle.

A set-associative cache fixes this by letting each address map to several lines ("ways") instead of just one — a 2-way, 4-way, 8-way, or higher associative cache. More ways means better average performance, at the cost of a more complex (and more power-hungry) lookup.

With multiple ways, the controller needs a replacement policy to decide which way to evict on a miss:

Policy	Idea
FIFO	evict whichever way was filled longest ago (the only option for a direct-mapped cache)
LRU	evict the least-recently-used way
LFU	evict the least-frequently-used way
NMRU	keep the most-recently-used way; evict among the rest
Pseudorandom	evict a randomly chosen way

Multilevel Caches

There's a fundamental trade-off between cache size and latency: bigger caches have higher hit rates, but can't physically sit as close to the core, so they're slower. Most systems resolve this by stacking levels — check the small, fast L1 first; fall back to the larger, slower L2; only hit main RAM if both miss. Some PCs go as far as L3/L4 given how expensive a full main-RAM round trip has become.

Instruction Cache and Data Cache

Code gets cached too. The instruction cache (I-cache, I$)** preloads machine code about to execute; the **data cache (D-cache, D$) speeds up the data that code reads and writes. At L1, these are always physically separate caches, so an instruction fetch can never accidentally evict useful data (or vice versa). Higher levels usually stop making this distinction, since their larger size makes the collision far less likely.

Write Policy

• Write-through — every write to the cache is immediately mirrored to main RAM. Simple, but every write pays the main-RAM cost.
• Write-back (copy-back) — writes land in the cache only; the line is flushed to main RAM later, when it's evicted (if "dirty") or explicitly flushed. Faster for repeated writes to the same location, at the cost of extra bookkeeping.

Cache Coherency

When multiple cores each have their own L1 cache but share main RAM (and often an L2), the system has to guarantee cache coherency: every core's view of memory has to behave as if it matches every other core's and main RAM's, even though the underlying caches aren't literally always in sync. The common protocols — MESI, MOESI, MESIF — track each cache line's state (modified / exclusive / shared / invalid, plus protocol-specific extras) to make that guarantee cheaply. This becomes central once we get to multicore programming, and is worth its own dedicated treatment later.

Avoiding Cache Misses

The practical payoff of all of the above:

• For D-cache: keep data small and contiguous, and walk it sequentially. Contiguous data means one miss loads everything relevant; small data means it's more likely to fit in one line (or a minimum number of them); sequential access avoids re-evicting and re-loading the same lines repeatedly.
• For I-cache: keep hot loops physically small, avoid calling out to large functions from inside them, and use inline judiciously — inlining a small hot function is a clear win, but over-inlining bloats code size until the loop no longer fits in I-cache at all, which can make things slower.

Check: cache hierarchies

Quiz

Cache lines, associativity, and write policy

1 / 3

Why do caches move data in whole cache lines rather than individual bytes?

Pick an answer to see if you are right.

---

Key takeaways

• Memory-mapped I/O means "reading/writing an address" and "talking to a peripheral" can literally be the same instruction.
• Virtual memory adds one layer of indirection (MMU + page table, sped up by the TLB) so every process gets a private, contiguous-looking address space regardless of what's physically installed or where.
• The memory gap — main RAM latency measured in hundreds of cycles versus single-digit-to-low-double-digit cycles for a register op — is why cache-aware data layout is a performance technique in its own right, not a micro-optimization.
• A cache line, not a byte, is the real unit of memory access cost — designing data structures around cache-line-sized, sequential access is the highest-leverage layout decision you can make.

Nonuniform Memory Access (NUMA)

Multiprocessor systems face a choice between two memory architectures:

• Uniform memory access (UMA) — one shared bank of main RAM, equally visible (and equally addressed) from every core, typically backed by a shared cache hierarchy. The downside is contention: cores compete for the same L2/main-RAM bandwidth. The PS4, for example, has eight cores in two clusters of four; each core has a private L1, each cluster shares an L2, and all eight share main RAM — so cores within (and across) clusters can contend for the same resources.
• Nonuniform memory access (NUMA) — each core gets its own small, fast local store: dedicated RAM on the same die as the core, invisible to every other core. Unlike a cache, a local store isn't managed automatically — moving data into and out of it is explicit, sometimes via ordinary memory-mapped addressing, sometimes via a dedicated DMA controller (DMAC).

SPU Local Stores on the PS3

The PS3's Cell Broadband Architecture is a textbook NUMA design: one PPU (main CPU) with exclusive access to 256 MiB of system RAM, an NVIDIA RSX GPU with exclusive access to 256 MiB of VRAM, and eight SPUs (synergistic processing units — six available to games, one reserved for the OS, one disabled to absorb fabrication defects), each with its own private 256 KiB local store. These address spaces are completely isolated: the PPU can't touch VRAM or any SPU's local store, and an SPU can only see its own local store — nothing else, not even another SPU's. Everything else moves via DMA.

PS2 Scratchpad (SPR)

The PS2's main CPU (the emotion engine, EE) had a 16 KiB scratchpad (SPR) alongside its regular L1 I-cache and D-cache. The scratchpad is memory-mapped — it looks to a programmer like an ordinary range of main-RAM addresses — but it's uncached, so accesses bypass the L1 cache entirely and go direct.

The scratchpad's real value wasn't its low latency (comparable to L1 anyway) — it was that CPU accesses to it don't use the system bus at all. That let the EE crunch data sitting in scratchpad while a completely separate DMA transfer moved data between main RAM and the PS2's two vector units, all in parallel, without any of that traffic contending with the CPU's own memory accesses. It's an early, concrete example of a strategy every subsequent NUMA-style console design has leaned on: give the CPU somewhere fast to work that doesn't compete for the same wires everything else is using.

Check: UMA vs. NUMA

Quiz

Uniform vs. nonuniform memory access

1 / 3

What is the core difference between a UMA and a NUMA memory architecture?

Pick an answer to see if you are right.

---

Further Reading

• Jason Gregory — Game Engine Architecture, 3rd ed. (CRC Press, 2018), Chapter 3, Section 3.5
• Ulrich Drepper — "What Every Programmer Should Know About Memory"
• John L. Hennessy & David A. Patterson — Computer Architecture: A Quantitative Approach