Skip to main content

Text versus Bytes

Source: Fluent Python, 2nd Edition — Chapter 4 by Luciano Ramalho

Humans use text. Computers store bytes. Python 3 keeps those worlds apart: str is Unicode characters; bytes / bytearray are integers 0–255. You convert with an encoding. Implicit mix-ups are how you get mojibake.

This chapter covers:

  • Code points vs byte sequences
  • bytes, bytearray, and why b[0] is an int
  • Codecs, encode/decode errors, and the Unicode sandwich
  • Normalization (NFC) and casefold
  • unicodedata names and combining marks
  • Dual-mode APIs (re on str vs bytes)

Characters, code points, bytes​

A Unicode character has a code point (U+0041 is A). An encoding turns code points into bytes and back. UTF-8 is the usual web default. The same character can be one byte in UTF-8 and two bytes in UTF-16LE.

s = "caf\u00e9"           # 4 characters
b = s.encode("utf-8")     # 5 bytes: é is two UTF-8 bytes
text = b.decode("utf-8")

Memory aid: decode bytes → human str. encode str → machine bytes.

You cannot reliably detect an encoding from bytes alone. HTTP/XML headers or an agreed codec must tell you. UTF-8 is strict: junk often raises UnicodeDecodeError. Many legacy 8-bit codecs silently turn any bytes into the wrong letters.

🧪 str length vs UTF-8 length

Mini quiz — encode and decode​

Quiz

Encode and decode

1 / 2

s.encode("utf-8") returns

Pick an answer to see if you are right.

Challenge: byte length​

Read a word. Print two integers: number of characters, then number of UTF-8 bytes.

Binary sequences​

bytes is immutable. bytearray is mutable. There is no bytearray literal; the repr looks like bytearray(b'...').

Each index is an int 0–255. A slice, even of length 1, is still bytes / bytearray. That differs from str, where s[0] == s[:1].

Printable ASCII shows as characters in the repr; other values use \xNN. bytes.fromhex("31 4B") builds from hex pairs.

Most str methods that do not need Unicode data also exist on binary sequences (replace, strip, endswith, …) but they take bytes arguments, not str.

UTF-16 (and UTF-32) care about endianness. A BOM (b'\\xff\\xfe' or b'\\xfe\\xff') marks byte order. UTF-8 does not need a BOM; utf-8-sig is for files that might start with ef bb bf. Prefer writing UTF-8 without a BOM for scripts (#! must stay b'#!').

🧪 Index vs slice

Mini quiz — bytes​

Quiz

bytes

1 / 2

type(b"ab"[0]) is

Pick an answer to see if you are right.

Challenge: first byte​

Read ASCII text. Print the integer of the first byte, then the repr of a one-byte slice (Python’s repr).

Encode/decode errors​

  • UnicodeEncodeError — str → bytes, character missing from the codec.
  • UnicodeDecodeError — bytes → str, invalid sequence for that codec.
  • Source files default to UTF-8. A # coding: cp1252 header is a last resort; converting the file to UTF-8 is better.

Handlers: strict (default), ignore (silent loss), replace (? on encode, � on decode), xmlcharrefreplace on encode.

str.isascii() is True only if every character is ASCII — then almost any codec can encode it.

Unicode sandwich: decode to str as soon as data enters; keep business logic on str; encode as late as possible. Always pass encoding= to open() for text. Locale defaults differ (Windows vs macOS/Linux) and change when stdout is redirected.

🧪 replace on decode

Mini quiz — errors and files​

Quiz

Errors and files

1 / 2

Opening a UTF-8 file on Windows without encoding= may

Pick an answer to see if you are right.

Challenge: safe decode​

Read a line of space-separated integers 0–255. Build bytes, decode as UTF-8 with errors="replace", print the resulting string.

Normalization and case folding​

é can be one code point or e + combining acute. They look the same but == is False until you unicodedata.normalize.

  • NFC — compose (shortest usual form; good default).
  • NFD — decompose to base + marks.
  • NFKC / NFKD — compatibility; can change meaning (½ → 1⁄2). Use for search indexes, not archival storage.
from unicodedata import normalize
normalize("NFC", a) == normalize("NFC", b)

str.casefold() is stronger than lower() for case-insensitive match (ß → ss).

Diacritic stripping (NFD, drop combining characters, NFC) is a search hack, not true equality. It can change meaning.

Locale sorting (locale.strxfrm) needs a working OS locale. Cross-platform collation is a library (UCA / ICU), not something we install in this sandbox.

🧪 NFC makes café equal

Mini quiz — NFC​

Quiz

Normalization

1 / 2

NFC is mainly for

Pick an answer to see if you are right.

Challenge: same word​

Two tokens. Print yes if they are NFC-equal, else no.

Unicode database​

unicodedata.name(ch) is the official name. unicodedata.numeric(ch) knows the human value of many number-like characters. str.isdecimal / isnumeric / isalpha use that database.

A tiny finder: walk chr(code), skip unnamed points, test whether query words are a subset of name.split().

re.compile(r"\\d+") on str matches more than ASCII digits. The same pattern as bytes (rb"\\d+") stays ASCII-only. os functions accept str or bytes paths so you can survive invalid filenames on Unix.

🧪 name and combining

Mini quiz — unicodedata​

Quiz

unicodedata

1 / 2

unicodedata.combining(c) is nonzero when c is

Pick an answer to see if you are right.

Challenge: official name​

Read one character (the first of the line). Print its Unicode name in uppercase (as unicodedata.name already is).

Chapter summary​

Key takeaways
  1. str is code points; bytes is 0–255. Convert with a named codec.
  2. b[i] is int; b[i:i+1] is bytes.
  3. Decode early, encode late, always pass encoding= for text files.
  4. NFC (and casefold when needed) before comparing user text.
  5. unicodedata is the catalog behind names, categories, and numeric values.
  6. re / os behave differently on str vs bytes.

Exercises​

Activity: ASCII check

Read a line. Print ascii if str.isascii() else unicode.