Skip to main content

Design a Notification System

Adapted from: Alex Xu, System Design Interview – An Insider's Guide (2020), Chapter 10 — rewritten and condensed for teaching.

Notifications deliver breaking news, product updates, events, offers, and payment confirmations, and almost every app has them. A notification is more than a phone buzz. This lesson covers three formats:

  • Mobile push notifications (iOS and Android)
  • SMS messages
  • Email

The design leans heavily on two earlier ideas: message queues to decouple the system, and rate limiting to protect users from being overwhelmed.

What you'll be able to do after this lesson:

  • Explain how APNs, FCM, SMS gateways, and email services fit into the delivery path.
  • Evolve a single-server design into a queue-based, horizontally scalable one.
  • Reason about reliability: no data loss, retries, and why exactly-once delivery isn't possible.
  • Implement templates, a dispatcher that checks settings, deduplicates, and rate-limits, and retrying workers.

Step 1: Understand the Problem and Establish Scope​

Candidate asksInterviewer answers
Which notification types?Push, SMS, and email
Real-time?Soft real-time: deliver as soon as possible, but a slight delay under heavy load is acceptable
Supported devices?iOS, Android, and laptop/desktop
What triggers notifications?Client applications, or server-side schedules
Can users opt out?Yes. Opted-out users receive nothing
Daily volume?10 M push, 1 M SMS, 5 M email

A quick estimate (assuming an even spread over the day):

ChannelPer dayAverage per second
Push10 M≈ 116
SMS1 M≈ 12
Email5 M≈ 58
Total16 M≈ 185

Notification traffic is bursty: a marketing campaign or breaking news can send millions at once. That burstiness is why queues as buffers will matter.

Step 2: High-Level Design​

How each notification type works​

iOS push​

flowchart LR
  P[Provider] -- "device token + payload" --> APNS[APNs] --> IOS[iOS device]
  • Provider: our server. It builds the notification request and sends it to APNs (Apple Push Notification service). It needs:
    • a device token, the unique identifier of the target app install, and
    • a payload, a JSON dictionary such as:
{
  "aps": {
    "alert": {
      "title": "Game Request",
      "body": "Bob wants to play chess",
      "action-loc-key": "PLAY"
    },
    "badge": 5
  }
}
  • APNs: Apple's remote service that delivers push notifications to iOS devices.
  • iOS device: the client that receives the notification.

Android push​

The flow is the same, except Firebase Cloud Messaging (FCM) replaces APNs.

flowchart LR
  P[Provider] --> FCM[FCM] --> AND[Android device]

SMS​

Use a commercial SMS gateway such as Twilio or Nexmo (now Vonage).

Email​

Companies can run their own mail servers, but most use commercial services such as SendGrid or Mailchimp, which offer better deliverability and analytics.

flowchart LR
  P[Provider] --> APNS[APNs] --> IOS[iOS]
  P --> FCM[FCM] --> AND[Android]
  P --> SMSS["SMS service<br/>(Twilio, …)"] --> SMS[SMS]
  P --> ES["Email service<br/>(SendGrid, …)"] --> EM[Email]

Contact info gathering​

To notify anyone, we need their device tokens, phone numbers, or email addresses. When a user installs the app or signs up, the API servers collect this information and store it:

flowchart LR
  U["User installs app / signs up"] --> LB[Load balancer] --> API[API servers] --> DB[(Database)]
userdevice
user_idbigintidbigint
emailvarchardevice_tokenvarchar
country_codeintegeruser_idbigint
phone_numberintegerlast_logged_in_attimestamp
created_attimestamp

Email and phone number live on the user table. Device tokens live on a separate device table, because one user can have many devices, and a push notification can go to all of them.

First design: a single notification server​

flowchart LR
  S1[Service 1] & S2[Service 2] & SN[Service N] --> NS["Notification server<br/>(+ DB, cache)"]
  NS --> APNS[APNs] --> IOS[iOS]
  NS --> FCM[FCM] --> AND[Android]
  NS --> SMSS[SMS service] --> SMS[SMS]
  NS --> ES[Email service] --> EM[Email]
  • Services 1…N: anything that triggers a notification, such as a microservice, a cron job, or another distributed system. For example, a billing service emails payment reminders, and a shopping site texts "your package arrives tomorrow."
  • Notification system: here, a single server. It exposes APIs to the services and builds payloads for the third parties.
  • Third-party services: deliver the notifications. Design for extensibility, so providers are easy to plug in or remove. A provider may also be unavailable in some markets: FCM doesn't work in China, where alternatives such as JPush or Pushy are used.

Three problems:

  1. Single point of failure. One server.
  2. Hard to scale. Database, cache, and processing all share one box, so none can scale independently.
  3. Performance bottleneck. Rendering HTML and waiting on slow third-party APIs ties up the server, which overloads at peak times.

Improved design​

Fixes:

  • Move the database and cache out of the notification server.
  • Add more notification servers with automatic horizontal scaling.
  • Add message queues to decouple components.
flowchart LR
  S1[Service 1] & S2[Service 2] & SN[Service N] --> NS[Notification servers]
  NS <--> C[(Cache)]
  NS <--> DB[(DB)]
  NS --> QI[[iOS PN queue]] --> WI[Workers] --> APNS[APNs] --> IOS[iOS]
  NS --> QA[[Android PN queue]] --> WA[Workers] --> FCM[FCM] --> AND[Android]
  NS --> QS[[SMS queue]] --> WS[Workers] --> SMSS[SMS service] --> SMS[SMS]
  NS --> QE[[Email queue]] --> WE[Workers] --> ES[Email service] --> EM[Email]

Reading left to right:

  • Services 1…N call the notification servers' APIs.
  • Notification servers:
    • expose send APIs, internal or verified clients only, to prevent spam;
    • validate basics such as email and phone formats;
    • fetch the data needed to render the notification from the cache or database;
    • put the notification on the right message queue for parallel processing.
  • Cache holds user info, device info, and templates. The database holds users, notifications, and settings.
  • Message queues remove dependencies between components and buffer bursts. There is one queue per notification type, so an outage at one provider (say, the SMS gateway) doesn't block email or push.
  • Workers pull events from the queues and call the third-party services.

An example send API (the book labels its email example with an sms path; the request shape is what matters):

POST https://api.example.com/v/email/send
{
  "to": [{ "user_id": 123456 }],
  "from": { "email": "from_address@example.com" },
  "subject": "Hello, World!",
  "content": [{ "type": "text/plain", "value": "Hello, World!" }]
}

End-to-end flow:

  1. A service calls the notification servers' API.
  2. The notification servers fetch metadata (user info, device token, notification settings) from the cache or database.
  3. The event goes onto the matching queue. For example, an iOS push goes to the iOS PN queue.
  4. Workers pull events from the queue.
  5. Workers send them to the third-party service.
  6. The third party delivers to the user's device.

Check: high-level design​

Quiz

Queues and providers

1 / 3

The SMS provider has a two-hour outage. Why do push and email notifications keep flowing in the improved design?

Pick an answer to see if you are right.

Step 3: Design Deep Dive​

Reliability​

Preventing data loss​

A notification system can delay or reorder notifications, but it must never lose them. To guarantee that:

  • Persist every notification in a notification log database before attempting delivery.
  • Retry failed sends.
flowchart LR
  Q[[Queue]] --> W[Workers]
  W --> NL[(Notification log)]
  W --> TP[Third-party service]
  TP -. failure .-> W

Will recipients get exactly one copy?​

No. Delivery is exactly-once most of the time, but in a distributed system duplicates can happen. A worker might send successfully and then crash before recording that it did, so the retry sends the notification again. Exactly-once delivery isn't achievable in general. What you can do is reduce duplicates:

  • Give every notification event an event ID.
  • When an event arrives, check whether its ID has been seen before. If so, discard it. Otherwise send it and record the ID.

This is the same end-to-end idempotency idea covered in Correctness and Integrity in Dataflow Systems.

Additional components​

Notification templates​

Millions of notifications a day mostly follow a few shapes, so don't build each one from scratch. A template is a preformatted notification with parameters to fill in:

BODY: You dreamed of it. We dared it. [ITEM NAME] is back — only until [DATE].
CTA:  Order Now. Or, Save My [ITEM NAME]

Templates give a consistent format, fewer mistakes, and faster authoring.

Practice: render a template​

Implement render(template, params):

  • Replace each [PLACEHOLDER] (uppercase letters and spaces inside square brackets) with params[PLACEHOLDER].
  • If any placeholder has no value, raise KeyError with the missing names joined by ", ", in first-seen order and without repeats. Never send a notification that says "[DATE]".

Input: line 1 is the template, and each following line is KEY=value.

Notification settings​

People get far too many notifications, so apps offer fine-grained control. A notification setting table records it:

user_id   bigint
channel   varchar   # push notification, email, or SMS
opt_in    boolean   # opted in to receive this channel?

Before sending anything, check that the user has opted in to that channel.

Rate limiting​

Cap how many notifications a single user can receive. Bombard people and they'll turn notifications off entirely. This is per-user frequency capping, built from the same algorithms as Lesson 4.

Practice: the notification server's dispatcher​

Implement dispatch(events, opted_out, limit). Process events in order, and for each (event_id, user, channel), apply the checks in this order:

  1. Dedupe: if event_id has been seen, log "<id>: duplicate". Otherwise mark it seen.
  2. Settings: if (user, channel) is in opted_out, log "<id>: opted out".
  3. Rate limit: if the user has already had limit notifications queued, log "<id>: rate limited".
  4. Otherwise route it to its channel's queue (ios_queue, android_queue, sms_queue, email_queue), count it against the user's limit, and log "<id>: -> <queue>".

Return (log, queues), where queues maps each queue name to its list of event IDs.

Input: line 1 is limit, line 2 is the opted-out pairs user:channel (or - for none), and each remaining line is event_id user channel.

Retry mechanism​

When a third-party service fails to deliver, put the notification back on its queue and try again. If it keeps failing after a set number of attempts, alert the developers.

Practice: workers with retries​

Implement run_workers(jobs, results, max_attempts). Jobs start in a FIFO queue at attempt 1. Pop a job and take its next scripted provider result from results[job]:

  • ok: log "<job> attempt <n>: sent".
  • fail with n < max_attempts: log "<job> attempt <n>: failed, requeue" and append the job back to the queue with attempt n + 1.
  • fail on the last attempt: log "<job> attempt <n>: failed, ALERT".

Input: line 1 is max_attempts, and each following line is job result1,result2,….

Requeueing at the back keeps one flaky notification from blocking everything behind it. Real systems also add backoff between attempts, so they don't hammer a provider that is already struggling.

Security in push notifications​

Push APIs are protected with an appKey/appSecret pair. Only authenticated, verified clients can send notifications through our APIs.

Monitor queued notifications​

The key health metric is the number of notifications waiting in the queues. If the backlog keeps growing, workers aren't keeping up, so add workers before users notice the delay.

Events tracking​

Open rate, click rate, and engagement show how users respond. The notification system reports events to an analytics service:

stateDiagram-v2
  [*] --> start
  start --> pending
  pending --> sent
  pending --> error
  sent --> delivered
  delivered --> click
  delivered --> unsubscribe
  error --> pending: retry

Updated design​

flowchart LR
  S[Services 1…N] --> NS["Notification servers<br/>auth + rate limiting"]
  T[(Notification templates)] --> NS
  NS <--> C[(Cache)]
  NS <--> DB[(DB)]
  NS --> Q[["Per-channel queues"]]
  Q --> W[Workers]
  W --> NL[(Notification log)]
  W -- "retry on error" --> Q
  W --> TP["Third-party services<br/>APNs · FCM · SMS · Email"]
  TP --> D[Devices]
  NS & W & TP -.-> AN[Analytics / monitoring]

What's new compared with the improved design:

  • Notification servers gain authentication and rate limiting.
  • A retry mechanism: failed notifications go back on the queue and are retried a set number of times.
  • Templates make notification creation consistent and efficient.
  • Monitoring and tracking for health checks and future improvements.

Check: reliability and extras​

Quiz

Delivery guarantees

1 / 4

A worker sends a push successfully but crashes before marking it done. On restart, the notification is retried. What does the user experience, and what limits it?

Pick an answer to see if you are right.

Step 4: Wrap Up​

We designed a scalable system for push, SMS, and email, using message queues to decouple components. Beyond the high-level design:

  • Reliability: persistence plus a robust retry mechanism to minimize failures.
  • Security: an appKey/appSecret pair, so only verified clients can send.
  • Tracking and monitoring at every stage of the flow.
  • Respect user settings: check opt-in before sending.
  • Rate limiting: users appreciate a cap on how often they're notified.

Key Takeaways​

  • Push goes through APNs (iOS) or FCM (Android). SMS and email go through commercial gateways. Treat all of them as pluggable providers.
  • Evolve from one server to stateless notification servers → per-channel queues → workers → providers, with the database and cache separated out.
  • Never lose, may delay or duplicate: a notification log plus retries, and event-ID dedupe to make duplicates rare. Exactly-once delivery isn't achievable.
  • Check opt-in, apply per-user rate limits, and render from templates before queueing.
  • Watch queue depth, and track delivery and engagement events.

Previous lesson: Design a Web Crawler.

Next lesson: Design a News Feed System — fanout on write vs. read, the hybrid model for celebrities, ID-only feed caches, and hydration.

Reference Materials​

  1. Twilio SMS
  2. Nexmo SMS
  3. SendGrid
  4. Mailchimp
  5. You Cannot Have Exactly-Once Delivery
  6. Security in Push Notifications — IBM Cloud
  7. RabbitMQ queue monitoring example