Overview
Design a notification system that:
- Sends notifications via push (mobile), email, SMS, and in-app
- Handles 10M notifications/day (~115/second, peaks at 1000/second)
- Deduplicates: don't send the same notification twice
- Respects user preferences (opted-out channels, quiet hours, frequency caps)
- Tracks delivery: sent → delivered → opened
- Retries on failure with exponential backoff
Architecture
PRODUCERS (event sources)
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│Order Svc │ │Marketing │ │ Alert │ │ System │
│ │ │ Campaigns│ │ Service │ │ Events │
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │ │
└────────────┴────────────┴────────────┘
│
▼ Notification Events
┌──────────────────────┐
│ Kafka / SQS │
│ notification.queue │
└──────────┬───────────┘
│
▼
┌──────────────────────────────────────────────────────┐
│ Notification Service │
│ │
│ 1. Fetch user preferences (opted-out? quiet hours?) │
│ 2. Deduplicate (Redis: seen this notif ID before?) │
│ 3. Apply frequency cap (max 5 push/day per user) │
│ 4. Template rendering (Handlebars / i18n) │
│ 5. Route to channel workers │
└──────────────────────────────────────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌────────────┐
│ Push │ │ Email │ │ SMS │ │ In-App │
│ Worker │ │ Worker │ │ Worker │ │ Worker │
│ │ │ │ │ │ │ │
│ FCM │ │ SES / │ │ Twilio/ │ │ WebSocket │
│ APNs │ │ SendGrid│ │ SNS │ │ or SSE │
└────┬────┘ └────┬────┘ └────┬────┘ └─────┬──────┘
│ │ │ │
└───────────┴───────────┴─────────────┘
│
┌─────────▼──────────┐
│ Delivery Tracking │
│ (Cassandra) │
│ │
│ notif_id │
│ user_id │
│ channel │
│ status: sent/ │
│ delivered/opened │
│ timestamps │
└─────────────────────┘
RETRY FLOW:
┌────────────────────────────────────────────┐
│ Failed delivery → Dead Letter Queue (DLQ) │
│ Retry consumer: exponential backoff │
│ attempt 1: +30s, 2: +2min, 3: +10min │
│ After 3 fails → mark failed, alert ops │
└────────────────────────────────────────────┘
Technical Implementation
Notification Event Schema
// Producers send this event to Kafka
interface NotificationEvent {
id: string; // idempotency key — dedup on this
type: 'order_shipped' | 'password_reset' | 'marketing' | 'alert';
userId: string;
channels: Array<'push' | 'email' | 'sms' | 'in_app'>;
priority: 'high' | 'normal' | 'low';
templateId: string;
variables: Record<string, string>; // { orderNumber: '12345', trackingUrl: '...' }
scheduledAt?: string; // ISO string — for scheduled notifications
expiresAt?: string; // don't send after this time
}
Notification Service — Core Orchestration
// src/notification.service.ts
export class NotificationService {
async process(event: NotificationEvent): Promise<void> {
// 1. Check expiry
if (event.expiresAt && new Date(event.expiresAt) < new Date()) {
return; // Notification has expired — drop silently
}
// 2. Deduplicate — Redis SETNX with TTL
const dedupeKey = `notif:sent:${event.id}`;
const isNew = await this.redis.setnx(dedupeKey, '1');
if (!isNew) return; // Already processed
await this.redis.expire(dedupeKey, 7 * 24 * 3600); // 7-day dedup window
// 3. Load user preferences
const prefs = await this.userPrefsService.get(event.userId);
if (!prefs.notificationsEnabled) return;
// 4. Filter channels by user opt-out
const activeChannels = event.channels.filter(
(ch) => !prefs.optedOutChannels.includes(ch)
);
if (activeChannels.length === 0) return;
// 5. Quiet hours check for non-urgent notifications
if (event.priority !== 'high' && this.isQuietHours(prefs)) {
await this.scheduleForLater(event, prefs.quietHoursEnd);
return;
}
// 6. Frequency cap — max 5 push notifications per user per day
if (activeChannels.includes('push')) {
const pushCount = await this.getCap(event.userId, 'push', 'day');
if (pushCount >= 5) {
activeChannels.splice(activeChannels.indexOf('push'), 1);
}
}
// 7. Render template
const rendered = await this.templateService.render(event.templateId, event.variables, prefs.locale);
// 8. Fan out to channel workers (parallel)
await Promise.allSettled(
activeChannels.map((channel) =>
this.dispatchToWorker(channel, event.userId, rendered, event.id)
)
);
}
private async dispatchToWorker(
channel: string,
userId: string,
content: RenderedNotification,
notifId: string
): Promise<void> {
await this.kafka.emit(`notifications.${channel}`, { userId, content, notifId });
}
}
Push Worker — FCM + APNs
// src/workers/push.worker.ts
@KafkaEventHandler('notifications.push')
export class PushWorker {
async handle(event: PushWorkerEvent): Promise<void> {
const device = await this.deviceService.getActiveDevice(event.userId);
if (!device) return;
try {
if (device.platform === 'android') {
await this.sendFCM(device.token, event.content);
} else {
await this.sendAPNs(device.token, event.content);
}
await this.trackDelivery(event.notifId, event.userId, 'push', 'sent');
} catch (err) {
await this.trackDelivery(event.notifId, event.userId, 'push', 'failed');
throw err; // Kafka consumer retries with backoff
}
}
private async sendFCM(token: string, content: RenderedNotification): Promise<void> {
await admin.messaging().send({
token,
notification: { title: content.title, body: content.body },
data: content.data,
android: { priority: 'high', ttl: 3600 * 1000 },
});
}
}
Delivery Tracking Schema (Cassandra)
CREATE TABLE notification_delivery (
notification_id UUID,
user_id UUID,
channel TEXT, -- push, email, sms, in_app
status TEXT, -- queued, sent, delivered, opened, failed
sent_at TIMESTAMP,
delivered_at TIMESTAMP,
opened_at TIMESTAMP,
failure_reason TEXT,
PRIMARY KEY ((notification_id), user_id, channel)
);
-- Query: did user open this notification?
SELECT status, opened_at FROM notification_delivery
WHERE notification_id = ? AND user_id = ? AND channel = 'push';
-- Query: unread in-app notifications for a user (different access pattern)
-- Use a separate table partitioned by user_id for this
CREATE TABLE user_notifications (
user_id UUID,
notification_id TIMEUUID,
title TEXT,
body TEXT,
read_at TIMESTAMP,
PRIMARY KEY (user_id, notification_id)
) WITH CLUSTERING ORDER BY (notification_id DESC);
Interview Preparation
Q: How do you prevent sending the same notification twice?
Idempotency key on every event (id field). On processing, SETNX notif:sent:{id} in Redis with 7-day TTL. If SETNX returns 0 (key already exists), skip. The producer must generate a stable, deterministic ID (e.g., hash of {type}:{userId}:{triggerId}).
Q: How do you implement quiet hours?
Store quietHoursStart and quietHoursEnd (e.g., 22:00–08:00) in user preferences. On process, check current time in user's timezone (store timezone in user profile). For non-urgent notifications, schedule re-delivery at quietHoursEnd via a delayed Kafka message or a scheduled job queue (Bull/SQS delayed messages).
Q: How do you handle channel failures?
Each channel worker catches exceptions and re-throws. Kafka consumer's retry policy handles backoff: 3 retries with exponential delay. After 3 failures, message goes to DLQ (notifications.push.dlq). DLQ consumer alerts on-call and marks the notification as failed in the tracking table.
Q: How does the "delivered" vs "opened" tracking work for push?
- Sent = FCM/APNs API call succeeded
- Delivered = device confirmed receipt (FCM delivery receipts, APNs feedback service)
- Opened = app is foregrounded by tapping the notification → app sends
ackevent to your API → updateopened_at
FCM delivery receipts are not 100% reliable. "Delivered" is a best-effort signal.
Q: How do you handle 10M+ users in a marketing blast?
Don't call the notification service once per user. Campaign service writes a campaign_id + segment filter to a job queue. A fan-out worker queries users matching the segment in pages (1000 at a time), emitting one Kafka event per user. This distributes the load over time and avoids spikes.
Learning Resources
- Firebase Cloud Messaging Docs
- Apple Push Notification Service
- Amazon SNS — managed push/SMS/email
- Twilio SMS API
- System Design Interview Vol. 1 — Alex Xu, Chapter 10
- Airbnb: Notification System — real-world scaling