Overview
Design a chat system that:
- Supports 1:1 and group messages (up to 500 members)
- Delivers messages in real-time with < 100ms latency
- Persists all messages with infinite history
- Shows online presence and "last seen"
- Handles 50M daily active users, 10B messages/day (~116K messages/second)
Architecture
┌─────────────────────────────────────────────────────────────┐
│ Client Devices │
│ (Mobile / Web — maintain WebSocket connection) │
└────────────────────────┬────────────────────────────────────┘
│ WebSocket (persistent)
▼
┌─────────────────────────────────────────────────────────────┐
│ WebSocket Gateway Layer │
│ │
│ WS Node 1 ──▶ 50K concurrent connections per node │
│ WS Node 2 ──▶ 50K concurrent connections per node │
│ WS Node N ──▶ ... │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Connection Registry (Redis) │ │
│ │ userId → { nodeId, socketId } │ │
│ │ (where is this user connected right now?) │ │
│ └──────────────────────────────────────────────────────┘ │
└────────────────────────┬────────────────────────────────────┘
│ Publish to message queue
▼
┌─────────────────────────────────────────────────────────────┐
│ Message Broker (Kafka) │
│ │
│ Topic: messages.1-1 Topic: messages.groups │
│ Partition by chatId Partition by groupId │
│ (ordering within chat) (ordering within group) │
└────────────┬──────────────────────────┬────────────────────┘
│ │
┌─────────▼──────────┐ ┌──────────▼─────────┐
│ Delivery Service │ │ Message Store │
│ │ │ (async write) │
│ - look up │ │ │
│ recipient nodeId│ │ Cassandra │
│ - forward via │ │ PK: chatId │
│ Redis pub/sub │ │ CC: messageId (timeuuid)
│ to correct WS │ │ message, senderId, │
│ node │ │ timestamp │
│ - queue if offline│ └─────────────────────┘
└────────────────────┘
│
┌─────────▼──────────┐
│ Push Notification │
│ Service │
│ (FCM / APNs) │
│ for offline users │
└────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ Presence Service │
│ │
│ Heartbeat: client sends ping every 30s │
│ Redis: SET presence:{userId} 1 EX 35 │
│ (auto-expires if client disconnects) │
│ │
│ Last seen: write to Cassandra on disconnect │
└─────────────────────────────────────────────────────────────┘
Technical Implementation
WebSocket Connection Manager
// src/gateway/websocket.gateway.ts
import { WebSocketGateway, OnGatewayConnection, OnGatewayDisconnect } from '@nestjs/websockets';
import { Socket } from 'socket.io';
@WebSocketGateway({ cors: true })
export class ChatGateway implements OnGatewayConnection, OnGatewayDisconnect {
constructor(
private redis: RedisService,
private presence: PresenceService,
) {}
async handleConnection(socket: Socket) {
const userId = await this.authenticate(socket);
if (!userId) return socket.disconnect();
// Register: which WS node this user is on
await this.redis.hset(`connection:${userId}`, {
nodeId: process.env.NODE_ID,
socketId: socket.id,
});
await this.redis.expire(`connection:${userId}`, 300);
await this.presence.setOnline(userId);
// Join their personal channel for direct delivery
socket.join(`user:${userId}`);
// Join all their group chat rooms
const groups = await this.groupService.getUserGroups(userId);
groups.forEach((g) => socket.join(`group:${g.id}`));
}
async handleDisconnect(socket: Socket) {
const userId = socket.data.userId;
await this.redis.del(`connection:${userId}`);
await this.presence.setOffline(userId);
}
}
Message Send Flow
// src/messages/message.service.ts
export class MessageService {
async send(senderId: string, chatId: string, content: string): Promise<Message> {
const message: Message = {
id: uuidv1(), // time-based UUID — sortable, Cassandra-friendly
chatId,
senderId,
content,
sentAt: Date.now(),
};
// 1. Persist to Cassandra immediately (before delivery — don't lose messages)
await this.messageStore.save(message);
// 2. Publish to Kafka for fan-out delivery
await this.kafka.emit('messages', {
key: chatId, // same key → same partition → ordering within chat
value: message,
});
return message;
}
}
// Kafka consumer — delivers to connected recipients
@KafkaEventHandler('messages')
async deliverMessage(message: Message) {
const recipients = await this.chatService.getRecipients(message.chatId);
for (const recipientId of recipients) {
if (recipientId === message.senderId) continue;
const connection = await this.redis.hgetall(`connection:${recipientId}`);
if (connection.nodeId) {
if (connection.nodeId === process.env.NODE_ID) {
// Recipient on this node — deliver directly via socket
this.io.to(`user:${recipientId}`).emit('message', message);
} else {
// Recipient on another node — use Redis pub/sub to forward
await this.redis.publish(`node:${connection.nodeId}`, JSON.stringify({
type: 'deliver',
userId: recipientId,
message,
}));
}
} else {
// User offline — queue for push notification
await this.pushQueue.add('send-push', { userId: recipientId, message });
}
}
}
Cassandra Schema — Messages
-- Optimized for "get messages for chatId, paginated by time"
CREATE TABLE messages (
chat_id UUID,
message_id TIMEUUID, -- time-based UUID: sortable, globally unique
sender_id UUID,
content TEXT,
media_url TEXT,
PRIMARY KEY (chat_id, message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);
-- Fetch latest 50 messages, paginated with page_state
SELECT * FROM messages
WHERE chat_id = ?
ORDER BY message_id DESC
LIMIT 50;
Presence Service
// src/presence/presence.service.ts
export class PresenceService {
private readonly TTL = 35; // seconds — slightly longer than 30s heartbeat
async setOnline(userId: string): Promise<void> {
await this.redis.set(`presence:${userId}`, '1', 'EX', this.TTL);
}
async setOffline(userId: string): Promise<void> {
await this.redis.del(`presence:${userId}`);
await this.cassandra.execute(
'UPDATE users SET last_seen = ? WHERE id = ?',
[new Date(), userId]
);
}
async isOnline(userId: string): Promise<boolean> {
return (await this.redis.exists(`presence:${userId}`)) === 1;
}
// Bulk check for group chats
async getBulkPresence(userIds: string[]): Promise<Record<string, boolean>> {
const pipeline = this.redis.pipeline();
userIds.forEach((id) => pipeline.exists(`presence:${id}`));
const results = await pipeline.exec();
return Object.fromEntries(
userIds.map((id, i) => [id, results![i][1] === 1])
);
}
// Client heartbeat — called every 30s to keep presence alive
async heartbeat(userId: string): Promise<void> {
await this.redis.expire(`presence:${userId}`, this.TTL);
}
}
Interview Preparation
Q: How do you deliver a message when the recipient is on a different WebSocket node?
Redis pub/sub as an inter-node message bus. Each WS node subscribes to its own channel (node:{nodeId}). When the delivery service determines the recipient is on node-2, it publishes to node:node-2. Node-2 receives it and emits to the recipient's socket directly.
Q: What happens to messages if the user is offline?
Messages are persisted to Cassandra immediately on send (before any delivery attempt). When the user reconnects, they fetch missed messages by querying Cassandra with message_id > last_seen_message_id. Push notifications (FCM/APNs) are sent for offline users via a separate queue.
Q: How do you guarantee message ordering?
Within a chat: Kafka partitioned by chatId ensures all messages for a chat go through the same partition in order. Cassandra uses TIMEUUID as the clustering key — sorts by time automatically. On the client, messages are sorted by message_id (time-based UUID).
Q: How do you handle the "read receipt" feature?
Client sends an ack event when a message is displayed. Server updates a last_read_message_id per (userId, chatId). Derive unread count as messages after last_read_message_id. Store in Redis for fast lookup — eventually persist to Cassandra.
Q: How does group chat differ from 1:1?
1:1: two recipients, simple. Group: fan-out to N recipients. For large groups (500 members), fan-out in the delivery service is 500 Redis lookups + 500 socket deliveries. Optimize with Socket.io rooms — each group is a room; server broadcasts to the room once, Socket.io handles per-member delivery within the node.
Learning Resources
- How Discord Stores Billions of Messages — Cassandra at scale
- Slack Engineering: Real-Time Messaging
- Socket.io Docs — Node.js WebSocket library
- Cassandra Data Modeling — official guide
- System Design Interview Vol. 2 — Alex Xu, Chapter 12