Overview
When a business operation spans multiple microservices (order → inventory → payment → notification), the services must coordinate. Two patterns define how that coordination happens:
- Orchestration: a central service directs the workflow (like a conductor)
- Choreography: each service reacts to events and emits new events (like dancers following music)
Both implement the Saga pattern for distributed transactions — a sequence of local transactions, each publishing an event that triggers the next step. Failed steps trigger compensating transactions to undo prior steps.
Architecture
ORCHESTRATION (Central Conductor)
════════════════════════════════════════════════════════
Order Service (Orchestrator)
│
│ 1. POST /reserve → Inventory Service
│◀─ reserved: true ──────────
│
│ 2. POST /charge → Payment Service
│◀─ charged: true ──────────
│
│ 3. POST /send-confirmation → Notification Service
│◀─ sent: true ──────────────
│
✓ Order complete
FAILURE ROLLBACK:
Payment fails →
Orchestrator calls: POST /release-reservation → Inventory
Orchestrator marks order: FAILED
Pro: easy to follow, centralized visibility, explicit business logic
Con: orchestrator becomes a God Service — knows everything, couples to all
CHOREOGRAPHY (Event-Driven — No Central Coordinator)
════════════════════════════════════════════════════════
Order Service Inventory Service Payment Service
│ │ │
│ emit: OrderPlaced ──▶│ │
│ │ reserve stock │
│ │ emit: StockReserved ─▶│
│ │ │ charge card
│ │ │ emit: PaymentProcessed
│◀─────────────────────────────────────────── │
│ update order status │
│ emit: OrderConfirmed │
▼
Notification Service
(listens to OrderConfirmed)
FAILURE ROLLBACK:
PaymentFailed event →
Inventory Service listens → releases reservation
Order Service listens → marks order FAILED → emits OrderCancelled
Pro: loose coupling, each service only knows events, scales independently
Con: hard to trace a single request, complex failure paths, eventual consistency
SAGA PATTERN COMPARISON:
════════════════════════════════════════════════════════
Traditional 2PC (Two-Phase Commit):
┌──────────────────────────────────────┐
│ Coordinator sends PREPARE to all │
│ All services LOCK resources │ ← blocking, slow
│ Coordinator sends COMMIT or ABORT │
│ All services release locks │
└──────────────────────────────────────┘
Works for single DB, fails across microservices (different DBs)
Saga (no distributed locks):
┌──────────────────────────────────────┐
│ T1 → T2 → T3 (forward — success) │
│ T3 fails: │
│ C3 → C2 → C1 (compensate — undo) │
└──────────────────────────────────────┘
Each Ti is a local transaction, Ci is its compensating transaction
No distributed locks — eventual consistency
WHEN TO USE EACH:
Orchestration → complex workflows with many conditional branches
workflows that need audit trail
BPM tools (AWS Step Functions, Temporal, Camunda)
Choreography → simple linear flows
high decoupling required
event streaming architecture (Kafka-native)
Technical Implementation
Orchestration with AWS Step Functions
// State machine definition (CDK)
import { StateMachine, Chain, Task } from 'aws-cdk-lib/aws-stepfunctions';
import { LambdaInvoke } from 'aws-cdk-lib/aws-stepfunctions-tasks';
const reserveInventory = new LambdaInvoke(this, 'ReserveInventory', {
lambdaFunction: inventoryLambda,
resultPath: '$.inventory',
});
const chargePayment = new LambdaInvoke(this, 'ChargePayment', {
lambdaFunction: paymentLambda,
resultPath: '$.payment',
});
const rollbackInventory = new LambdaInvoke(this, 'RollbackInventory', {
lambdaFunction: inventoryRollbackLambda,
});
// Payment failure triggers rollback
const handlePaymentFailure = rollbackInventory
.next(new Fail(this, 'OrderFailed', { error: 'PaymentFailed' }));
chargePayment.addCatch(handlePaymentFailure, {
errors: ['PaymentError'],
resultPath: '$.error',
});
const definition = Chain.start(reserveInventory)
.next(chargePayment)
.next(new LambdaInvoke(this, 'SendConfirmation', {
lambdaFunction: notificationLambda,
}));
new StateMachine(this, 'OrderSaga', { definition });
Choreography with Kafka
// Order Service — emits events, doesn't know who processes them
export class OrderService {
constructor(
private readonly producer: KafkaProducer,
private readonly db: Database,
) {}
async placeOrder(order: Order): Promise<void> {
await this.db.transaction(async (tx) => {
await tx.orders.create({ ...order, status: 'PENDING' });
// Transactional outbox: write event to DB before publishing
await tx.outbox.insert({
topic: 'order.placed',
key: order.id,
payload: JSON.stringify(order),
});
});
// Outbox relay publishes to Kafka asynchronously
}
}
// Inventory Service — reacts to OrderPlaced
export class InventoryConsumer {
async onOrderPlaced(order: Order): Promise<void> {
const reserved = await this.inventory.reserve(order.items);
if (reserved) {
await this.producer.send({
topic: 'inventory.reserved',
key: order.id,
value: JSON.stringify({ orderId: order.id, items: order.items }),
});
} else {
await this.producer.send({
topic: 'inventory.insufficient',
key: order.id,
value: JSON.stringify({ orderId: order.id, reason: 'out_of_stock' }),
});
}
}
}
// Order Service listens for downstream events to update order status
export class OrderEventConsumer {
@KafkaListener('payment.processed')
async onPaymentProcessed({ orderId }: { orderId: string }): Promise<void> {
await this.db.orders.update(orderId, { status: 'CONFIRMED' });
await this.producer.send({ topic: 'order.confirmed', key: orderId, value: ... });
}
@KafkaListener('payment.failed')
async onPaymentFailed({ orderId }: { orderId: string }): Promise<void> {
await this.db.orders.update(orderId, { status: 'FAILED' });
// Compensating event — inventory consumer will release reservation
await this.producer.send({ topic: 'order.cancelled', key: orderId, value: ... });
}
}
Transactional Outbox Pattern
// Prevents "published event but DB rolled back" or vice versa
// Write event to outbox table atomically with state change
async function processOrderWithOutbox(order: Order): Promise<void> {
await db.transaction(async (tx) => {
// 1. Write business state
await tx.execute(
'INSERT INTO orders (id, status) VALUES (?, ?)',
[order.id, 'PENDING']
);
// 2. Write event to outbox — same transaction
await tx.execute(
'INSERT INTO outbox (id, topic, payload, published) VALUES (?, ?, ?, false)',
[uuid(), 'order.placed', JSON.stringify(order)]
);
// Both commit or both rollback — atomically
});
}
// Relay process: polls outbox, publishes to Kafka, marks published
// Or: Debezium CDC connector reads outbox table from DB binlog → Kafka
Interview Preparation
Q: What is the difference between the Saga pattern and 2PC (two-phase commit)?
2PC requires all participants to hold locks until the coordinator decides commit/abort. Works within a single database but fails across microservices with different databases — locks across network calls lead to deadlock and unavailability. Saga uses only local transactions: each step commits immediately, and if a later step fails, compensating transactions undo previous steps. No distributed locks. Trade-off: Sagas have intermediate states where the system is temporarily inconsistent (order reserved but not yet paid).
Q: How do you debug a choreography flow when something goes wrong?
Distributed tracing. Every event carries a correlation-id (the original request ID, propagated through all downstream events). OpenTelemetry / Jaeger collects spans across services. You query "show me all events with correlation-id=abc" and see the full event chain and timings. Without a correlation ID propagated through all events, choreography is nearly impossible to debug in production.
Q: What is the Transactional Outbox pattern and why do you need it?
Without outbox, a service might commit its DB transaction but then fail before publishing the Kafka event (crash, network timeout). Or it publishes the event but the DB transaction rolls back. Either way: inconsistency. The outbox pattern writes the event to an outbox table in the same database transaction as the business state change. A relay process (or Debezium CDC) reads the outbox and publishes to Kafka, marking events as published. This guarantees at-least-once delivery: the event is published if and only if the DB committed.
Q: When would you choose orchestration over choreography?
Orchestration when: the workflow has many conditional branches (if payment fails, retry with different method; if second method fails, offer credit); when you need a visual audit trail of every step; when using a workflow engine that handles retries, timeouts, and human approvals (AWS Step Functions, Temporal). Choreography when: you have a simple linear event chain; loose coupling is critical (inventory team shouldn't know about payment); the system is built on Kafka and teams prefer event-native APIs.
Learning Resources
- Saga Pattern — Chris Richardson
- Transactional Outbox Pattern
- AWS Step Functions — managed orchestration
- Temporal.io — open-source workflow engine
- Building Microservices — Sam Newman, Chapter 6 (workflow)
- Designing Distributed Systems — Burns, Chapter 8