Consistency: At-Most Once, At-Least Once, Exactly Once

Overview

In distributed systems, networks are unreliable. A message send can fail after the receiver processed it (so you don't know if it succeeded) or before. The three delivery semantics define what guarantee the system provides:

Architecture

AT-MOST ONCE (fire and forget)
════════════════════════════════════════════════════════
  Producer          Network          Consumer
     │── message ──────────────────────▶│ (success)
     │── message ──────x                │ (lost — network failure)
                    ^^^
                    No retry → message lost forever

AT-LEAST ONCE (retry until ACK)
════════════════════════════════════════════════════════
  Producer          Network          Consumer
     │── message ──────────────────────▶│
     │◀─ ACK ─────────────────────────  │
     │── message ─────────x             │ (network failure after processing)
     │   (no ACK received)              │
     │── RETRY message ────────────────▶│ ← processed AGAIN (duplicate!)
                                    ^^^
                    Consumer must handle duplicates (idempotency)

EXACTLY ONCE (two-phase commit or idempotent producer + transaction)
════════════════════════════════════════════════════════
  Producer                                Consumer
     │                                       │
     │── BEGIN TRANSACTION ─────────────────▶│
     │── message (seq#=5) ─────────────────▶│
     │   (if already processed seq#=5 → skip)│
     │── COMMIT ───────────────────────────▶│
     │◀─ ACK ─────────────────────────────  │
     │                                       │
  Deduplication key = producer_id + sequence_number

OPTIMISTIC LOCKING (prevent lost updates):
════════════════════════════════════════════════════════
  Tx A reads row (version=5, balance=100)
  Tx B reads row (version=5, balance=100)

  Tx A: UPDATE ... SET balance=50, version=6 WHERE id=1 AND version=5
  → Succeeds (version matched), row now: version=6, balance=50

  Tx B: UPDATE ... SET balance=80, version=6 WHERE id=1 AND version=5
  → Fails (version=5 no longer exists — A already changed to 6)
  → Tx B retries from scratch

CAP THEOREM:
════════════════════════════════════════════════════════
  In presence of a network Partition, choose:

  Consistency  ─────────────────────┐
  "All nodes see same data           │
   or return error"                  │
                                     │ You must choose
  Availability ─────────────────────┤ one during partition
  "System always returns             │
   a response (possibly stale)"      │
                                     │
  Partition tolerance ──────────────┘
  (required — networks fail)

  CP systems: ZooKeeper, HBase, etcd
  AP systems: DynamoDB, Cassandra, CouchDB
  CA systems: single-node PostgreSQL (no partition tolerance)

CONSISTENCY MODELS (from strongest to weakest):
  Linearizable   ← "appears instantaneous" — like a single machine
  Sequential     ← consistent order but allows delay
  Causal         ← respects cause/effect ordering
  Eventual       ← "will converge eventually" — DNS, Cassandra default

Technical Implementation

Idempotent API (Stripe pattern)

// Client generates idempotency key — stable for retries
const idempotencyKey = `payment-${userId}-${orderId}`;

const response = await fetch('/api/payments', {
  method: 'POST',
  headers: {
    'Idempotency-Key': idempotencyKey,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ amount: 2999, currency: 'USD' }),
});

// Server — deduplicate using idempotency key
async function processPayment(req: Request) {
  const key = req.headers['idempotency-key'];

  // Check if already processed (Redis or DB)
  const cached = await redis.get(`idem:${key}`);
  if (cached) {
    return JSON.parse(cached);  // return same response as original
  }

  // Process (within a transaction)
  const result = await db.transaction(async (tx) => {
    const payment = await tx.payments.create({ ... });
    await tx.ledger.debit({ ... });
    return payment;
  });

  // Store result (24-hour window)
  await redis.setex(`idem:${key}`, 86400, JSON.stringify(result));
  return result;
}

Optimistic Locking in Java/Spring

@Entity
@Table(name = "accounts")
public class Account {
    @Id private String id;
    private BigDecimal balance;

    @Version  // JPA adds version column, auto-increments on UPDATE
    private Long version;
}

@Service
public class PaymentService {

    @Retryable(
        value = OptimisticLockingFailureException.class,
        maxAttempts = 3,
        backoff = @Backoff(delay = 100, multiplier = 2)
    )
    @Transactional
    public void transfer(String fromId, String toId, BigDecimal amount) {
        Account from = accountRepo.findById(fromId).orElseThrow();
        Account to = accountRepo.findById(toId).orElseThrow();

        if (from.getBalance().compareTo(amount) < 0)
            throw new InsufficientFundsException();

        from.setBalance(from.getBalance().subtract(amount));
        to.setBalance(to.getBalance().add(amount));
        // If version changed since read, JPA throws OptimisticLockingFailureException
        // @Retryable catches it and retries the whole method
    }
}

Kafka Exactly-Once Processing

// Producer with exactly-once semantics
Properties props = new Properties();
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);  // auto-assign producer ID + seq#
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "payment-processor-1");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

try {
    producer.beginTransaction();

    // Read from input topic, process, write to output topic
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        String processed = process(record.value());
        producer.send(new ProducerRecord<>("output-topic", record.key(), processed));
    }

    // Commit offsets AND produced records atomically
    producer.sendOffsetsToTransaction(
        getOffsets(records), consumer.groupMetadata()
    );
    producer.commitTransaction();

} catch (Exception e) {
    producer.abortTransaction();  // rolls back both offsets and produced messages
}

Interview Preparation

Q: How do you implement exactly-once in a payment system without Kafka transactions?

Idempotency keys + database transactions. Client generates a UUID for each payment attempt. Server: (1) check if key exists in processed_payments table, (2) if yes return cached result, (3) if no, process payment in DB transaction and insert into processed_payments atomically. Retries get the same response without recharging.

Q: CAP theorem — can you give a concrete example?

Cassandra is AP: during a network partition, nodes on either side of the partition keep serving reads and writes (high availability). When the partition heals, conflicting writes are resolved via last-write-wins or vector clocks. You get availability but temporary inconsistency. ZooKeeper is CP: during a partition, if a node can't confirm majority quorum, it stops serving requests (returns error) to stay consistent.

Q: What is the difference between optimistic and pessimistic locking?

Optimistic: read data with version number, update with WHERE version = N, retry on conflict. No DB locks held — good for low-contention scenarios. Pessimistic: use SELECT FOR UPDATE to acquire an exclusive lock before reading — no other transaction can modify the row until you release it. Better for high-contention (e.g., seat reservation where you know conflicts will happen).

Q: What is eventual consistency and when is it acceptable?

Eventual consistency means all replicas will eventually converge to the same value, but may be temporarily out of sync. Acceptable for: social media likes/counts (off by a few for a second), DNS (propagation delay is expected), shopping cart (temporary inconsistency vs higher availability trade-off). Not acceptable for: bank balances, inventory counts where overselling is costly.

Learning Resources

• Designing Data-Intensive Applications — Kleppmann, Chapters 7-9
• Kafka Exactly-Once Semantics
• Stripe Idempotency Keys
• CAP Theorem — Gilbert & Lynch — original paper