AWS Step Functions: Durable Workflow Orchestration

Overview

AWS Step Functions is a serverless state machine service. You define states in Amazon States Language (ASL — JSON), and AWS handles retries, timeouts, state persistence, and execution history.

State types:

Standard vs Express Workflows:

Technical Implementation

State Machine Definition (ASL)

{
  "Comment": "Membership Provisioning — Standard Workflow",
  "StartAt": "ValidateRequest",
  "States": {
    "ValidateRequest": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:${AWS::AccountId}:function:validate-member",
      "ResultPath": "$.validation",
      "Next": "ProvisionInParallel",
      "Retry": [{
        "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
        "IntervalSeconds": 2,
        "MaxAttempts": 3,
        "BackoffRate": 2.0
      }],
      "Catch": [{
        "ErrorEquals": ["ValidationError"],
        "ResultPath": "$.error",
        "Next": "NotifyFailure"
      }]
    },
    "ProvisionInParallel": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "ProvisionLoyalty",
          "States": {
            "ProvisionLoyalty": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:us-east-1:${AWS::AccountId}:function:provision-loyalty",
              "End": true,
              "Retry": [{ "ErrorEquals": ["States.ALL"], "MaxAttempts": 2 }]
            }
          }
        },
        {
          "StartAt": "ProvisionEntitlements",
          "States": {
            "ProvisionEntitlements": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:us-east-1:${AWS::AccountId}:function:provision-entitlements",
              "End": true,
              "Retry": [{ "ErrorEquals": ["States.ALL"], "MaxAttempts": 2 }]
            }
          }
        },
        {
          "StartAt": "ProvisionDigitalPass",
          "States": {
            "ProvisionDigitalPass": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:us-east-1:${AWS::AccountId}:function:provision-digital-pass",
              "End": true,
              "Retry": [{ "ErrorEquals": ["States.ALL"], "MaxAttempts": 2 }]
            }
          }
        }
      ],
      "Next": "NotifySuccess"
    },
    "NotifySuccess": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:${AWS::AccountId}:function:notify-success",
      "End": true
    },
    "NotifyFailure": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:${AWS::AccountId}:function:notify-failure",
      "End": true
    }
  }
}

Starting an Execution (TypeScript)

import { SFNClient, StartExecutionCommand } from '@aws-sdk/client-sfn';

const sfn = new SFNClient({ region: process.env.AWS_REGION });

export async function startProvisioning(memberId: string, plan: string): Promise<string> {
  const command = new StartExecutionCommand({
    stateMachineArn: process.env.PROVISIONING_STATE_MACHINE_ARN!,
    // Name must be unique per execution — include memberId + timestamp
    name: `provision-${memberId}-${Date.now()}`,
    input: JSON.stringify({ memberId, plan, requestedAt: new Date().toISOString() }),
  });

  const { executionArn } = await sfn.send(command);
  return executionArn;
}

Bypassing the 256KB Payload Limit

State data is limited to 256KB. For user profiles (with purchase history, entitlements), this is easy to exceed.

// Lambda: store large payload in S3, pass reference only
export const handler: Handler = async (event) => {
  const s3Key = `sf-executions/${event.memberId}/profile.json`;
  
  await s3.putObject({
    Bucket: process.env.EXECUTION_BUCKET,
    Key: s3Key,
    Body: JSON.stringify(event.fullProfile),  // large object
    ContentType: 'application/json',
  }).promise();

  // Return S3 reference — tiny, safe for state machine
  return {
    memberId: event.memberId,
    profileRef: { bucket: process.env.EXECUTION_BUCKET, key: s3Key }
  };
};

Downstream Lambdas fetch from S3 using the reference. Delete the S3 object in the terminal state.

Architecture

Challenges

1. 256KB state data limit hit in production

When I added full user profile data (including 90-day purchase history for loyalty tier calculation) to the state input, executions failed with StateMachineExecutionLimitExceededException for ~8% of members with large purchase histories.

Fix: the S3 reference pattern described above. Now state data is always < 5KB regardless of profile size.

2. Execution naming collisions

The execution name must be unique within the state machine. During a load test, concurrent provisioning requests for the same member (idempotency retry scenario) hit a ExecutionAlreadyExists error.

Fix: added idempotency key in the name: provision-${memberId}-${idempotencyKey} where the idempotency key is a hash of the request payload. Duplicate requests reuse the same execution.

3. Cold start chains

Validate → Parallel(3 Lambdas) → Notify = 5 Lambda invocations per workflow, each potentially a cold start. Under low traffic, p99 was 8s with cold starts.

Fix: Lambda provisioned concurrency on the critical path Lambdas (validate + provision-entitlements). Reduced cold start p99 from 2.1s to 180ms per Lambda.

My Responsibilities

• Designed the membership provisioning state machine — defined the ASL, parallel branches, retry/catch configuration.
• Implemented the S3 reference pattern to bypass the 256KB limit — a reusable pattern now used by 2 other state machines.
• Owned the migration from SQS-chained Lambdas to Step Functions — defined the cutover plan, ran parallel execution for 2 weeks to verify correctness.
• Set up X-Ray distributed tracing across the state machine and all Lambdas for full execution visibility.
• Operated the workflow in production — debugged execution failures via the Step Functions console, tuned retry policies based on downstream error patterns.

References

• AWS Step Functions Developer Guide
• Amazon States Language (ASL) — state machine JSON spec
• Step Functions Best Practices
• AWS SDK v3 — SFN Client
• Lambda Provisioned Concurrency
• Related: Microservices