Field note / ai-engineering

publishedupdated 2026-05-01#agents#tool-calling#orchestration#kafka#sqs

AI Agent Orchestration

Tool calling, multi-step reasoning loops, event-driven AI pipelines with Kafka/SQS, and human-in-the-loop checkpoints for production agent systems.

Overview

An AI agent is an LLM that can take actions — call tools (APIs, databases, code executors), observe results, and decide the next step. Unlike a single LLM call (question → answer), an agent runs a loop: Think → Act → Observe → Think → Act... until it reaches a goal or hits a stop condition.

Production agents need:

  • Tool registry — defined functions the model can call
  • Execution engine — runs tool calls safely (sandboxed, rate-limited)
  • State management — persists conversation + tool call history
  • Guardrails — max iterations, timeout, cost limit, human escalation
  • Observability — trace every reasoning step for debugging

Architecture

┌──────────────────────────────────────────────────────────────┐
│                    Agent Orchestration Layer                  │
│                                                               │
│   User Request / Event Trigger                                │
│         │                                                     │
│         ▼                                                     │
│   ┌─────────────┐      ┌────────────────────────────────┐   │
│   │  Agent      │      │         Tool Registry          │   │
│   │  Controller │      │                                │   │
│   │             │      │  search_knowledge_base()        │   │
│   │  - max 10   │      │  create_jira_ticket()           │   │
│   │    iterations│     │  query_database(sql)            │   │
│   │  - $0.50    │      │  send_email(to, subject, body)  │   │
│   │    cost cap │      │  call_api(method, url, body)    │   │
│   │  - 60s      │      │  run_code(language, code)       │   │
│   │    timeout  │      └────────────┬───────────────────┘   │
│   └──────┬──────┘                   │ tool definitions        │
│          │                          │ (JSON schema)           │
│          ▼                          ▼                         │
│   ┌─────────────────────────────────────────────────────┐   │
│   │                  LLM (Claude / GPT-4o)              │   │
│   │  Input:  conversation history + tool definitions    │   │
│   │  Output: text OR tool_call{name, args}              │   │
│   └───────────────────────┬─────────────────────────────┘   │
│                           │                                   │
│            ┌──────────────▼──────────────┐                  │
│            │      Tool call?             │                  │
│            │  YES ──▶ Execute Tool       │                  │
│            │           │                │                  │
│            │           ▼                │                  │
│            │       Append result        │                  │
│            │       to history           │                  │
│            │           │                │                  │
│            │           └──▶ Loop back   │                  │
│            │  NO  ──▶  Final Answer     │                  │
│            └────────────────────────────┘                  │
└──────────────────────────────────────────────────────────────┘

━━━━━━━━━  EVENT-DRIVEN AI PIPELINE  ━━━━━━━━━

  Kafka Topic                  Agent Service           Downstream
  ┌────────────┐               ┌──────────────┐        ┌──────────┐
  │ user.events│──────────────▶│              │───────▶│  CRM API │
  │ (SQS/Kafka)│               │  LLM Agent   │        └──────────┘
  └────────────┘               │              │        ┌──────────┐
                               │  - classify  │───────▶│  Slack   │
  ┌────────────┐               │  - enrich    │        └──────────┘
  │ order.fail │──────────────▶│  - route     │        ┌──────────┐
  └────────────┘               │  - respond   │───────▶│  Jira    │
                               └──────────────┘        └──────────┘

Technical Implementation

Tool Definition and Registration (TypeScript)

// src/agents/tools/index.ts
import { Tool } from '@anthropic-ai/sdk';

export const tools: Tool[] = [
  {
    name: 'search_knowledge_base',
    description: 'Search the internal knowledge base for relevant documents. Use this when the user asks a question that requires internal documentation.',
    input_schema: {
      type: 'object',
      properties: {
        query: { type: 'string', description: 'The search query' },
        topK: { type: 'number', description: 'Number of results to return (default 5)', default: 5 },
      },
      required: ['query'],
    },
  },
  {
    name: 'create_jira_ticket',
    description: 'Create a Jira ticket for a bug or feature request.',
    input_schema: {
      type: 'object',
      properties: {
        title: { type: 'string' },
        description: { type: 'string' },
        priority: { type: 'string', enum: ['low', 'medium', 'high', 'critical'] },
        assignee: { type: 'string', description: 'Jira username' },
      },
      required: ['title', 'description', 'priority'],
    },
  },
  {
    name: 'query_database',
    description: 'Run a read-only SQL query against the analytics database. Only SELECT statements allowed.',
    input_schema: {
      type: 'object',
      properties: {
        sql: { type: 'string', description: 'SQL SELECT statement' },
      },
      required: ['sql'],
    },
  },
];

Agent Execution Loop

// src/agents/agent.service.ts
import Anthropic from '@anthropic-ai/sdk';

export class AgentService {
  private client = new Anthropic();
  private readonly MAX_ITERATIONS = 10;
  private readonly MAX_COST_USD = 0.50;

  async run(userMessage: string, context: AgentContext): Promise<AgentResult> {
    const messages: Anthropic.MessageParam[] = [
      { role: 'user', content: userMessage }
    ];

    let iterations = 0;
    let totalCostUsd = 0;

    while (iterations < this.MAX_ITERATIONS) {
      iterations++;

      const response = await this.client.messages.create({
        model: 'claude-opus-4-7',
        max_tokens: 4096,
        system: this.buildSystemPrompt(context),
        tools,
        messages,
      });

      // Track cost
      totalCostUsd += this.calculateCost(response.usage);
      if (totalCostUsd > this.MAX_COST_USD) {
        return this.buildResult(messages, 'COST_LIMIT_EXCEEDED');
      }

      // End of reasoning — final answer
      if (response.stop_reason === 'end_turn') {
        const textBlock = response.content.find((b) => b.type === 'text');
        return this.buildResult(messages, 'SUCCESS', textBlock?.text);
      }

      // Model wants to call tools
      if (response.stop_reason === 'tool_use') {
        messages.push({ role: 'assistant', content: response.content });

        // Execute all tool calls in parallel
        const toolResults = await Promise.all(
          response.content
            .filter((b): b is Anthropic.ToolUseBlock => b.type === 'tool_use')
            .map((toolUse) => this.executeTool(toolUse, context))
        );

        messages.push({ role: 'user', content: toolResults });
      }
    }

    return this.buildResult(messages, 'MAX_ITERATIONS_EXCEEDED');
  }

  private async executeTool(
    toolUse: Anthropic.ToolUseBlock,
    context: AgentContext
  ): Promise<Anthropic.ToolResultBlockParam> {
    try {
      let result: unknown;

      switch (toolUse.name) {
        case 'search_knowledge_base':
          result = await this.ragService.query(
            (toolUse.input as { query: string }).query, context.tenantId
          );
          break;
        case 'create_jira_ticket':
          result = await this.jiraService.create(toolUse.input as JiraTicketInput);
          break;
        case 'query_database':
          // Validate SQL is read-only before execution
          const sql = (toolUse.input as { sql: string }).sql;
          if (!/^SELECT/i.test(sql.trim())) {
            throw new Error('Only SELECT queries are allowed');
          }
          result = await this.analyticsDb.query(sql);
          break;
        default:
          throw new Error(`Unknown tool: ${toolUse.name}`);
      }

      return {
        type: 'tool_result',
        tool_use_id: toolUse.id,
        content: JSON.stringify(result),
      };
    } catch (err) {
      return {
        type: 'tool_result',
        tool_use_id: toolUse.id,
        is_error: true,
        content: `Tool error: ${(err as Error).message}`,
      };
    }
  }
}

Human-in-the-Loop Checkpoint

// src/agents/hitl.service.ts
// For high-stakes actions, pause and get human approval before executing

export class HITLService {
  private pendingApprovals = new Map<string, PendingApproval>();

  async requireApproval(
    action: ToolCallAction,
    context: AgentContext
  ): Promise<'approved' | 'rejected'> {
    const approvalId = crypto.randomUUID();

    // Notify human reviewer (Slack, email, etc.)
    await this.notifier.sendApprovalRequest({
      id: approvalId,
      action: action.name,
      args: action.args,
      summary: action.humanReadableSummary,
      approveUrl: `${process.env.APP_URL}/approvals/${approvalId}/approve`,
      rejectUrl: `${process.env.APP_URL}/approvals/${approvalId}/reject`,
      expiresAt: new Date(Date.now() + 30 * 60 * 1000),  // 30 min timeout
    });

    // Poll for decision (store approval state in Redis/DB)
    return this.waitForDecision(approvalId, 30 * 60 * 1000);
  }

  // Actions that always require human approval
  static requiresApproval(toolName: string): boolean {
    return ['send_email', 'create_payment', 'delete_record', 'deploy_code'].includes(toolName);
  }
}

Event-Driven Agent Pipeline (Kafka)

// src/agents/event-agent.consumer.ts
// Agent triggered by Kafka events — processes async, no user waiting

@Injectable()
export class EventAgentConsumer {
  @KafkaEventHandler('user.support.request')
  async handleSupportRequest(event: SupportRequestEvent): Promise<void> {
    const result = await this.agentService.run(
      `Classify and respond to this support request: "${event.message}"
       User tier: ${event.userTier}. Order ID: ${event.orderId ?? 'none'}`,
      { tenantId: event.tenantId, userId: event.userId }
    );

    if (result.status === 'SUCCESS') {
      // Route based on agent's classification
      const classification = JSON.parse(result.finalAnswer);

      if (classification.requiresHuman) {
        await this.crmService.createTicket(event, classification);
      } else {
        await this.emailService.sendResponse(event.userId, classification.response);
      }

      await this.producer.emit('support.request.resolved', {
        requestId: event.id,
        resolution: classification.category,
        agentIterations: result.iterations,
      });
    }
  }
}

Interview Preparation

Q: What's the difference between a chain and an agent?

A chain is a fixed sequence of LLM calls (prompt A → output → prompt B). An agent has a dynamic loop — the model decides at each step whether to call a tool or produce a final answer. Chains are predictable and testable; agents are flexible but harder to control. Use chains when the steps are known; agents when the steps depend on intermediate results.

Q: How do you prevent an agent from looping forever?

Hard limits enforced outside the LLM: max iteration count (10–20), wall-clock timeout (60s), cost cap (e.g., $0.50 per run). Also: detect repeated tool calls with identical args (the agent is stuck), and break with an error.

Q: How do you test agent behavior?

  1. Unit test tool executors independently — mock the LLM, verify tool logic
  2. Snapshot test conversation traces — given this history, assert the agent calls the right tool
  3. Eval dataset — 50–100 representative user inputs with expected outcomes; measure success rate
  4. Red-team — explicitly try to get the agent to call dangerous tools or ignore its system prompt

Q: When does a tool call need human approval?

Any action that is irreversible or high-consequence: sending emails to customers, creating charges, modifying production data, deploying code. The pattern: before executing the tool, emit an approval request event and pause the agent's execution coroutine until approved (or timed out → default to reject).

Q: How do you handle tool failures in an agent loop?

Return the error as a tool_result with is_error: true. The model is trained to handle errors and will typically either retry with different args, try a different tool, or tell the user it couldn't complete the task. Don't throw exceptions that kill the loop — graceful degradation matters.


Learning Resources