LLM Integration in Production

Overview

Integrating LLMs into production systems is not just an API call. At scale, you need:

• Latency management — LLM calls are 200ms–10s; your clients cannot block a thread waiting
• Cost control — GPT-4 at $30/1M output tokens adds up fast at production QPS
• Reliability — model APIs have rate limits, outages, and non-deterministic outputs
• Safety — prompt injection, PII leakage, hallucinated facts must be caught before delivery

The pattern that works: build an LLM Gateway Service that sits between your product APIs and all model providers. It owns rate limiting, caching, cost tracking, prompt templating, and output validation.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Client Applications                      │
│         (Web UI / Mobile / Internal Services)                │
└──────────────────────────┬──────────────────────────────────┘
                           │ REST / gRPC
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                   LLM Gateway Service                        │
│                                                              │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │Auth & AuthZ │  │Rate Limiter  │  │ Prompt Templates │   │
│  │(JWT / API   │  │(token bucket │  │ (Handlebars /    │   │
│  │ key)        │  │ per tenant)  │  │  Mustache)       │   │
│  └─────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │Input Guard  │  │ Model Router │  │ Output Validator │   │
│  │(PII detect, │  │(cost/latency │  │ (schema, PII     │   │
│  │ injection)  │  │ heuristic)   │  │  strip, safety)  │   │
│  └─────────────┘  └──────────────┘  └──────────────────┘   │
│                                                              │
│  ┌─────────────────────────────────────────────────────┐   │
│  │           Token Usage & Cost Tracker                 │   │
│  │     (DynamoDB / Redis — per tenant, per model)       │   │
│  └─────────────────────────────────────────────────────┘   │
└────────┬─────────────────┬──────────────────┬──────────────┘
         │                 │                  │
         ▼                 ▼                  ▼
  ┌────────────┐   ┌──────────────┐   ┌──────────────┐
  │  OpenAI    │   │  Anthropic   │   │  AWS Bedrock │
  │  GPT-4o    │   │  Claude 3.5  │   │  (Titan,     │
  │            │   │              │   │   Llama 3)   │
  └────────────┘   └──────────────┘   └──────────────┘
         │                 │                  │
         └─────────────────┴──────────────────┘
                           │
              ┌────────────▼────────────┐
              │    Observability Layer  │
              │  DataDog / CloudWatch   │
              │  token_usage, latency,  │
              │  error_rate per model   │
              └─────────────────────────┘

Technical Implementation

Node.js LLM Gateway (Express + TypeScript)

// src/services/llm-gateway.service.ts
import Anthropic from '@anthropic-ai/sdk';
import OpenAI from 'openai';
import { BedrockRuntimeClient, InvokeModelCommand } from '@aws-sdk/client-bedrock-runtime';
import { RateLimiter } from './rate-limiter';
import { CostTracker } from './cost-tracker';
import { PromptTemplate } from './prompt-template';
import { OutputValidator } from './output-validator';

export interface LLMRequest {
  tenantId: string;
  templateId: string;
  variables: Record<string, string>;
  modelPreference?: 'fast' | 'smart' | 'cheap';
  maxTokens?: number;
  stream?: boolean;
}

export interface LLMResponse {
  content: string;
  model: string;
  usage: { inputTokens: number; outputTokens: number; costUsd: number };
  latencyMs: number;
}

export class LLMGatewayService {
  private anthropic = new Anthropic();
  private openai = new OpenAI();
  private rateLimiter = new RateLimiter();
  private costTracker = new CostTracker();

  async complete(req: LLMRequest): Promise<LLMResponse> {
    // 1. Rate limit check
    await this.rateLimiter.checkAndConsume(req.tenantId, req.maxTokens ?? 1000);

    // 2. Build prompt from template
    const { system, user } = await PromptTemplate.render(req.templateId, req.variables);

    // 3. Route to model based on preference
    const model = this.selectModel(req.modelPreference);

    // 4. Call model
    const start = Date.now();
    const raw = await this.callModel(model, system, user, req.maxTokens ?? 1024);
    const latencyMs = Date.now() - start;

    // 5. Validate output
    const content = await OutputValidator.validate(raw.content, req.templateId);

    // 6. Track cost
    const costUsd = this.costTracker.record(req.tenantId, model, raw.usage);

    return { content, model, usage: { ...raw.usage, costUsd }, latencyMs };
  }

  private selectModel(pref?: string): string {
    const routing: Record<string, string> = {
      fast: 'claude-haiku-4-5-20251001',
      smart: 'claude-opus-4-7',
      cheap: 'claude-haiku-4-5-20251001',
    };
    return routing[pref ?? 'smart'];
  }

  private async callModel(model: string, system: string, user: string, maxTokens: number) {
    const msg = await this.anthropic.messages.create({
      model,
      max_tokens: maxTokens,
      system,
      messages: [{ role: 'user', content: user }],
    });
    const content = msg.content[0].type === 'text' ? msg.content[0].text : '';
    return {
      content,
      usage: { inputTokens: msg.usage.input_tokens, outputTokens: msg.usage.output_tokens },
    };
  }
}

Token Bucket Rate Limiter (Redis)

// src/services/rate-limiter.ts
import { Redis } from 'ioredis';

export class RateLimiter {
  private redis: Redis;
  private readonly TOKENS_PER_MINUTE = 100_000;  // 100K tokens/min per tenant

  async checkAndConsume(tenantId: string, tokensNeeded: number): Promise<void> {
    const key = `rate:${tenantId}:${Math.floor(Date.now() / 60_000)}`;

    const pipeline = this.redis.pipeline();
    pipeline.incrby(key, tokensNeeded);
    pipeline.expire(key, 120);
    const [[, used]] = await pipeline.exec() as [[null, number]];

    if (used > this.TOKENS_PER_MINUTE) {
      throw new RateLimitExceededError(`Tenant ${tenantId} exceeded ${this.TOKENS_PER_MINUTE} tokens/min`);
    }
  }
}

Prompt Template with Variable Injection

// src/services/prompt-template.ts
// Templates stored in DB or S3 — NOT hardcoded in app
export class PromptTemplate {
  static async render(templateId: string, vars: Record<string, string>) {
    const template = await TemplateRepository.findById(templateId);

    // Sanitize variables — prevent prompt injection
    const sanitized = Object.fromEntries(
      Object.entries(vars).map(([k, v]) => [k, this.sanitize(v)])
    );

    return {
      system: this.interpolate(template.system, sanitized),
      user: this.interpolate(template.user, sanitized),
    };
  }

  private static sanitize(input: string): string {
    // Strip any attempt to override the system prompt
    return input
      .replace(/ignore (all |previous |above )?instructions/gi, '[BLOCKED]')
      .replace(/system prompt/gi, '[BLOCKED]')
      .substring(0, 4000);  // hard token cap on user-provided content
  }

  private static interpolate(template: string, vars: Record<string, string>): string {
    return template.replace(/\{\{(\w+)\}\}/g, (_, key) => vars[key] ?? '');
  }
}

Spring WebFlux LLM Client (Java)

// LlmGatewayClient.java
@Service
@RequiredArgsConstructor
public class LlmGatewayClient {

    private final WebClient webClient;
    private final MeterRegistry meterRegistry;

    public Mono<LlmResponse> complete(LlmRequest request) {
        return webClient.post()
            .uri("/v1/complete")
            .header("X-Tenant-Id", request.getTenantId())
            .bodyValue(request)
            .retrieve()
            .onStatus(HttpStatus::is4xxClientError, resp ->
                resp.bodyToMono(ErrorResponse.class)
                    .flatMap(err -> Mono.error(new LlmGatewayException(err.getMessage())))
            )
            .bodyToMono(LlmResponse.class)
            .timeout(Duration.ofSeconds(30))
            .doOnSuccess(resp -> recordMetrics(resp, request.getTenantId()))
            .retryWhen(Retry.backoff(2, Duration.ofMillis(500))
                .filter(ex -> ex instanceof WebClientResponseException.ServiceUnavailable));
    }

    private void recordMetrics(LlmResponse resp, String tenantId) {
        meterRegistry.counter("llm.tokens.input",
                "tenant", tenantId, "model", resp.getModel())
            .increment(resp.getUsage().getInputTokens());
        meterRegistry.counter("llm.tokens.output",
                "tenant", tenantId, "model", resp.getModel())
            .increment(resp.getUsage().getOutputTokens());
        meterRegistry.timer("llm.latency", "model", resp.getModel())
            .record(Duration.ofMillis(resp.getLatencyMs()));
    }
}

Output Validator — Schema + Safety

// src/services/output-validator.ts
import { z } from 'zod';

const validators: Record<string, z.ZodSchema> = {
  'product-summary': z.object({
    summary: z.string().min(10).max(500),
    sentiment: z.enum(['positive', 'neutral', 'negative']),
    keyFeatures: z.array(z.string()).min(1).max(5),
  }),
};

export class OutputValidator {
  static async validate(rawOutput: string, templateId: string): Promise<string> {
    // Check for PII patterns
    if (this.containsPII(rawOutput)) {
      throw new OutputValidationError('PII detected in LLM output');
    }

    // If template expects structured JSON, parse and validate
    const schema = validators[templateId];
    if (schema) {
      const parsed = JSON.parse(rawOutput);
      schema.parse(parsed);  // throws ZodError if invalid
    }

    return rawOutput;
  }

  private static containsPII(text: string): boolean {
    const patterns = [
      /\b\d{3}-\d{2}-\d{4}\b/,          // SSN
      /\b\d{16}\b/,                       // credit card
      /\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b/i,  // email
    ];
    return patterns.some((p) => p.test(text));
  }
}

Interview Preparation

Q: How do you handle LLM latency in a synchronous API?

Never block a synchronous request on an LLM call. Options in order of preference:

1. Async + polling — return a jobId immediately, client polls /status/{jobId}
2. SSE streaming — stream tokens as they arrive, first token in ~200ms
3. Background + push — fire-and-forget, push result via WebSocket/SNS when done

Q: How do you prevent prompt injection?

Three layers: (1) sanitize user input before injection into templates — strip override patterns, cap length; (2) use system prompt that explicitly instructs the model to ignore injection attempts; (3) validate output against expected schema — if output doesn't match, it was likely hijacked.

Q: How do you control cost at scale?

• Token budget per tenant per minute (token bucket in Redis)
• Model routing: use cheap/fast model (Haiku, GPT-3.5) for simple classification; expensive model only for complex generation
• Semantic caching: embed the user query, search a vector cache — if a similar query was answered recently, return cached response without calling the model
• Max token limits enforced at gateway, not trusted from client

Q: How do you make LLM calls reliable?

• Retry with exponential backoff on 429/503 (never on 400 — that's your prompt that's broken)
• Circuit breaker: if error rate for a model > 20% in 60s, switch to fallback model
• Timeout + fallback: 30s hard timeout, fall back to degraded response rather than hang
• Multi-model router: primary Anthropic, fallback OpenAI, fallback Bedrock

Q: Difference between fine-tuning and RAG?

Use RAG for knowledge-base Q&A, internal docs, product search. Fine-tune for domain-specific tone, classification, or structured output format.

Learning Resources

• Anthropic API Docs — Claude models, tool use, streaming
• OpenAI API Reference — completions, function calling
• AWS Bedrock Developer Guide — managed inference
• LangChain4j — Java LLM framework with built-in tools, memory, RAG
• LangChain.js — Node.js equivalent
• OWASP LLM Top 10 — security vulnerabilities
• Building LLM-Powered Apps — O'Reilly book