Overview
AI observability is harder than traditional software observability because outputs are probabilistic — the same input can produce different outputs, and "correctness" often requires semantic understanding to evaluate. The discipline covers: operational telemetry (latency, cost, token usage, error rates), output quality evaluation (evals), and regression detection (does a prompt change break existing behavior?).
Architecture
AI OBSERVABILITY STACK
════════════════════════════════════════════════════════
User Request
│
┌────▼────────────────────────────────────────────────┐
│ LLM Application │
│ RAG → Prompt Assembly → LLM Call → Response │
└────┬────────────────────────────────────────────────┘
│
┌────▼────────────────────────────────────────────────┐
│ Observability Layer │
│ │
│ Traces: request → retrieval → llm → response │
│ Metrics: latency p99, tokens/req, cost/req │
│ Evals: answer quality, hallucination, relevance │
└────┬────────────────────────────────────────────────┘
│
┌────▼─────────────────────────────────────────────────────────┐
│ Tools: LangSmith, Langfuse, Arize Phoenix, Helicone │
└──────────────────────────────────────────────────────────────┘
EVAL TYPES
════════════════════════════════════════════════════════
Rule-based evals (fast, cheap, deterministic):
├── Format check: does response contain valid JSON?
├── Length check: response between 50-500 chars?
├── Contains check: does answer mention required terms?
└── SQL safety: does generated SQL have any DROP/DELETE?
LLM-as-Judge (flexible, medium cost, non-deterministic):
├── Faithfulness: is the answer grounded in the context?
├── Relevance: does the answer address the question?
├── Harmlessness: does the answer contain harmful content?
└── Completeness: are all parts of the question answered?
Human evals (ground truth, expensive, slow):
├── Golden dataset: curated Q&A pairs with known correct answers
├── Side-by-side: raters compare model A vs model B
└── Thumbs up/down: user feedback in production
PROMPT REGRESSION DETECTION
════════════════════════════════════════════════════════
Before deploying a prompt change:
1. Run old prompt on golden dataset → save scores
2. Run new prompt on same golden dataset → new scores
3. Compare: did any eval metric drop > threshold?
4. Gate: fail deployment if faithfulness drops > 5%
"Prompt CI/CD" — same discipline as code testing
Technical Implementation
OpenTelemetry Tracing for LLM Calls
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('ai-service');
export async function answerQuestion(question: string, userId: string): Promise<string> {
return tracer.startActiveSpan('rag.answer', async (span) => {
span.setAttributes({
'user.id': userId,
'question.length': question.length,
});
try {
// Span: retrieval
const docs = await tracer.startActiveSpan('rag.retrieve', async (retrievalSpan) => {
const results = await vectorStore.search(question, { topK: 5 });
retrievalSpan.setAttributes({
'retrieval.count': results.length,
'retrieval.query': question,
});
retrievalSpan.end();
return results;
});
// Span: LLM call
const response = await tracer.startActiveSpan('llm.call', async (llmSpan) => {
const start = Date.now();
const result = await anthropic.messages.create({
model: 'claude-sonnet-4-6',
messages: [buildMessage(question, docs)],
max_tokens: 1024,
});
llmSpan.setAttributes({
'llm.model': result.model,
'llm.input_tokens': result.usage.input_tokens,
'llm.output_tokens': result.usage.output_tokens,
'llm.latency_ms': Date.now() - start,
'llm.cost_usd': calculateCost(result.usage),
'llm.stop_reason': result.stop_reason,
});
llmSpan.end();
return result.content[0].type === 'text' ? result.content[0].text : '';
});
span.setStatus({ code: SpanStatusCode.OK });
return response;
} catch (error) {
span.recordException(error as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw error;
} finally {
span.end();
}
});
}
LLM-as-Judge Eval Framework
// Evaluate answer quality: faithfulness (is it grounded in the context?)
interface EvalResult {
score: number; // 0-1
reasoning: string;
passed: boolean;
}
async function evaluateFaithfulness(
question: string,
context: string,
answer: string,
): Promise<EvalResult> {
const judgePrompt = `
You are an expert evaluator. Rate how faithful this answer is to the provided context.
QUESTION: ${question}
CONTEXT:
${context}
ANSWER: ${answer}
EVALUATION CRITERIA:
- Score 1.0: All claims in the answer are directly supported by the context
- Score 0.7: Most claims supported, minor extrapolation
- Score 0.4: Some claims unsupported or contradicted by context
- Score 0.0: Answer contradicts context or makes things up
Respond with JSON: {"score": 0.0-1.0, "reasoning": "1 sentence explanation"}`;
const response = await anthropic.messages.create({
model: 'claude-haiku-4-5-20251001', // fast, cheap for evals
messages: [{ role: 'user', content: judgePrompt }],
max_tokens: 256,
});
const text = response.content[0].type === 'text' ? response.content[0].text : '';
const result = JSON.parse(text.match(/\{.*\}/s)?.[0] ?? '{"score":0,"reasoning":"parse error"}');
return {
score: result.score,
reasoning: result.reasoning,
passed: result.score >= 0.7,
};
}
// Batch evaluation against golden dataset
async function runEvalSuite(goldenDataset: GoldenExample[]): Promise<EvalReport> {
const results = await Promise.all(
goldenDataset.map(async (example) => {
const answer = await answerQuestion(example.question, 'eval-user');
const faithfulness = await evaluateFaithfulness(
example.question, example.context, answer
);
const isCorrect = answer.toLowerCase().includes(
example.expectedKeywords.join('|').toLowerCase()
);
return { ...example, answer, faithfulness, isCorrect };
})
);
const avgFaithfulness = results.reduce((s, r) => s + r.faithfulness.score, 0) / results.length;
const accuracy = results.filter(r => r.isCorrect).length / results.length;
return {
totalExamples: goldenDataset.length,
avgFaithfulness,
accuracy,
failures: results.filter(r => !r.faithfulness.passed || !r.isCorrect),
};
}
Cost and Token Usage Monitoring
// Track cost per request — LLM costs add up quickly at scale
const PRICING = {
'claude-sonnet-4-6': { input: 3.00, output: 15.00 }, // per 1M tokens
'claude-haiku-4-5-20251001': { input: 0.25, output: 1.25 },
'gpt-4o': { input: 5.00, output: 15.00 },
} as const;
function calculateCost(model: string, inputTokens: number, outputTokens: number): number {
const pricing = PRICING[model as keyof typeof PRICING];
if (!pricing) return 0;
return (inputTokens / 1_000_000) * pricing.input
+ (outputTokens / 1_000_000) * pricing.output;
}
// Prometheus metrics for LLM usage
export function recordLLMCall(
model: string, inputTokens: number, outputTokens: number,
latencyMs: number, success: boolean
): void {
const cost = calculateCost(model, inputTokens, outputTokens);
llmTokensTotal.inc({ model, type: 'input' }, inputTokens);
llmTokensTotal.inc({ model, type: 'output' }, outputTokens);
llmCostDollarsTotal.inc({ model }, cost);
llmLatencyHistogram.observe({ model }, latencyMs / 1000);
llmCallsTotal.inc({ model, status: success ? 'success' : 'error' });
}
// Alert if daily cost > $100 or avg latency > 3s
Interview Preparation
Q: Why is evaluating LLM outputs harder than evaluating traditional software?
Traditional software has deterministic outputs — the same input always produces the same output, and you can assert equality. LLM outputs are probabilistic (different runs may produce different valid responses), open-ended (many correct phrasings exist), and semantic (correctness requires meaning, not string matching). You can't write assert response == expected_string. Evaluation requires either: rule-based checks (format, length, presence of required terms), reference-based scoring (BLEU/ROUGE — compares to human-written reference, imperfect), LLM-as-judge (use another LLM to evaluate quality — scalable but non-deterministic), or human evaluation (ground truth, expensive). Production AI systems typically use all four at different frequencies.
Q: What is a golden dataset and how do you build one?
A golden dataset is a curated collection of input-output pairs that represent the expected behavior of the system. Each example includes: a query/prompt, the relevant context (for RAG), the expected answer or key facts that should be present, and quality criteria. Build it by: (1) Sampling real production queries (diverse, representative), (2) Having domain experts write or validate expected answers, (3) Including edge cases and known failure modes, (4) Covering all major use cases (happy path, empty context, ambiguous queries). A good golden dataset has 100-500 examples. It's used to: baseline current performance, detect regressions before deploying prompt changes, and compare different model/prompt combinations.
Q: What is prompt regression testing and how does it work in CI/CD?
Prompt regression testing runs your eval suite before and after a prompt change to detect quality degradation. The flow: (1) Developer modifies a prompt. (2) CI runs the eval suite against the golden dataset using both old and new prompts. (3) Scores are compared: if faithfulness drops > 5%, accuracy drops > 3%, or any critical eval fails, the CI pipeline blocks the merge. (4) Results are logged to an eval tracking tool (LangSmith, Langfuse) for trend analysis. This is analogous to unit tests for code, but for prompt behavior. The challenge: evals are probabilistic — run 3+ times and average scores to reduce variance, and set thresholds that account for natural variation.
Learning Resources
- LangSmith — LLM tracing and eval platform
- Langfuse — open-source LLM observability
- Arize Phoenix — open-source AI/LLM observability
- RAGAS — RAG evaluation framework
- Anthropic Evals — eval methodology from Anthropic