Overview
RAG grounds LLM responses in your private data. Without it, the model can only answer from training knowledge (stale) and hallucinate facts it doesn't know. With RAG, you retrieve relevant documents at query time and inject them into the prompt as context.
Two-phase system:
- Indexing (offline) — chunk documents, embed each chunk, store vectors in a vector DB
- Retrieval (online, per query) — embed the query, search the vector DB for nearest chunks, assemble context, call LLM
Architecture
━━━━━━━━━━━━━━━━━ INDEXING PIPELINE (offline) ━━━━━━━━━━━━━━━━━
┌──────────────┐ ┌────────────┐ ┌─────────────────────┐
│ Raw Sources │ │ Chunker │ │ Embedding Model │
│ │───▶│ │───▶│ (text-embed-3- │
│ - S3 docs │ │ 512 token │ │ small / Cohere) │
│ - Confluence│ │ chunks, │ │ │
│ - Web crawl │ │ 20% overlap│ │ chunk → float[1536]│
└──────────────┘ └────────────┘ └──────────┬──────────┘
│
▼
┌───────────────────────┐
│ Vector Store │
│ (Pinecone / pgvector │
│ / OpenSearch) │
│ │
│ id | vector | metadata│
│ ---|--------|-------- │
│ 1 | [0.2,] | {doc, │
│ | ... │ page, │
│ | │ chunk}│
└───────────────────────┘
━━━━━━━━━━━━━━━━━ QUERY PIPELINE (online) ━━━━━━━━━━━━━━━━━━━
User Query
│
▼
┌──────────────┐
│Query Rewriter│ (optional — expand abbreviations, decompose
│ │ multi-part questions into sub-queries)
└──────┬───────┘
│
▼
┌──────────────┐ ┌───────────────────────┐
│ Embed query │ │ Vector Store │
│ (same model │───▶│ ANN search k=5 │
│ as indexing│ │ cosine similarity │
│ pipeline) │ │ returns top-k chunks │
└──────────────┘ └──────────┬────────────┘
│
▼
┌──────────────────────┐
│ Reranker (optional) │
│ cross-encoder model │
│ reorders k results │
│ by relevance │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ Context Assembler │
│ - deduplicate │
│ - token budget mgmt │
│ - citation tracking │
└──────────┬───────────┘
│
┌───────────▼──────────┐
│ Prompt Builder │
│ system: "Answer │
│ based on CONTEXT" │
│ context: <chunks> │
│ user: <query> │
└───────────┬──────────┘
│
▼
┌────────────┐
│ LLM │
│ (Claude / │
│ GPT-4o) │
└─────┬──────┘
│
▼
┌──────────────────────┐
│ Response + Citations│
│ [1] Source: doc.pdf │
│ [2] Source: wiki/X │
└──────────────────────┘
Technical Implementation
Document Chunking Strategy
Chunking is the most impactful tuning knob in RAG. Bad chunks = bad retrieval.
// src/indexing/chunker.ts
export class DocumentChunker {
// Recursive character text splitter — tries to split on \n\n, then \n, then ' '
static chunk(text: string, options = { chunkSize: 512, overlap: 100 }): string[] {
const { chunkSize, overlap } = options;
const separators = ['\n\n', '\n', '. ', ' ', ''];
return this.splitRecursively(text, separators, chunkSize, overlap);
}
// For code files — split by function/class boundaries, not character count
static chunkCode(code: string, language: 'java' | 'typescript'): string[] {
const functionPattern = language === 'java'
? /(?:public|private|protected)\s+(?:static\s+)?\w+\s+\w+\s*\(/g
: /(?:function|const|async function)\s+\w+\s*[=(]/g;
// Split at function boundaries, preserve context
return this.splitAtBoundaries(code, functionPattern, 1024);
}
// Sentence-aware chunking — never cut mid-sentence
static chunkSentences(text: string, maxChunk = 512): string[] {
const sentences = text.match(/[^.!?]+[.!?]+/g) ?? [text];
const chunks: string[] = [];
let current = '';
for (const sentence of sentences) {
if ((current + sentence).length > maxChunk && current) {
chunks.push(current.trim());
current = sentence;
} else {
current += sentence;
}
}
if (current) chunks.push(current.trim());
return chunks;
}
}
Embedding + Indexing Pipeline (Node.js)
// src/indexing/pipeline.ts
import { OpenAI } from 'openai';
import { Pinecone } from '@pinecone-database/pinecone';
const openai = new OpenAI();
const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
export async function indexDocument(doc: RawDocument): Promise<void> {
const index = pinecone.index('knowledge-base');
// 1. Chunk
const chunks = DocumentChunker.chunkSentences(doc.content, 512);
// 2. Embed in batches (rate limit: 2048 inputs per request for text-embedding-3-small)
const batchSize = 100;
for (let i = 0; i < chunks.length; i += batchSize) {
const batch = chunks.slice(i, i + batchSize);
const embeddingResp = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: batch,
dimensions: 1536,
});
// 3. Upsert vectors with metadata
const vectors = batch.map((chunk, j) => ({
id: `${doc.id}-chunk-${i + j}`,
values: embeddingResp.data[j].embedding,
metadata: {
docId: doc.id,
source: doc.source,
chunkIndex: i + j,
text: chunk, // store raw text for retrieval (no re-fetch needed)
updatedAt: doc.updatedAt,
},
}));
await index.upsert(vectors);
}
}
Query Pipeline — Full RAG Flow
// src/retrieval/rag.service.ts
export class RagService {
async query(userQuery: string, tenantId: string): Promise<RagResponse> {
// 1. Embed the query (same model as indexing — critical)
const queryEmbedding = await this.embed(userQuery);
// 2. Vector search — top 8 candidates
const candidates = await this.vectorSearch(queryEmbedding, { topK: 8, tenantId });
// 3. Rerank — optional but improves precision significantly
const reranked = await this.rerank(userQuery, candidates);
const topK = reranked.slice(0, 4); // take top 4 after reranking
// 4. Assemble context within token budget
const context = this.assembleContext(topK, maxTokens: 3000);
// 5. Build prompt
const prompt = this.buildPrompt(userQuery, context);
// 6. Call LLM
const response = await this.llm.complete(prompt);
// 7. Return with citations
return {
answer: response.content,
citations: topK.map((c) => ({ source: c.metadata.source, text: c.metadata.text })),
};
}
private assembleContext(chunks: VectorResult[], maxTokens: number): string {
let tokenCount = 0;
const selected: string[] = [];
for (const chunk of chunks) {
const chunkTokens = Math.ceil(chunk.metadata.text.length / 4); // rough estimate
if (tokenCount + chunkTokens > maxTokens) break;
selected.push(`[${selected.length + 1}] ${chunk.metadata.text}`);
tokenCount += chunkTokens;
}
return selected.join('\n\n---\n\n');
}
private buildPrompt(query: string, context: string): string {
return `You are a helpful assistant. Answer the question based ONLY on the provided context.
If the answer is not in the context, say "I don't have information about that."
Do not use prior knowledge. Cite sources using [1], [2], etc.
CONTEXT:
${context}
QUESTION: ${query}
ANSWER:`;
}
}
pgvector with Spring / Java
// For teams already on PostgreSQL — pgvector avoids a separate vector DB
@Repository
public class VectorSearchRepository {
private final JdbcTemplate jdbcTemplate;
// Store chunk embeddings in pg
public void upsertEmbedding(String id, float[] embedding, String text, String source) {
jdbcTemplate.update("""
INSERT INTO embeddings (id, embedding, text, source, updated_at)
VALUES (?, ?::vector, ?, ?, NOW())
ON CONFLICT (id) DO UPDATE
SET embedding = EXCLUDED.embedding, text = EXCLUDED.text
""",
id, Arrays.toString(embedding), text, source
);
}
// ANN search using pgvector cosine distance (<=>)
public List<ChunkResult> searchSimilar(float[] queryEmbedding, int topK, String tenantId) {
return jdbcTemplate.query("""
SELECT id, text, source, 1 - (embedding <=> ?::vector) AS similarity
FROM embeddings
WHERE tenant_id = ?
ORDER BY embedding <=> ?::vector
LIMIT ?
""",
(rs, i) -> new ChunkResult(rs.getString("id"), rs.getString("text"),
rs.getString("source"), rs.getDouble("similarity")),
Arrays.toString(queryEmbedding), tenantId,
Arrays.toString(queryEmbedding), topK
);
}
}
Interview Preparation
Q: Walk me through a RAG pipeline end to end.
Two phases: offline indexing (chunk → embed → store in vector DB) and online retrieval (embed query → ANN search → rerank → assemble context → LLM call → response with citations). The key insight: embedding model must be identical in both phases — you're measuring cosine similarity between vectors from the same embedding space.
Q: How do you choose chunk size?
Depends on query type:
- Short factual queries ("What is the refund policy?") → small chunks (256–512 tokens), high precision
- Summarization / long-form → larger chunks (1024 tokens), more context per chunk
- Code → split at function/class boundaries, not character count
Always overlap chunks (10–20%) to avoid cutting mid-sentence. Evaluate with a test set — measure retrieval recall before tuning LLM prompt.
Q: What is a reranker and when do you need one?
ANN search ranks by embedding similarity (fast, approximate). A reranker is a cross-encoder model that takes (query, chunk) pairs and scores relevance more precisely — but is O(k) LLM calls. Use it when:
- Top-k candidates are good but in wrong order
- Queries are longer / more nuanced
- You can afford 50–200ms extra latency
Common rerankers: Cohere Rerank, cross-encoder/ms-marco-MiniLM-L-6-v2 (open source).
Q: How do you handle stale/updated documents?
Delete-and-reindex on content change. Track updatedAt in vector metadata. A background job compares document checksums against indexed versions, triggers re-chunking and re-embedding for changed docs.
Q: RAG vs Fine-tuning — which and when?
Use RAG when the information changes (product docs, policies, real-time data) or is proprietary. Use fine-tuning when you need the model to change how it responds (tone, structured format, domain vocabulary) not what it knows.
Learning Resources
- Pinecone Learning Center — RAG, embeddings, vector search
- LlamaIndex Docs — Python RAG framework, great for understanding patterns
- pgvector GitHub — Postgres vector extension
- Cohere Rerank API — reranking service
- OpenAI Embeddings Guide — text-embedding-3 models
- BEIR Benchmark — evaluate retrieval quality
- Designing Machine Learning Systems — Chip Huyen (O'Reilly) — ch. on feature stores / serving