Building Production RAG Systems: Hard-Won Lessons from 1200 Hours of Enterprise Development

After spending 1200+ hours building enterprise RAG systems—from clones of GleanAI and ChatGPT Enterprise to custom internal tools—I've learned one brutal truth: most RAG implementations that work beautifully in demos fail catastrophically in production.

The gap between academic benchmarks and messy enterprise reality is a chasm. While papers promise 50% retrieval improvements on clean datasets, production data is chaotic, user queries are terrible, and the "one weird trick" vendors sell you often creates more problems than it solves.

Here's what actually works after burning months of engineering time on approaches that didn't.

Enterprise data is a mess. You've got PDFs from 2012, PowerPoints with embedded charts, Notion pages with nested databases, and Confluence spaces that haven't been organized since the Bush administration. Before your RAG system can retrieve anything useful, you need to normalize this chaos.

The Markdown Standard

Base models—whether GPT-4, Claude, or open-source alternatives—process markdown exceptionally well. It's plain text with just enough structural hints (headers, lists, code blocks) to convey hierarchy without the formatting noise of raw PDFs or Word docs.

My pipeline converts everything to GitHub Flavored Markdown (GFM):

• PDFs → text extraction with layout preservation
• Office docs → structured markdown with table support
• Images → OCR to markdown (for diagrams with text)
• Notion/Confluence → API export to markdown

Late Chunking: Why Timing Matters

Traditional chunking cuts documents into arbitrary 512-token pieces, then embeds each piece independently. The problem? A sentence at the end of chunk 1 and a sentence at the start of chunk 2 might be semantically inseparable, but your embedding model sees them as unrelated.

Late chunking flips the order:

1. Embed larger semantic units (full sections or pages)
2. Use the embedding to guide intelligent chunk boundaries
3. Preserve paragraph and sentence integrity

Think of it like tearing a map. Traditional chunking rips randomly across streets and landmarks. Late chunking finds the seams—rivers, district boundaries—and tears there. Both give you pieces, but one is actually useful for navigation.

Users are terrible at searching. They ask "why is the login broken?" when the documentation discusses "OAuth2 token expiration workflows." Semantic search fails here because the vocabulary mismatch is severe.

Hypothetical Document Embeddings (HyDE) solves this by bridging the vocabulary gap:

# Traditional search: embed the raw query (fails)
query_embedding = embed("why is the login broken?")

# HyDE: generate hypothetical answer, then embed that
hypothetical_answer = llm.generate(
    "Answer this question with technical details: why is the login broken?"
)
# hypothetical_answer now contains "OAuth2 token expiration..."
hyde_embedding = embed(hypothetical_answer)

The LLM expands the user's vague complaint into technical terminology that matches your documentation. Now your semantic search finds the OAuth2 troubleshooting guide instead of returning random login-related pages.

Implementation in Production:

async def retrieve_with_hyde(query: str, top_k: int = 5):
    # Generate hypothetical document
    hyde_prompt = f"""Answer this question using technical terminology 
    that would appear in documentation: {query}"""
    
    hypothetical_doc = await llm.complete(hyde_prompt, max_tokens=200)
    
    # Embed the hypothetical answer, not the raw query
    hyde_embedding = embedding_model.encode(hypothetical_doc)
    
    # Two-stage hierarchical retrieval
    # Stage 1: Find relevant documents using summary embeddings
    doc_candidates = await db.query("""
        SELECT document_id, summary_embedding <-> $1 as distance
        FROM document_summaries
        ORDER BY distance
        LIMIT 20
    """, [hyde_embedding])
    
    # Stage 2: Search chunks within candidate documents
    chunks = await db.query("""
        SELECT content, embedding <-> $1 as semantic_score
        FROM document_chunks
        WHERE document_id = ANY($2)
        ORDER BY semantic_score
        LIMIT $3
    """, [hyde_embedding, [d.id for d in doc_candidates], top_k])
    
    return chunks

The cost? One extra LLM call per query. The benefit? 34% improvement in retrieval relevance in my production tests.

Academic papers love reranking. You retrieve 100 chunks with a fast bi-encoder, then use a slow but accurate cross-encoder to rerank them and pick the top 5. In theory, you get the speed of approximate search with the accuracy of exact comparison.

Production reality: On messy enterprise data with inconsistent formatting, partial matches, and domain-specific terminology, reranking gave me a 7% accuracy boost at the cost of 400ms additional latency. For a real-time chat interface, that's unacceptable.

The techniques that matter more than reranking:

• Better chunking (late chunking, semantic boundaries)
• Query expansion (HyDE)
• Metadata filtering (date ranges, document types)
• Hybrid keyword + semantic scoring

Skip the fancy reranker. Invest in your ingestion pipeline and query preprocessing instead.

Before you launch your RAG system to real users, verify:

• [ ] Late chunking with semantic boundary detection (not fixed token counts)
• [ ] HyDE query expansion for bridging vocabulary gaps
• [ ] Hierarchical retrieval (document → chunk two-stage search)
• [ ] Markdown normalization from heterogeneous data sources
• [ ] Hybrid scoring combining keyword BM25 + semantic embeddings
• [ ] Metadata filtering by date, source, document type
• [ ] Query preprocessing handling typos, acronyms, and domain terms
• [ ] Fallback strategies when retrieval confidence is low
• [ ] Evaluation on real user queries, not just clean test sets

RAG isn't magic. It's a data pipeline problem disguised as an AI problem. Get your ingestion right, implement hierarchical retrieval with HyDE, and ignore the vendors promising "one weird trick" solutions. Your users—and your sanity—will thank you.