RAG Basics That Actually Move the Needle

Retrieval-augmented generation fails quietly. The model still writes a confident paragraph; it just writes it from the wrong three chunks. Before you touch the prompt, measure whether the right context ever arrived.

Chunk on structure, not character count

Fixed 500-character windows cut tables in half and orphan headings from their content. Split on document structure — headings, sections, list groups — then merge small neighbours up to a target size. Carry the heading path into each chunk's text so a section titled "Refunds" is still findable when the body never repeats the word.

Hybrid search beats pure vectors

Embeddings capture meaning and miss exact tokens: product codes, error strings, surnames, version numbers. Keyword search does the opposite. Run both, fuse the ranks, and you recover the queries each method drops on its own.

score = 1 / (60 + rank_vector) + 1 / (60 + rank_keyword)

Reciprocal rank fusion needs no tuning and no shared score scale, which makes it the right default before you invest in anything learned.

Rerank the top 50 down to five

First-stage retrieval optimises for recall. A cross-encoder reranker reads the query and candidate together and reorders them for precision. This is usually the single largest quality jump available, and it is the step most teams skip.

Measure retrieval separately

1. 1Collect 50 real questions with the passage that answers each one.
2. 2Track recall@k for the first stage — is the answer anywhere in the top k?
3. 3Track precision after reranking — is it in the top three?
4. 4Only debug the generation prompt once retrieval clears both bars.

Ground the generation step

• Number the chunks and require inline citations to those numbers.
• Instruct the model to answer "not in the provided context" rather than infer.
• Show retrieved sources in the UI so users can verify without trusting.
• Log query, retrieved IDs, and answer together for every request.

A RAG system without retrieval metrics is a search engine you cannot debug wearing a chatbot costume.