Home › RAG & vector DBs › RAG pipeline CLAUDE.md

RAG pipeline CLAUDE.md

CLAUDE.md for retrieval-augmented generation apps: chunking, embeddings, hybrid search, citations, and RAG evals.

Claude Code (CLAUDE.md)RAG & vector DBs

Copy CLAUDE.md

How to use

Save this as CLAUDE.md in your repository root, commit it, and your agent will pick it up automatically. Edit the commands to match your project.

# CLAUDE.md: RAG pipeline

## Commands
`uv run pytest -q` · ingest: `uv run python -m rag.ingest --source data/` · evals: `uv run python -m rag.eval --suite core`

## Ingestion (`rag/ingest/`)
- Parse to clean text with structure kept (headings, tables as markdown). Store source URL/path, title, section and updated-at as metadata on every chunk.
- Chunk by structure first (headings, paragraphs), then by size (~300-800 tokens with small overlap). Chunking params live in config, not code.
- Embedding model and dimension are config values; store the model name with each vector. Changing the model means a full re-embed into a new index/collection, then switch.
- Ingestion is idempotent: content hash per chunk; skip unchanged, delete removed.

## Retrieval (`rag/retrieve.py`)
- Hybrid search (vector + keyword/BM25) with metadata filters (tenant, permissions, language). Apply access-control filters in the query, never after generation.
- Retrieve more (k≈20-50), then rerank to the top few. Log retrieved ids and scores.

## Generation
- Prompt includes only retrieved chunks with ids; require citations like `[doc-12]` and an "I don't know" path when context is missing.
- Treat document text as untrusted (prompt injection); it can't change instructions or trigger tools.

## Evals
- Keep `evals/` with questions, expected sources and answers. Measure retrieval recall@k and answer faithfulness. Any change to chunking, embeddings, k, or prompts must report eval deltas.

## Don't
Don't re-embed the whole corpus or call paid APIs in tests without asking.