RAG Corpus Curator & Deduper Prompt

Curate the corpus below for RAG indexing.
Steps:
1. Dedupe near-identical documents (state the similarity threshold).
2. Detect low-quality docs (boilerplate, nav, scraped UI text) and mark for removal.
3. Propose a chunking strategy (by section, by fixed size, or by semantic boundary) with the chunk size.
4. State which docs need metadata (title, date, source) for filtering before retrieval.

Corpus (file list + sizes):
{{corpus}}

Replace the {{fields}} with your own context and tighten the rules to match your domain.