Get retrieval right before agent orchestration
Wrapping ChatGPT on a drive is not RAG. An internal knowledge MVP must answer inside ACL with citations—and refuse when unsure. We spend discovery on corpus and evals, not vendor logos. Below: six-week scope covering questions, docs, chunking, authz, measurement. Fold this into KSX Studio discover→design→build→launch: named owner, Definition of Done, staging proof, and rollback so verbal alignment does not drift.
Pick one work-ticket scenario
Examples: RFP answers, policy lookup, runbooks. Narrow scenarios make corpus and evals tractable. Document roles and refuse zones. Fold this into KSX discover→design→build→launch checks on staging so verbal alignment does not drop the requirement. Fold this into KSX Studio discover→design→build→launch: named owner, Definition of Done, staging proof, and rollback so verbal alignment does not drift.
Corpus inventory and freshness
List sources, refresh cadence, owners. Ownerless docs stay out. Define delete/expiry policies. Fold this into KSX discover→design→build→launch checks on staging so verbal alignment does not drop the requirement. Fold this into KSX Studio discover→design→build→launch: named owner, Definition of Done, staging proof, and rollback so verbal alignment does not drift.
Chunking and metadata beat model brand
Chunk by headings and table integrity; keep path, ACL, updated-at. Test bilingual corpora. Start hybrid search before complex rerankers. Fold this into KSX discover→design→build→launch checks on staging so verbal alignment does not drop the requirement. Fold this into KSX Studio discover→design→build→launch: named owner, Definition of Done, staging proof, and rollback so verbal alignment does not drift.
ACL at retrieval
Inject user-visible spaces into retrieval—not post filters. Redact logs. Treat cross-ACL answers as security tests. Fold this into KSX discover→design→build→launch checks on staging so verbal alignment does not drop the requirement. Fold this into KSX Studio discover→design→build→launch: named owner, Definition of Done, staging proof, and rollback so verbal alignment does not drift.
Answers must cite sources
Show sources and links; refuse on no-hit. Block parametric invention of policy. Citation CTR is a health metric. Fold this into KSX discover→design→build→launch checks on staging so verbal alignment does not drop the requirement. Fold this into KSX Studio discover→design→build→launch: named owner, Definition of Done, staging proof, and rollback so verbal alignment does not drift.
Eval set: launch gate
50–100 real questions with gold sources. Regress on embedding/prompt/chunk changes. RAG without evals is theater. Fold this into KSX discover→design→build→launch checks on staging so verbal alignment does not drop the requirement. Fold this into KSX Studio discover→design→build→launch: named owner, Definition of Done, staging proof, and rollback so verbal alignment does not drift.
Six-week cadence
W1 scenario/corpus; W2 pipeline/ACL; W3 retrieval; W4 cite/refuse UI; W5 evals/security; W6 pilot. Expand corpus after success. Fold this into KSX discover→design→build→launch checks on staging so verbal alignment does not drop the requirement. Fold this into KSX Studio discover→design→build→launch: named owner, Definition of Done, staging proof, and rollback so verbal alignment does not drift.
Checklist
- 1Lock one scenario and refuse boundaries
- 2Corpus owners + refresh table
- 3Mandatory ACL at retrieval
- 4UI citations + no-hit refusal
- 5≥50-item eval set in CI
Key takeaways
- RAG MVPs are ACL’d retrieval and citations.
- Eval sets equal functional tests as launch gates.
- Narrow pilots before corpus expansion.
FAQ
- Vector DB mandatory?
- Small corpora can prove value on managed search first; choose infra later.
- Whole company drive?
- Not for MVP. Whitelist directories per scenario.
- Open-source or API?
- Depends on residency, cost, latency. Compare on evals. Generators swap easier than retrieval quality.