Case study
Making role search accurate with RAG
Candidates searching for roles got results that were only roughly aligned with what they searched for. I led the team that fixed it, and I built it hands-on alongside them.
- internal documents chunked and indexed
- 400
- gold standard searches with expected results
- 300
- the index behind every retrieved passage
- Doc · page · line
01 The problem
Close was not close enough
Candidates searching for roles got results that were only roughly aligned with what they searched for.
02 What I built
Retrieval grounded to the line
I led the team that built a retrieval-augmented generation pipeline on AWS Bedrock, and I built it hands-on alongside them. We split 400 internal documents into chunks and indexed every chunk by document, page and line. When a candidate searches, the LLM receives a prompt grounded in the exact passages that matter, and each passage traces back to its source.
- 01 Documents 400 internal documents
- 02 Chunks indexed by document, page and line
- 03 Retrieval the passages that match the search
- 04 LLM prompt grounded in those exact passages
- 05 Results close matches for the candidate
Test loop Every change ran against 300 gold standard searches.
03 How we proved it worked
A test set, not a gut check
I created a gold standard set of 300 searches, each with its expected results. Every change ran against that set, so we tested result quality reliably instead of judging it by eye.
04 The result
From roughly aligned to closely matched
Results went from roughly aligned with candidate searches to closely matching what candidates were looking for. The 300 gold standard searches let us confirm that on every change.
Building search or RAG that has to be right?
I am open to discussing engineering leadership opportunities.