Sample case study. It shows the structure every case study should follow; the client, numbers and details are illustrative. Replace it with a real project.
AI / LLM · RAG
RAG support assistant for a SaaS help desk
Cut average support resolution time by 40% with an assistant that answers from 12,000 help articles and cites every source.
- Client
- B2B SaaS, 200 staff
- My role
- Lead architect and engineer
- Duration
- 8 weeks
- Stack
- Python, FastAPI, Postgres + pgvector, Redis, Langfuse
Context
The support team answered about 3,000 tickets a week, and most answers already existed somewhere in the help centre. Agents spent much of their time searching instead of solving.
Problem and constraints
- Answers had to be traceable to a source article; no invented steps.
- Customer data could not leave the client’s cloud region.
- Median response time under 3 seconds, and a fixed monthly LLM budget.
- The help centre changes daily, so the index had to stay fresh.
Architecture
A single assistant API sits between the help widget and the model. It owns guardrails, caching and retrieval, so the model provider can change without touching the product.
The request path
- The widget sends the question with the customer’s product plan.
- The API checks a semantic cache; about a third of questions are near-duplicates.
- On a miss it retrieves the top 8 chunks, filtered by plan, from the vector index.
- The model answers using only those chunks and must cite each one.
- A post-check drops any answer whose citations do not support it, and hands over to a human.
Key decisions
ADR-001
Postgres + pgvector over a managed vector database
- Options considered
- Managed vector DB; Elasticsearch; Postgres + pgvector
- Decision
- Postgres + pgvector in the existing cluster.
- Why
- Data stays in-region with the client’s existing backups and access controls. At 12k articles, performance was not a constraint.
ADR-002
Citation check before every answer
- Options considered
- Trust the model; LLM-as-judge on a sample; check every answer
- Decision
- A cheap second pass verifies each citation supports the claim.
- Why
- One wrong answer costs more trust than a slower one. The check adds about 300 ms and removed most unsupported answers in testing.
ADR-003
Evaluation set before the first prompt
- Options considered
- Manual spot checks; user feedback only; a fixed eval set
- Decision
- 250 real questions with expected sources, run on every change.
- Why
- Prompt and model changes are measured, not argued about. It also made a later model switch a one-day task.
Results
| Measure | Before | After |
|---|---|---|
| Median resolution time | 11 h | 6.5 h |
| Tickets solved without an agent | 0% | 22% |
| Answers with a valid citation | n/a | 97% |
| LLM cost per answered ticket | n/a | $0.012 |
Lessons and what I would change
The evaluation set was the most valuable artefact of the project; I would build it in week one again. I would also add plan-aware retrieval earlier, since most early wrong answers came from articles for a different product tier.
[Client quote about this project.]