I built a full RAG pipeline. Here is what actually matters

Everyone has a RAG demo. Load some PDFs, embed them, ask a question, get an answer. It works on the first try and breaks on the fifth. The gap between that demo and something a company would put in front of employees is where all the interesting engineering lives.

So I built the whole thing for an IT service desk assistant, end to end, on about 16,000 resolved IT support tickets from Kaggle. Every stage is measured, and CI blocks any change that makes quality worse. The code is open: github.com/madmecodes/rag-pipeline-project.

This post walks through every step: what it does, why it exists, and how it is built. One running example throughout: Riya, in the support group, asks “My VPN keeps disconnecting, how do I fix it?”

01 What RAG is, in one paragraph

An LLM learned from the public internet. It has never seen your company's internal fixes, policies, or past tickets, so when asked about them it guesses, and a confident guess is a hallucination. Retrieval-Augmented Generation fixes this in three moves: retrieve the few relevant documents, augment the prompt by pasting them in, and generate an answer that may only use those documents, with citations.

Think of an open-book exam. The LLM is a clever student who never read your company's book. RAG hands them the right pages right before they answer, and makes them point to the page for every claim.

02 The whole pipeline on one screen

Four parts: build the knowledge once (A), answer every question (B), prove it works (C), and ship it (D). Arrows marked <-- show where an encoder or the LLM runs.

======================================================================
                 PART A: OFFLINE - BUILD THE KNOWLEDGE
               (python -m rag.ingest | rag/ingest.py)
======================================================================

  Kaggle CSV: 28k IT tickets (EN + DE)
               |
               v
  [A1] LOAD ............ pandas, keep English, drop empty -> ~16k tickets
               |
               v
  [A2] CLEAN ........... regex: fix "\n", extra spaces
               |
               v
  [A3] MASK PII ........ regex: email/phone/card/IP -> <email> <phone>
               |
               v
  [A4] BUILD DOCUMENT .. "Issue: ... Resolution: ..."
               |           + metadata (queue, type, priority, tags)
               |           + ACL (HR -> [hr], Billing -> [finance,support],
               |                  rest -> [everyone])
               |           + doc_id + content_hash
               v
  [A5] INCREMENTAL CHECK  same hash -> skip
               |           changed  -> delete old chunks, redo
               |           removed  -> delete
               v
  [A6] CHUNK ........... split: paragraph > line > sentence > word
               |           1200 chars, 200 overlap
               |           + header "[IT Support / Incident] VPN down"
               v
        +------+-------+
        v              v
  [A7] DENSE       [A8] SPARSE              <-- ENCODERS
  bge-small-en     BM25 (Qdrant/bm25)
  bi-encoder       {word: weight}
  384 numbers      = exact keywords
  = meaning
        |              |
        +------+-------+
               v
  [A9] STORE IN QDRANT  (16,641 chunks)
        dense vector + sparse vector + payload
        (acl, queue, chunk_text, parent_text)
        HNSW index, int8 quantization, IDF computed live


======================================================================
                 PART B: ONLINE - ANSWER A QUESTION
          (POST /ask | rag/api.py -> rag/pipeline.py)
======================================================================

  Riya: "My VPN keeps disconnecting"   +  groups: [support, everyone]
               |
               v
  [B1] INPUT GUARD ........ rag/guard.py
               |   injection regex match? ---------> 400 BLOCKED
               |   length check, mask PII
               v
  [B2] REWRITE ............ rag/generate.py (gpt-4.1, only if history)
               |   "what about Mac?" -> "VPN disconnecting on Mac"
               v
  [B3] SEMANTIC CACHE ..... rag/cache.py          <-- bi-encoder
               |   similar question, same groups? -> return cached answer
               v (miss)
  [B4] HYBRID SEARCH ...... rag/retrieve.py (inside Qdrant)
               |   ACL filter applied INSIDE the search
        +------+-------+
        v              v
   dense top 40    BM25 top 40                <-- bi-encoder + sparse
        +------+-------+
               v
          RRF FUSION -> top 30       (merge by rank position)
               |
               v
  [B5] RERANK ............. ms-marco-MiniLM   <-- CROSS-ENCODER
               |   reads question + chunk together -> score
               |   keep top 5, one per ticket, return full ticket
               |
               |   (future: LightGBM learning-to-rank would sit here,
               |    combining BM25 / dense / rerank scores + clicks)
               v
  [B6] CONFIDENCE GATE .... rag/pipeline.py
               |   best score < -4.0? ---> "I don't know" + escalate
               |                           (no LLM call, no cost)
               v
  [B7] GENERATE ........... rag/generate.py (Azure Foundry gpt-4.1)
               |   sources stripped of injection lines      <-- DECODER
               |   "answer ONLY from sources, cite [1]..[5]"
               v
  [B8] OUTPUT GUARD ....... rag/guard.py
               |   no citation / fake citation? ---> escalate
               v
  [B9] RESPOND + RECORD
          answer + sources + escalate flag + trace_id
          |-> trace    -> logs/traces.jsonl  (time per step, docs, tokens)
          |-> metrics  -> GET /metrics       (Prometheus counters)
          |-> cache    <- store good answers
          |-> feedback <- POST /feedback     (thumbs up/down)


======================================================================
                 PART C: EVALUATE - PROVE IT WORKS
                       (eval/ | make golden eval)
======================================================================

  [C1] GOLDEN SET .......... gpt-4.1 writes 145 user questions,
               |              each linked to its correct ticket
               v
  [C2] RETRIEVAL TEST ...... recall@5 / MRR
               |    BM25 0.46 | Dense 0.51 | Hybrid 0.50 | +Rerank 0.58
               v
  [C3] SECURITY TEST ....... HR questions asked as "everyone" -> 0 leaks
               v
  [C4] ANSWER TEST ......... LLM judge: faithfulness 1.0, grounded 1.0
               v
  [C5] GATE ................ below eval/thresholds.json? -> build FAILS


======================================================================
                 PART D: SHIP AND RUN
======================================================================

  git push -> GitHub Actions: unit tests (12) -> rebuild index -> eval gate
                                   |
                                   v
  docker compose:  [ API :8000 (FastAPI, models baked in) ]
                          |                    |
                          v                    v
                   [ Qdrant :6333 ]     [ Azure gpt-4.1 ]

  Endpoints: /ask  /feedback  /healthz  /metrics  /admin/reindex

The same thing in one line:

Tickets  -> clean -> chunk -> encode (dense + BM25) -> Qdrant
Question -> guard -> rewrite -> cache -> hybrid search -> RRF -> rerank
         -> confidence gate -> LLM with citations -> check -> answer + trace

03 Encoders: the models that turn text into numbers

Computers cannot compare meaning directly. An encoder reads text and outputs numbers that represent it. Almost everything about search quality comes down to which encoder runs where.

Modern encoders are transformers, the same family as GPT, but built to read and summarise rather than write. GPT is a decoder: it produces the next word. BERT-style models are encoders: they produce one representation of the whole input. Here is what happens inside:

"VPN keeps disconnecting"
   |
1. Tokenize      ["[CLS]", "vp", "##n", "keeps", "disconnect", "##ing", "[SEP]"]
   |             (word pieces from a fixed 30,000-entry vocabulary)
2. Look up       each piece gets a starting list of numbers from a learned table
   |
3. Transformer   12 layers; in each one every word "looks at" every other word
   layers        and adjusts its numbers by context ("bank" + "river" vs "bank" + "loan")
   |
4. Pool          the [CLS] slot has absorbed the whole sentence; take it
   |
5. Normalize     scale to length 1 so comparisons are fair
   |
[0.21, -0.53, 0.88, ...]   384 numbers

It learned this through contrastive learning: hundreds of millions of text pairs that belong together, such as a question and its answer, pulled close; unrelated pairs pushed apart. Afterwards “VPN disconnects” and “connection keeps dropping” land near each other with no shared words. The steps are fully known; what each of the 384 numbers means is not human-readable. That is why we measure it.

Why both a bi-encoder and a cross-encoder? The bi-encoder is cheap enough to search all 16,641 chunks in milliseconds but shallow. The cross-encoder is accurate but would take minutes over everything. So the bi-encoder narrows 16,641 to 30, and the cross-encoder orders those 30 and keeps 5. That is two-stage retrieval, and almost every production search system works this way.

04 Part A: building the knowledge

Runs with python -m rag.ingest. Nothing here is manual, and it only re-processes what changed.

A1 to A3: load, clean, mask PII

pandas loads the CSV, keeps English rows (the encoder is English only), and drops tickets without a resolution: 28,587 rows become about 16k usable tickets. Regex rules fix line breaks and spacing, because chunking splits on paragraphs and dirty breaks make bad chunks. Then regex masks PII: "call 98765 43210" becomes "call <phone>", and the same for emails, cards, and IPs, so personal data is never indexed or sent to an LLM provider. Regex misses names and addresses; production adds an ML detector like Microsoft Presidio.

A4: build the document, with permissions

Each ticket becomes "Issue: ... Resolution: ..." plus metadata, a stable doc_id, a content_hash, and an ACL: HR tickets visible only to hr, billing to finance and support, the rest to everyone. In a real company these are copied from the source system's own permissions.

A5: incremental sync

Compare each document's hash with what is indexed. Same: skip. Changed: delete the old chunks and re-embed. Gone from the source: delete. Forgetting deletions is a real data leak, because revoked content stays searchable. The second run here embedded 0 documents in 1.8 seconds.

A6: chunking

Pieces of at most 1,200 characters with 200 characters of overlap, split recursively by paragraph, then line, then sentence, then word. Small pieces keep search precise and prompts short; overlap stops a fact being cut in half. Each chunk also gets a header like [IT Support / Incident] VPN down, a cheap form of contextual retrieval, so even a small piece knows what it is about. Result: 16,641 chunks.

A7 and A8: two encodings per chunk

Dense: bge-small turns each chunk into 384 numbers capturing meaning, locally, with no API cost and no data leaving the machine. Sparse: BM25 turns it into {word: weight}.

BM25 scores a chunk as the sum, over query words, of IDF times TF, adjusted for chunk length. IDF is how rare the word is across all documents: “VPN” counts a lot, “please” almost nothing. TF is how often it appears in this chunk, with diminishing returns. Embeddings are fuzzy on exact strings; for error 0x80070005, BM25 finds the ticket containing exactly that code. Qdrant computes IDF at query time, so it stays correct as documents change.

A9: the vector database

Each chunk becomes one point in Qdrant: a dense vector, a sparse vector, and a payload with the ACL, queue, chunk text, and full parent ticket. A vector database answers “which stored vectors are closest to this one” across millions of points in milliseconds using an HNSW graph, a layered network of shortcuts, highways then streets, that checks a few hundred points instead of all of them. It is approximate: a tiny chance of missing the exact best match, for huge speed.

Why Qdrant: dense and sparse vectors in one collection, built-in fusion, filters applied during the search (essential for permissions), int8 quantization, and an embedded mode that runs without Docker. Weaviate, Milvus, Pinecone, pgvector, Elasticsearch, and Azure AI Search are all reasonable alternatives.

05 Part B: answering a question

Every POST /ask runs one function, ask() in rag/pipeline.py. About 60 lines; reading it top to bottom is the fastest way to understand the system.

Tracing earned its place on day one. A Mac follow-up returned “I don't know”. The trace showed the rewrite worked and retrieval scored 3.42, so the problem was not the pipeline: no ticket covered Mac VPN specifically. The model refused correctly. A correct non-answer is a feature, not a bug.

06 The reranker, in depth

A reranker is a second, more careful model that re-orders the first search's results. In practice it is almost always a cross-encoder, and “cross” is the whole point:

Bi-encoder (search):                 Cross-encoder (reranker):
  "VPN fails in office" -> [vec A]     "[CLS] VPN fails in office [SEP] Office firewall blocks VPN ports [SEP]"
  "Firewall blocks VPN" -> [vec B]                       |
  compare A and B                         one transformer reads it all at once
                                                         |
                                                    score: 6.2

Because question and document are one input, attention flows across them: “office” in the question attends directly to “office firewall” in the document. A bi-encoder compresses each side into a vector first and loses that.

Not any model can rerank. Options run from small local cross-encoders (ours, or bge-reranker-base), to large multilingual ones like bge-reranker-v2-m3, to hosted APIs like Cohere Rerank, Voyage, and Jina, to using an LLM as the ranker, which is very good and 10 to 100 times slower and costlier. Swapping is one setting; then you rerun the eval and let the numbers decide.

07 Where LightGBM would fit

Gradient boosting builds many small decision trees, each correcting the mistakes of the trees before it. LightGBM is Microsoft's fast implementation. It is excellent on tabular data and does not read raw text.

Predicting a house price of 80:
  tree 1 (size)      guesses 60      still off by 20
  tree 2 (location)  adds   +15      now 75, off by 5
  tree 3 (age)       adds    +4      now 79, off by 1
  ... 100 trees      about 80

I did not use it, and that is the right call on day one. Its natural place is learning-to-rank right after the reranker: each (question, document) pair becomes a row of BM25 score, dense score, rerank score, document age, past clicks, same department, labeled by whether users found it helpful. LGBMRanker learns that a slightly lower rerank score but recent and often clicked beats an old, never-clicked ticket. It also fits query routing, escalation prediction, and tuning the confidence gate. But it needs labeled feedback, which only exists after weeks of real traffic.

hybrid search -> RRF -> cross-encoder rerank -> [ LightGBM ranker ] -> top 5 -> LLM

08 The numbers

You cannot feel whether RAG is good. gpt-4.1 wrote 145 realistic questions from random tickets, each tied to its source ticket. Then each stage is scored separately. recall@5 is the share of questions whose correct ticket lands in the top 5, which caps answer quality because the LLM sees 5 sources. MRR rewards ranking it high.

Answers judged by an LLM (n=30): faithfulness 1.00, relevance 1.00, grounded 1.00. Permission leak test: 0 leaks across 24 HR and billing questions asked as a plain employee. All of it runs in CI, and the build fails if recall@5 drops below 0.52 or any ACL leak appears.

09 What is still missing for production

The design is production-shaped. Before real users it would need:

• Real identity: groups come from a header anyone can set; production reads a verified SSO token from Okta or Entra.
• Live connectors: a static CSV today; production syncs Confluence, SharePoint, and ServiceNow with their real permissions.
• Scale: a Qdrant cluster, a Redis cache, rate limiting, and graceful fallback when the LLM is down.
• Latency: a GPU or hosted reranker and streaming responses to bring 1.5 seconds down.
• Stronger guardrails: ML classifiers for injection and PII instead of regex alone.
• Richer data: long policy PDFs next to tickets, where chunking really matters, and a multilingual encoder for the German half.
• Real evaluation data: questions from real users labeled by agents, and business metrics like deflection rate, CSAT, and escalations.
• Agents: tools for actions like resetting a password or opening a ticket, with the model deciding when to search and when to act.

10 The takeaway

The LLM call is the easy 10%. What makes RAG trustworthy is everything around it: retrieval you can measure, permissions enforced inside the search, a reranker that reads carefully, a gate that knows when to say “I don't know”, citations you can check, and traces that tell you which step failed. Build those, measure each one, and the model mostly takes care of itself.

Code, eval, and the full walkthrough: github.com/madmecodes/rag-pipeline-project.