Engineering a Governed Local RAG Stack for Edge-Native AI

From Lexical Retrieval to Grounded Local Intelligence

At PeachBot AI, we are exploring a fundamental question: Can an AI knowledge-assistance system remain useful when it must operate locally, offline, under resource constraints, and with explicit evidence governance?

The answer cannot simply be to add a larger language model.

As part of the Medi AI+ Sprint 5 engineering programme, we have been progressively building and evaluating a local Retrieval-Augmented Generation (RAG) architecture around a simple principle:

Retrieve governed evidence first. Generate only from that evidence.

The objective is not to build another chatbot. The objective is to establish a reproducible engineering foundation for local, privacy-conscious and edge-oriented AI systems.

1. Establishing a Frozen Lexical Baseline

Before introducing semantic embeddings, we established a deterministic lexical retrieval baseline.

This provides a fixed reference point against which semantic and hybrid retrieval can be evaluated.

Without a frozen baseline, it becomes difficult to determine whether an embedding model actually improves retrieval or whether changes in the evaluation environment are responsible for the observed result.

The benchmark corpus and evaluation harness were therefore frozen and recorded as reproducible evidence.

2. Benchmark B1: Evaluating Local Embedding Candidates

We evaluated three compact embedding models under the same controlled workload:

• BAAI/bge-small-en-v1.5
• intfloat/e5-small-v2
• thenlper/gte-small

All three candidates achieved perfect MRR and Recall@3 on the controlled semantic evaluation cases.

However, retrieval quality was only one part of the decision. We also measured model loading, query latency, memory characteristics and reproducibility.

BGE-small produced 384-dimensional embeddings and achieved the following measurements in our controlled benchmark environment:

• Mean query latency: approximately 25.6 ms
• p95 query latency: approximately 31.1 ms
• Recall@3: 1.0
• MRR: 1.0

These measurements were subsequently hardened into a reproducible three-replicate benchmark.

3. Model Governance: Licence, SBOM and Attribution

Before using the selected embedding model in the vector-backend evaluation, we introduced a dedicated licence, SBOM and attribution gate.

The BGE model was pinned to an exact revision rather than relying on a mutable model reference.

This means the benchmark identifies not only the model being evaluated, but the precise model revision used to generate the vectors.

For an AI system intended to become auditable, dependency provenance should be part of the engineering process rather than an afterthought.

4. Benchmark B2: FAISS vs Qdrant vs sqlite-vec

With the embedding model frozen, we generated a frozen vector corpus containing:

• 12 documents
• 38 queries
• 384-dimensional vectors

The identical vectors were evaluated against three local vector backends:

• FAISS IndexFlatIP
• Qdrant local
• sqlite-vec

All three backends produced deterministic results and achieved 1.0 exact top-k agreement against the reference ranking in the controlled benchmark.

FAISS

• p50: approximately 0.0086 ms
• p95: approximately 0.0092 ms
• Median QPS: approximately 110,818
• RSS: approximately 102 MiB

Qdrant

• p50: approximately 0.553 ms
• p95: approximately 1.141 ms
• Median QPS: approximately 1,517
• RSS: approximately 103 MiB

sqlite-vec

• p50: approximately 0.327 ms
• p95: approximately 0.765 ms
• Median QPS: approximately 2,586
• RSS: approximately 100 MiB

These figures should not be interpreted as universal vector-database benchmarks. The corpus was deliberately small and synthetic, and the measurements were obtained in a constrained benchmark environment.

Their purpose is architectural: the same frozen vectors and workload can be compared reproducibly across different local retrieval backends.

5. Benchmark C: Hybrid Retrieval

The next stage combined lexical and dense retrieval rather than replacing one with the other.

The controlled Sprint 5 retrieval evaluation produced:

This was an important architectural result.

Dense retrieval improved semantic coverage, while hybrid retrieval combined the strengths of lexical and semantic matching.

The resulting architecture therefore uses retrieval fusion rather than treating embeddings as a replacement for deterministic retrieval.

6. Benchmark D: Grounded Generation

Retrieval is only half of a RAG system.

Even when the correct evidence is retrieved, a language model can introduce information that is not contained in that evidence.

Benchmark D therefore introduced a grounded-generation contract.

The local model was instructed to:

• Use only supplied governed evidence.
• Avoid unsupported facts.
• Avoid invented mechanisms or operational details.
• Avoid unsupported permissions or security properties.
• Avoid diagnoses and treatment recommendations.
• Avoid adding background knowledge.
• Explicitly cite the supplied evidence.

The hardened controlled evaluation achieved:

• 100% citation rate
• 100% simulation-boundary rate

This does not establish universal model safety. It demonstrates that the generation behaviour can be converted into a measurable engineering contract.

7. Benchmark E: Discovering the Resource Boundary

One of the most useful experiments was also one of the simplest: deliberately constrain the available memory.

The local RAG stack was first tested under a 2 GiB memory limit.

The container was OOM-killed with exit code 137.

Instead of hiding the failure, we preserved it as benchmark evidence.

We then repeated the constrained evaluation with a 3 GiB memory limit.

The 3 GiB constrained profile passed the defined benchmark gate and recorded approximately:

• Retrieval p50: approximately 100 ms
• Retrieval p95: approximately 899 ms
• Single grounded generation: approximately 59.4 seconds
• Peak RSS: approximately 2.69 GiB

The engineering conclusion is therefore more useful than simply saying that the system "works":

The tested local RAG configuration exceeded the 2 GiB memory profile, while passing the tested 3 GiB constrained-container profile on the benchmark host.

This gives us a measurable resource boundary and a concrete optimisation target.

8. The Architecture We Are Building Toward

Governed Corpus
       |
       +----------------------+
       |                      |
       v                      v
Lexical Retrieval      Dense Retrieval
       |                      |
       +----------+-----------+
                  |
                  v
             Rank Fusion
                  |
                  v
             Evidence Set
                  |
                  v
              Local LLM
                  |
                  v
        Grounded Response
                  |
                  v
            Source Citation

The important property of this architecture is that each layer can be evaluated independently.

We can therefore measure retrieval quality, ranking behaviour, generation grounding, citation behaviour and resource consumption without treating the entire system as an opaque black box.

9. What These Benchmarks Do Not Establish

It is important to be precise about what the current evidence means.

These benchmarks do not constitute:

• Clinical validation
• Medical-device approval
• Regulatory approval
• Production approval
• Real-world clinical performance
• Security certification
• Universal vector-database performance
• General-purpose LLM quality evaluation

The current benchmark corpus is synthetic and deliberately controlled. This is intentional: reproducibility comes before real-world validation.

10. What Sprint 5 Has Established

Sprint 5 has progressed through a sequence of increasingly demanding engineering gates:

1. Frozen lexical retrieval baseline
2. Local embedding evaluation
3. Pinned model revision
4. Licence, SBOM and attribution gate
5. Frozen vector corpus
6. FAISS, Qdrant and sqlite-vec comparison
7. Hybrid retrieval evaluation
8. Grounded generation evaluation
9. Constrained-resource evaluation

This is more than a demonstration of RAG functionality. It is the beginning of an evidence-controlled local AI runtime whose behaviour can be measured at each layer.

What Comes Next

The next engineering priorities are focused on expanding the evaluation and reducing the resource requirements of the local runtime.

• Broader retrieval evaluation.
• Local-generation memory optimisation.
• More realistic edge-resource constraints.
• Expanded dependency and model provenance.
• Additional reproducibility checks.
• Manual engineering review.

The Bigger Lesson

The most important result from this work is not which embedding model or vector backend is fastest.

It is that local RAG is a systems problem.

A governed local AI pipeline requires control over the entire chain:

Data → Retrieval → Ranking → Evidence → Generation → Citation → Resources

If those layers cannot be measured independently, it becomes difficult to understand why a system succeeded—or why it failed.

At PeachBot AI, we are building toward an architecture where that pipeline can operate locally, remain evidence-driven, and expose its behavioural and resource boundaries rather than hiding them.

Local AI should not merely run. It should be measurable.