A Local, Multi-Tenant Graph RAG System with LightRAG

The standard way to build a RAG system is to chunk the whole database, throw it into a vector store, and wire it up to a large cloud model. Then the bill arrives, and you notice something worse than the cost: the cloud model, brilliant as it is at reasoning, gets lost in the middle when you feed it thousands of tokens of irrelevant context. You end up paying a premium to confuse the smartest models on the market.

The fix isn’t to stop using cloud models. It’s to stop using them for the dirty work. Shift indexing, embedding, and initial retrieval to a local environment with tools like Ollama and LightRAG, and you can build a precise, multi-tenant knowledge graph that only forwards the most relevant context to the expensive model. Here’s the local, hub-and-spoke Graph RAG system we built on top of PostgreSQL, and what we learned building it.

The architecture: hub-and-spoke Graph RAG

Traditional RAG relies on vector databases: dense retrieval that’s good at finding semantically similar sentences but bad at understanding relationships. Ask “how does feature X affect my billing?” and a pure vector search can pull up a document about feature X and a separate one about billing, and completely miss the bridge between them.

That’s the gap LightRAG closes. It builds a knowledge graph, extracting entities as nodes and their relationships as edges, instead of just storing embedded text. Our architecture uses a central Python gateway to move data from an existing PostgreSQL database into isolated LightRAG workspaces:

• Source of truth: PostgreSQL, holding raw app data, customer logs, and markdown docs.
• The engine: Ollama running llama3.1 for local reasoning and extraction, and nomic-embed-text for embeddings.
• The graph: LightRAG, running in a Dockerized container.
• The gateway: a FastAPI layer that enforces strict workspace routing.

A daily ETL job pulls rows from Postgres, formats them into structured markdown blocks, and pushes them into workspace-specific LightRAG instances (customer_facing, internal_dev, product_codex).

Why this approach holds up

Data partitioning matters as much in RAG as it does in database design. One of the more common mistakes in enterprise AI is a single monolithic vector database holding API docs, product roadmaps, and unrelated customer content side by side. Once you mix everything into one index, the model’s retrieval gets muddy. Multi-workspace isolation applies the same separation-of-concerns principle the gateway already knows from routing regular API traffic: a customer asking a grammar question only queries customer_facing, a developer debugging an endpoint only queries internal_dev. That alone cuts hallucination and limits unauthorized data access.

Graph retrieval beats vector retrieval for code and docs specifically because code is relational. A function calls another function; a feature requires a specific schema. When LightRAG ingests a document, it uses the local Ollama model to actually read the text and extract a graph, not just store it. Ingestion is slow, local models work hard to build those relationships, but querying afterward is fast, because a query becomes hybrid retrieval: BM25 keyword search, vector similarity, and graph traversal together.

Compute costs split naturally along the same lines. The cheap local model does the heavy lifting of reading and mapping the data during the graph-extraction phase. Expensive cloud models only get invoked for the reasoning-heavy question at the end, and even then, the local system hands over an 800-token context block instead of the raw source. That turned what would have been a sizeable monthly prompt bill into pennies.

Scaling it further

The Docker Compose stack is solid locally, but scaling it up needs a few more pieces.

Semantic caching. Right now, if a hundred users ask “how do I reset my password,” the system runs the full graph-retrieval and generation pipeline a hundred times. A semantic cache in front of the gateway, keyed on embedded intent, can return a cached answer when a new query matches a recent one closely enough, dropping latency from seconds to milliseconds.

Agentic routing. Routing is currently deterministic (if user_type == 'dev'), which works until the number of workspaces grows. A small model at the gateway level acting as a router can read the prompt and decide which workspace, or workspaces, to query, letting a cross-functional question (“did the new endpoint cause customer complaints?”) pull from both internal_dev and customer_facing and synthesize an answer.

Event-driven ingestion. Batch syncing via cron leaves the knowledge base up to a day stale. Moving from an ETL script to an event-driven pipeline, Postgres triggers or a message broker like RabbitMQ or Kafka, lets a merged pull request or an updated PRD fire a webhook straight into LightRAG, keeping the graph close to real time.

The point

Building AI systems well isn’t about calling the smartest API, it’s systems engineering: treat the LLM as one modular component in a traditional architecture, use local models for the processing and graph generation, and control data flow strictly through workspaces. That combination is cheaper to run and more accurate, and the bigger context windows get, the more that discipline matters rather than less.