Building a self-hosted RAG workspace with hybrid search
How Nexus turns a folder of documents into cited answers — queue-based indexing, pgvector + full-text retrieval, streaming WebSockets, and the retrieval decisions that mattered more than the model.
Every "chat with your documents" product makes the same pitch: upload a PDF, get answers. Almost all of them also make the same demand: give us your documents. For contracts, research notes, internal runbooks, or anything under NDA, that is a non-starter. And even when it is acceptable, you inherit someone else's model choice, someone else's chunk size, and someone else's pricing.
So I built Nexus: a self-hosted AI RAG workspace. Upload documents into isolated workspaces, index them asynchronously, and chat with them over a real-time stream with citations you can verify. Everything runs on your infrastructure — Docker Compose and a .env file. Bring a frontier API key, or point it at a local model.
This post is about the engineering decisions that made it feel good to use.
The pipeline, not the prompt
The LLM gets all the attention, but the quality of a RAG system is decided before the model sees a token. Nexus has four stages, each decoupled:
Frontend (React + tRPC) ── HTTP /trpc + WS /ws ──▶ Express API ──▶ BullMQ (Redis)
│ │ │
│ tRPC router embedding.worker
│ + uploads │
│ │ chunk + embed
│ ▼ ▼
└────────────────────────────── PostgreSQL 16 + pgvector ◀────────┘
(documents, chunks, chat)
- Upload stages and validates files (PDF, TXT, Markdown, CSV, JSON — 25 MB each) and enqueues them.
- Index runs in a BullMQ worker: extract pages, recursively chunk (1000-token chunks, 200-token overlap), embed in batches of 100 with exponential-backoff retries.
- Retrieve fuses vector and keyword search under per-workspace ownership checks.
- Answer streams grounded tokens with inline citations over a WebSocket.
The queue is the part users never see and always feel. Uploading a 200-page PDF returns immediately. One corrupt document fails alone instead of blocking the queue. Retries are sane. If indexing were synchronous inside the request, the product would feel broken at exactly the moment it should feel powerful.
Hybrid search is not a buzzword — it is damage control
The first version was pure vector search. It was impressive in demos and wrong in practice. Embeddings are excellent at "find me the passage about the deployment process" and terrible at "find the file where I wrote DATABASE_URL". Lexical tokens — identifiers, error codes, product names — fall through the cracks of semantic similarity.
Nexus fuses the two:
- Semantic: pgvector cosine distance over 1536-dimensional embeddings.
- Lexical: PostgreSQL full-text ranking (
ts_rank). - Fusion: configurable weights, default 0.6 vector / 0.4 keyword, top 6 chunks.
- Fallback: if one side returns nothing, the other still answers.
The weights are exposed in the Settings panel because retrieval quality is data-dependent. A legal contract corpus wants more keyword; a support-ticket archive wants more semantic. Shipping a fixed magic number would have been easier and worse.
Streaming with sources first
The chat interface is a tRPC subscription over WebSocket with a deliberate event order:
sources— the retrieved chunks, with titles and scores, rendered as numbered references.token— the answer, streamed as it is generated.done— final citations and persistence.
Showing sources before the answer changes how the result is read. Instead of asking "do I trust this paragraph?", the user can see which documents were consulted while the answer is still forming. If the sources are wrong, the answer is wrong, and they know it immediately.
The system prompt has two jobs: ground every claim in the provided context with [1], [2] references, and refuse to answer when the context does not contain the answer. A RAG system that guesses is worse than one that says "I don't know", because it launders hallucination through the credibility of your own documents.
Settings belong in the product
Most self-hosted tools make you edit .env and restart. Nexus moves the knobs into the UI: chunk size, top-K, model, temperature, even the OpenAI key.
The precedence rule is UI → environment → default, and secrets are encrypted at rest with AES-256-GCM under a SETTINGS_SECRET. You can change your embedding model and re-index without a redeploy. That is not just convenience — it makes the system safe to experiment with, which is how you find the retrieval settings that actually work for your corpus.
Ownership checks everywhere
Multi-tenancy bugs in RAG are subtle: retrieval scoped correctly but chat history leaking across workspaces, or a document fetch that forgets the workspace clause. Nexus scopes every document and query to its workspace at the data layer, and authentication is optional by deployment: set AUTH_PASSWORD for a login screen with httpOnly session cookies (30-day TTL), or leave it unset for single-user local mode.
Operational hygiene as a feature
Self-hosted software is judged as much by its docker-compose.yml as its UI. Nexus ships:
- GHCR images for backend and frontend, plus
docker-compose.prod.yml. - CI that runs typecheck, build, lint, Vitest, gitleaks, CodeQL, and
pnpm audit. - Optional auth with no external identity provider required.
If deploying it is a weekend project, nobody will deploy it.
What I would do differently
- Measure retrieval, not vibes. I now want a small evaluation set — question, expected source, expected answer — wired into CI. "It looked good in the demo" is not a metric.
- Think about embeddings migration earlier. Changing models means re-embedding everything; a versioned embedding column would have simplified that.
- Treat chunking as product design. Overlap and chunk size visibly change answer quality. They deserve the same care as prompt engineering.
Nexus is MIT-licensed. Clone it, point it at a folder of documents you would never upload to a SaaS, and see what your own corpus knows.