Building a Private Chat Assistant with Open WebUI, Qdrant, and Ollama

I already had Ollama running for AI-assisted alert analysis (see AI-Powered Infrastructure Monitoring with n8n, Ollama, and Zabbix), but that was a narrow, single-purpose workflow. What I wanted next was something more general: a private chat interface I could actually use day to day, that could also answer questions grounded in my own homelab documentation instead of just guessing from whatever the model already knew.

The Stack#

• Ollama, running on an LXC container on one of the Proxmox nodes, serving qwen2.5:3b-instruct-q4_K_M for general chat, qwen2.5-coder:7b-instruct-q4_K_M for coding questions, and nomic-embed-text for embeddings
• Open WebUI, deployed as a container on docker02, exposed through Traefik and Authentik SSO at a subdomain, with DNS handled locally by Pi-hole
• Qdrant, a standalone vector database also on docker02, holding a collection of embedded homelab documentation
• n8n, running a nightly workflow that pulls the same documentation set an existing sync script already uses, chunks it, embeds it through Ollama, and upserts the vectors into Qdrant

All of it stays on local hardware. No API keys, no per-token billing, no homelab documentation leaving the network to reach a third party.

Why Not Something Bigger#

I looked at heavier all-in-one platforms before settling on this combination, and ruled them out mainly on resource grounds. A full multi-container platform with its own database, search index, and worker processes was more footprint than any of my Proxmox or Docker hosts had headroom for, especially with a GPU already carrying real Plex and Ollama workloads. Open WebUI plus a standalone Qdrant instance gets most of the same value, a chat interface with retrieval-augmented answers, for a fraction of the resource cost.

Wiring Up Retrieval#

The retrieval half took more iteration than the chat half. The n8n sync workflow itself came together quickly: pull the docs, chunk them, embed each chunk through Ollama's embedding model, and push the vectors into Qdrant's homelab_docs collection on a nightly schedule.

The part that wasn't obvious until I tested it directly was whether Open WebUI's own Knowledge feature, its built-in way of attaching a document collection to a chat, could actually read a collection that n8n had populated from the outside rather than one uploaded through Open WebUI's own UI. It couldn't, at least not reliably. The fix was switching from Knowledge to Open WebUI's Tool interface instead, which calls out to Qdrant directly at query time rather than expecting a pre-registered collection. Once that switch was made, RAG worked end to end: ask a question about a specific piece of homelab infrastructure, and the response pulls in the actual documented configuration rather than a generic guess.

A Second Project Riding on the Same Stack#

Once Ollama and the existing Zabbix integration were already in place, extending the same idea to Telegram was a natural next step. I built a status bot that polls Telegram for messages from one authorized chat, classifies the intent (a fast regex check first, falling back to Ollama only when the regex doesn't match) into either "what's currently broken" or "how's this specific host doing," queries Zabbix accordingly, and has the model write a natural-language reply. It's effectively "ask the homelab a question from your phone," without needing to open a laptop or a dashboard.

Two real bugs showed up building that piece, both n8n-specific rather than AI-specific: a workflow imported through the CLI doesn't automatically get its version tracking or trigger count set the way one created through the UI does, and static data used for tracking a Telegram polling offset turned out to be unreliable once the workflow got force-activated rather than toggled on normally. Both were fixable once identified, but neither would have been obvious without hitting them live.

Keeping the Blast Radius Small#

Everything talking to Zabbix from this stack uses a dedicated, purpose-built account with read-only permissions scoped to the relevant host groups, not the admin account an earlier workflow had been using by default. I tested this the way I'd want any permission boundary tested: confirmed reads succeed and confirmed a write attempt gets rejected with a real permissions error, not just trusted the role assignment on faith.

That same caution extends to the design of the whole stack. It's read-only and advisory by design. It can tell me what's wrong or answer a question about a host, but it has no path to actually change anything in the infrastructure it's reporting on. If that ever changes, it gets a separate, deliberately scoped-down credential and a human confirming anything destructive before it happens, never the same access a monitoring query uses.

Where It's Rough Around the Edges#

Tool-calling reliability is the one place this setup still falls short. The local model doesn't consistently wrap its tool-call syntax the way Open WebUI expects, so RAG and Tool-based lookups work well but aren't perfectly dependable on every single query. I've left this as a known limitation rather than chasing a fix, since the failure mode is a slightly wrong or generic answer, not anything worse, and it's not worth the model upgrade it would probably take to fully resolve.

Where It Landed#

A private chat assistant, reachable from anywhere through the same Cloudflare Tunnel and Authentik SSO the rest of the sites use, backed by two purpose-built models for general chat and coding, with retrieval grounded in my own documentation instead of the model's raw training data. Combined with the Telegram status bot, it's become a genuinely useful second interface into the homelab, not just a chat toy running in the background.

For the alert-analysis half of this AI setup, see AI-Powered Infrastructure Monitoring with n8n, Ollama, and Zabbix.