Built Jan 2024 – Nov 2024 · In production
Sangfor IMD Internal GenAI Support Assistant (RAG)
Built and operates a self-hosted RAG assistant for internal engineers, channel partners, and frontline staff. Retrieval uses Dify's vector database with a Cohere rerank model; Ollama, vLLM, OpenAI, Gemini, and Claude were benchmarked before settling on a hosted model.
How it was built
Problem
Our product knowledge base was entirely in Chinese, while frontline staff work in English. Documented issues were escalated to Support Engineers simply because the documentation wasn't readable at first contact.
Model selection
I benchmarked self hosted open source models (Ollama, vLLM) against hosted models (OpenAI, Gemini, Claude). Local models failed on long form technical translation, producing truncation and residual untranslated text, which is worse than no answer. I validated the hosted option on a controlled translation set, using an existing commercial agreement with training opt out.
Cross language retrieval
A Python translation and terminology normalisation layer runs at ingestion rather than query time. Product terminology stays consistent across the corpus, since inconsistent terms break retrieval, and translation latency disappears from every query.
Grounding and evaluation
I iterated prompt versions against a fixed set of expected answers, checking whether citations traced to the correct source article rather than whether the answer read well. Fluent answers with wrong citations were the failure mode that mattered. In production: user thumbs up and down, scheduled spot checks, and weekly review with support engineers and product specialists.
Ingestion
Knowledge sync runs on n8n. A webhook from the KB triggers chunking, embedding, and write back, so new articles become queryable immediately. Success and failure both notify to email and MS Teams, surfacing silent sync failures before users hit them.
Data handling
Public product documentation flows through this pipeline. Content with customer data is routed to a separate de-identification pipeline and never enters this one, a classification decision reviewed and approved before build.