How to Build a Custom AI Chatbot Trained on Your Company Docs

Build a custom AI chatbot for business using your own documents: architecture, ingestion, retrieval, permissions, evaluation and a realistic cost breakdown.

By Danyal Ahmed - Data Scientist · · 4 min read

A custom AI chatbot for business that answers from your own documents is built with retrieval-augmented generation (RAG), not by "training" a model on your files. You load your documents into a searchable index, retrieve the most relevant passages for each question, and have a language model write an answer grounded in them, with citations. The work is mostly in cleaning the content, tuning retrieval, enforcing permissions and testing, not in the model.

This is the architecture and process we use. If you want the concepts first, read what is RAG and fine-tuning vs RAG vs prompting. For why RAG usually beats fine-tuning for this job, see RAG vs fine-tuning for a support chatbot.

Key takeaways

  • "Training on your docs" means retrieval. You index documents and fetch relevant passages at question time.
  • Content quality sets the ceiling. Outdated or contradictory documents produce bad answers.
  • Permissions must be enforced at retrieval, so the bot never surfaces content a user should not see.
  • Evaluate with real questions and measure retrieval and answer quality separately.
  • Plan for ongoing freshness. Documents change, so indexing is a recurring job.

Architecture

textDocuments (PDFs, wiki, tickets, web pages)
        ↓
Ingestion: extract text → clean → chunk → embed → store
        ↓
Vector store + keyword index (with metadata and permissions)

User question
        ↓
Retrieve (hybrid search + rerank, filtered by user access)
        ↓
LLM answers using only retrieved passages, with citations
        ↓
Log, feedback, evaluation

Step 1: Audit your content

Before building anything, list what knowledge exists, where it lives, how current it is and who may see it. Remove duplicates and obsolete files. Decide which source of truth wins when documents disagree. This step often improves answers more than any technical change.

Step 2: Ingestion pipeline

Connectors pull documents from where they live (drive folders, a wiki, a help centre, a ticketing system, a database). The pipeline then:

  • Extracts text from PDFs, slides and web pages. Tables and scanned documents need extra handling (OCR, layout parsing). See document processing automation.
  • Cleans boilerplate, headers and duplicates.
  • Chunks content into passages that each hold one idea with enough context to stand alone, keeping headings and source information as metadata.
  • Embeds each chunk as a vector. See embeddings explained.
  • Stores vectors with metadata in a vector database. See vector databases explained.

Schedule re-indexing, or update on change, so the bot does not serve stale answers.

Step 3: Retrieval

Good retrieval is the difference between a helpful bot and a frustrating one.

  • Hybrid search combines vector similarity with keyword matching, which helps with product names, codes and exact terms.
  • Reranking reorders the top results with a more precise model.
  • Query rewriting turns a chatty question into a better search query.
  • Metadata filters limit results by product, region, date or access level.

Step 4: Generation with guardrails

Prompt the model to answer only from the supplied passages, cite them and say it does not know when nothing relevant is found. Add checks for unsupported claims and define handoff to a person for sensitive topics. See reducing LLM hallucinations and AI guardrails and safety.

Step 5: Permissions and privacy

If different people may see different documents, enforce that when retrieving, not after generating. Store access rules as metadata and filter the search per user or tenant. Decide which model provider handles your data, what is logged and how long it is kept, and check this against your contracts and any regulations that apply.

Step 6: Interface

Options include a chat widget on your site, an internal tool, Slack or Teams, or an API into another product. Show citations so users can verify answers, and give a way to flag a wrong one.

Step 7: Evaluation

Create a test set of real questions with known good answers and measure:

  • Retrieval quality: did the right passage come back?
  • Answer quality: is it correct, complete and grounded?
  • Refusal behaviour: does it decline when it should?

Re-run the set after every change to prompts, chunking or models. See LLM evaluation.

Cost breakdown

Costs fall into build effort and running costs.

Build effort covers content audit, ingestion pipeline, retrieval tuning, the generation layer, permissions, interface and evaluation. Our planning starts an AI agent or RAG project at about three weeks for a focused first version, with custom vector indexing adding about a week, authentication and multi-tenancy about half a week and dashboards about half a week. Try your combination in the estimator.

Running costs include model usage per question (which grows with the amount of context you send), the vector database and hosting, re-indexing and monitoring. Caching and routing easy questions to smaller models cut spend. See LLM cost optimization.

Common mistakes

  1. Indexing everything. More documents is not better. Curate.
  2. Ignoring permissions until a leak makes the point.
  3. No test set, so quality is judged by anecdotes.
  4. Chunking carelessly, splitting tables and steps mid-way.
  5. Launching without a feedback loop to catch failures.

Build it with us

We build document-grounded chatbots as a service. Tell us where your knowledge lives and who will use it, and we will propose a first version. See our RAG chatbot development service, or get a scope estimate for your company chatbot.

FAQ

Can I train an AI chatbot on my own documents?

Yes, through retrieval-augmented generation. You index your documents and give relevant passages to a language model at question time, rather than retraining the model.

How much does a custom AI chatbot cost?

It depends on the number and quality of sources, permissions, interface and quality bar. Size it in weeks and add running costs for model usage and hosting.

Is my data safe?

It depends on your design and providers. Choose providers and settings that meet your requirements, enforce access control at retrieval and limit logging and retention.

How accurate will it be?

As accurate as your content, retrieval and evaluation make it. Measure on real questions and let the bot decline when it is unsure.

How do I keep it up to date?

Re-index changed documents on a schedule or on change and remove outdated sources.

Sources and further reading

Primary documentation, standards and research on the topics covered in this article: