AI Assistants That Employees Actually Trust

Every HR team has the same story. The same handful of questions come in over and over: What's the sign-on bonus at this level? How many leave days do I get after five years? What's the policy on career breaks? These aren't hard questions, the answers are sitting in a policy document somewhere. But finding them, every time, for every employee, eats hours a week that HR teams would rather spend on actual people problems.

The obvious fix is an AI assistant that reads your documents and answers these questions directly. The hard part is making that assistant reliable enough that people stop double-checking it.

This post is about how we close that gap, and the specific engineering discipline we use to keep it closed.

Most Retrieval-Augmented Generation (RAG) systems are built the same way: connect an AI model to a document store, test it on a few sample questions, and ship it once the answers look reasonable.

The problem with "looks reasonable" is that it doesn't scale, and it doesn't hold up under real use. A system that answers three test questions correctly can still fail silently on the fourth, fifth, and sixth, and nobody notices until an employee gets a wrong answer about their own compensation.

So we built something most teams skip: a quantitative testing framework that scores every answer the system gives, across dozens of real questions, on multiple dimensions of quality, before any change goes live.

Every optimization in this post wasn't a guess. It was found, measured, and verified through that framework.

For HR, compensation, and policy content, that answer needs to be "nowhere." Our production architecture runs entirely on your own infrastructure:

• The document search engine, on-premise
• The language model that generates answers, a locally hosted, open-weight model, not a cloud API
• The conversation history and session data, stored in your own database
• The ranking and retrieval logic, all local compute, no external calls

No employee question, no retrieved policy excerpt, no compensation figure is ever transmitted to an outside service during normal operation. The only exception is an isolated, offline testing environment we use internally to grade system quality during development, and that environment never sees live employee queries, only historical test questions used to check our own work.

1. Not All "Chunks" of a Document Are Useful

When you feed a policy document into an AI system, it gets broken into smaller pieces ("chunks") so the system can search through them efficiently. Our first production version was splitting documents at every section heading, which sounds sensible, but in practice created a huge number of near-empty fragments: a heading like "Salary Structure" with almost no actual content attached to it.

The result: when someone asked about salary, the system sometimes matched a bare heading instead of the paragraph that actually explained the salary structure. We measured it directly, chunks were averaging under 50 words, far too little to answer a real question.

2. One Search Method Isn't Enough

People don't ask questions the same way twice. "What's the PF contribution?" and "How much does the company put into my provident fund?" mean the same thing but share almost no words. "Maturnity leave" is a typo, but the intent is obvious to a human.

A single search method, pure keyword matching, or pure AI-based semantic search, will always miss some of these. So our system runs three different search strategies at once: one built for meaning, one built for exact terminology, and one built for typos and near-matches. The results are merged and ranked together, so no single blind spot sinks the answer.

3. Making Sure One Document Doesn't Drown Out the Others

For cross-cutting questions, say, something touching both leave policy and health benefits, we found the system was sometimes pulling almost all of its supporting material from a single document, even when other relevant documents existed. We added a rule that caps how much any one source document can dominate a single answer, guaranteeing the system pulls from multiple sources when a question genuinely spans them.

4. Catching a Blind Spot in Our Own Testing Process

While testing the system, we hit a case where the AI gave a completely correct answer, but our own automated quality-checking process scored it as a failure, flagging it as unsupported by the source material. On investigation, the problem wasn't the AI assistant at all. It was our testing tool: it had been given a truncated version of the retrieved content, cut off right before the relevant sentence.

In other words, our grading process was marking correct answers as wrong, for reasons that had nothing to do with the actual system. We caught this by manually tracing a discrepancy instead of trusting the automated score, and it's exactly the kind of blind spot that leads teams to spend weeks "fixing" a part of the system that was never actually broken.

We fixed the underlying issue and now treat this kind of cross-check as a standing part of how we validate every change.

5. Making Testing Itself Fast Enough to Be Useful

A quality-testing process that takes 40 minutes to run doesn't get run often. We found our own testing pipeline was needlessly sequential, waiting for one step to finish before starting the next, even when the steps didn't actually depend on each other.

We restructured it to run in parallel wherever there was no real dependency, cutting a typical test cycle from around 40 minutes down to 12–18 minutes. That's not a technical curiosity, it's the difference between testing five configuration changes in a day versus one.

6. Giving the "Double-Check" Step Enough to Work With

After initial search results come back, we run a second, more precise ranking pass over them, think of it as a specialist reviewing the shortlist before the final answer is written. Early on, that specialist was only shown the same five results the first search step had already narrowed things down to, which meant it could only reorder what it was given, not rescue a good answer that had been ranked slightly too low upstream.

7. Matching the Right AI Model to the Right Job

Not every AI model that scores relevance is solving the same problem. Some are trained to judge "does this text imply something related to the question?", which sounds useful, but can rank a tangentially related paragraph above the paragraph that actually states the policy answer. We found and fixed exactly this: a model trained on the wrong task was quietly demoting the correct working-hours policy below a document about time-tracking procedures, because the wrong model was answering the wrong question.

Swapping in a model actually trained for "does this passage answer this query", the right task for the right job, fixed it immediately, and we now test model choices for this step the same rigorous way we test every other parameter.

8. Updating One Sentence Shouldn't Mean Reprocessing the Whole Document

HR content changes, a leave entitlement figure gets corrected, a policy sentence gets clarified. Our original system treated any edit as a reason to reprocess the entire document from scratch, even when 90% of it hadn't changed.

We measured the waste directly: editing one sentence in an employee handbook was triggering full reprocessing of ten unrelated sections. We rebuilt the update logic so it identifies content by what it says, not where it sits in the document, so an edit to one paragraph only touches that paragraph. On our own documents, this cut reprocessing time by 75–90% per edit. For a system using a metered cloud AI service instead of local infrastructure, the same fix translates directly into lower cost per update, the percentage saved holds regardless of which one it shows up as.

9. Locking Down the System Before Wider Rollout

Once quality is proven, we run a completely separate audit, not about accuracy, but about who can access what. Every request now requires authentication. Cross-origin access is restricted to approved sources. File uploads and deletions are checked to prevent any attempt to manipulate paths outside the intended document store. Rate limiting is enabled and made visible on a live dashboard, not just present in the code and silently switched off.

10. Knowing Exactly What's Running, and Being Able to Undo It

The last piece is one clients rarely ask about upfront but always care about once something goes wrong: can we tell exactly what configuration produced a given answer, and can we roll it back in minutes, not days?

Every setting that shapes an answer, prompt wording, search tuning, model choice, now lives in one version-controlled file, so a change is a reviewable diff and a bad change is a one-command revert. Every single answer the assistant has ever given can be traced back to the exact content it retrieved and the exact instructions the model was given, looked up directly from a user's feedback report. And every change to this configuration is automatically tested against a real search-quality check before it's allowed to merge, catching regressions before they reach anyone.

Chunk quality roughly quadrupled. Response payloads shrank by correctly capping results after, not before, combining search methods. Testing cycles got fast enough to run far more experiments per day. And editing one section now reprocesses only that section, whether the saving shows up as compute time or cloud cost.

Most vendors will show you a demo. Fewer will show you the process behind keeping that demo accurate once it's answering real questions from real employees, every day, as your documents change.

What we've described here isn't a one-time build, it's a discipline: measure everything, question your own testing tools as hard as you question the system itself, keep the data on your own infrastructure, and make sure you can always answer "what's running, why did it say that, and how do we undo it if it's wrong."

Interested in what this would look like for your own documents, HR policies, compliance manuals, internal knowledge bases, or something else entirely? We're happy to walk through it.

Discuss Your RAG System    Back to Blog