AuditPilot — Internal Audit Policy Navigator

• GIF

Inspiration

I work as a data science associate at Mitsubishi Power Americas, and I deal with policies a lot. Policies are long, full of numbered clauses, and the answers people need are small and specific: a time limit, an approval requirement, an escalation rule. Finding them means searching through pages of text, and it is easy to land on a similar-looking clause and get the wrong number.

I wanted a tool where someone can simply ask, "How long does AP have to review a potential duplicate invoice?" and get an answer taken directly from the policy. That became AuditPilot: Internal Audit Policy Navigator.

Why RAG, and why locally

Many general-purpose AI tools run in the cloud. Pasting a policy or any internal document into them means sending that text to a third party, which can expose confidential information. They also answer from their general training, so they can sound confident while being wrong about your policy.

Retrieval-augmented generation (RAG) helps address the problem: the model is only shown the passages retrieved from the actual document and is told to answer from them, or say it doesn't know. I handled the first problem by running every piece locally:

• the embedding model runs on my machine,
• the search index is a local file,
• the language model runs through Ollama, so no document text is sent to an outside service.

RAG gives grounded answers, and running it locally keeps the document private.

What it does

AuditPilot answers plain-English questions about an Accounts Payable policy. It finds the most relevant parts of the policy, writes an answer using only those parts, and says when the answer is not in the document. Users can download the full policy from the app to check the answer themselves.

How I built it

1. Extract: read the 19-page policy PDF with PyMuPDF and removed the repeated page headers.
2. Chunk: split the text into chunks of 1200 characters with 150 overlap, using LangChain's RecursiveCharacterTextSplitter.
3. Embed and index: turn each chunk into a vector with the BAAI/bge-small-en-v1.5 embedding model and store them in a FAISS index.
4. Retrieve: embed the question the same way and fetch the 3 closest chunks. By default, FAISS ranks chunks by distance, where a smaller distance means a closer match:

$$d(\mathbf{q}, \mathbf{c}) = \lVert \mathbf{q} - \mathbf{c} \rVert_2$$

1. Answer: send the question and the retrieved chunks to a local qwen2.5:7b model through Ollama, with rules to answer only from the excerpts and not to invent policy names or numbers.
2. Interface: a Streamlit app with a question box, an answer card, information cards, and a sidebar that shows whether the index and the model are ready.

Challenges I ran into

• No cloud subscription. The tutorial I followed used Azure AI Foundry. I replaced it with free local models.
• Chunk size. My first setting of 800 characters could split one rule across two chunks. Many clauses in the policy are longer, so I moved to 1200.
• PDF noise. PyMuPDF put a line break inside every page header, so my first cleanup pattern matched nothing. Fixing it removed about 1,000 characters of noise.
• Model choice. A small 3B model can pick the wrong number from similar clauses. I compared it with a 7B model and used the larger one.
• Tooling problems. I worked through a misplaced requirements file, a missing package, a stale PATH that hid the Ollama command, and a Streamlit crash caused by its file watcher.
• Wrong policy names. In early testing, the model could mix up a section heading with a policy name, so I added explicit rules to the prompt.

What I learned

• I gained a clear understanding of the complete RAG workflow and learned which models and tools to use at each stage, from embeddings and retrieval to answer generation.
• Test retrieval before adding a language model. If the wrong chunks come back, no model can answer correctly.
• Chunk size and PDF cleanup affect answer quality as much as the model does.
• The embedding model used to build the index must be the same one used for questions.
• A "say you don't know" rule matters for policy work, because an unsupported answer is worse than no answer.

What's next

• Show the section or page behind each answer, which means keeping page information on each chunk.
• Support several policy documents at once.

Built With

• bge-small-en-v1.5
• faiss
• hugging-face
• langchain
• ollama
• python
• qwen2.5