SecureStep-RAG: Building Trustworthy Multi-Hop Retrieval-Augmented Generation

Inspiration

Retrieval-Augmented Generation (RAG) has made AI systems more knowledgeable by allowing Large Language Models (LLMs) to retrieve information from external documents. However, this introduces a serious security challenge: retrieved documents can contain malicious instructions, prompt injections, misleading information, or content designed to manipulate an AI system.

We were inspired to explore an important question: How can we make RAG systems not only intelligent, but also secure and trustworthy?

Traditional RAG pipelines often focus on retrieving relevant documents, but relevance alone does not guarantee trustworthiness. A document can appear relevant while containing malicious instructions. This risk becomes even more significant in multi-hop RAG, where information retrieved in one step influences subsequent retrieval decisions.

This motivated us to develop SecureStep-RAG.

What We Built

SecureStep-RAG is a security-focused framework that protects multi-hop Retrieval-Augmented Generation pipelines against adversarial document injection and retrieval manipulation.

Instead of blindly passing retrieved documents to an LLM, our framework evaluates each document through a multi-signal trust filtering mechanism.

The system analyzes four major dimensions:

1. Semantic Similarity: Measures how closely a retrieved document relates to the original query.
2. Source Credibility: Evaluates the credibility information associated with the document's source.
3. Injection Risk: Detects suspicious instructions, prompt injection patterns, and potential retrieval redirection.
4. Hop Drift: Monitors whether successive retrieval steps are moving away from the original user intent.

These signals contribute to a weighted trust score:

[ T(d)=0.30S+0.25C+0.25I+0.20H ]

where (S), (C), (I), and (H) represent semantic similarity, source credibility, injection safety, and hop consistency scores.

Documents that satisfy the configured trust threshold can proceed through the pipeline, while suspicious content can be blocked or flagged.

We also incorporate position-aware guardrails, confidence checks, cross-hop consistency validation, and bounded retrieval iterations to improve security throughout the generation process.

How We Built It

We designed SecureStep-RAG as a modular architecture using:

• LangGraph: Orchestrates the multi-hop retrieval and generation workflow.
• BGE-M3: Supports semantic embedding and document-query similarity analysis.
• Qdrant: Provides vector storage and retrieval.
• NeMo Guardrails: Supports conversational and retrieval-transition safety checks.
• Python and FastAPI: Power the backend and pipeline integration.
• React, TypeScript, and Tailwind CSS: Support the interactive dashboard.
• RAGAS and DeepEval: Support retrieval and generation evaluation.
• Ollama and external LLM providers: Provide flexible model integration.

We also designed an adversarial benchmarking workflow to investigate different attack scenarios, including retrieval hijacking, query drift, instruction injection, and multi-hop attack amplification.

Our goal was to make security a continuous part of the retrieval process rather than adding a simple prompt filter at the beginning or end.

Challenges We Faced

One of our biggest challenges was distinguishing genuinely useful documents from documents that could manipulate the LLM. A document may have high semantic similarity but still contain malicious instructions.

We addressed this by combining multiple trust signals instead of depending on a single similarity score.

Another challenge was detecting retrieval manipulation across multiple hops. We explored query-transition validation and hop-drift monitoring to help preserve the original user intent.

We also considered the trade-off between security and performance. Additional filtering and guardrail checks introduce latency, so we designed the architecture to support configurable thresholds and bounded retrieval iterations.

What We Learned

Through this project, we explored the security limitations of conventional RAG architectures and gained practical experience with:

• Multi-hop retrieval and agentic workflow orchestration.
• Prompt injection and adversarial document attacks.
• Trust scoring and multi-signal document evaluation.
• Vector databases and embedding-based retrieval.
• AI guardrails and responsible LLM application design.
• Benchmarking, evaluation, and security-focused testing.

Most importantly, we learned that a trustworthy AI system must evaluate not only what information it retrieves, but also where that information comes from, whether it is safe, and how it influences subsequent reasoning.

Our Vision

SecureStep-RAG aims to contribute to a future where organizations can use RAG-based AI applications with greater confidence.

We envision extending this framework with adaptive trust scoring, stronger provenance verification, explainable security decisions, and more comprehensive adversarial evaluations.

Our mission is simple: Don't just retrieve information. Verify trust before allowing it to influence AI decisions.

Built With

• ai-security
• artificial-intelligence
• bge-m3
• cybersecurity
• fastapi
• langgraph
• large-language-models
• machine-learning
• nemo-guardrails
• prompt-injection
• python
• qdrant
• rag
• retrieval-augmented-generation