Observability for RAG Chatbots Using OpenTelemetry

Inspiration

As AI applications grow more complex, especially those powered by LangChain and Retrieval-Augmented Generation (RAG), it's crucial to understand what’s happening behind the scenes. When things go wrong — like slow responses, poor retrievals, or LLM hallucinations — developers need deeper visibility. This project was inspired by the need to bring observability to AI pipelines using OpenTelemetry and AWS X-Ray.

What it does

This project enables end-to-end tracing for LangChain-based RAG chatbots, integrating OpenTelemetry to capture granular telemetry data and sending it to AWS X-Ray via AWS Distro for OpenTelemetry (ADOT).

It helps developers:

-Trace every user query through the system -Monitor latencies in vector search, Bedrock inference, and context retrieval -Debug failures in real-time with X-Ray trace maps and spans

How we built it

-Developed a Streamlit-based chatbot UI powered by Amazon Bedrock + OpenSearch Vector Search -Integrated OpenTelemetry SDK into the LangChain code to instrument request/response flows -Deployed ADOT Collector on EC2 using Docker to forward telemetry to AWS X-Ray -Created a Knowledge Base in Amazon Bedrock with Titan Embeddings + OpenSearch Serverless for vector retrieval -Tagged and traced spans in retrieval and generation pipelines for visibility and performance monitoring

Challenges we ran into

-Fine-tuning the OTEL config to properly export traces to AWS X-Ray -Getting streamlit + LangChain + Bedrock working together with minimal latency -IAM permission setup between Bedrock Knowledge Base, S3, and OpenSearch

Accomplishments that we're proud of

-Successfully built a fully observable RAG chatbot pipeline -Visualized meaningful spans and traces in AWS X-Ray’s service map

What we learned

-The power of instrumenting AI workloads with OpenTelemetry

• How AWS X-Ray can provide deep insights even for modern LLM applications

What's next for Observability for RAG Chatbots Using OpenTelemetry

-Add support for metrics and logs, not just traces -Extend tracing to LangChain agents, tools, and multi-step chains

Built With

• bedrock
• claude
• ec2
• langchain
• opentelemetry
• python
• streamlit
• xray