Logs talk to me

What if you could ask, "Tell me about last 5 unsuccessful login attempts into my AWS account"? This projects crestes conversational interface for CloudTrail logs using Lambda Functions and Bedrock.

Comment

Inspiration

As along time networker and security engineer, I used to work with logs a lot. Then after I start to work with AWS, I kinda struggled with that. JSON is great, but the CloudTrail logs can be sometimes difficult to read and explain. So I built a project, that explains the CloudTrail logs in multiple ways.

What it does

LTTMv2 is a serverless conversational pipeline for CloudTrail log analysis. It lets users ask questions like: Show failed console logins in the last 24 hours. Which IP addresses accessed IAM roles recently?

and receive human-readable summaries of the logs.

It consists of three Lambda functions: lttm-vs-lambnda-intent: Classifies user intent and extracts slots using Amazon Bedrock. lttm-vs-lambnda-query: Builds and runs CloudTrail Lake SQL queries based on the intent and slots. lttm-vs-lambnda-summarizer: Summarizes raw query results into a concise, user-friendly explanation.

How I built it

Infrastructure as Code:

  • I used AWS CloudFormation to deploy Lambda functions, IAM roles, and CloudTrail Lake. Deployment can be done progtramaticaly with bash script deploy_cd.sh.
  • Lambdas are written in python 3.12.
  • Amazon Bedrock: Integrated using the AWS SDK to invoke models securely with structured prompts.
  • CLI Integration: A shell script (alexandra.sh) allows direct CLI-based interaction with the Lambda chain for fast testing and usage.
  • CloudWatch Logging: Each Lambda function emits clear, structured logs for debugging and observability.
  • Layer Build Pipeline: Used layer_build to bundle common Python dependencies for Lambda layers efficiently. ## Challenges I ran into
  • Designing structured few-shot prompts for intent detection and SQL generation to ensure stable parsing.
  • Managing prompt size and latency constraints while chaining multiple Bedrock calls across Lambdas.
  • Handling dynamic slot extraction (like time ranges or service names) without false positives during intent detection.
  • Aligning CloudTrail Lake SQL syntax with real user questions while maintaining flexibility for various request styles.
  • Keeping the pipeline clean while supporting CLI interaction and potential future Slack/Chatbot integrations.

Accomplishments that I am proud of

  • Building a fully serverless conversational log analysis pipeline using native AWS services.
  • Achieving clear slot extraction and stable, structured intent detection with Bedrock for complex user queries.
  • Seamlessly chaining Lambda invocations with direct payload passing, enabling modular debugging and enhancements.
  • Using few-shot learning with Bedrock for SQL generation, avoiding hard-coded mappings for every possible user question.
  • Maintaining clean CloudFormation deployment, making the solution reproducible across accounts for anyone.

What I learned

  • Prompt Engineering Matters: Small prompt adjustments can significantly improve stability and output quality when using Bedrock for intent detection and summarization.
  • Structured Logging Speeds Debugging: Adding clear stages and markers in CloudWatch logs saved significant debugging time when chaining Lambda executions.
  • CloudTrail Lake is powerful: Using it for near-real-time querying with structured queries greatly simplifies AWS account monitoring.
  • Serverless chaining is viable: With thoughtful payload design, chaining multiple Lambdas for LLM-powered pipelines remains efficient without requiring step functions or external orchestrators.
  • Iterative Deployment with CloudFormation ensures rapid, controlled infrastructure updates without drift during high-paced hackathon building.

What's next for Logs talk to me

Not ti stay in CloudTrail, I want to integrate AWS Config, AWS CloudWatch, WAF, and onprem.

Built With

Share this project:

Updates