Llola

Inspiration

Large language models are powerful but fragile. We saw teams ship bots with strict policies only to watch them break under clever prompts. Unauthorized refunds, leaked info, made up policies, you name it. Chaos engineering transformed reliability for infrastructure. We wanted the same for LLM guardrails.

What it does

Llola runs automated chaos tests on any LLM. It generates adversarial prompts, executes them at scale, and scores every response for safety, hallucination, and policy compliance. You instantly see which guardrails failed and how they were bypassed.

How we built it

We use Claude to generate domain specific attacks, Daytona to run tests in parallel sandboxes, and Galileo to score every response. A Next.js dashboard tracks runs, results, and history using Prisma and SQLite. Everything works together to simulate real world LLM pressure.

Challenges we ran into

Coordinating distributed tests in many sandboxes, keeping execution stable, and aligning three sponsor APIs was tricky. Designing clean scoring logic and storing accurate traces also took careful work. Building a simple UI for a complex system pushed our design choices.

Accomplishments that we're proud of

We built a working chaos testing platform for LLMs in under forty eight hours. It runs real adversarial attacks, logs to Galileo in real time, and produces clear pass and fail insights. It feels like an actual production tool, not a hackathon prototype.

What we learned

We learned how fragile LLM guardrails can be, how to design evaluation pipelines, and how to integrate observability tools. We also learned how to orchestrate distributed execution and how to make red teaming repeatable instead of manual.

What's next for Llola

We want continuous nightly testing, automatic guardrail tuning, model comparison, and compliance oriented checks. Over time Llola can evolve into the standard way teams validate safety and reliability before deploying any LLM to customers.

Built With

• anthropic
• daytona
• intel-galileo