FOLKLORE.AI

What inspired us Traditional, impactful storytelling and comic creation are expensive processes, often inaccessible to educators, activists, and nonprofits who need them most. We wanted to build a bridge over that gap. During the EnviroCast GEO: From Insight to Impact Hackathon, we set out to create a platform that lowers the barrier to entry, empowering users to turn simple insights into visually stunning, cinematic stories. By amplifying underrepresented voices and democratizing art creation, Folklore AI gives social-good organizations a powerful, free tool to communicate their missions.

What it does Folklore AI is a premium, cinematic scrollytelling and AI-native comic generation platform. Users provide a simple text prompt (e.g., "A robot climbs a tree in a futuristic jungle || retro sci-fi"), and our narrative orchestration engine automatically slices it into a complete script—determining panel count, camera angles, and dialogue. The platform then generates the images using Advanced Stable Diffusion techniques, maintains character consistency across the story, automatically places text bubbles so they don't obscure faces, and allows users to export the final piece as a .pdf or .cbz comic book.

How we built it We architected Folklore AI using a robust split-stack infrastructure:

Frontend: Built with Next.js 14 (App Router), Tailwind CSS, and Framer Motion. The UI is highly optimized for aesthetics, utilizing a cinematic HTML5 Canvas sequence bound to scroll values for a "scrollytelling" experience.

Backend Orchestration: Powered by FastAPI (Python) and LangChain for robust AI inference orchestration. We utilized Supabase for PostgreSQL database management and JWT authentication.

ML Pipeline & Inference: Our primary generation pipeline runs on local GPU-accelerated inference using diffusers to control Stable Diffusion XL (SDXL). We incorporated an inference fallback chain utilizing the Featherless AI inference platform, Pollinations, and Together APIs.

Consistent Character Generation: To ensure characters look the exact same across every generated panel, we integrated StoryDiffusion. We monkey-patched the UNet attention mechanisms in the PyTorch diffusers library to implement Consistent Self-Attention. Instead of standard self-attention, we project keys and values from previously generated reference frames alongside the current frame.

Mathematically, standard attention is defined as: Attention

For Consistent Self-Attention, we concatenate the Key and Value tensors of the current generation with those of a consistent reference batch, shifting the attention weights across time:

Smart Layout Engine: We implemented Ultralytics YOLO (object detection) to map out focal points (like characters' faces) in the generated images, calculating safe bubble coordinates dynamically so dialogue never blocks the art.

Challenges we faced Character Consistency: One of the most notorious problems with Generative AI is retaining a character's identity between different generations. Implementing the StoryDiffusion adapter and manually patching the UNet attention layers was a massive technical hurdle. Dynamic Text Placement: Overlaying text procedurally onto generated images often resulted in speech bubbles covering up the subject's face. Integrating an object-detection model (YOLO) into the downstream pipeline fixed this but took careful orchestration. Inference Latency & Fallbacks: Running heavy SDXL pipelines takes time and heavy compute. We built out a robust fallback architecture including a "Fast Free Mode" that procedurally draws comic art using pure math and Python's Pillow library to serve instantaneous debug placeholders while the heavy ML models load. Ethical Generation: We wanted to ensure the platform adhered strictly to ethical AI practices. We built a TruthInTech transparency pipeline that visibly watermarks all generated media in the studio workspace. What we learned Advanced Diffusers Control: We gained deep, hands-on experience manipulating the internal attention layers of Stable Diffusion models using PyTorch, rather than just prompting APIs. Cinematic Web Animations: We learned how to synchronize heavy HTML5 Canvas image sequences (CanvasSequence.tsx) dynamically with useScroll and useTransform in Framer Motion to create high-end, zero-lag web experiences. Orchestration Architecture: We discovered how to effectively structure multi-model workflows, chaining LLMs (for script generation), Vision Models (YOLO for layout), and Diffusion Models (SDXL for art) into a single, cohesive asynchronous FastAPI backend.