Inspiration
The initial idea stemmed from the idea of deconstructing the troubles into comics, and then I discussed a lot with AI (mainly Gemini), and many interaction forms and designs were completed under my discussions with AI. I think that the multimodal large model is like a synesthete who is good at synesthesia, and how to make comprehensive use of the five senses is a problem worth considering.
What it does
You can put your troubles into words. Traveler A is able to navigate the reality transformed by your diary, he will overcome everything in his adventures, and you will need to guide and help him. You can also feed your worries to a goldfish that only remembers for seven seconds.
How we built it
FrameShift is built on a Feature-based Architecture using React 19 and Vite, deeply integrated with the Google Gemini Ecosystem. The Cognitive Core (gemini-3-flash-preview): We use Gemini 3's high-speed reasoning capabilities to act as a "CBT Therapist." It analyzes the user's "worry text," deconstructs it into psychological symbols (e.g., "Burnout" → "Heavy Gravity Zone"), and drafts a 6-panel narrative arc. Project A.N.C.H.O.R. (The Visual Engine): To solve the notorious consistency problem in GenAI comics, we built a proprietary pipeline called Neural Linkage: We created a hidden "Darkroom" component that renders SVG skeletal poses (React components) off-screen. We capture these skeletons using html2canvas and inject them as Reference Images into gemini-2.5-flash-image. This forces the AI to adhere to strict anatomical consistency for "Traveler A" while allowing it to hallucinate wild, surreal backgrounds. The Synesthetic Layer: Audio: We use gemini-2.5-flash-preview-tts to generate real-time, emotional narration for each panel. Physics: We implemented interactive rituals (Wiping fog via Canvas API, Connecting stars via SVG/Framer Motion) to make therapy tangible.
Challenges we ran into
The "Shapeshifting" Protagonist: In early tests, "Traveler A" would change height, style, or lose his iconic orange scarf between panels. Pure text prompts were not stable enough. Solution: We shifted from "Prompt Engineering" to "Visual Injection." By feeding the model a code-generated skeleton (SVG) as a reference image for every single panel, we achieved 99% character consistency without model fine-tuning. Balancing Latency vs. Immersion: Generating 6 images + audio + text analysis takes time. Solution: We implemented a Streaming Waterfall architecture. Panel 1 generates and displays immediately while Panels 2-6 generate in the background. We also used gemini-3-flash for its incredible speed to handle the logic layer instantly. The "Therapy" Boundary: Ensuring the AI offers comfort without pretending to be a doctor. Solution: We designed the "Fishbowl" companion (Gemini 3) with a specific persona prompt: a goldfish with a 7-second memory. This naturally limits the AI to providing "ephemeral emotional companionship" rather than deep medical advice, keeping the experience safe and poetic.
Accomplishments that we're proud of
"Neural Linkage" Consistency: We successfully made a Generative AI comic strip where the character feels like the same character in every frame, purely through frontend code and API orchestration. Gamified CBT Rituals: We turned abstract psychological concepts into physical interactions. The "Fog Wipe" (Canvas erasure) and "Star Connect" (SVG pathing) rituals feel incredibly satisfying and grounding. The "Void" Aesthetic: We are proud of the UI design—the Magnetic Tuning header, the Stardust Stream particles, and the Ghost Subtitles. It feels like a high-tech artifact from a sci-fi movie, not just a standard web app. Multimodal Synesthesia: Successfully blending Text, Image, Audio, and Haptic Feedback (vibration) into a single, cohesive flow.
What we learned
Gemini is a Reasoner, not just a Chatbot: We learned to use Gemini 3 not just to write text, but to control game logic (selecting which "Ritual" to trigger based on user emotion). Constraints Breed Creativity: By constraining the AI with our SVG skeletons (Project ANCHOR), we actually got better artistic results than letting it generate freely. Structure enables freedom. The Power of "Clean" Data: We learned that visual metaphors (Sci-Fi tropes) are often easier for users to process than raw clinical language. It's easier to say "I'm in a high-gravity zone" than "I am experiencing depressive symptoms."
What's next for FrameShift
- Interactive Narrative Exploration (交互式叙事探索) Currently, users read the comic linearly. We plan to make specific items within the generated panels clickable. Example: If a "Broken Lantern" appears in Panel 3, clicking it could trigger a side dialogue or a specific memory retrieval, allowing users to "investigate" their own subconscious landscape rather than just watching it.
- Extended Story Arcs & Granular Choices (延长剧情与分支选择) We aim to break the 6-panel limit to support multi-chapter journeys. Users will face more frequent, granular choices (e.g., "Do you pick up the stone or leave it?") that directly affect the environment's physics and the ending, creating a true "Choose Your Own Adventure" therapy session.
- Expanded Ritual Library (扩展疗愈仪式库) We will introduce more gamified CBT mechanics beyond "Wipe" and "Connect": "Balance": Using the device's gyroscope to steady a trembling bridge. "Breath": Using the microphone to physically blow away storm clouds. "Kintsugi": A puzzle game where users repair broken pottery with gold seams to practice self-acceptance.
- Visual Evolution & Deep Lore (视觉升级与世界观完善) We plan to refine the "Void" aesthetic with higher fidelity animations and consistent environmental effects. We will also flesh out the lore of "Traveler A" and the "Goldfish," turning the platform from a tool into a living, breathing universe where users feel genuinely inhabited.
Log in or sign up for Devpost to join the conversation.