Inspiration
My career has been a lot of helping others bring their ideas to life. Working at Rooster Teeth Animation wearing multiple hats in film and animation instilled this undying desire to see people achieve even in the worst conditions. Pre-pandemic Magic Leap taught me having a divergent approach to adapting emerging technology to different industries and I maintained that approach in my career. Now, after wrapping up my time at Apple, it was time for me to bring my own idea to life and blend together my video production work and software development work together to create a unique type of interactive experience that is impactful to each user.
What it does
Dreams.ai lets you craft and experience personalized narrative journeys. Inspired by the beloved 'Choose-Your-Own-Adventure' books and early interactive videos, Dreams.ai blends captivating short-form media with dynamic interactivity to create minute-long, shareable video experiences. Simply describe the adventure or world you want to explore, and our team of AI agents – the Director, Story Writer, and Cinematographer – work together seamlessly to bring your dream to life. As the scene unfolds, you become an active participant, guiding the story with your choices and shaping a narrative uniquely tailored to your imagination. Share your creations and see how others navigate your world using Dreams.ai’s innovative ‘imagine files,’ opening up endless possibilities for collaborative storytelling.
How I built it
For back-end: I build the agents that utilized a locally running LLM on my PC and organized with LangGraph. Each agent has access to tools to handle different parts of the production pipeline and the creation of an imagine file (.imn) that is used to store information about the dream in production. The next step is to create a new tool set to utilize fluxpipeline to generate out images and also first - last frame generation for video.
*For front-end: * I utilized bolt and figma to design a landing page for dreams.ai. I wanted to establish a unique brand identity and it's own abstract art form that highlighted how bizarre dreams can feel. I landed on "Cosmic Corporation" with it being this blend of nebulous clouds of space mixed with a 1980s Wall Street appeal. Marble and Brass against the Pillars of Creation. Next steps are to connect the landing page to the back end, My next major milestone is to take the prompt and handle a simple image generation based upon the input of three separate agents working together to achieve a product. By adding in additional checks and balances in the form of The Cinematographer and The Story-Writer that are fact checking each other while The Director makes the final decision, it creates a more focused output based on the initial user input.
Challenges I ran into
I made the mistake of starting on the last weekend thinking it was just a weekend hackathon and a month long but I'm happy with the progress I've made in just a little under 48 hours. Adapting a parallel production pipeline was based on something similar back at Rooster Teeth, but needed a way to have each agent utilize the same information simultaneously in order for each agent to run in parallel with each other and opened the door to an entire social and revenue based model that can be added to the platform.
Accomplishments that I'm proud of
This agentic parallel video production pipeline and the creation of the imagine filetype elevated this project into becoming something truly fantastic. The agentic workflow of how data is handled between each agent required a lot of notes, sketches and sticky notes but it paid off.
What I learned
Time is not on your side and don't procrastinate. I'll be the first to admit this is an insane idea. But it's become such a tangible dream that I'm bringing to life because I can see it so clearly in my mind. It lead to different pit falls like being hung up on the art style for too long when I should of been debugging agentic workflows.
What's next for dreams.ai
I can boil down my next goal in 3 steps:
- Finding additional funds to host with AWS: I'm working with Ollama using a LLM I trained and fine-tuned myself for each agent. I'm limited by my income, or lack of in this case, to provide additional resources and to scale in size. Taking on additional work and what ever extra funds I can gather will be dealing with that expense to keep building.
- Creating The Editor Agent: The forth and final agent needs to be added to the pipeline. The editor will utilize FFMPEG to handle all video output from The Cinematographer and used to splice together multiple videos.
- The Story-Writer interactivity: The second agent needs to understand the current frame and subject matter of the scene to understand how the user's action plays into it's narrative. Taking the 2D vector coordinates from the user's input and individual frames from The Editor lets The Story-Writer adapt the story to the user's prompt.
I ain't stopping on this anytime soon. I've worked with developers building content for a lot of digital content creation platforms, Minecraft, Snapchat, and most recently with Apple on the Vision Pro. I truly think this can become a natural evolution of what we are seeing with TikTok and Instagram and creator focused developer content creation platforms. Short-form interactive fiction blends together the dopamine fix from things like Tetras and Solitaire on your phone and the rapid consumption from short videos. I can visualize the impact it can have on a new wave of creators and an entire revenue platform filled with people who can create and share viral experiences hand crafted from just a few words. I'll be continuing on this every day until someone takes it from my hands.
Log in or sign up for Devpost to join the conversation.