Inspiration
Let's be real: almost no one reads anymore. In a survey conducted in 2025, 40% of American adults reported reading 0 books the entire calendar year. Noticing trends of increased online activity and increasing consumption of social media content, we created Sceneweaver to bridge the growing gap between traditional literary media and digital platforms that would be both educational and engaging.
What it does
Sceneweaver takes any input of literature, including, but not limited to, novels, novellas, non-fiction, and short stories. After receiving a .pdf or .txt input, Sceneweaver takes a .pdf or .txt, finds the most visually important scenes, and turns them into narrated 3D worlds. The scenes are generated as panoramas, converted into Gaussian splats, and rendered into an interactive experience that follows the structure of the original text.
How we built it
We parse the book, use GPT-5-nano to identify scene-rich passages, and build paired text-panorama data for fine-tuning. We then fine-tune a LoRA adapter on a world generation model so it better understands literary scene descriptions. At inference time, the model outputs SPZ Gaussian splats, Cartesia generates narration, and Spark renders the final world.
Challenges we ran into
The hardest part was fine-tuning. This was an expensive and confusing process because there is no simple reward signal for whether a generated scene actually matches a book well. A scene could look visually strong while still missing the tone, setting, or key details from the text. That made evaluation difficult. We had to balance visual quality, literary faithfulness, and consistency across scenes, but none of those are easy to score automatically, so iteration was much slower than a standard generation pipeline.
Accomplishments that we're proud of
We were initially concerned that Sceneweaver may struggle with generating scenes from different genres or time periods, but we were pleasantly surprised by how well the system reacted to various inputs.
What we learned
While learning the balancing of the fine-tuning process was a significant learning point, there were other major points of growth. Having different, asynchronous API calls go on simultaneously for different parts of the generation (GPT-5, Cartesia, world generation model) forced us to design flexible endpoints.
What's next for Sceneweaver
In the future, we'd like to further optimize our fine-tuning to improve visual clarity and add an authentication system for users. We plan on adding full VR compatibility to Sceneweaver as a full 3D VR experience would significantly elevate the level of immersion. We also plan on better orienting the worlds created for Unity to increase the amount of possibilities with each generation.
Log in or sign up for Devpost to join the conversation.