-
-
Scrub the story with a fingertip
-
Stand in 221B Baker Street with Holmes and Watson
-
Warm your hands at the Time Machine's forest fire
-
Step into Little Red Riding Hood's 360 wood
-
Browse your library hands-first: no controllers needed
-
Watch the sun set on the far future
-
Explore a tunnel in The Time Machine with 6DOF
-
The Eve of St. Agnes: a world built from the poem
Full Volume: step into the story
Inspiration
Full Volume sits where three of my long-standing interests meet.
The first is 3D reconstruction from images. That's my background, and I've followed the recent explosion of work on Gaussian splatting with real excitement. For the first time, photographs and paintings can become 3D spaces you can actually stand inside, and those spaces render in real time.
The second is VR. I've been hooked since my Oculus Rift DK1. I've seen the medium grow from a tethered prototype to the standalone Quest 3, and I've always wanted to make something for it that isn't a game (although I play many VR games myself) and thus expands the types of applications available to Meta headset users broadening the appeal and depth of content to users.
The third is audiobooks, which fill a lot of my spare time. A great narrator can conjure a world, but you still only hear it.
So the question became: what if you could step inside the book while it's read to you? You'd stand in Baker Street as Sherlock Holmes reads a letter, sit at the Time Traveller's dinner table, or follow Red Riding Hood into the wood. Full Volume is my answer: a hands-first VR audiobook player in which every scene of a book is a fully immersive, 360°, six-degree-of-freedom world, narrated by a full cast with spatial audio.
What it does
- Every scene is a place. As the story moves on, you're surrounded by the scene it describes, rendered as Gaussian splats you can walk around, lean into and immerse yourself in.
- Full-cast narration of the whole book. Each character has their own voice, and with spatial audio each voice comes from where that character stands.
- Sound effects in sync with the text. Trumpets herald the feast, door knocks, rings, hooves on cobble stones. Sound effects for every occasion. Additionally, ambient sound beds give a sense of presence to each scene like birds chirping in a forest or the sound of waves at a beach
- Hands-first interaction. A lectern within arm's reach lets you browse the shelf, choose a book, play, pause and seek with your fingertips. Controllers are optional.
How I built it
Full Volume has two halves: a book-processing pipeline (a desktop Studio app plus a CLI) and a WebXR player that runs in the Quest browser.
The pipeline: from EPUB to explorable world
The goal from day one was a pipeline, not a hand-made demo. If each book needs weeks of manual work, the library never grows. So everything is built to run as automatically as possible, with a human reviewing at each stage rather than doing the work.
- Parse. An EPUB, which is the text of the book, is split into chapters and paragraphs.
- Understand the book. Language models extract the cast of characters, the locations and the recurring props, and build a reference library of how each one looks.
- Find the scenes. The text is scanned for the points where the story moves somewhere new or something visually distinct happens. Each of these becomes a scene, anchored to its place in the text.
- Design the look. Each book gets an art direction suited to it, from a photorealistic film still to an oil painting. Characters, locations and props get reference images so they stay consistent from scene to scene.
- Paint the scenes. Each location gets a fixed, empty "set plate", and every scene there is generated as an edit of that plate. The room stays the same room, with only the people and props changing. Each result is checked automatically (did the camera move? is anyone cropped by the frame?) and regenerated if needed.
- Narrate. Speaker attribution decides who says every line. Each character gets a designed voice, the whole book is synthesised, and every line is transcribed back and compared with the text to catch mistakes. (Existing human-narrated audiobooks are supported too, aligned to the text.)
- Add sound. Sound effects and ambient beds are detected from the text, generated, and placed precisely on the narration timeline.
- Build the worlds. Each scene image becomes a 360° Gaussian-splat world. Outdoor locations can be built as full surround scenes.
The player: WebXR, so it's just a web page
I built the player on WebXR, so Full Volume runs entirely in the web browser on a Meta VR headset. You don't install anything or sideload an APK, and you don't need a store listing: open a link in the Quest browser, tap Enter VR and you're in the library. That makes it effortless to share with judges and testers, and a new book or a fix reaches everyone the moment it's published.
It's written in TypeScript and built with Vite, on:
- React and Three.js, via React Three Fiber, for the 3D scene;
- @react-three/xr for the WebXR session, hand tracking and touch pointers;
- Spark, a Gaussian-splat renderer for Three.js, to draw the scenes;
- drei and react-spring for 3D text and animation;
- Zustand for player state;
- Howler.js with the Web Audio API for narration, sound effects and spatial audio.
It's designed hands-first throughout: touch targets you press with a fingertip, sound and visual feedback instead of haptics, a pointing-hand onboarding demo, and UI text drawn in WebXR compositor layers so it stays crisp. Narration, sound effects and character voices are mixed through a spatial-audio graph, so each voice sits where its character is in the scene.
To develop it without wearing a headset all day, the player runs against a simulated Quest 3 on the desktop. A suite of unit tests and Playwright end-to-end tests drives that simulated headset: steering a fingertip onto the lectern and checking what was actually pressed. Everything is then verified on a real Quest 3.
Hosting on AWS
The player is hosted on AWS at vr.bitwise.ventures, all defined as infrastructure as code with CloudFormation:
- Amazon S3 (private) holds the player app and the published books: scene splats, images, narration and sound.
- Amazon CloudFront serves it all worldwide over HTTPS, with long-lived caching for the large, content-hashed assets.
- A CloudFront Function backed by a CloudFront KeyValueStore is an access gate on every request. It lets me open the site to judges, hand out short-lived guest codes, or lock it down, instantly and without a redeploy.
- Lightweight, anonymous usage beacons go into CloudFront's logs and are turned into usage reports and a CloudWatch dashboard.
Publishing a book is a single command: the Studio exports only the files the player reads, and syncs them to S3.
Challenges
Convincing splats from AI-generated scenes
This was the hardest problem by far, and it's very much an unsolved problem in general. Reconstruction normally needs many views of a real place. I had a single image, and a painted one at that. Off-the-shelf approaches produced worlds that looked fine from the exact capture point but fell apart as soon as you moved. Characters turned into flat cut-outs, rooms filled with invented clutter, and pictures sometimes came out as a framed print on a wall.
What moved this forward was exploiting the constraints of my own use case:
- I control how every image is made, so I can make images that reconstruct well. Each location is a single space with a fixed camera, the people stay whole and in the middle ground, and the images are full-bleed, with no borders.
- I know exactly where the viewer stands and roughly where they'll look.
- Every scene at a location shares the same underlying room.
Building on those guarantees, I fuse a high-fidelity, single-image reconstruction of the painting with a generated surrounding environment, and treat the join between them as a registration problem. That involves camera re-calibration (including roll and focal length), a non-rigid warp at the seam, depth fusion that respects object boundaries, and colour harmonisation, so the two halves agree on geometry and appearance where they meet. For open outdoor locations, the whole 360° surround is built outward from the painting instead. The result is a world that stays faithful to the artwork straight ahead and holds together as you turn and lean. It's a big step on from where I started, and there are clear further improvements on the roadmap.
Rendering on Quest 3
Gaussian splats are expensive. Rendering millions of them, at a stereo VR frame rate, on a mobile GPU, is a constant fight. I measured rather than guessed, and some results were surprising:
- The cost follows the number of splats, not the number of pixels. Lowering the render resolution barely helped, while thinning the splat count did.
- Turning off multisample anti-aliasing roughly doubled the frame rate without any visible loss.
- Moving UI text into compositor layers made it sharp at no frame cost.
There's still work to do. The frame rate isn't ideal in some of the densest scenes. Those measurements point to the next steps: smarter per-scene splat budgets, level-of-detail that suits the way splats are actually rendered, and simplifying parts of a scene the viewer will never get close to. Significant rework on how the splats are generated and making it custom to this application will also yield big speedups.
Automation without losing quality
Every manual fix that doesn't scale is a book that never ships. A lot of the work went into automatic checks, so the pipeline notices its own mistakes and fixes them: a camera that moved, a figure cut off by the frame, a mis-spoken line, a stray signature on a painting, a border around an image.
What I learned
I learned an enormous amount about what it really takes to go from a text file to a world you can stand in. That covers language-model analysis of a book's structure and cast, consistent image generation across hundreds of scenes, full-cast speech synthesis and its quality control, synchronised sound design, single-image 3D reconstruction, and the realities of shipping real-time splats on a standalone headset.
The biggest lesson was that the hard part isn't any one model. It's the pipeline: making each stage's output a reliable input to the next, catching failures automatically, and keeping a human in the loop only where taste matters. That's what turns a one-off demo into a library that keeps growing, with Treasure Island, The Mysterious Affair at Styles, All Quiet on the Western Front and The Great Gatsby already on the way.
What's next for Full Volume
- Further improvements to splat quality from generated scenes, which is the core of the experience
- More Quest performance work, so every scene holds a comfortable frame rate
- A continuous release pipeline for new books
Built With
- amazon-web-services
- gaussian-splatting
- meta-quest
- python
- pytorch
- react
- react-three-fiber
- three.js
- typescript
- webxr