Inspiration
We both come from backgrounds where we edit on our iPhones on a daily basis, but to edit even a simple video with cool effects is a tedious process, not to mention the learning curve to make cool clips with clean transitions and effects.
What it does
Morpho is a live video-editor that allows you to see the video on one screen, while showing the edited AI version on the other screen of the iPhone Duo, which is edited by voice commands. The lower screen is treated as the controller, whereas the upper screen is treated as the visualizer.
How we built it
Challenges we ran into
The challenges that we faced or ran into were:
- The audio API, the local audio tool embedded within Apple Intelligence, had a really very hard time connecting, and we had a very hard time segmenting our frames and performing audio inference on top of it.
- Running continuous segmentation inference on top of live video frames to infer shapes that were of interest to us.
- Piping everything from audio transcription into Apple's on-device LLM prompt engineering and then passing it all into the Decart Lucy API to continuously stream our edited video.
Accomplishments that we're proud of
We were successfully able to use Apple's on-device intelligence across audio transcription and their local LLMs to prompt engineer. We were able to connect all this to Decart Lucy's video editing API that allowed us to edit videos in real time while recording the videos live. We were also able to use Apple's Vision APIs to successfully segment objects of interest across the entire video for users to be able to simply tap, speak and edit with less than 25ms latency across our frames.
What we learned
Building Morpho for the new iPhone Duo taught us to treat the fold as a layout input: Apple's new ArrangementView and reserved-region APIs place my live camera feed and controls around the hinge automatically, while the onHingeChange lets the fold angle tilt a merged 3D scene like a diorama. I also used Duo's new dual-display and dual-camera APIs, streaming both front cameras at once and showing the subject a live preview on the outer screen while I film from the inner one. With Swift, we were able to combined iOS 27's on-device Speech and Foundation Models, and the app still runs fully offline when no AI or API access is available. Bitrig made all of this possible without Duo hardware, as its in-built iPhone Duo simulator let us test each fold position and an AI agent that commits each change made the project history read like a changelog.
What's next for Morpho
We want to incorporate a dual camera view to then merge them into sort of an 3d reality based video form factor to allow users to augment via AI on video frames merged together from multiple camera feeds.
Log in or sign up for Devpost to join the conversation.