ARC React

Summary
AR/VR/Multitouch Summary

Inspiration

Vibe coding has enabled developers everywhere to write code by both manually editing and writing natural language prompts.

Code editors on the phone are an underexplored market but rapidly expanding market. Mobile phone adoption is massive and there is no go to option yet for a mobile IDE.

What it does

We have decided to improve upon existing options by providing an Augmented reality IDE experience to view your code files in 3D space. View your codebases entire architecture even on a smaller screen. Use hand gestures, speech to text or just edit code with your keyboard.

The AR mode also allows you to use spatial scenes to change your distracting background to a list of different scenes such as Cambridge Gardens.

You can also check your app's mongoDB connection and stream your channges optionally to the intellij plugin or just save to your github repo from your phone for version control.

It has everything you might want in an IDE, you can view terminal output, debug etc.

How we built it

Arc React is split into four moving parts: the iOS app, a FastAPI backend on Vercel, a JetBrains IntelliJ plugin, all stitched together by MongoDB Atlas, OpenAI, and ElevenLabs.

iOS app (Swift, SwiftUI, ARKit, RealityKit). The whole IDE JetBrains-styled toolbar, Monaco editor in a WKWebView, Junie chat panel, file tree runs natively on iPhone in SwiftUI.

AR mode boots an ARSession with horizontal-plane detection, then RealityKit anchors floating 3D PanelEntity objects to the user's desk. We render every panel's content (editor, preview, terminal, MongoDB graph) into Core Graphics → CGImage → TextureResource so each surface is a real-time-updating texture on a flat 3D entity.

Hand tracking runs through MediaPipe + the iOS Vision framework (21 joints), interpreted into pinch / palm / wave gestures by a custom GestureInterpreter that drives the existing grabbedPanel infrastructure for pinch-and-drag on AR entities.

Shake detection uses UIKit's motionEnded.

Voice in/out: iOS SFSpeechRecognizer for STT, ElevenLabs streaming TTS for JARVIS replies, queued through a custom JarvisVoice audio pipeline (NSObject + AVAudioPlayerDelegate) so multiple lines don't cut each other off.

The backend uses a python fast api service to send code updates from the app to mongoDB. It also supports sending the images / diagrams to the openai API platform to embed images into vectors and save them in MongoDB vector db.

The intellij plugin built in kotlin, calls the backend too, to get access to code changes and images saved in the vector db.

Challenges we ran into

Hand gesture tracking on the app was a signifigant challenge we overcame. Adding in different "scenes" and making sure file windows stayed still in space was another challenge.