Nimbus

tech stack diagram

Nimbus puts the power of precise drone flight into the hands (or lack thereof, it's hands-free!) of anyone. When given a command, it leverages Freesolo's LLM fine-tuning to handle contextual prompts like: "fly to the pillar in the distance" and convert verbal instructions into drone actions.

Visual input: gimbal-mounted drone camera, audio input: Airpods microphone, spatial input: drone and Airpods accelerometers, computing: iPhone neural engine + cloud compute.

Every prompt triggered by "Hey Nimbus" goes to ElevenLabs to be converted into a string; we then feed it into a Freesolo-trained LLM fine-tuned to understand user intent in the context of drones, which returns a strict JSON of drone actions. In the case that image input is needed (e.g., an object in real life is referred to as a reference point), Freesolo recognizes this and delegates the job to Google's Gemini to use as a VLM to feed bounding box coordinates.

Nimbus by default is in "follow mode", where it uses a local YOLOv8 model on the iPhone neural engine to keep track of and follow the drone operator (assisted in facing the right direction by Airpods accelerometers). When prompted, Nimbus looks up and takes in the scene in front of you to see what you see and, with that information, your intent.