Inspiration
Almost every AI app today looks the same: a chat box where you type text and get a wall of text back.
I wanted to explore a different question: What if an AI didn’t reply with paragraphs, but instead expressed itself through sound, light, motion, and color?
VieWs uses weather as a sensory canvas. You type how you feel, or a place you miss, and instead of chatting back, the AI translates your words into a living atmosphere you can see and hear.
What it does
A single structured response from NVIDIA Nemotron-3.5-Lightning on Nebius Token Factory simultaneously drives five sensory output layers:
- Real-Time Procedural Audio (
AVAudioSourceNode): Synthesizes rain, wind gusts, thunder rumbles, snow chimes, and harmonic pads directly from math on the audio thread, no looping MP3s. - Particle Physics (
SpriteKit): Five dynamic emitters (rain, snow, fog, golden motes, and lightning) react in real time to the AI's numeric parameters. - Adaptive Color & Light: Shifts the sky's color temperature and contrast so particles and typography stay legible across light pastel or deep moody scenes.
- Real-World Grounding (
Tavily AI Search): When you mention a place ("i miss NY" or "walking through Kyoto"), Tavily pulls the live weather and temperature there right now so the AI grounds your emotional sky in physical reality. - Minimalist Poetic Copy: A two-word title ("weary dusk", "muted drizzle") and one short sensory line—nothing more.
How I built it
- Nebius Token Factory (
nvidia/Nemotron-3_5-Lightning): Instead of generating conversational text, Nemotron outputs a strict 9-field JSON schema (precipitation,windSpeed,fogDensity,snowIntensity,clearness,lightningFrequency,colorTemperature,sceneTitle,atmosphereDescription) that plugs straight into the visual and audio engines in ~1 second. - Tavily AI Search: Detects geographic references in the prompt, fetches current local conditions in under 400ms, and feeds that live context into Nemotron.
- Native Swift 6, SwiftUI, SpriteKit & AVAudioEngine: Built originally as an on-device experiment with Apple's 3B model, then re-architected during the hackathon to run on Nebius cloud inference with real-time web grounding.
Challenges I ran into
- Taming Reasoning Tokens vs. Structured JSON on Nemotron: Calling
nvidia/Nemotron-3_5-Lightningwith default settings burned through the token budget inside chain-of-thought (reasoning_content), truncating the JSON or echoing system prompt examples. Disabling thinking ("enable_thinking": false) and removing unsupportedstrict: trueschema keywords (minimum/maximum, which caused silent HTTP 400 downgrades) made structured output fast and reliable. - Prompt Engineering Over Model Size: Packing weather rules and 25 lines of colour-theory instructions into one prompt overloaded the model's attention and produced literal one-word titles like "exhausted" instead of "weary dusk". Leaning out the prompt and moving to a flat schema with clear field descriptions let Nemotron-3.5-Lightning deliver poetic results in ~1 second without needing a huge model.
- Filtering Forecast Noise from Tavily: Web search snippets for weather often mix monthly averages ("October has 19 sunny days") or rate-limited domain disclaimers with live conditions. I tuned the Tavily query (
exclude_domains,include_answer: true) and built a sentence filter so Nemotron only grounds the scene in what the sky is doing right now.
What I learned
- AI as a Controller, Not a Chatbot: When an LLM outputs structured parameters instead of prose, it can orchestrate physics engines, DSP synthesizers, and shaders at once.
- Lean Prompts Beat Bigger Models: A fast lightning model with a clean, focused schema outperforms a bloated prompt every time.
Built With
- avaudioengine
- dsp
- ios
- json-schema
- macos
- nebius
- nvidia-nemotron
- procedural-generation
- sprite-kit
- swift
- swiftui
- tavily
