Inspiration

Almost every AI app today looks the same: a chat box where you type text and get a wall of text back.

I wanted to explore a different question: What if an AI didn’t reply with paragraphs, but instead expressed itself through sound, light, motion, and color?

VieWs uses weather as a sensory canvas. You type how you feel, or a place you miss, and instead of chatting back, the AI translates your words into a living atmosphere you can see and hear.


What it does

A single structured response from NVIDIA Nemotron-3.5-Lightning on Nebius Token Factory simultaneously drives five sensory output layers:

  1. Real-Time Procedural Audio (AVAudioSourceNode): Synthesizes rain, wind gusts, thunder rumbles, snow chimes, and harmonic pads directly from math on the audio thread, no looping MP3s.
  2. Particle Physics (SpriteKit): Five dynamic emitters (rain, snow, fog, golden motes, and lightning) react in real time to the AI's numeric parameters.
  3. Adaptive Color & Light: Shifts the sky's color temperature and contrast so particles and typography stay legible across light pastel or deep moody scenes.
  4. Real-World Grounding (Tavily AI Search): When you mention a place ("i miss NY" or "walking through Kyoto"), Tavily pulls the live weather and temperature there right now so the AI grounds your emotional sky in physical reality.
  5. Minimalist Poetic Copy: A two-word title ("weary dusk", "muted drizzle") and one short sensory line—nothing more.

How I built it

  • Nebius Token Factory (nvidia/Nemotron-3_5-Lightning): Instead of generating conversational text, Nemotron outputs a strict 9-field JSON schema (precipitation, windSpeed, fogDensity, snowIntensity, clearness, lightningFrequency, colorTemperature, sceneTitle, atmosphereDescription) that plugs straight into the visual and audio engines in ~1 second.
  • Tavily AI Search: Detects geographic references in the prompt, fetches current local conditions in under 400ms, and feeds that live context into Nemotron.
  • Native Swift 6, SwiftUI, SpriteKit & AVAudioEngine: Built originally as an on-device experiment with Apple's 3B model, then re-architected during the hackathon to run on Nebius cloud inference with real-time web grounding.

Challenges I ran into

  • Taming Reasoning Tokens vs. Structured JSON on Nemotron: Calling nvidia/Nemotron-3_5-Lightning with default settings burned through the token budget inside chain-of-thought (reasoning_content), truncating the JSON or echoing system prompt examples. Disabling thinking ("enable_thinking": false) and removing unsupported strict: true schema keywords (minimum/maximum, which caused silent HTTP 400 downgrades) made structured output fast and reliable.
  • Prompt Engineering Over Model Size: Packing weather rules and 25 lines of colour-theory instructions into one prompt overloaded the model's attention and produced literal one-word titles like "exhausted" instead of "weary dusk". Leaning out the prompt and moving to a flat schema with clear field descriptions let Nemotron-3.5-Lightning deliver poetic results in ~1 second without needing a huge model.
  • Filtering Forecast Noise from Tavily: Web search snippets for weather often mix monthly averages ("October has 19 sunny days") or rate-limited domain disclaimers with live conditions. I tuned the Tavily query (exclude_domains, include_answer: true) and built a sentence filter so Nemotron only grounds the scene in what the sky is doing right now.

What I learned

  • AI as a Controller, Not a Chatbot: When an LLM outputs structured parameters instead of prose, it can orchestrate physics engines, DSP synthesizers, and shaders at once.
  • Lean Prompts Beat Bigger Models: A fast lightning model with a clean, focused schema outperforms a bloated prompt every time.

Built With

  • avaudioengine
  • dsp
  • ios
  • json-schema
  • macos
  • nebius
  • nvidia-nemotron
  • procedural-generation
  • sprite-kit
  • swift
  • swiftui
  • tavily
Share this project:

Updates

Submission history