sunken cipher

sunken cipher is a gemini based story teller

Comment

The Sunken Cipher // Devpost Project Submission

Inspiration

The ocean covers over 70% of Earth's surface, yet over 80% of it remains unmapped and unexplored. Hidden beneath thousands of meters of crushing pressure and eternal darkness lie forgotten artifacts, lost shipwrecks, and speculative histories waiting to be uncovered.

The Sunken Cipher was born from a simple question: What if any ordinary object around us was actually an ancient, corroded relic recovered from the deepest abyssal trenches of the ocean floor?

We wanted to create more than just a typical AI chat widget or image descriptor. We envisioned a dark, immersive sci-fi deep-sea research terminal aboard the fictional research submersible Nautilus-IX—where Google Gemini acts as an active expedition director orchestrating text narrative, WebGL underwater shaders, HUD target overlays, and Web Audio sonar frequencies in real time.

What it does

The Sunken Cipher transforms any user-uploaded photo into an interactive, multimodal deep-sea archaeological expedition:

  1. Submerged Relic Transformation: When a user uploads or captures a photo, the app submerges the object into a live WebGL viewport complete with procedural marine snow, caustic sunlight reflections, mineral encrustation, water ripple refraction, and sonar sweeps.
  2. AI Relic Identification & Backstory: Powered by Gemini, the system analyzes visual cues on the object to generate a structured artifact card detailing its speculative relic name, estimated oceanic depth (down to 11,000 meters), trench location, structural integrity percentage, and a speculative backstory.
  3. Interactive Sub-Comms Interrogation: Users chat directly with Dr. Aris Thorne, Chief Historian & Deep-Sea Archaeologist aboard the Nautilus-IX.
  4. Gemini as an Interface Director: As Dr. Thorne responds, Gemini returns structured JSON directing 4 real-time modalities simultaneously:
    • In-Character Narrative: Rich dialogue exploring the object's materials and history.
    • WebGL Shader Controls: Dynamically shifts u_zoom, target u_focal_point, color spectrum (SONAR_GREEN, THERMAL_AMBER, XRAY_BLUE, UV_SPECTRUM), and u_noise_glitch.
    • HUD Canvas Overlays: Draws dynamic SVG bounding boxes, coordinate targets, and glyph translation labels directly over the object.
    • Web Audio Sonar Engine: Triggers procedurally synthesized sonar pings with custom base frequencies, pulse rates, and wave shapes.

How we built it

The system architecture seamlessly bridges frontend graphics, browser audio synthesis, and server-side AI model orchestration:

                      +-----------------------------+
                      |     User Photo Input        |
                      +--------------+--------------+
                                     |
                                     v
                      +-----------------------------+
                      |   WebGL Underwater Shader   |
                      |  (Marine Snow / Palette Pass|
                      +--------------+--------------+
                                     |
                                     v
                      +-----------------------------+
                      |     Gemini Vision API       |
                      | (Multimodal Artifact Scan)  |
                      +--------------+--------------+
                                     |
                                     v
                      +-----------------------------+
                      | Structured Relic Metadata   |
                      | (JSON Schema + Zod Spec)    |
                      +--------------+--------------+
                                     |
                                     v
+-------------------------------------------------------------------------+
|                  The Sub-Comms Terminal (Interrogation Loop)            |
|                                                                         |
|  User Input --------> Gemini API (Expedition Director)                  |
|                             |                                           |
|                             +---> Text Dialogue (Dr. Aris Thorne)       |
|                             +---> WebGL Uniforms (Zoom, Focal, Palette) |
|                             +---> HUD SVG Overlay Annotations           |
|                             +---> Web Audio Sonar Ping Triggers         |
+-------------------------------------------------------------------------+
  • Frontend & UI Architecture: Built with Next.js (App Router), React 19, and strict TypeScript. The design follows UI_THEME.md with deep-sea glassmorphism tokens, bioluminescent cyan highlights, and retro terminal typography.
  • Custom WebGL Shader Engine: Hand-crafted GLSL shaders in components/ScanViewport.tsx implementing fractal Brownian motion (fBm) water distortion, procedurally drifting marine snow particles, caustic light propagation, scanlines, and instant palette mapping.
  • Procedural Web Audio Engine: Built natively in components/AbyssalAudioEngine.ts using the Web Audio API to create dynamic sonar ping synthesis with exponential gain decay envelopes without relying on pre-recorded sample files.
  • Structured Gemini API Integration: In app/api/gemini.ts, we connect to Gemini (gemini-3.5-flash / gemini-1.5-flash) using strict response_mime_type: "application/json" schemas validated on both ends via Zod schemas (directorPayloadSchema).
  • Security First: The Google AI API key is kept strictly on the server side via Next.js Route Handlers (/api/scan and /api/interrogate), ensuring zero client-side key exposure.

Challenges we ran into

  1. Enforcing Dual-Output Schema Strictness: Standard LLM prompts often hallucinate non-valid JSON or return invalid enum values when asked to generate story text and math/shader coordinates simultaneously. We overcame this by writing rigorous JSON Schemas (response_schema) inside app/api/gemini.ts and validating all payloads with Zod before passing them to the rendering loop.
  2. 60 FPS Real-time WebGL Interpolation: Updating GLSL uniforms dynamically based on AI responses can lead to harsh visual snaps. We implemented smooth uniform interpolation loops in components/ScanViewport.tsx to gently pan camera focal points and transition zoom levels over time.
  3. Browser Audio Autoplay Policies: Modern browsers block audio context initialization until explicit user interaction. We designed an arm() gesture listener in components/AbyssalAudioEngine.ts to seamlessly initialize the AudioContext during the user's initial image selection or click.
  4. Grounded vs. Speculative Prompt Engineering: Balancing creative sci-fi story generation with accurate visual grounding proved tricky. We fine-tuned the master prompt system instructions to instruct Gemini to anchor all speculative lore in visible material details (rust, glass, brass, shapes) present in the user's photograph.

Accomplishments that we're proud of

  • True Multimodal AI Direction: Moving beyond simple text generation—Gemini acts as an active director over text, graphics shaders, HUD visuals, and synthesizer audio in a single unified API payload.
  • Atmospheric Deep-Sea Experience: The introductory dive descent sequence, glassmorphism UI, sonar ping acoustics, and dark marine aesthetic make users feel like they are operating a real underwater ROV terminal.
  • Zero Heavy 3D Dependencies: Built custom GLSL shaders and procedural Web Audio engines directly on browser primitives without heavy external libraries like Three.js or Howler.
  • Robust Type Safety & Security: 100% strict TypeScript with Zod runtime validation and secure server-proxied API keys.

What we learned

  • How to leverage Gemini's structured output capability (response_schema) to control non-text application state (shaders, audio parameters, coordinate overlays).
  • Advanced GLSL fragment shader techniques, including procedural noise generation, marine snow particle math, and caustic light simulation.
  • Crafting system instructions that maintain consistent roleplay persona while remaining strictly bounded within numerical limits.

What's next for sunken cipher

  • Expedition Catalog & Relic Museum: Allowing researchers to save analyzed relics into a personal offline-first Abyssal Museum logbook.
  • 2D-to-3D Gaussian Splatting / NeRF: Converting 2D photographs into fully rotatable 3D underwater relics inside the WebGL canvas.
  • Co-Op Submarine Terminals: Real-time WebSocket multi-user interrogation sessions where multiple researchers can inspect the same artifact together.
  • Dynamic Ambient Hydrophone Engine: Adding ambient deep-ocean acoustics (hydrothermal vent hiss, deep hull creaks, distant bio-sonar calls) procedurally modulated by target depth meters.

https://github.com/BeanieMen/sunken-cipher

Built With

Share this project:

Updates