TidePool — Cursor for Audio Engineering

Every sound, yours to shape.

Inspiration

AI has transformed how we write code, generate images, and create content, yet audio editing remains surprisingly complicated. Once a song or video is exported, its individual sounds are mixed together, making even simple changes difficult without professional software or access to the original recordings.

We wanted to change that.

What if you could remix your favorite song without its studio tracks? Click on an object in a movie and change its sound? Or replace every punch in an action scene with a cartoon sound effect, simply by describing what you want?

Inspired by tools like Cursor, we wanted to bring the same intuitive, conversational experience to audio engineering and entertainment.

That's why we built TidePool — Cursor for Audio Engineering.

By combining audio understanding, computer vision, generative AI, and natural-language reasoning, we're making sound design accessible to anyone with a creative idea.

What it does

TidePool is an AI-powered audio and video editor that lets users isolate, modify, generate, and replace individual sounds within existing media.

Instead of treating audio as one unchangeable recording, TidePool transforms it into editable layers that users can manipulate manually or through conversation.

Key features

  • Intelligent audio separation: Separate vocals, drums, bass, instruments, dialogue, and environmental sounds. A drummer can remove the drums from a favorite song to create a custom backing track.

  • Conversational editing: Describe an edit in plain English, such as "Add reverb to the vocals" or "Remove the barking between 10 and 15 seconds," and TidePool translates it into editing operations.

  • Computer vision and object-aware audio: Using YOLO, ByteTrack, and Meta's Segment Anything models, TidePool can track visible objects and use them to guide audio editing. Users can select a dog to target its barking or request "Play a gong every time this person appears."

  • AI-generated sound effects and replacement: Generate custom sounds using Stable Audio and replace existing audio events. For example, replace repeated punches with cartoon effects while preserving the surrounding soundtrack.

  • Speech-aware editing: Using word-level transcription, TidePool can trigger edits based on spoken dialogue. Users can request "Play a sound every time someone says 'what'" or automatically bleep profanity.

  • Creative mixing and accessibility: Independently adjust sound layers, apply effects, amplify dialogue, and reduce distracting background audio.

TidePool is designed for musicians, filmmakers, content creators, video editors, and anyone who wants more creative control over existing media.

How we built it

We built TidePool as a full-stack, multimodal AI application using React, TypeScript, Vite, and Tailwind CSS for the frontend and Python, FastAPI, and SQLite for the backend. FFmpeg and NumPy handle media processing, audio effects, synchronization, and exports.

Rather than relying on one AI model, we developed a modular pipeline that combines specialized systems for audio detection, computer vision, natural-language reasoning, and sound generation.

Audio understanding and separation

We integrated YAMNet to identify audio events such as barking, footsteps, speech, and impacts, along with approximate timestamps.

For music classification, we explored PANNs and AST-based models to help determine whether audio should be routed to a specialized music separator.

We use AudioSep and Meta's SAM Audio for text-guided and experimental visual-guided sound isolation, alongside Demucs and RoFormer-based models for music stem separation.

We benchmarked multiple approaches to compare separation quality, inference speed, and GPU memory requirements, allowing us to make more informed model-selection decisions.

Computer vision and scene understanding

Our computer vision pipeline combines YOLO, ByteTrack, and Meta's Segment Anything models to detect, track, and segment objects across video frames.

We also integrated Qwen3-VL-2B-Instruct to analyze sampled frames and describe visible actions, providing additional context for creative editing.

Together, these components allow TidePool to connect visual objects with audio operations, including object-guided sound isolation and effects triggered by tracked appearances.

OpenAI and LangChain

We use OpenAI models with structured outputs to interpret natural-language instructions and translate them into validated editing plans.

We also integrated LangChain to resolve conversational references to existing sound effects.

For example, after inserting several duck-quacking effects, a user might say, "Replace all those quacks with loud gongs."

Using ChatOpenAI, structured outputs, and Pydantic validation, LangChain matches the user's reference against actual effect labels stored in the project. It returns a match, no match, or ambiguity, allowing the backend to select the correct layers without inventing nonexistent sounds.

This separates AI reasoning from deterministic editing execution, making conversational workflows more reliable.

Generative audio and speech recognition

We integrated Stable Audio through a configurable sound-generation workflow to create custom effects from text prompts. These effects can be inserted or used to replace existing sounds.

We also use faster-whisper for word-level speech transcription, allowing users to trigger edits based on specific words or phrases.

End-to-end pipeline

  1. Upload: Users upload audio or video, which FFmpeg preprocesses.
  2. Understand: YAMNet, music classifiers, computer vision, and optional Qwen3-VL analysis provide relevant media context.
  3. Plan: OpenAI interprets editing instructions, while LangChain resolves references to existing effects when needed.
  4. Execute: Specialized audio models separate sounds, Stable Audio generates effects, and FFmpeg/NumPy render modifications.
  5. Preview and export: Users compare results, apply or undo edits, and export their finished media.

We used Docker and RunPod GPU infrastructure to support computationally intensive model inference.

Challenges we ran into

One of our biggest challenges was achieving reliable audio separation. Real recordings contain overlapping voices, instruments, and environmental sounds, making it difficult to isolate one source without introducing artifacts or affecting surrounding audio.

We experimented with different models, inference windows, separation passes, and refinement techniques to improve results.

Another major challenge was connecting computer vision with audio. Detecting an object doesn't necessarily mean it's producing sound, so we had to carefully synchronize object tracking, segmentation masks, audio timestamps, and separation models.

We also faced GPU memory constraints, model dependency conflicts, and inference latency when integrating multiple large AI systems.

Finally, making conversational editing reliable required grounding AI-generated instructions in actual project data, validating operations, and keeping edits synchronized and reversible.

Accomplishments that we're proud of

We're proud that TidePool goes beyond being another audio separator with a chatbot attached.

We built a multimodal editing system that combines audio detection, computer vision, generative audio, and conversational reasoning into one creative workflow.

Some of our biggest accomplishments include:

  • Integrating object tracking with audio editing, enabling visual-guided sound manipulation and object-triggered effects.
  • Building a sound-replacement pipeline capable of identifying and replacing recurring audio events.
  • Benchmarking multiple audio separation models and developing specialized processing routes.
  • Implementing conversational editing using OpenAI and LangChain with structured, validated operations.
  • Creating a multitrack editor with synchronized playback, effects, previews, undo, and media export.

Most importantly, we made complex audio engineering capabilities more approachable for everyday creators.

What we learned

Building TidePool taught us that integrating AI models is only one part of creating a useful product.

Audio models process waveforms, computer vision models analyze frames, speech recognition produces word timestamps, and language models interpret instructions. Connecting these different representations into one coherent editing workflow required careful engineering.

We also learned that different models excel at different tasks. Specialized music separation models can outperform general-purpose sound separators, making intelligent model selection essential.

Working with LangChain and structured outputs taught us how to translate flexible conversational instructions into grounded, validated actions.

Above all, we learned that the best creative AI tools combine intelligent automation with precise human control.

What's next for TidePool

Our goal is to evolve TidePool into a more holistic, AI-powered media editing platform that brings audio and video editing together in one seamless workspace.

We plan to expand beyond audio manipulation by integrating video editing capabilities, allowing users to modify visuals, synchronize effects, and transform entire scenes through natural-language instructions.

We're also focused on fine-tuning our audio segmentation and source separation models to improve accuracy, reduce artifacts, and better isolate overlapping sounds. By refining our sound detection and replacement pipelines, we aim to make recurring sound replacements more precise, consistent, and natural.

Ultimately, we envision TidePool as an all-in-one creative environment where anyone can edit, remix, and reimagine audio and video without needing professional production expertise.

Built With

Share this project:

Updates

Submission history