METAL — Your Gaming Memories, One Search Away

Inspiration

Some of the best gaming moments happen when you least expect them. Maybe your friend says something hilarious, someone pulls off an unbelievable play, or your entire team starts laughing.

The problem is finding those moments afterward.

Even if you recorded everything, that moment might be buried somewhere in hours of gameplay. You remember what happened, but you don't remember which video it's in or when it happened.

That inspired me to build METAL, a gaming memory platform designed to make those moments easier to find, relive, and share.

I wanted to go beyond traditional clipping tools and create something that understands more about the recordings themselves.

What It Does

METAL combines AI-powered search, gameplay timelines, multiple perspectives, and video editing into one application.

Recall — Search Your Gaming Memories

Recall lets you search your recordings using natural language.

If you remember something your friend said, you can search for it and jump directly to that moment. METAL uses speech transcription and semantic search, so your search doesn't always have to match the original dialogue word for word.

Recall also includes experimental visual scene search. You can describe what a scene looks like, such as "jungle cliff" or "a dark hallway with a monster," and METAL searches indexed video frames for visually similar moments.

Visual results are labeled as approximate because the model can make mistakes.

Time Machine — Relive Your Sessions

Time Machine lets you revisit your gaming history through organized sessions and recordings.

Instead of digging through individual video files, you can navigate your past gaming moments through a timeline and jump back into the footage.

Multi-POV — Experience the Same Moment From Different Perspectives

When two friends record the same moment, they usually capture it from completely different angles.

METAL lets you group those recordings, align their timelines, and watch both perspectives together.

You can adjust the synchronization and switch between individual views or side-by-side playback.

Story Studio — Turn Memories Into Videos

Once you've found a moment, you can bring it into Story Studio.

You can trim the clip, add captions, preview the result, and export an actual MP4.

For synchronized recordings, METAL can combine both perspectives into a single split-screen video.

The final result is a real rendered file, not just a visual effect inside the browser.

A Guided Experience

METAL also includes a polished interface and a guided demo to help new users understand the features. I wanted the application to feel approachable even though it handles several technical workflows.

How I Built It

I built METAL using Next.js, React, TypeScript, Node.js, SQLite, and Drizzle ORM.

For the AI features, I integrated local models:

• Whisper transcribes speech from gameplay recordings.
• MiniLM creates text embeddings for semantic transcript search.
• CLIP creates visual embeddings so scene descriptions can be compared with sampled video frames.

The application stores recording metadata, transcripts, sessions, synchronized POV groups, and editing projects in SQLite.

A separate Node.js worker handles background processing, including media indexing and video exports, so the web application can remain responsive.

For video processing, I used FFmpeg to extract frames, trim recordings, synchronize footage, apply captions, and render finished MP4 files.

I also built the interface around a consistent dark gaming aesthetic, with clear navigation between Recall, Time Machine, Multi-POV, and Story Studio.

Challenges I Ran Into

Getting different recordings synchronized

Two players might record the same event, but their videos don't start at the same time.

I had to build a shared timeline system using recording offsets so both perspectives could play together and remain aligned in the final export.

Making AI search useful and reliable

Speech search worked well with semantic matching, but visual search was much harder.

Some visual descriptions matched unrelated scenes, while short searches sometimes returned nothing.

I experimented with different approaches and confidence thresholds. Rather than pretending the model recognizes every action correctly, I kept visual search experimental and added clearer confidence labels and search guidance.

Processing video without slowing down the application

Transcription, visual indexing, and video rendering can be resource-intensive.

Separating the background worker from the application helped keep the interface responsive while those tasks ran.

Building a connected user experience

I didn't want METAL to feel like several unrelated tools.

One of the biggest challenges was making the whole flow feel natural: find a moment, open the recording, explore another perspective, and turn it into a shareable video.

Accomplishments I'm Proud Of

I'm especially proud that METAL's main features work together as one application.

• Built semantic speech search that can retrieve dialogue even when the search wording differs.
• Added experimental visual scene retrieval using real gameplay frames.
• Built synchronized playback for multiple players' recordings.
• Created genuine split-screen MP4 exports with captions.
• Developed a guided experience that connects the main features.
• Tested the application with real gameplay footage, database migrations, browser automation, and actual rendered video files.

One of my favorite parts is the Multi-POV workflow. Being able to take two separate gameplay recordings, line up the same moment, and export them as one video made the project feel like a complete product.

What I Learned

This project taught me a lot about connecting different technologies into one working system.

I learned more about semantic search, video embeddings, media processing, background workers, database design, and synchronization.

I also learned that adding AI to a project isn't enough by itself. The AI needs to be evaluated carefully, and the interface needs to communicate its limitations.

Most importantly, I learned how much work goes into turning a technical idea into something people can actually use.

What's Next for METAL

There are several features I'd love to build next:

• More accurate visual search that understands actions across multiple video frames.
• Easier synchronization between friends' recordings.
• Shared gaming libraries and collaborative clip creation.
• Better automatic highlight discovery.
• Expanded editing tools and publishing options.
• Support for more users, games, and devices.

Right now, METAL is a working local-first prototype with a focus on demonstrating the full experience.

My long-term goal is to make gaming memories as easy to search and revisit as photos on your phone.

Don't just record the moment. Remember it.

Built With

• clip
• drizzle-orm
• ffmpeg
• minilm
• next.js
• node.js
• obs
• react
• sqlite
• typescript
• whisper