FFprompt

• 

  Screenshot of FFprompt UI

Inspiration

FFmpeg is a complicated program that can do incredible things, but sometimes you need to do small tasks, fast, without the hassle of remembering the very complicated cli flags. The inspiration came from wanting to convert videos to gifs on the regular.

What it does

Use natural language to do some common ffmpeg commands entirely in the browser and without ever sending any data to a server.

You attach a video and chat with it e.g. "gif me", "get rid of audio", "scale up".

How we built it

Using the Prompt API I used the intitialPromps option to set a system message and give it some n-shot prompt examples so it could map back the query to a Map of some common ffmpeg tasks. Once the ffmpeg command was determined it utilised ffmpeg.wasm project to run it inside the browser as a wasm module.

Challenges we ran into

Initially I tried to get the Prompt API to return ffmpeg commands directly but it wildly hallucinated those, returning legit looking cli flags. I pivoted to use stored cli flags of known working ones to map the natural language query to my own string as the closest match.

Accomplishments that we're proud of

That is actually works! It's amazing what you can do on the client without any servers involved.

What we learned

N-shot prompting is key to getting consistent responses and the bigger prompt context window will help explore more complex structured data approaches.

What's next for FFPrompt

I want to improve the parsing of queries to allow it to parse out things like time, resolution, and keywords like loop in order to do more complex ffmpeg operations e.g. trim a video at 00:02 to 00:30, scale video to 768x480.

I also want to explore using a project like TypeChat to add some schema validation and checking of response with the likes of zod.

Another area is ffmpeg multi threaded version that requires COEP and COOP headers to utilise SharedArrayBuffers as well as look into using the WORKERFS mounting to allow large files to be loaded while avoiding OOM errors.

Built With

• chromeai
• ffmpeg
• prompt
• react
• shadcn/ui
• tailwindcss
• typescript
• wasm