VidFetch — Devpost project page

Inspiration

Every app that works with video eventually needs to get a video from a URL. The first time, you write twenty lines around yt-dlp. The second time you discover that yt-dlp breaks weekly, that YouTube changes things that break extraction, and that the machine running the download needs ffmpeg, disk headroom, and a way to not fall over when someone submits a four-hour video.

I kept rebuilding the same ingestion pipeline for different products. VidFetch is that pipeline, extracted into an API so the next product can just call it.

What it does

One API call: give it a URL, get the file in your own storage bucket.

• Fetch video or MP3 from a URL
• Trim server-side before delivery
• Deliver directly to S3, R2, or GCS
• No extraction infrastructure to run yourself

How I built it

The API surface is deliberately small, because the value is in what happens behind it.

1. Preflight, then work. Extraction failures are usually discoverable before you download anything — the video is region-locked, it is longer than the plan allows, or above the size cap. Checking first turns a two-minute failure into an immediate, meaningful error, and stops storage costs from accumulating on jobs that were never going to succeed.

2. Streaming straight to storage. The naive implementation downloads to local disk, then uploads. For a large file on a small server this fails at the worst possible moment — halfway through, with a full disk. The pipeline streams from the extractor into the destination bucket, so the peak disk usage is a buffer rather than the file size.

3. Server-side trimming. Trimming before delivery means the user gets only the part they asked for and never pays for the rest. ffmpeg does the trim during the pipeline rather than as a second download-and-reupload round trip.

4. Signed delivery. The response includes a signed URL to the object in the user's own bucket. The file belongs to the user's storage, not to mine, which keeps the cost model honest and the data ownership unambiguous.

5. Plan-aware limits. Minimum billed duration, maximum length, and size caps are enforced before extraction. Quotas that are checked after the work is done are quotas that lose money.

Challenges I ran into

Extraction is a moving target. Anything built on top of YouTube has a maintenance cost that never goes to zero. Treating extraction as an isolated, replaceable stage rather than something woven through the pipeline is the only way this stays maintainable — when it breaks, one adapter changes and the rest keeps working.

Disk is the failure mode nobody plans for. Concurrency plus large files equals a full disk, and a full disk takes down everything on the machine, not just the failed job. Streaming to storage and bounding concurrent downloads solved a problem that only appears under load.

Datacenter IPs get treated differently. Requests from cloud IP ranges meet bot checks that a home connection does not. This is an operational reality rather than a code bug, and it has to be handled with a fallback path and honest error reporting rather than silently retried until the user's credits are gone.

Accomplishments that I'm proud of

• Streaming directly to the user's bucket. It removes the entire class of disk-full failures and makes the cost model predictable.
• Preflight checks that fail fast and refund cleanly instead of burning time and storage.
• A small, boring API surface that does not leak the complexity of what it sits on.

What I learned

Wrappers around unreliable things need to expose the unreliability, not hide it. My first instinct was to retry silently. That is worse: the user waits, burns time, and gets a vague failure. Reporting "this specific video is blocked" in two seconds is a better product than retrying eight times and failing slowly.

Billing has to be designed with the pipeline, not bolted on. Minimum durations, caps, and refund-on-failure are not billing features — they are what stops a working pipeline from losing money on cheap jobs.

What's next for VidFetch

• Webhook completion. Poll for a job status today; a webhook is better for anything long-running.
• Batch requests. Submit a list of URLs and get one callback when the set completes.
• Audio-only post-processing. Normalisation and format conversion during the pipeline, so the output is publish-ready rather than just extracted.
• Self-hosted option. Some teams cannot send URLs to a third party. Packaging the pipeline for their own infrastructure is the natural answer.

Built With

• react