Media transcode

MediaTranscode is a C++20 and FFmpeg-backed media transcode framework centered on the graph DAG architecture. The entire project is implemented using Codex.

Comment

Inspiration

During OpenAI Build Week, I wanted to explore the boundaries of AI-assisted software engineering in a highly technical and performance-critical domain. I have long been interested in media processing pipelines that power modern video platforms. At the same time, I was fascinated by Codex harness engineering — the systematic use of structured prompting, iteration, and validation to let AI models build complex systems.

This led to the question: Can Codex, guided by a well-designed harness, create a media transcoding engine that meets or even exceeds industrial standards? The combination of deep media systems engineering and autonomous AI coding made this an exciting and ambitious experiment.

What it does

The project is an AI-native media transcoding engine that converts and optimizes video and audio files. It supports:

  • Format conversion (H.264, H.265, AV1, VP9, etc.)
  • Resolution scaling and bitrate control
  • Audio encoding and stream multiplexing
  • Basic filtering and post-processing

The engine aims to deliver high perceptual quality while maintaining strong performance characteristics, with the long-term goal of reaching or surpassing traditional industrial-grade transcoding solutions.

How we built it

I built the entire project using Codex harness engineering, with zero manual code writing. The development followed a structured iterative loop:

  1. Defining project functionality, coding constraints, validation requirements, quality scoring criteria, and entropy/cleanup protocols via AGENTS.md.
  2. Establishing the project's overall framework via ARCHITECTURE.md.
  3. Tracking scores via QUALITY_SCORE.md, a critical component of the project's autonomous iteration loop.
  4. Assigning distinct sub-agents to handle planning, coding, and code reviews.
  5. Codex generated individual modules (parsers, encoders, pipelines, optimizers) in small, testable units.
  6. Automated tests and quality benchmarks were run, with failures fed back to Codex for refactoring.
  7. I performed only subjective human evaluation (visual and audio A/B testing) and provided feedback through the harness.

This created a closed-loop system where the AI handled implementation and iteration, while I focused on direction and quality validation.

Challenges we ran into

  • Zero manual coding constraint: Every line of code had to come from Codex, making debugging and optimization significantly harder.
  • Context and codebase management: Maintaining coherence as the project grew required advanced prompt engineering and summarization.
  • Performance optimization: Achieving low-level efficiency (memory layout, SIMD, GPU acceleration) through natural language guidance demanded many refinement cycles.
  • Perceptual quality consistency: Ensuring output matched industry standards across edge cases relied heavily on repeated subjective testing.
  • Long-term complexity: Building a truly competitive engine is a multi-month effort, not a one-week sprint.

Accomplishments that we're proud of

  • Successfully built a functional media transcoding engine without writing a single line of code manually.
  • Established a reusable Codex harness framework for complex systems engineering.
  • Through AI-driven iterative optimization, impressive results have been achieved in both local and real-time video transcoding.
  • Demonstrated that AI can tackle deep technical domains when properly scaffolded.

What we learned

  • The real skill in AI-assisted development shifts from writing code to designing excellent requirements, feedback loops, and evaluation systems.
  • Media transcoding is an outstanding benchmark for AI coding capabilities due to its mix of algorithmic complexity and human-perceptible results.
  • Disciplined, iterative prompting with strong validation is essential to prevent drift and maintain quality.
  • Autonomous AI engineering shows tremendous potential but still requires thoughtful human oversight and patience.

What's next for Media transcode

This is a long-term project. Future plans include:

  • Expanding codec support and advanced features ( Multiple forms of hardware acceleration, adaptive streaming, AI-enhanced upscaling)
  • Improving automation and self-optimization capabilities within the harness
  • Rigorous benchmarking against industry leaders (FFmpeg, commercial encoders)
  • Exploring distributed transcoding and cloud-native architectures
  • Continuing to push the boundaries of what fully AI-generated systems can achieve in performance-critical software

I’m excited to keep iterating and evolving this engine well beyond Build Week.

Built With

Share this project:

Updates