-
-
Model Rankings Table: Empirical pass rates comparing Claude, DeepSeek, Llama 3, Mistral, Gemini, ChatGPT, and Grok with zero vibes.
-
Task Matrix: Binary PASS/FAIL test taxonomy evaluating IEEE-754 precision, regex state mutation, calendar drift, and race conditions.
-
Failure Forensics: Detailed breakdown uncovering architectural blind spots where frontier models fail silently in production code.
-
Execution Telemetry: Isolated test harness output verifying deterministic binary assertions without subjective LLM-as-a-judge scoring.
-
FrontierBench Leaderboard: Live interactive dashboard tracking frontier LLM code correctness across real-world engineering edge cases.
Inspiration
AI coding assistants are confidently wrong. When developers ask frontier large language models (ChatGPT, Claude, Gemini, DeepSeek, Grok) to generate production code, the generated code often:
Passes visual human inspection: It looks clean, idiomatic, well-commented, and structurally sound. Fails catastrophically in production: It harbors silent IEEE-754 precision loss, stateful regex mutations, calendar month rollover drift, and async race conditions.
Existing LLM benchmarks (like HumanEval or SWE-bench) either rely on synthetic interview puzzles or subjective "LLM-as-a-judge" grading. We built FrontierBench for SLAY HACKS to establish a deterministic, zero-vibes benchmark: evaluating actual frontier model outputs against real-world engineering edge cases with strict binary PASS/FAIL execution.
What it does
FrontierBench is an open-source evaluation harness and live interactive leaderboard comparing frontier AI models on real-world edge-case resilience:
Deterministic Multi-Model Stress-Testing: Evaluates leading frontier models (Claude 3.5 Sonnet, DeepSeek-V2.5, Meta Llama 3.1 405B, Mistral Large 2, Gemini 1.5 Pro, ChatGPT-4o, and Grok 2) across 7 critical production blind spots.
Zero-Vibes Binary Verification: No LLM grading or heuristic scoring. Every test case executes raw generated code in isolated test harnesses with deterministic unit assertions.
Key Empirical Discoveries:
No Model Scored 100%: Every single frontier and open-weight model tested exhibits distinct architectural blind spots. Claude & DeepSeek fail on 64-bit Snowflake ID deduplication, coercing large integer IDs into IEEE-754 floats and silently dropping real database records. ChatGPT fails on stateful RegExp global flags (/g), causing repeated validations on identical input to alternate between true and false. Grok suffered the highest failure rate, failing 4 out of 7 critical engineering tasks. Public Interactive Leaderboard: A lightweight, dark-mode web leaderboard displaying model pass rates, detailed task taxonomies, and reproducible test outputs.
How we built it
Language & Test Runner: Python 3.12 and Node.js deterministic test runners. Benchmark Tasks: Handcrafted edge cases targeting OWASP CWE-601 open redirects, floating-point penny leakage, async race conditions, and leap-year calendar rollover. Data Collection: Direct responses collected from the native interfaces of all 7 frontier models without synthetic modification. Interactive Leaderboard: Clean HTML5/CSS/JavaScript cockpit deployed live on GitHub Pages.
Challenges we ran into
Eliminating Benchmark Leakage: Crafting prompt variations that don't match standard LeetCode or HumanEval training corpora so models cannot simply recite memorized solutions. Multi-Runtime Execution Isolation: Running tests across Python and JavaScript runtimes while ensuring strict environment isolation and sub-millisecond execution. Objective Metric Design: Structuring unit tests so that differences in coding style or library imports never cause false failures—only genuine algorithmic or semantic flaws fail the test.
Accomplishments that we're proud of
Uncovering real architectural blind spots in frontier models like Claude 3.5 Sonnet and GPT-4o that have never been documented on standard benchmarks. 100% Deterministic Reproducibility: Every benchmark prompt and raw model output is version-controlled and verifiable by any engineer. Live Public Dashboard: Instant interactive leaderboard accessible with zero setup.
What we learned
More parameters do not automatically mean fewer edge-case bugs. Even 400B+ parameter models consistently fail on fundamental CS concepts like floating-point precision and stateful regex caching when prompts look deceptively simple.
What's next for FrontierBench: Deterministic Frontier AI Benchmark
Automated CI/CD Regression Action: Allowing engineering teams to run FrontierBench against their fine-tuned coding models before deploying them into developer IDEs. Expanding Edge-Case Taxonomy: Adding concurrency deadlock tests, SQL injection escaping, and memory leak vectors across Go, Rust, and C++.
Built With
- ai
- benchmarking
- chatgpt
- claude
- deepseek
- gemini
- javascript
- llm-evaluation
- machine-learning
- python
