-
-
Empirical benchmark scorecard showing pass rates and failure modes across 7 frontier LLMs.
-
Overall model rankings across 7 deterministic production engineering edge cases.
-
Aggregate benchmark metrics and automated AST repair validation rates.
-
Detailed failure matrix exposing silent architectural blind spots in Claude, ChatGPT, and Gemini.
-
Root cause analysis of IEEE-754 precision loss and stateful regex mutation bugs.
-
Interactive leaderboard tracking deterministic code safety and execution reliability.
Inspiration
AI code generation benchmarks like HumanEval and SWE-bench have become saturated, often testing for surface-level fluency rather than deep engineering robustness. In production, AI coding assistants generate code that passes human visual inspection and basic syntax checks, yet fails catastrophically under edge-case runtime conditions — silently truncating 64-bit database identifiers, introducing stateful regular expression bugs, or leaking security redirects.
Furthermore, popular evaluation approaches rely on "LLM-as-a-judge", introducing non-deterministic vibes and model self-preference bias. We built FrontierBench to provide an uncompromising, deterministic evaluation harness that tests frontier language models against real production failure modes with binary (PASS/FAIL) execution verdicts.
What it does
FrontierBench is an open-source evaluation and benchmarking harness that stress-tests leading frontier models (Claude 3.5 Sonnet, DeepSeek V2.5, Meta Llama 3, Mistral Large, Google Gemini, OpenAI ChatGPT, and xAI Grok) across 7 critical software engineering edge cases:
IEEE-754 64-Bit Precision (snowflake_dedupe): Tests whether models preserve 64-bit integer precision when deduplicating large Snowflake IDs, or silently coerce them to JavaScript doubles resulting in corrupted primary keys. Stateful RegExp Mutation (regex_filter): Evaluates whether models handle global /g regex flags safely, preventing lastIndex state mutation that causes alternating boolean evaluation on identical strings. Tab-Whitespace Evasion (safe_redirect): Verifies whether relative URI sanitizers detect tab-prefixed domains (/\t/evil.com) used in modern browser redirect attacks. Calendar Rollover (billing_dates): Tests month-end date math to verify models avoid silent JavaScript month-skipping (e.g., advancing Jan 31 into March). JSON Extraction (extract_json): Tests strict JSON payload parsing from mixed conversational text without truncation. Financial Minor Unit Allocation (allocate_minor_units): Verifies discrete cent allocation to avoid fractional rounding loss in payment distributions. Race Condition Search Resolution (search_latest_wins): Tests asynchronous race-condition safety in query handlers. Key Benchmark Discoveries: Zero Models Achieved 100%: Every frontier model exhibits distinct architectural blind spots. Claude 3.5 Sonnet & DeepSeek (86% Pass Rate): Both failed on 64-bit integer precision, silently dropping deduplicated records. Meta Llama 3 & Gemini (71–86% Pass Rate): Failed on URI tab-whitespace evasion sanitization. OpenAI ChatGPT (57% Pass Rate): Failed on stateful regular expression mutation. xAI Grok (43% Pass Rate): Exhibited the highest failure rate across financial allocations, race conditions, and precision arithmetic.
How we built it
Deterministic Test Harness: Built in pure Node.js and Python (run_eval.py), executing raw model responses in isolated sandboxes with zero external judge dependencies. Empirical Web Dataset: Collected raw responses directly from the production web interfaces of each of the 7 frontier models (stored transparently in responses/). Interactive Leaderboard: Developed a dark-mode titanium dashboard (leaderboard.html) deployed live via GitHub Pages featuring model comparison matrices, failure logs, and pass rates. Automated Repair Engine: Integrated an AST-based repair pipeline verifying reproducible bug fixes.
Challenges we ran into
Eliminating Non-Determinism: Standard benchmark runners often suffer from flaky async timeouts. We engineered the harness to run synchronous AST parsing and isolated VM memory execution to guarantee 100% reproducible test outputs. Designing Genuine Trap Scenarios: Creating test cases that appear straightforward to an LLM's next-token predictor but inherently test deep semantic invariants required studying actual production post-mortems from enterprise codebases.
Accomplishments that we're proud of
100% Deterministic Reproducibility: Binary pass/fail criteria with zero subjective grading. Uncovering Real Flaws in Top Models: Documented concrete edge cases where Claude, ChatGPT, and Gemini fail silently. Live Leaderboard & Video Demo: Deployed an interactive public leaderboard and produced a detailed video walkthrough of the evaluation suite.
What we learned
LLM coding benchmarks must test execution semantics, not lexical similarity. A model can produce beautifully commented code that immediately introduces a critical CVE or data loss in production.
What's next for FrontierBench
Scaling to 100+ Edge Invariants: Expanding the test harness to cover concurrency deadlocks, memory leaks, and distributed consensus failures. Automated GitHub Action: Releasing a pre-commit GitHub Action to automatically benchmark AI-generated pull requests before merging into production.
Built With
- ai-benchmarking
- automated-testing
- chatgpt
- claude
- data-science
- deepseek
- gemini
- javascript
- large-language-models
- node.js
- python
