EvalShield

• 

  Working of EvalShield

Inspiration

With the rise of on-chain AI agent commerce (ERC-8004 and ERC-8183), autonomous agents can hire and pay one another using a third agent as an "evaluator." However, a glaring security gap exists in the standard: the evaluator is simply a wallet address. Nobody has to prove they can't be bribed, nobody has to prove they aren't secretly colluding with the provider, and there is absolutely no financial penalty if they approve fraudulent work. We realized that while current solutions focus on making evaluators smarter, nothing is making them honest.

What it does

EvalShield acts as an economic accountability layer to close four major agent commerce attack vectors: bribery, collusion, Sybil clusters, and front-running. It introduces four core mechanisms:

1. Staking & Slashing: Evaluators lock collateral that gets slashed if they approve fraudulent work.
2. Verifiable Random Assignment: Panels are drawn randomly with strict diversity constraints, ensuring no two evaluators share a registered operator identity.
3. Multi-Model Attestation: Drawn panelists independently judge the work (using different LLMs) and sign their verdicts on-chain.
4. Adversarial Challenge Market: Any third party can stake a bond to dispute a finalized job. If a second, independent panel overturns the original decision, the initial panel is slashed, their ERC-8004 reputation drops, and the challenger receives a bounty.

How I built it

I built the core smart contracts using Solidity, Foundry, and OpenZeppelin. I implemented a custom commit reveal randomness scheme with an on-chain diversity constraint to simulate secure panel selection on a local anvil instance. For the actual evaluation logic, we bypassed expensive APIs and integrated Ollama, running three different open-weight models (Llama 3, Mistral, Phi-3) completely locally to judge job evidence and feed verdicts back into the EVM.

Challenges I ran into

The hardest technical challenge was implementing an efficient diversity constraint during the random panel draw. I had to ensure that not only was the draw mathematically random, but that the transaction would securely revert if the randomness selected two evaluators belonging to the same Sybil cluster (sharing an operator ID). Securing the state-machine transitions across the Escrow, Pool, and Challenge contracts with proper reentrancy guards also required rigorous negative testing.

Accomplishments that I'm proud of

This entire topic, agent-to-agent commerce, on-chain escrow security, and game-theoretic mechanism design was completely new to me. I'm proud that I was able to go from zero knowledge of ERC-8004/ERC-8183 to a working, tested smart-contract system with a live adversarial challenge market in a single hackathon sprint. I did take the help of AI. Getting the commit reveal randomness scheme to correctly enforce Sybil-diversity constraints on-chain, and wiring up three local LLMs to feed independent verdicts back into the EVM, were particularly rewarding milestones.

What I learned

• How ERC-8004 and ERC-8183 define the primitives for autonomous agent commerce and escrow evaluation.
• Designing gas-efficient state machines across multiple interacting contracts (Escrow ↔ Pool ↔ Challenge) with proper reentrancy guards.
• Implementing a commit-reveal randomness scheme with on-chain diversity constraints to prevent Sybil panel stacking.
• Running multiple open-weight LLMs (Llama 3, Mistral, Phi-3) locally via Ollama and piping their outputs into Foundry scripts.
• Writing rigorous negative tests in Foundry to ensure invalid state transitions revert correctly.
• The difference between solving evaluator miscalibration (multi-model voting) vs. evaluator dishonesty (economic incentives via staking/slashing).

What's next for EvalShield

For Stage 2, we plan to upgrade our commit-reveal randomness scheme to true Chainlink VRF on the Base Sepolia testnet. We also plan to build out a full Vite/React frontend with RainbowKit integration, and use web3.storage to anchor evaluation reasoning evidence permanently on IPFS.

Built With

• anvil
• forge
• foundry
• node.js
• ollama
• openzeppelin
• solidity