Verity AI

Inspiration

Reinforcement learning has emerged as the dominant way AI models are post-trained to reason, code and act as agents, and labs are reportedly spending billions on the environments they train in. In these environments, a model learns whatever the grader approves or rewards, and so if the grader has a loophole, the model learns to cheat using the loophole instead of learning the actual task. We built Verity to catch these loopholes within an environment, and ensure that models actually learn the task and gain the reasoning skills intended from these environments during training.

What it does

Verity works in two stages. First, an attacker model probes the training environment for loopholes that earn the reward without actually solving the task, and Verity turns all the hacks it confirms into an audit report. Then, during the actual post-training/fine-tuning stage, a reward monitor uses that report to watch each of the model's attempts and penalizes any that follow a known hack. Because cheating actively punishes the model, training pushes the model's weights away from reward hacking and toward actually solving the task, which directly leads to reasoning gain in deployment.

How we built it

We built Verity mainly in Python. For the attacker model setup, we call Claude Opus through the OpenRouter API as the attacker. It explores each environment, and any reward hack it finds is replayed to confirm it, then saved to a JSON audit report. During post-training/fine-tuning, the reward monitor reads that report and checks every attempt while we fine-tune Qwen3-8B with multi-turn GRPO using Hugging Face TRL and vLLM. We ran training on RunPod GPUs and UW Allen School compute.

Challenges we ran into

Our first audit reports didn't catch every reward hack in an environment, and so we gave the attacker more attempts on an environment and had each new round look for a different exploit than the ones already found. We also saw that after the first round of fine-tuning, the model still had traces of reward-hacking intent, so we fine-tuned it further, and the cheating dropped off much more.

Accomplishments that we're proud of

We built a full pipeline in one weekend that goes from auditing an environment to training a model on it. Our attacker found real reward hacks without being told what to look for, and every hack it reported was confirmed by replaying it. Most importantly, the model trained with Verity actually learned the task, while the same model trained without it learned to cheat.

What we learned

A rising reward doesn't mean a model is learning, because a model that cheats can earn just as much reward as one that does the task. We learned that you have to measure whether the task was actually done, separately from the reward. We also learned that catching cheating works best when you check the result of what the model did, not the specific chain-of-thought it took, since there are too many ways to cheat around it's portrayed thinking process.

What's next for Verity

We plan to launch Verity as a startup that independently audits training environments for AI labs and environment vendors. We plan to expand it to many more tasks and environments, strengthen the attacker, improve the reward monitor, and benchmark against state-of-the-art baselines.

Built With

• ai
• developer-optimization
• machine-learning
• model-misalignment
• python
• reinforcement-learning
• reward-hacking
• training-efficiency