Inspiration
ML is fundamentally a data problem.
Anyone can harvest their own data- But it is prone to overfitting and is sample inefficient from redundant cases.
Anyone can generate their own synthetic data. But it suffers from the aforementioned reasons, and it is also difficult to verify its validity and quality.
This data and environment problem can be solved through an emerging technique:
- ASP: Asymmetric Self-Play grounds a self-play curriculum in a real goalpost problem instead of open-ended generation. Two models are being trained in parallel, a teacher and a student. The teacher's goal is to build non redundant tasks that have headroom for the student. The students goal is to simply solve these tasks. This mutualistic relationship challenges each other to converge towards a goal post.
The pipeline e2e
- A CLI scans a repo for OpenAI call sites, records real traffic, and builds it into a task profile (prompt template, schema, inferred goal).
- That profile seeds a GRPO self-play loop: a generator writes tasks, a small student solves them, both train every iteration.
- The curriculum follows GASP's lemma/lift structure, targeting a difficulty band instead of raw difficulty.
- Eval sets are frozen before training starts, not tuned against afterward.
- The loops repeat as the student converges towards the goal post.
- The model can be extracted and is ready for use.
How we built it
- Reward. ASP's lemma and lift rewards, as a function of the student's live pass rate p on a candidate task:
$$r_{\text{lemma}}(p) = \big(4p(1-p)\big)^5, \quad p \in [0.3,\ 0.7]$$
$$r_{\text{lift}}(p) = 10p\left(\frac{1-p}{0.9}\right)^9, \quad p \in [0.1,\ 0.5]$$
Outside either band: a flat -0.5 penalty. Lift's band sits harder than lemma's on purpose. That asymmetry is what pushes the student's frontier forward instead of parking at 50% pass rate forever.
- Novelty gate. A candidate is rejected if it's too close to anything already accepted, by cosine similarity on
text-embedding-3-smallembeddings:
$$\max_{e \in B} \cos(v_c, v_e) \geq 0.95 \implies \text{reject}$$
- Group scoring. Implemented against Flash's
score_episodeshook (its GRPO group-scoring path) instead of per-sample scoring, so a whole rollout group gets graded in one judge call using the standard group-relative advantage:
$$A_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G)}$$
Citations
[1] S. Jana, C. Sancaktar, T. Daniš, M. Georg, A. Orvieto, and K. Pavel, “GASP: Guided Asymmetric Self-Play for Coding LLMs,” arXiv.org, 2026. https://arxiv.org/abs/2603.15957 (accessed July 19, 2026).
[2] Z. Ye et al., “Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play,” arXiv.org, 2024. https://arxiv.org/abs/2411.00062 (accessed July 19, 2026).
[3] L. Bailey, K. Wen, K. Dong, T. Hashimoto, and T. Ma, “Scaling Self-Play with Self-Guidance,” arXiv.org, 2026. https://arxiv.org/abs/2604.20209 (accessed July 19, 2026).
Built With
- freesolo
- langchain
- python
Log in or sign up for Devpost to join the conversation.