System1

Inspiration

Small "System 1" models like TypeSafe's Jev are a big unlock for teams. They answer routine decisions (route this ticket, approve this refund) in milliseconds, for a fraction of a cent. But on tasks that depend on a team's own rules and history, they don't perform well. They haven't seen your policy or your edge cases.

The fix is a custom model, but most business users can't build one. It takes data work, training infrastructure and ML expertise that a small team doesn't have. We wanted to close that gap, so the people who own the decision can get a model that beats the general one on their own data.

What it does

system1 is a full pipeline, from generating data to training a model to serving it:

1. Define: an onboarding agent turns a repeated team decision into written rules and a label set.
2. Data: it generates a practice dataset, and the user reviews it before anything is trained.
3. Train: a LoRA on Laya is trained with Unsloth on Modal, with no ML knowledge needed.
4. Use: the trained model is served behind a Jev-compatible API (POST /v1/systemone), so teams can swap it in without changing their integration.
5. Improve: corrections feed back into the next training round.

The business case is better performance without hiring a technical team. Because sys1 is a small model, it is cheaper to run than Jev and faster.

How we built it

• Model: LoRA fine-tune of Laya using Unsloth's decision-training API, run on Modal.
• Server: FastAPI, with the Jev-compatible gateway and per-organization model serving.
• Client: React.
• Auth: Clerk handles all sign-ins. We use Clerk organizations and admin roles, so only admins can delete models.
• Deployment: the server also serves the client, deployed on Railway, with per-org caps on how many models each org can hold.
• Data: BANKING77, which simulates a real use case.
• Evaluation: calibration is computed from the probabilities field, since Jev and Laya define confidence differently. Jev is pinned to a versioned id (jev-1.13.0), and we log the model field from every response.

Challenges we ran into

• Making training easy. Hiding data generation, training and evaluation behind a flow a non-technical user can follow.
• Different datasets, different problems. Training reliably across very different decision tasks and label sets is hard, because a setup that works for one dataset doesn't automatically work for another.
• Clean labels. We deliberately never used Jev's answers as training labels. TypeSafe's customer agreement forbids it, and it would cap sys1 at Jev's quality.

Accomplishments that we're proud of

• A working end-to-end pipeline, from dataset creation to a trained model served live behind a Jev-compatible API.
• Per-user custom models with real organization and admin controls through Clerk.
• A fair benchmark: a frozen test split that was never trained on, calibrated on, or used to tune prompts.
• Beating Jev on BANKING77 with a tiny model.

What we learned

• Automated model training end to end: dataset creation, training runs, evaluation and calibration.
• Storing and loading the large LoRA adapter files that each user's trained model produces.
• Benchmark hygiene matters: a frozen test split, pinned model versions and logged model ids.

What's next for System1

Improving the model training and data creation pipelines.

Built With

• bun
• clerk
• fastapi
• jev
• laya
• modal
• unsloth
• vite