Home Page

Inspiration Modern AI systems are powerful, but they still struggle with ambiguous images, sarcasm, subjective comparisons, and other cases where human judgment matters. Most data-labeling platforms treat workers like anonymous clickers and provide little transparency into how their work is evaluated or rewarded.

We built Taskmaster around a different idea: AI should handle the obvious work and route uncertain cases to people. Humans remain the source of truth, while AI acts as a monitor—not an unquestionable grader. Workers should also be able to see how much they earned and verify every completed payment onchain. What it does Taskmaster is a human-in-the-loop data-labeling platform for:

Image classification
Text sentiment analysis
Pairwise evaluation of AI responses Gemini first analyzes each task and produces a proposed answer with a confidence score. High-confidence tasks can be handled automatically, while uncertain tasks are sent to human workers.

For human-submitted labels, Taskmaster combines several quality signals:

Gemini confidence and agreement
Hidden reference tasks
Peer consensus
Submission latency
Repeated-answer detection
Periodic batch audits Gemini disagreement alone does not automatically make a human wrong. If an answer is defensible, it can still be accepted. This was important to us because the entire purpose of human labeling is to handle cases where models may be uncertain or mistaken.

Workers build a trust score between 0 and 100. Their recent performance updates the score as an exponential moving average: [ T_{new} = (1-\alpha)T_{old} + \alpha Q ] where (Q) represents the latest quality signal. Higher-trust workers receive a larger payment multiplier, while suspicious behavior can pause rewards.

Approved work is paid in Solana Testnet SOL. A portion is sent immediately, while the remainder enters a short onchain escrow period. Every confirmed payment includes its transaction signature and a direct Solscan link. How we built it The application uses Next.js 16, TypeScript, React, and Tailwind CSS. We designed the interface around focused, flashcard-style task decks so workers can label data quickly without feeling like they are filling out a spreadsheet.

Gemini provides structured JSON responses for:

Initial task routing and confidence estimation
Submission-time coherence monitoring
Periodic audits across multiple labels We use Tiger Cloud and TimescaleDB to store label events, payout records, trust changes, and audit results. Continuous aggregates power the live quality dashboard and allow us to track labels, approval rates, response latency, and worker behavior over time.

The Solana integration uses @solana/kit, Sign-In With Solana, server-side treasury transfers, and derived escrow accounts. The payment flow is:

A worker submits an approved label.
The reward is split into an immediate payment and escrow.
The immediate portion is transferred to the worker.
The escrow portion is released after the demo vesting period.
Confirmed transactions become clickable Solscan receipts. Server-Sent Events provide live payout and audit updates without repeatedly refreshing the page.

Taskmaster is containerized with Docker, served through Caddy with automatic HTTPS, and deployed on a Vultr VM. A GitHub Actions pipeline runs type checks, linting, tests, a production build, database migrations, and deployment smoke tests before accepting a release. Challenges we faced One of our largest conceptual challenges was deciding how much authority to give the AI evaluator. Our first approach treated model agreement like a correct answer key. That defeats the purpose of human-in-the-loop labeling: if the model already knows the answer, there is little reason to ask a person.

We redesigned the system so Gemini contributes a confidence signal while human labels remain first-class data. Verified reference labels outrank model predictions, and reasonable human disagreements are retained.

Data quality was another challenge. Early seeded image tasks occasionally had answer choices that did not match the image. We replaced the problematic dataset, added reviewed images, and allowed workers to enter a custom text label when none of the suggestions are correct.

The payment lifecycle was also more complicated than simply sending SOL. We had to distinguish queued, submitted, confirmed, escrowed, released, and failed transactions. We learned that a user-facing payment system must expose these states clearly; otherwise, a pending database record can look like money that already exists onchain.

Finally, keeping the wallet adapter, backend RPC, treasury, and Solscan links on the same Solana network required careful configuration. We centralized the network settings and added a deployment assertion so mixed Testnet and Devnet releases fail visibly. What we learned We learned that effective human-in-the-loop systems are less about replacing judgment and more about deciding when judgment is needed.

We also learned that confidence is not correctness. A model’s confidence can help route work and identify unusual submissions, but it should not silently erase legitimate human disagreement.

On the engineering side, we gained experience with:

Structured Gemini outputs
Behavioral quality monitoring
TimescaleDB continuous aggregates
Solana wallet authentication and transfers
Onchain escrow mechanics
Real-time event streaming
Docker-based continuous deployment What’s next Our next step is replacing the custodial escrow mechanism with an Anchor smart contract, allowing workers to claim vested funds without relying on the Taskmaster server. We also want to introduce multi-worker consensus, dispute resolution, customer-uploaded datasets, and reputation that can be verified across labeling projects.

Ultimately, Taskmaster aims to create a transparent marketplace where people are paid for the judgment AI still cannot reliably provide.