AIOAC (AI on a Click)

Build AI models via simple chat with AIOAC. See costs upfront and pay via PayPal only when verified results are delivered.

Comment

1. Project overview

Project name: AIOaC (AI on a Click)

Elevator pitch (under 200 characters): 65% of businesses have data they never use. What if you could build an AI model just by describing your problem, and only pay when verified results are delivered, controlled by PayPal?

2. Project details

About the project (paste as Markdown)

Inspiration

Training a machine-learning model still needs an ML engineer, a cloud account, and a lot of guessing about cost. We tried platforms like AWS SageMaker Canvas, but they were far too complex and overwhelming for a non-technical person. People who just want an answer ("which customers will leave?") give up. We wanted it to feel like Gamma for presentations: you say what you want in plain English, an AI agent does the technical heavy lifting, and you only approve the few decisions that matter. The hardest part is trust—nobody wants an AI agent near their wallet. So we made PayPal the ultimate safety rail.

What it does

AIOaC takes a plain-language goal and walks one path from start to finish:

  1. Describe. You state the goal. The agent asks one short question if needed.
  2. Data. Upload a CSV, paste a dataset link, or say "I have no data" and the agent searches open datasets (hand-checked list, UCI, OpenML, Hugging Face) and explains the top three in plain words.
  3. Readiness. A readiness grade with plain OK/NOTE findings (leaky columns, lopsided answers, too few rows).
  4. Free test, pilot and forecast. A free baseline on held-out data, then a free pilot on 5-10% of the rows that forecasts the full-run result against your target, with a confidence. If we cannot honestly sell it, we say so ("Not eligible").
  5. Quote. One price: estimated compute plus a 5% success fee, a range, and a hard cap. Everything is labeled an estimate.
  6. Pay. PayPal places a hold for the cap. Nothing is charged yet. The server confirms the hold with PayPal; a browser success screen is never proof.
  7. Train. A capped job runs on AWS SageMaker with live progress and a Stop button.
  8. Result. The model is checked once on locked test rows it never saw. The server then captures the actual compute, captures the success fee only if the target was verified, and voids the rest of the hold. If our own code fails, the whole hold is voided and you pay $0.
  9. Use. Try the model on one record, run a CSV, download it, share a read-only result link, or start a free "Improve" round.

Task types: classification, regression, forecasting, grouping and unusual-record detection.

How we use PayPal

PayPal sandbox is the core spending control of the product, not just a checkout bolted on at the end:

  • Orders v2 authorize places a strict hold for the cap. The server confirms it before any paid AWS resources are spun up.
  • Payments v2 capture (with final_capture and a PayPal-Request-Id on every call for idempotency) captures the actual compute cost first. Then, a second capture on the exact same hold takes the success fee, but only if the locked-test check passed.
  • Void releases the unused remainder back to the user instantly.
  • Webhooks: Signature verified through PayPal's verify-webhook-signature API. Events are stored once in an append-only ledger and used as a cross-check. A webhook can never create a hold or mark a project paid on its own.
  • Ledger Protection: An internal ledger records every capture; the application strictly refuses to capture above the hold or above the compute cap.

How we use AI

AI does the work an ML engineer would otherwise do, but it is deeply restricted from touching the wallet.

1. The agent (LLM through Azure AI Foundry or OpenRouter). A custom tool-calling loop that:

  • Finds data. Runs discover_datasets, ranks results, and explains the top three in one plain sentence.
  • Checks the data. Provides a readiness grade, profiles columns, flags anomalies (e.g., numbers stored as text), and does a side-by-side comparison of model types.
  • Runs the free steps. Executes a free test and pilot forecasting against the user's target.
  • Explains. Translates quotes, baseline metrics, and results into plain English.

2. The agent proposes, the system disposes (Permission Grants). A permission gate (ordinary code) sits between the model and every tool based on strict levels:

  • Level 0 Tools: Free, read-only tasks (searching data, profiling) that run independently.
  • Level 1 Tools: Actions that change state (choosing a dataset, applying a data fix). These require explicit user approval via a UI card click.
  • Level 2 Tools (Payment Protection): Requesting payment or starting cloud training. These are strictly protected. They require an approved, unexpired quote and a server-confirmed PayPal hold. The gate reads stored database state, never what the model claims. If the user approves a card, the server runs the exact API call stored securely on the card, meaning the model cannot talk its way around financial approvals.

3. Context engineering. Context is injected dynamically based on the exact state and step the user is on. At each step, the model sees a short rules prompt, a live state block rebuilt directly from the database (status, chosen dataset, grade, free test, quote, next step), and only the tools that fit the current stage. Eleven short skill playbooks (handling leakage, lopsided answers, choosing target columns) load dynamically on demand.

4. Guards around the model. Tool arguments are checked against strict schemas (Pydantic). The target accuracy must come from the user's own words. A reply guard replaces any final answer that hallucinates a payment, training run, or settlement. Limits are strictly enforced per turn (tokens, deadlines, looping) and per project.

5. Evaluation. An eval harness runs scripted and real models against fixed cases with a mock payment provider. It checks that no paid work ran without holds, ensuring complete safety.

6. Classic ML that is honest. scikit-learn on tabular data. The paid run trains the pinned model type, scores it once on locked test rows, and exactly matches the code in the SageMaker training script so local and cloud environments cannot drift.

How we built it

Architecture. The dependency direction is strictly enforced: routes -> services -> repositories and integrations -> database and interfaces. Services contain absolutely no SQL; only repositories touch the database. Every outside system (PayPal, AWS, LLM) has its own folder behind an interface (ports.py). An architecture test fails the CI/CD build if a service imports SQLAlchemy or the app imports a mock double. Next.js (TypeScript) powers the frontend, FastAPI (Python) powers the backend, and Postgres/SQLite handles the data.

The money path. A project moves through a strict state machine (described -> data_chosen -> quoted -> awaiting_payment -> paid -> training -> done). Steps cannot be skipped. The paid state is unlocked only after the server asks PayPal to authorize the order and verifies the amount. Settlement saves each step (compute capture, fee capture, void) sequentially. If the server crashes during settlement, it can resume without moving money twice thanks to idempotency keys.

Training. POST /train returns immediately, running the SageMaker job in the background. On SageMaker, the quote's cap literally becomes the MaxRuntimeInSeconds parameter (minus a grace margin). AWS itself stops the job at the cap even if our server dies. Actual cost is calculated by AWS billable seconds, ensuring the user is never overcharged.

Testing

Because AIOaC moves money and runs an AI agent, testing was the core of the project, not an afterthought. We tested in four layers, and the rule was: payment provider, ledger and cost estimate get tests before they count as done.

1. Four test layers

Layer What it proves Uses
Unit Cost math, ledger, permission gate, lifecycle, readiness, agent guards, architecture rules In-memory SQLite, no outside systems
Contract Our code against PayPal, AWS SageMaker, Azure/OpenRouter, Hugging Face, UCI, OpenML Mocked HTTP and boto3
End-to-end Full flows through the real API: approve, hold, train, verify, capture, void Mock PayPal, scripted model, local training
Live integration Real PayPal sandbox, real AWS, real LLM, real database Opt-in only (RUN_REAL=1)

Hundreds of offline tests run with no network and no keys. Paid tests need a second switch (RUN_PAID=1), so a normal run can never start a paid job. A one-command demo check (pytest tests/e2e/test_payments_and_demo.py) runs the whole money path and must pass before any task is called finished.

2. Safety checks that must always pass

  • No paid action runs without an approved, unexpired quote.
  • Training never starts before a server-confirmed PayPal hold.
  • The success fee is never captured unless locked-test verification passed.
  • A capture or void repeated with the same request id moves no money a second time.
  • Total captures never exceed the hold, and compute never exceeds the compute cap.
  • A repeated webhook changes nothing the second time.
  • Our own failure voids the whole hold ($0 for the user).
  • One user can never see another user's project (404, not 403). A test lists every route and fails if a new one isn't classified.
  • A test checks that the public share link returns an exact allow-list of fields and never money, emails or storage keys.

3. Live PayPal sandbox

We ran the real lifecycle against the PayPal sandbox: authorize the order into a hold, repeated authorize (same authorization), a partial capture for compute, a second capture on the same hold for the fee, repeated capture with the same PayPal-Request-Id (returns the first result), void of the remainder, repeated void (harmless), and a refused capture after the void. All passed. (Sandbox money only; the buyer approval step needs a person.)

4. Live AWS SageMaker

Real capped jobs proved the cap and the cost formula: a job with a 60-cent cap finished in 69 billable seconds, scored 82% against 69% for a simple guess, and the reported cost matched our formula on AWS's billable seconds. A test also fails if any job is left running.

A real bug the live run caught: a model trained on SageMaker's scikit-learn 1.4.2 loaded on our app's 1.9.1 with a warning, then crashed on predict. We pinned training to the 1.9-0 image, and a new test now fails if the trained model can't load and predict in the app's environment.

5. Evaluating the AI agent

LLM output varies, so we measure a pass rate, not a single green tick. An eval harness (python -m evals.run) runs the real model against 15 fixed cases (dataset discovery, vague goals, non-English goals, uploaded files with known problems) with a mock payment provider and a throwaway database. Per run it checks: no paid work ran, no false claims about payment or training, plain language, no accuracy promise, correct tool order, forbidden tools never called, and limits respected.

Results on real models (Azure AI Foundry):

  • gpt-5.4-mini: 15 of 22 runs passed, 9 of 11 cases ok.
  • gpt-5.4: 22 of 22 runs passed, 11 of 11 cases ok, 0 safety violations.

Findings we fixed from the first runs: the agent chose the target accuracy itself, so a code guard now requires it to come from the user's own words; it used jargon and markdown, so wording rules and plain-language fields were added; and after the user approved a card it asked again, so the server now runs the stored approved call itself.

6. Testing the skills and the data

We built synthetic files with known problems (3% in the smaller group, only 80 rows, an exact copy of the answer, a near copy) to check the agent loads the right playbook. Rewriting skill descriptions took unneeded skill loads from 12 of 12 runs to 0 of 12 and loads per run from 8.8 to 3.7, with the needed skill loaded in 12 of 12 runs.

These files also exposed a real bug: readiness said a copy-of-the-answer column "will be ignored", but training still used it and reported 100%. We fixed it, and the same file now scores 69% against 52% for a simple guess.

7. Browser tests and cost accuracy

  • Playwright drives the real UI against a test launcher (scripted model, mock PayPal, local training): signup, the full describe-to-result path, project pages, sharing, isolation and error states.
  • Cost accuracy: every finished job stores estimate, range and actual cost, and a report shows how often the actual fell inside the quoted range. This is how we found that our baseline timing underestimated larger datasets.

Honest limits

Eval runs are small samples (2-3 repeats per case, one model family), so "22 of 22" means "clearly better than mini", not a guarantee. Webhook signature verification is tested on mocked HTTP but not yet against a public URL, and the full browser flow with real PayPal plus SageMaker was not run end to end.

The Future / What's Next: Crowdfunding Big AI

The big vision for AIOaC goes far beyond tabular data. We want to support full Machine Learning (ML), Deep Learning (DL), Large Language Models (LLM), and Text-to-Speech (TTS) architectures.

Fundraising for Cloud Architecture Training: Training large DL or LLM models requires massive GPU compute and a lot of money. A non-technical founder might have the dataset and the vision, but they cannot afford the AWS GPU bill. AIOaC's future scope includes a fundraising feature: users can open a campaign where backers pledge funds directly into PayPal holds. Once the estimated GPU cost cap is reached, AIOaC orchestrates the massive cloud architecture training. Compute is captured at actual cost across the backers' holds, and anything unused is automatically voided and released. The entire hold, capture, and void ledger we built for this hackathon is exactly the foundation needed to make crowdfunded AI training a reality.

Built With

Share this project:

Updates

Submission history