TokenFlash

Inspiration

Everyone says "just use GPT‑4 or Claude." But those are cloud APIs that charge per token, and the bill adds up fast — especially for students and small teams. What bothered me most: the "cost optimizer" tools out there still call an expensive cloud model to do the optimizing. You literally spend tokens to save tokens.

So I flipped the question: what if none of it touched the cloud at all? Every engineer already has a capable laptop. If the model runs locally, the marginal cost of every request is $0 — and then token optimization becomes about making a small local model punch above its weight, not about shaving a cloud bill.

What it does

TokenFlash is a fully local AI assistant with a dual‑pane web UI. It runs open‑source models on your own machine via Ollama — no cloud, no per‑token bill — and layers three things on top so a small model does more:

1. Pre‑Flight Scoping — instead of forwarding a vague prompt, it interviews you first, one question at a time, to lock down exact requirements. A bad, expensive prompt never gets processed.
2. Token Compression — LLMLingua‑2‑style extractive compression drops low‑signal filler while keeping technical terms. You watch the dropped tokens struck through live.
3. Cost‑Aware Routing — the work is split into subtasks and routed to the cheapest local model that fits (a light Llama 3.2 3B vs. a heavier Qwen 2.5 Coder 7B).

And it does two kinds of work automatically: it builds software (a website, CLI tool, API, script) or performs a task (research, structured notes, a summary, a poem) — it detects which you want and produces the actual result. The right pane reports real token counts measured on‑device and the cloud cost you avoided ($0 actually paid).

How we built it

• Frontend: Next.js + React, a dual‑pane UI streaming over the Vercel AI SDK data‑stream protocol.
  • Backend: FastAPI (Python) orchestrating the pipeline — elicitation, a heuristic LLMLingua‑2‑style compression engine, an intent classifier (build vs. perform), a task decomposer, and a cost‑aware router that runs subtasks in parallel.
  • Local models: Ollama running Llama 3.2 3B (light tier) and Qwen 2.5 Coder 7B (heavy tier). The backend reads the models' real prompt_eval_count / eval_count so the token panel is ground truth, not an estimate.
  • Provider abstraction: a dispatch layer means it's local by default, with AWS Bedrock as an optional fallback — the same pipeline works either way. ## Challenges we ran into
• Spending tokens to save tokens. My first version ran the whole optimizer on a paid cloud model, which defeated the point. Pivoting the entire pipeline to run locally on Ollama — while keeping the token accounting honest — was the biggest rework.
  • Small models are weaker at instruction‑following. A 3B model would sometimes ignore "scope only, don't produce the answer yet," or classify a task inconsistently. I fixed this structurally — deterministic keyword routing for obvious cases and a backend safety net — rather than trusting the small model to always behave.
  • Real token accounting. Making the UI show actual on‑device token counts (and the cloud cost avoided) instead of guesses meant plumbing usage data through every model call.
  • Being honest about savings. Output tokens dominate cost, so I built the UI to show that transparently instead of overselling input compression. ## Accomplishments that we're proud of
• It genuinely runs 100% locally at $0 — you can pull the WiFi cable and it still works.
  • Dual mode: the same tool builds working code and performs knowledge tasks like research and notes.
  • Real, on‑device token measurement and a transparent "cloud cost avoided" breakdown — no hand‑wavy numbers.
  • A visible, educational pipeline (compression diff, model routing, token report) that shows the user how their prompt is optimized, instead of hiding it like an invisible backend proxy. ## What we learned
• For a laptop‑scale product, running locally is a far bigger cost lever than input compression — and being honest about that made the project stronger, not weaker.
  • Small local models need guardrails around them (deterministic routing, safety nets), not just clever prompts.
  • A 7B model on a modern laptop is genuinely capable for a huge range of real tasks — the barrier was never the hardware, it was the tooling.
  • Showing your work (live compression, real token counts) builds trust that an invisible optimizer never earns. ## What's next for TokenFlash
• Verified savings — run a compressed vs. uncompressed A/B and prove the output stays equivalent (e.g. the generated code still compiles), turning "cost saved" into a guarantee.
  • Quality‑gated compression — auto‑tune how hard to compress per request without degrading the result.
  • Smarter local routing — try a cheap model first, escalate to a heavier one only if the output fails a check.
  • Real research mode — let "perform" tasks fetch and summarize live sources (papers, docs) instead of relying only on the model's built‑in knowledge.
  • More model tiers and one‑click model management so anyone can run it in minutes.

Built With

• fastapi
• llama
• llmlingua
• local-llm
• next.js
• node.js
• ollama
• open-source-llm
• python
• qwen
• react
• tailwindcss
• typescript
• uvicorn
• vercel-ai-sdk