FrameKind

• 

  FP32 vs UINT8 benchmark on Apple M4 (Arm64): latency, model size, and detection agreement.

• 

  Responsive FrameKind interface showing local inference and the editable alt-text workflow.

Inspiration

Alt text makes the web more usable, but sending personal, unpublished, or sensitive images to a hosted vision API creates a privacy tradeoff. We wanted to see whether a compact open model could provide a useful starting point locally—and make the performance-versus-quality tradeoff transparent instead of hiding it behind a loading spinner.

What it does

FrameKind is a privacy-first accessibility tool that turns an image into an editable alt-text draft without uploading the image to a vision API.

A user can choose a PNG, JPEG, or WebP image—or try the bundled sample. FrameKind runs YOLOS Tiny locally through Transformers.js and ONNX Runtime Web, draws detected objects and confidence scores, and produces a concise spatial draft describing what appears on the left, near the center, and on the right. The draft stays editable and can be copied in one click. Images remain in browser memory; only the open model weights are fetched and cached.

The optimization story is part of the product. A judge can rerun a controlled FP32-versus-UINT8 comparison using the same pinned model, image, confidence threshold, runtime, and worker. FrameKind reports model size, session-load time, median inference, and label-and-box agreement, so speed is always presented with a quality guardrail.

Challenge track

FrameKind targets the Mobile AI track. It runs on-device inference in the browser and was validated on an Apple M4 Arm64 system. The project was created and meaningfully developed during the challenge period.

How we built it

FrameKind is a React and TypeScript application bundled with Vite. Transformers.js loads Xenova/yolos-tiny at pinned revision e2f9c7673f0fa61849efe2b56a0d7774779ebb9d, while ONNX Runtime Web executes its FP32 and UINT8 ONNX variants through the WASM backend. Inference runs inside a dedicated Web Worker so loading and detection do not block interface interactions. The worker serializes jobs to protect model lifecycle state, and the UI ignores stale results if the selected image changes.

Detections above a 0.5 confidence threshold are grouped by label and converted into a deterministic draft. Caption logic includes quantities and coarse horizontal positions while avoiding unsupported visual claims. The benchmark pre-caches both weight variants, creates a fresh session for each, performs one untimed warm-up, and records five measured runs on the same image. The production host sends cross-origin isolation headers so ONNX Runtime Web can use isolation-enabled WASM threads where supported.

Challenges we ran into

Browser vision inference has meaningful memory and latency costs, especially for full-precision models. We had to keep the interface responsive, manage the model lifecycle in a worker, distinguish session-loading time from inference time, and design a fair comparison that did not favor whichever weights were already cached.

We also needed a useful definition of quality agreement. FrameKind matches detections only when labels are identical and bounding boxes overlap with intersection-over-union of at least 0.5, then reports a symmetric match score across both output sets.

Accomplishments we are proud of

• Real, fully local object detection rather than a simulated demo
• An editable alt-text workflow that keeps a human in control
• A judge-rerunnable optimization benchmark embedded in the app
• A dedicated worker, responsive interface, pinned model revision, and five passing regression tests
• A 63.7% reduction in weight size and a 3.43 observed median speed ratio, with 86% detection agreement

Benchmark methodology and observed result

Environment: Safari 26.5 on an Apple M4 MacBook Air (Arm64), macOS 26.5.1, July 10, 2026 Input: Bundled 14481086 sample image
Models: YOLOS Tiny FP32 and UINT8 ONNX variants from the same pinned revision
Runtime: ONNX Runtime Web/WASM in one dedicated Web Worker
Threshold: 0.5 for both variants

After pre-caching each model, FrameKind creates fresh sessions, performs one untimed warm-up, and records five timed runs on the same image. The reported inference time is the median.

Observed on this device and input: 3.43 median speed ratio, 63.7% smaller weights, and 86% detection agreement. These are reproducible local observations, not universal device claims.

What we learned

Quantization should be evaluated as a product tradeoff, not merely a smaller file. Weight size, initialization, inference latency, and output agreement can move independently. Accessibility tooling also benefits from restraint: a factual draft that invites human review is more trustworthy than a fluent description that invents details.

What's next

• Publish benchmark profiles across more Arm phones and browsers
• Add richer but still evidence-grounded object relationships
• Evaluate additional compact detection and captioning models
• Add JSON benchmark export and offline PWA installation
• Conduct usability testing with screen-reader users and accessibility professionals

AI assistance disclosure

OpenAI Codex assisted with ideation, implementation, testing, design exploration, deployment, and documentation. YOLOS Tiny supplies object detection. The bundled sample image was AI-generated for demonstration and testing. Benchmark evidence, source code, and methodology are public in the repository.

Built With

• arm64
• netlify
• onnx
• react
• transformers.js
• typescript
• vite
• webassembly
• yolos