Inspiration
I'm shopping for a used car, and every listing raises the same question: what is the seller not showing me? A phone photo shows the paint, might be only good sides. It doesn't show an engine that misfires until it warms up, or trouble codes that were cleared the morning of the sale.
That's why I am building UpClose. You film a walkaround with your phone and get an evidence-linked condition report. A vision pipeline finds the car, then looks for rust, dents and scratches only on the car. Every finding links back to the frame it came from. While I was looking at cars, though, I kept running into the same limit: a camera only checks the outside.
Large auction and dealer operations use fixed inspection hardware to cover what a camera can't, such as engine-sound recordings, undercarriage imaging and drive-through camera arches. Small dealers, flippers and someone like me looking at a car in a driveway have none of that. So the goal for this weekend was to fit part of an inspection lane into a ~$50 sensor kit plus the phone you already carry, and to stay honest about anything it didn't check.
What it does
UpClose now follows a Sense → Understand → Act loop:
- Sense. The phone walkaround. A Grove sensor pod on the engine records vibration (~400 Hz), sound and temperature. An OBD snapshot reads trouble codes and readiness monitors. You can also add an engine audio clip and the VIN.
- Understand.
- A new engine-anomaly model learns this engine's healthy baseline in about 20 seconds and scores every half-second after that.
- Gemini reads all of the evidence together and writes questions for the seller and checks for a mechanic.
- Deterministic rules flag patterns like "codes recently cleared".
- The VIN check digit is validated and the VIN is decoded.
- Act.
- On the pod itself: while you're still standing at the car, the OLED switches to
MISFIRE SUSPECTED, the LED blinks and the buzzer chirps. - In the report: mechanical checks, a per-second vibration chart, the Gemini advisory and spoken narration by ElevenLabs.
- During capture: ElevenLabs voice guidance tells you what to film next.
- On the pod itself: while you're still standing at the car, the OLED switches to
The rule throughout is that "not checked" never counts as "fine." A kit you didn't use, a blurry clip or a bad sensor read shows up as not checked, never as a pass.
How I built it
Capture rig
- Firmware: Arduino C++ on a Seeed Grove Beginner Kit. It streams a line protocol at 250,000 baud. I measured 397 accelerometer samples/s with zero garbled lines.
- Python bridge and simulator: the simulator writes the exact same protocol and recording format as the board. Everything runs without hardware, and a real recording drops in with no code changes.
- Emulated OBD: I don't own an OBD dongle, so an ELM327 emulator runs in-process and the real
python-OBDlibrary talks to it over a loopback socket. That's the same path a Wi-Fi dongle would use. Clearing codes resets the distance counters and leaves the emissions monitors incomplete, and that is the fingerprint UpClose flags. - Engine-anomaly model (numpy only): it extracts features for each 0.5 s window: RMS, crest factor, kurtosis, band energy around the firing frequency and its sub-orders, and the amplitude spread from one firing to the next. For a four-stroke engine the firing frequency is
$$f_{\text{fire}} = \frac{\text{RPM}}{60}\cdot\frac{n_{\text{cyl}}}{2}$$
so a 4-cylinder idling at 780 RPM fires at 26 Hz. Features are standardized with median/MAD and scored by Mahalanobis distance under a Ledoit-Wolf-shrunk covariance:
$$d(\mathbf{z}) = \sqrt{\mathbf{z}^\top \hat{\Sigma}_{\text{LW}}^{-1}\,\mathbf{z}}, \qquad z_j = \frac{x_j - \operatorname{median}_j}{\operatorname{MAD}_j}$$
Half of the baseline fits the model. The other half sets the threshold at the 99th percentile of held-out scores. A recording is flagged only if at least 3 windows and at least 15% of all windows exceed that threshold. The model says "unlike this engine's baseline", never "faulty".
- Tiger Data: pod samples go into a Timescale hypertable with a per-second rollup, and the report's vibration chart reads from it.
- Digital twin: a three.js scene with damage decals at known positions. Playwright and ffmpeg render a deterministic 1080p/30 fps walkaround with per-frame ground-truth boxes from a GPU ID pass. That gives me video where I already know the right answer.
Additions to the UpClose platform
- Evidence-source API: each kit attaches its evidence to an inspection before the report is generated. The evidence is validated, summarized deterministically, hashed and frozen into the report, so the same inputs always give the same report.
- Gemini advisory: uses
generateContentwith a JSON response schema. Every question or check has to cite an evidence key that was actually provided, and items that cite anything else are dropped. The prompt tells it to ask questions, not to diagnose. - ElevenLabs: narrates the report and voices a fixed set of capture-guidance lines. Audio is synthesized on the server and cached, and captions are always shown.
- Web app: a capture-kit panel, engine-audio upload with a spoken engine check, and Visual / Mechanical / Overall category scores.
- Other: an offline VIN check digit, WMI and model-year decoder with an optional NHTSA vPIC lookup, and a Playwright harness that drives the whole flow end to end.
Stack: Arduino C++, Python, numpy, FastAPI, PostgreSQL, Tiger Cloud (Timescale), React, Vite, Tailwind, three.js, Playwright, ffmpeg, Docker, the Gemini API, ElevenLabs, and Meta SAM 3.1 + Muse Spark for the vision pipeline.
Challenges
- No car, no OBD dongle, no labeled fault data. I went simulation-first with one shared recording format, emulated the OBD adapter while keeping the real client library, and built a baseline-relative model instead of a classifier I had no data to train.
- Evaluating honestly. On 336 held-out simulator recordings, window ROC-AUC is 0.997 and 4% of healthy recordings raised a false alarm. That's synthetic only, not real-car accuracy. On the real board, a tap test raised the alert at 25.1 s and cleared it at 34.4 s, and a 48-second still run raised no false alarms.
- Keeping uncertainty visible through every layer. The sensors, the model, the LLM and the score all have to be able to say "I don't know". The Mechanical score shows as not checked without OBD data or an assessed pod verdict. The Overall score appears only when both Visual and Mechanical were assessed.
- One laptop doing everything. The Docker stack, headless renders, Playwright and several parallel coding agents lagged the machine badly, so I capped parallel work and gated every change on its tests before committing.
- Latency. A live vision run takes about 7–8 minutes for a short clip, so day-to-day work runs in offline fixture mode.
What I learned
- Without labeled faults, "unlike this engine's own baseline" is more honest and more useful than any "faulty / not faulty" label.
- Sellers clearing trouble codes leaves a detectable fingerprint: the codes are gone, the distance counters are zeroed and the emissions monitors are incomplete.
- LLM output only belongs in a report once it's forced into a schema and every claim is tied to evidence you can check.
- Rendering ground truth deterministically is harder than it looks: GPU backends, shader warm-up and depth-aware labeling all matter.
Synthetic data disclosure
Simulated inputs are labeled synthetic everywhere they appear in the product.
