Dialed

Read analog gauges from a phone photo. OpenCV 5 reads the dial or asks for a better photo; an agent logs it, names the next view, or holds a work order for a person.

Comment

For the judges: 60 seconds

  • Read a gauge: tap a sample or upload a photo; step through the nine OpenCV 5 stages. If it isn't sure, it reads the same photo again and says whether the readings agree.
  • Walk the round: TI-201 holds a work order from a week's drift; PI-104 with glare asks for the view from the other side.
  • Evidence: every held-out photo, every wrong answer, a risk-coverage chart, and the agent run 20 times per scenario.
  • Technical report (PDF) · Code (MIT) · Architecture

For the Agentic Vision Award: the workflow diagram (tools, guards, fallbacks), traces where the OpenCV 5 result decides the next tool call, and the evaluation: 359 of 360 model runs passed across 18 scenarios, plus the whole agent on 199 real photos (144 gauges, 55 to decline).

Inspiration

Plants are full of analog gauges that will never be replaced with connected sensors. They are read on rounds, by people with clipboards, and the numbers are typed up later, if at all. A bearing can warm for a week without ever leaving its normal band, and a filter's pressure drop lives in two rows nobody subtracts. Inspection robots read gauges commercially; we wanted it from the phone in the operator's pocket, with a reading someone can trust or an honest "take it again".

What it does

  • Reads a phone photo of a pressure gauge or dial thermometer with OpenCV 5 and returns a value with a calibrated confidence, or asks for a better photo and says what to fix.
  • When it isn't sure but nothing is wrong with the photo, it reads the same photo four more times, turned and rescaled, and gives the number only if four readings agree within 1% of the scale.
  • Shows its work in nine stages: the dial it found, the edges, the straightened face, the stroke map, the unrolled tick ring, the pivot, the needle, the numbers it read and the fitted scale.
  • On a round, an agent decides what happens next: log the reading; ask for a better photo and name the view it needs (glare on the needle: "step to the left"); flag a photo of the wrong gauge; or hold a maintenance work order when a limit, a week's drift or a filter's pressure drop is breached. A re-shoot is checked against the first photo before anything is logged.
  • Reads the operator's note: a note that reports smoke, heat or grinding raises the held order's priority one level, never lower.
  • Only someone with the supervisor passcode releases a work order, with the photo, the reading, the trend and the agent's full trace beside it.
  • On a phone, OpenCV.js 5 coaches the shot live (tilt, distance, glare, focus, steadiness) before the shutter unlocks.

One photo, decided: the OpenCV 5 reader, the confidence gate, the re-reads, the agent's choices and the guards in code

How we built it

Dialed on AWS: CloudFront, Lambda with OpenCV 5, Amazon Bedrock (Nova Micro), DynamoDB, S3, CloudWatch and X-Ray in one CloudFormation stack

  • OpenCV 5.0 in Python on AWS Lambda (arm64): Canny and fitEllipse find the dial; a warp maps the rim ellipse to a circle (a full projective rectification from the rim conic and the pivot is implemented and measured too); a black-hat stroke map keeps needle, ticks and print; warpPolar finds the tick ring; tick lines meet at the true pivot; HoughLinesP and HoughCircles find the needle and its hub, and the pointer end is the one whose unbroken ink run is longer.
  • Two OpenCV Model Zoo networks through cv.dnn: PP-OCRv3 finds the printed numbers and CRNN reads each box in its own orientation. OpenCV 5's new DNN engine runs the detector faster than the classic engine on CPU, while the classic engine runs the recogniser faster, so each model uses the engine that suits it.
  • The scale is fitted per ring of numbers with RANSAC over every plausible reading of each number (lost decimal points, lost minus signs), then interpolated between neighbours.
  • Trust: a logistic confidence model over the reader's own measurements, fitted on 600 rendered gauges; re-reads of the same photo for the readings it doubts; nothing far beyond the numbers read is ever given.
  • The agent: Amazon Nova Micro on Amazon Bedrock (Nova Lite as backup, a rule engine if neither answers) chooses among seven tools. Guards in code decide what each tool may do: a work order needs a measured breach, its priority can't go below the breach and only a note can raise it (one level), figures the model writes that aren't in the measurements are struck through, a re-shoot that disagrees with the first view can't be logged, text read from a dial never reaches the model.
  • AWS (Sydney): one CloudFormation stack with Lambda behind CloudFront (the only way in, with a secret origin header), DynamoDB with point-in-time recovery, S3, Bedrock, X-Ray, and CloudWatch metrics from one log line per request and per capture, a dashboard, and alarms on server errors and model fallbacks. Spending guards: 5 concurrent Lambda runs, 400 model-assisted captures a day, 600 tokens per model reply, 30 photos per visitor an hour.

Challenges we ran into

  • Real photos fight back: counterweights that look like pointers, needles at rest below the first printed number, two printed scales, numbers printed sideways, glare across the needle.
  • Honest evaluation. Real photos don't come with a true value, so we built a benchmark of 162 openly licensed photos from Wikimedia Commons and Flickr, read each one from the photo twice, independently, settled every disagreement on a zoomed crop, and set 63 aside. Every change was chosen on the other photos first.
  • Answering more often without answering wrong. An Oct 18 rule looked free on the development photos and added 2 wrong answers in 3 on the held-out ones, so it is off. The Oct 19 re-reads took development answers from 13 to 24 (11 to 20 right); on the held-out photos they added two answers, one right and one 3.3% off, so those photos neither confirm nor rule out the gain. The held-out photos and the agent scenarios also found two bugs in the re-reads, both fixed and published: they skipped the reader's rule against values far beyond the numbers read, and they "confirmed" a reading taken through glare on the needle, where only a new view can help.
  • Where the model helps. Letting the model choose the re-reads gave 22 answers where a fixed order gave 26, so re-reading is code. The model's own call is the operator's note, which code can't read.

Accomplishments that we're proud of

  • Real photos it was never tuned on (63 gauges): Dialed answered 7 times and 6 answers were within 2% of the scale (49–97%, 95% interval); on the other 56 it asked for a better photo. The reader alone answered 5 (5 within 2%); the version first submitted answered 10 with 6 within 2%.
  • Against a vision-language model: Amazon Nova Lite reading the same photos answered all 63, and 4 answers were within 2% (51 were off by 5% or more). It answered 53 of the 55 photos that should be declined; Dialed answered 1.
  • Against a classic OpenCV reader (Hough circle and longest Hough line, even told the true scale range): 10 of its 41 answers were within 2%.
  • Rendered gauges with exact answers: 99.4% of 180 accepted ordinary photos and 97.7% of 43 accepted bad photos were within 2% of the scale.
  • The agent, 20 runs per scenario with Amazon Nova Micro: 359 of 360 runs passed across 18 scenarios, including text printed on a dial that tells it to approve, notes that try to cancel or downgrade a work order, and two-photo active perception. With a note reporting smoke it raised the order's priority in 19 of 20 runs and kept it in 20 of 20 with a harmless note. About $0.00031 of model tokens per decision.

What we learned

  • Classical geometry still does the heavy lifting: tick lines and ellipses carry more truth about a dial than any single detector, and refusing is worth more than guessing.
  • A confidence fitted on synthetic data does not transfer to real photos by itself; you need real photos, labelled twice, and a held-out set you look at as little as possible. It stopped one rule and found two bugs.
  • The textbook fix is not always the measured fix: projective rectification is exact for a flat dial, but on real photos it did no better than an affine map.
  • Give the model the decisions code can't make. Re-reading is arithmetic; reading a person's note is judgement.

What's next

  • Coverage on real photos: per-gauge enrolment from one good photo, and a needle detector trained on real photos for counterweights and resting needles.
  • A real phone field set from a working plant, scored once, and a pilot.
  • Proactive rounds: the agent reminds the operator which gauges are due and which drift to watch.

Built With

Share this project:

Updates

Submission history