-
reference detection
-
ai segmentation
-
perspective correction
-
reference detection
Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned## Inspiration
OptiXAI was inspired by the OptiFrame challenge and the idea of giving recycled eyeglass lenses a second life.
In humanitarian and low-resource settings, corrective lenses may already be available through donations or reused glasses, but creating a frame that fits an individual recycled lens can still be difficult and expensive. Our goal was to replace specialized lens-measuring equipment with something much more accessible: a smartphone camera and computer vision.
What it does
OptiXAI is a mobile-friendly web application that measures the physical dimensions of an eyeglass lens from a photo.
The user places the lens inside a reference area containing four ArUco markers of known size and takes a picture. The application then:
- Detects the ArUco reference markers.
- Corrects camera perspective to create a top-down view.
- Calculates a pixels-to-millimeters calibration.
- Uses a pretrained SAM 2 segmentation model to isolate the transparent lens.
- Extracts the lens contour.
- Measures the lens width and height in millimeters.
- Displays the final measurement together with intermediate computer-vision images.
We also added fallback marker detection for photos where a reference marker is very close to the edge of the image.
Our repeated tests on several real lenses produced measurements generally within approximately 1 mm of physical measurements taken manually.
How we built it
The computer-vision pipeline was built in Python using OpenCV.
We use four ArUco markers as a geometric and physical reference. Since each marker has a known size, the application can calculate the relationship between image pixels and real-world millimeters.
After detecting the markers, we compute a perspective transformation that rectifies angled photographs into a top-down view.
The rectified image is then processed using Meta's pretrained Segment Anything Model 2 (SAM 2). Instead of training a segmentation network from scratch, we use SAM 2 to generate possible lens masks at different positions in the image. We filter those masks using physical size, shape, border contact, and segmentation confidence to select the most plausible lens.
The selected contour is converted from pixel coordinates to physical millimeter coordinates.
For the user interface, we built a Streamlit web application that displays the measured dimensions as well as the ArUco detection, rectified image, segmentation mask, and final measurement result.
Challenges we faced
The hardest part was segmenting a transparent object.
Eyeglass lenses have weak edges, reflections, glare, and very little contrast against many backgrounds. A simple threshold was not reliable enough.
Our first SAM implementation also assumed that the lens would always be centered in the image. When the lens moved to another part of the capture area, SAM sometimes segmented the background instead.
We solved this by searching multiple positions across the rectified image and automatically selecting the segmentation mask that best matches the physical and geometric characteristics of an eyeglass lens.
Another challenge occurred when ArUco markers were positioned close to the image boundary. Some images detected only three out of four markers. We added an automatic padding fallback that temporarily adds a white border around the image for marker detection and then converts the detected coordinates back to the original image coordinate system.
What we learned
We learned that reliable physical measurement requires more than object detection.
Segmentation tells us which pixels belong to the lens, but the reference markers and perspective correction are what allow those pixels to become meaningful real-world measurements.
We also learned the importance of testing the same physical object under different positions, camera angles, lighting conditions, and reflections. Instead of evaluating the system on only one ideal image, we tested batches of repeated photographs and measured average error, worst-case error, and measurement variation.
The project also gave us practical experience combining computer vision, pretrained machine-learning models, image geometry, physical calibration, Python, and web application development.
What's next
Our next development step is to preserve the complete lens contour in millimeters and use the left and right lens contours to automatically generate a custom 3D-printable eyeglass frame.
The intended consumer version would replace the ArUco capture reference with a common object of known dimensions, such as a standard bank card, making the system easier to use without specialized equipment.
Ultimately, OptiXAI aims to turn a phone camera into an accessible tool for measuring recycled lenses and creating affordable custom frames.
