Inspiration
An existing pair of lenses should be the starting point for a frame that fits them. The SNSF OptiFrame challenge inspired us to connect computer vision and practical fabrication in a workflow someone can use from a phone: photograph each lens, inspect its measured outline, generate a frame front, and export the files needed to check and print it.
Transparent lenses make this especially challenging. Their appearance changes with the background, lighting, reflections, and camera angle, while their boundaries can be barely visible. We developed OptiFrameAI around combining learned visual features with calibrated geometry, so that detecting a lens can lead to a physically meaningful outline.
What it does
OptiFrame organizes the process into six steps: Prepare, Capture, Measure, Design, Validate, and Export. Users capture or upload one original photograph for each eye with a known-size ArUco reference board. Our team’s hosted inference service returns separate lens contours, horizontal width A, vertical height B, perimeter, and verification overlays in millimeters.
The app turns those two potentially different outlines into a configurable frame front with lens rims, retaining grooves, a bridge, and temple attachment lugs. Users can adjust bridge spacing, clearance, and fabrication settings, inspect the model in 3D, and compare the lens contours with the generated seats. Validation tools record repeated-capture differences and optional caliper measurements.
Exports include:
- Left and right 1:1 SVG outlines with a print-scale check square.
- monture.stl, containing the generated frame front.
- A JSON report containing measurements and fabrication settings.
- Test rings with several clearances for printer calibration before committing to a full frame.
A saved real worked example shows reference detection, perspective correction, contour verification, and measurements.
OptiFrameAI provides the model architecture, training workflow, and metric evaluation pipeline for developing the lens-measurement component. Its full training package is prepared for an NVIDIA H100; that training run and its accuracy validation remain pending. The currently deployed service supports the application while this work continues.
How we built it
The interface uses React, TypeScript, and Vite. Three.js displays the generated frame, while Manifold’s WebAssembly geometry kernel builds the solid in a browser worker. IndexedDB saves progress on the current device.
The app is hosted on Vercel and calls our team’s inference service on Runpod through a server-side route, keeping the service credential out of the browser. Users need no account, installation, or personal API key.
We built against a defined analysis contract using synthetic fixtures while the model was being prepared, then connected the real service. The adapter preserves millimeter coordinates, eye orientation, model provenance, and per-eye failures. A failed result for one lens does not discard a successful result for the other.
We built OptiFrameAI in Python, using PyTorch and Hugging Face Transformers for segmentation, OpenCV for reference detection and geometric correction, and NumPy, SciPy, and scikit-image for contour processing and measurement. The implementation separates image restoration, segmentation, physical geometry, and evaluation so that each stage can be inspected and validated.
AI architecture
OptiFrameAI builds on heavily fine tuned L+GNet, DINOv3-S/16 and Swin-S with learned feature-fusion layers and a Mask2Former decoder.
DINOv3 provides general visual representations extracted from multiple transformer depths. Swin-S supplies a hierarchy of spatial features at different resolutions. Combining these streams gives the model access to both broader object structure and the finer image details needed to interpret a faint lens rim.
The fusion layers align the feature maps and apply learned projections with squeeze-and-excitation attention, allowing the network to adjust the contribution of different feature channels. Mask2Former then produces mask and class predictions that are combined into a dense lens-probability map.
The training configuration includes all approximately 116 million segmentation parameters: both backbones, the fusion layers, and the decoder. Lower learning rates allow the pretrained backbones to adapt while preserving useful visual representations.
A fixed, pretrained Real-ESRGAN x4plus model handles restoration before segmentation. The same segmentation network then runs twice: once to locate the lens in the overall scene and once to examine a higher-resolution crop.
AI pipeline
Orient and calibrate the photograph.
The pipeline applies EXIF orientation and corrects camera distortion when a matching calibration profile is available.Establish scale and correct perspective.
OpenCV detects ArUco markers using a physical dark-square edge length of 24.2 mm. Their geometry maps the photograph into a planar coordinate system measured in millimeters. Independently surveyed board coordinates support strict evaluation; an equal-square fitting method supports diagnostic processing when those coordinates are unavailable.Restore the image.
Real-ESRGAN performs 4× super-resolution using 256-pixel tiles with a 32-pixel context halo. Tiling controls memory use, while the surrounding context helps reduce tile-boundary artifacts. Training preparation, validation, and inference use the same restoration process.Find the lens.
A 768 × 768 view provides a global segmentation proposal. The image is resized without changing its aspect ratio.Examine the boundary.
The system crops around the proposal with a physical margin and runs the same model at 1024 × 1024. This dedicates more input resolution to the lens and its rim.Extract the metric contour.
The probability map is converted into a continuous outline. The pipeline reverses the crop and super-resolution coordinate transforms, converts the contour into millimeters, and applies controlled smoothing before measurement.Export evidence alongside measurements.
Evaluation produces contours, probability maps, overlays, structured measurements, and failure information. These outputs support inspection of both the detected boundary and the geometry used to measure it.
Training and optimization
We assembled 450 procedural lens scenes and 1,059 renders derived from public object shapes. Procedural scenes provide smooth contours with known physical geometry. Shape-derived renders add variety, but receive lower sampling weight because some contain polygonal rims that could introduce undesirable boundary characteristics.
We also implemented public-data training paths using Trans10K and Trans2K to expose the model to broader transparent-object appearances before lens-specific fine-tuning. Public images supervise segmentation only; physical losses require examples with valid metric annotations.
Our optimization choices address boundary accuracy, training stability, memory use, and data quality:
Boundary-sensitive losses. Mask2Former query supervision is combined with binary cross-entropy, Dice loss, additional boundary emphasis, and a modest metric perimeter regularizer. This makes rim placement part of the training objective alongside overall region overlap.
Controlled parameter updates. AdamW, learning-rate warmup, cosine decay, and gradient clipping are configured to stabilize training. Both backbones use one-tenth of the fusion and decoder learning rate.
Memory-efficient training. BF16 computation, activation checkpointing, and gradient accumulation support high-resolution inputs. Fine-tuning accumulates eight microbatches of two images for an effective batch size of 16. Sensitive probability losses run in FP32.
Balanced sampling. Sampling accounts for dataset source and source group, limiting domination by large collections of similar frames. Independently annotated real lens data receives higher priority when supplied.
Geometry-preserving augmentation. Rotations, flips, translations, and uniform scaling affect images and annotations together, with physical scale updated accordingly. Crops that cut off a metric lens are rejected.
Consistent appearance processing. Exposure changes, blur, noise, compression, and approximate glare are introduced before restoration. This preserves the intended processing order during training.
Validation-based selection. The experiment workflow compares public-data curricula and augmentation recipes, repeats the selected recipe with additional random seeds, and selects contour thresholds and smoothing using validation data.
These comparisons are implemented but remain to be executed in the full H100 run. They are designed to establish which choices improve the result.
Challenges and what we learned
The difficult part was preserving physical meaning across photographs, AI contours, and a manufacturable solid. Left/right orientation, coordinate axes, and scale must stay consistent throughout. The app uses horizontal and vertical A/B boxing dimensions, while the current AI evaluation reports the short and long sides of a minimum-area enclosing rectangle. Integrating the trained model must explicitly preserve the app’s measurement convention.
Transparent-object detection also requires more than identifying a visually plausible region. Small boundary displacements can produce significant perimeter errors even when a segmentation mask looks convincing. We therefore evaluate width, height, perimeter, and maximum outline deviation separately.
We learned that image sharpness and physical accuracy need separate validation. Super-resolution changes image appearance and sampling, but its effect on measurement accuracy still needs to be demonstrated. Camera distortion, marker scale, board geometry, and the lens rim’s height above the reference plane can also introduce errors that segmentation alone cannot resolve.
Synthetic data provides precise annotations, but its optical appearance is approximate. Public transparent-object datasets broaden appearance coverage, but do not supply calibrated lens measurements. This led us to use source weighting, group-aware splits, duplicate checks, and separate segmentation and metric supervision.
On the fabrication side, dense real contours exposed a degenerate geometry fragment, which we addressed and checked by exporting and reimporting the STL. Supporting original photo uploads also required a hosting route capable of carrying both images without exposing the inference credential.
Successful geometry generation and comfortable physical fit require different evidence. Adjustable clearance and test rings support calibration, but a real print and the actual lenses are needed to demonstrate insertion without force or play.
Results and current limitations
The deployed inference route successfully processed both supplied diagnostic photographs in approximately 1.56 seconds for one measured request. A separate transport check accepted two files padded to 20 MiB each in approximately 6.88 seconds. That check establishes upload capacity, not high-resolution image performance or complete phone-workflow timing.
The saved worked example reports A/B dimensions of 50.95 × 38.04 mm for the left example and 52.98 × 35.46 mm for the right. These are model outputs, not caliper ground truth. These measurements and timings belong to the currently deployed service and do not establish performance for the pending OptiFrameAI training run.
Automated application checks confirm that the saved real contours generate a single closed solid that survives a binary STL export/import round trip.
For OptiFrameAI, recorded verification includes 15 passing unit tests, static analysis, model construction and optimizer-coverage checks, and a combined data audit covering 7,325 lens and public-data records. The 30 real diagnostic photographs are excluded from training and model selection. Full H100 execution, training throughput, peak GPU memory use, and trained-model accuracy remain unverified.
Submillimeter accuracy has not yet been demonstrated. Strict evaluation requires matching camera calibration, independently surveyed board coordinates, independent lens dimensions and registered outlines, and verified capture geometry. Failed detections remain in the evaluation denominator, and missing references cannot produce an accuracy pass.
A fresh evaluation cohort is also needed because the existing diagnostic photographs have already informed development. Physical measurement accuracy, robustness on unfamiliar lenses, real-device end-to-end timing, and printed lens fit remain to be validated.
Sources, tools, and pre-existing work
The application uses React, TypeScript, Vite, Three.js, Manifold, Zod, Lucide, QRCode, and Fontsource fonts. The project includes third-party notices.
OptiFrameAI builds on L+GNet, DINOv3, Swin, Mask2Former, and Real-ESRGAN, using PyTorch, Hugging Face Transformers, OpenCV, NumPy, SciPy, and scikit-image. Its public-data training workflow uses Trans10K and Trans2K. These models, datasets, and libraries remain subject to their respective licenses and access terms.
The inference service and diagnostic worked-example assets were supplied by our team’s model developer. OpenAI Codex assisted with challenge translation, planning, application implementation, documentation, and tests.
Synthetic fixtures supported software development, and synthetic lens datasets support model training; neither establishes real-world measurement accuracy. Model licensing, final weight packaging, and reproduction documentation are still being consolidated.
What’s next
Run the prepared OptiFrameAI training experiments on the H100, complete the public-data preparation, and evaluate the selected checkpoint using independent real measurements. Integrate the trained checkpoint behind the application’s analysis contract while preserving eye orientation, physical coordinates, and A/B measurement conventions.
Complete caliper and repeated-capture evaluation, test the full workflow on real phones, calibrate clearance on the available printer, and print the frame front for a real lens insertion demonstration.
Package the source, model weights, configurations, calibration requirements, and evaluation results so the complete workflow can be reproduced.
Log in or sign up for Devpost to join the conversation.