theMnexus Lumen

An OpenCV 5 agent on AWS Lambda that turns bracketed real-estate photos into one true image per room. Every fix is measured, explained in a trace, or handed to a person.

Comment

Inspiration

Most home listings are shot by owners and small agents, not by professional editors. Interiors come out with white windows, dark corners, leaning walls, door frames bent by wide lenses, and a ghost of whoever walked through the bracket. The usual "AI enhance" fix repaints the room. For a listing that is the wrong answer: the photo is a promise about a real property.

We wanted the opposite: a tool that only corrects optics, uses only the pixels the camera recorded, and can say exactly what it changed.

Real 7-exposure bracket: the camera's middle exposure vs Lumen

What it does

You drop a folder of bracketed photos into the web page. Lumen:

  1. Groups the photos into rooms, with or without EXIF.
  2. Aligns each bracket to sub-pixel accuracy.
  3. Merges the exposures and removes ghosts of people or objects that moved.
  4. Brings the window back from the darker frames instead of inventing a sky, and refuses the fill if it would add edges no frame recorded.
  5. Self-calibrates the lens from the room's own straight lines.
  6. Stands the verticals up, removing camera roll and keystone.

An agent runs these steps. After every step it reads OpenCV measurements and decides: keep the result, revert it, try another tool, drop a frame that does not belong, or stop and ask a person. Every room comes back with a trace: the numbers, the rule that fired, the tool and the decision. A person can accept, reject, or drop a frame and re-run, and can switch optional steps off at upload; both are written into the trace.

How we built it

  • OpenCV 5 (5.0.0) does the vision work:
    • Grouping: phaseCorrelate on Sobel edge maps, with a dct signature as a second opinion.
    • Alignment: AlignMTB, then findTransformECCMultiScale with masks on both images, with SIFT + findHomography RANSAC as a fallback. A 4 × 4 tile phaseCorrelate residual tells the agent whether a frame belongs.
    • Merge: MergeMertens, and MergeDebevec + TonemapDrago; ghosts from per-channel exposure matching, absdiff, morphologyEx, connectedComponentsWithStats.
    • Lens: Canny + findContours edge chains, a plumb-line cost after undistortPoints, then initUndistortRectifyMap + remap. The model is one radial term, found by making long edges straight:

$$x_d = x_u \left(1 + k_1 r^2\right), \qquad k_1^{*} = \arg\min_{k_1} \sum_{\text{edges}} \text{RMS distance to the fitted line}$$

  • Tilt: createLineSegmentDetector, a vertical vanishing point, and warpPerspective.
    • Agent: a fixed rule table in Python. The same photos always give the same trace.
    • AWS: a Lambda container on arm64 (Graviton) behind a Function URL; the browser uploads straight to a private S3 bucket with presigned links; an asynchronous worker invocation does the heavy work; CloudWatch keeps the logs. One SAM template deploys it all, and a 7-photo room costs about US$0.0007.

One job, step by step

  • Measurement: a generator draws rooms with known faults (camera shake, lens k1, roll, a moving object, a clipping window) under two camera profiles, including a harsh "phone" profile. Real brackets come from the Empa HDR Image Database and the Cambridge deghosting stacks (both CC BY 4.0). bench/ regenerates every number in the report.

Challenges we ran into

  • Our own tests were too kind. The first synthetic camera used the same gamma our code assumed. A harsher phone profile broke alignment (ECC on only 60 % of frames, 9.7 px error). A 2 px blur before the gradient and a correlation limit fixed it: ECC on 100 % of frames, error ≤ 0.26 px.
  • ECC aligns two different rooms if you let it. We added tile correlation and the ECC coefficient as checks, so a stranger frame reads as an infinite residual and the agent drops it.
  • Ghost masks fought themselves. A free exposure curve learned from the moving object and flagged up to 62 % of a frame. Quantile matching per channel fixed it: 0 % on a static real room.
  • Real data humbled the agent. It once accepted a worse Debevec merge, measured a straight room as 1.7° off because of a chair, and blended a window into tree-shaped blotches. Each became a rule or a measurement fix, with a test.
  • Lambda memory. This account caps Lambda at 3008 MB, so the worker processes one room at a time and shrinks frames evenly to the budget. A 7-photo 16 MP room runs in 18.3 s with a 1362 MB peak.

Accomplishments that we're proud of

  • 192 of 192 normal and injected-fault cases handled correctly on both camera profiles.
  • On 36 real brackets with motion, ghost removal raised the match with the motion-free shot from 29.4 to 34.8 dB.
  • Three real 7-exposure rooms finished on their own, with verticals within 0.42°.
  • A person stays in charge, and the same OpenCV 5 tools are available to other agents through an MCP server.

What we learned

  • Measure before trusting a threshold: most of our bugs were thresholds chosen on clean data.
  • A good agent needs good signals. The rules are simple; the work is making OpenCV say "this frame does not belong" reliably.
  • Stopping with a reason is better than a confident wrong photo.

What's next

  • Measure on more real listings and tune the limits on them.
  • A fisheye model and a second radial term.
  • Benchmark COOL against stock OpenCV on Graviton.

Built With

Share this project:

Updates

Submission history