SpotCheck

• 

  presentation slide

Inspiration

• Real estate development projects are complicated and costly. The current method of monitoring progress is hopelessly analog: hire a consultant to travel to the site and perform a scheduled visit once per month.

What it does

• Spot Check provides daily monitoring and updates on construction progress, using drone technology and AI analysis to identify safety issues and steadily track construction progress.

How we built it

• First we stream the video output from the drone to an AWS EC2 server using Real time media protocol.
• We also used Operant AI to secure the media stream created by the drone.
• Then we process the frames per user query using OpenAI image processing pipeline.
• We use our MCP server to process the raw image data into structured JSON format.
• We used bem to analyze the drone footage and the data is local (from next door),
• And then we use WebRTC to have a voice interaction with the drone.
• Used MiniMax Audio to build a phone agent and Operant Woodpecker for our eval.

Challenges we ran into

• Stable live video stream from the drone with low latency < 5sec.
• How to find the right frequency of data processing to have seamless live analysis. We chose in this context to let the user query for that processing.
• We understand that the RTMP is not secured so we use Operant.ai to secure the whole communication workflow.
• From product and project management perspective, time constraints, coordinating parts of the project so we could dual or triple-track the deliverables and meet the deadline were the primary challenges

Accomplishments that we're proud of

From a technical perspective, we accomplished the following:

• We were able to produce live inference from drone footage for our custom use case.
• Voice interaction with the AI Agent to review the analysis.

From a product perspective, SpotCheck solves the following pain points:

• Reduces the need for inefficient and expensive monthly construction consultant visits,
• Provides timely and comprehensive insights to construction progress
• Alerts stakeholders of issues when they, thus minimizing the information reporting delay and increasing transparency of progress and problems

What we learned

• Security is crucial for that kind of pipeline : Drone footage > VPS > Client side interactions. One major learning curve is to understand how data flows for our use case.
• Handling multiple data streams (videos, json, voice, pdf, ...) in a comprehensible interface is challenging and we think we have the right user interface for it.

What's next for SpotCheck

• Make the voice interaction more comprehensive
• Create a chatbot interface for quick queries

Built With

• amazon-web-services
• dem
• docker
• dronestream
• ec2
• flask
• html/css
• minimax
• openai
• operant
• pypdf
• python
• rtmp
• webrtc