SERRATA

"Your AI Co-Worker on the Construction Site."

Comment

SERRATA: An Intelligent AI Co-Worker for Construction

Inspiration

Construction is one of the most demanding industries in the world. Every day, workers make hundreds of decisions while handling heavy equipment, following complex blueprints, and working in environments where even a small mistake can lead to serious consequences. Despite this, most critical information is still locked away in paper documents, mobile phones, or laptops that are difficult to access while actively working.

When we learned that Meta Smart Glasses could understand what a user is looking at, we immediately saw an opportunity. Instead of forcing workers to stop what they are doing to search for information, why not bring the information directly to them?

This idea became SERRATA—an AI construction co-worker that sees what the worker sees, understands the job site in real time, and provides intelligent, hands-free assistance whenever it is needed.

Our goal was to create something that feels less like software and more like an experienced colleague standing beside every worker on site.

What It Does

SERRATA transforms Meta Smart Glasses into an intelligent assistant that continuously understands the worker's surroundings and provides contextual guidance throughout the workday.

Instead of waiting for commands, SERRATA actively observes the environment and steps in whenever it detects something important.

Some of its core capabilities include:

  • Detecting missing safety equipment such as helmets, gloves, and reflective vests.
  • Identifying hazards including exposed electrical wiring, unsafe ladders, fall risks, restricted areas, and nearby heavy machinery.
  • Providing spoken, step-by-step installation guidance while workers keep both hands free.
  • Reading and explaining project drawings and blueprints in simple language.
  • Comparing completed work with construction plans to identify possible mistakes.
  • Detecting visible defects such as cracks, poor finishing, or misalignment.
  • Recognizing construction materials, tools, and equipment.
  • Recommending the correct tool for a specific task.
  • Answering technical questions using project documents, manuals, and standard operating procedures.
  • Tracking work progress automatically by recognizing completed installations.
  • Creating daily site reports and inspection logs without manual paperwork.
  • Translating conversations between workers who speak different languages.
  • Allowing remote supervisors or experts to assist workers through live video and annotations.
  • Predicting potential safety risks before an accident occurs by analyzing the surrounding environment.

Rather than functioning as a chatbot, SERRATA behaves like an experienced site supervisor that quietly monitors the work and speaks only when its assistance is valuable.

How We Built It

SERRATA combines computer vision, speech recognition, large language models, and document retrieval into a single system.

Using the camera on Meta Smart Glasses, the system continuously captures the worker's point of view. Computer vision models analyze the scene to identify safety equipment, tools, materials, machinery, structural components, and possible hazards.

Workers interact naturally through voice. Questions such as "How do I install this?" or "Is this the correct component?" are converted into text and processed by a multimodal language model capable of understanding both the visual scene and the spoken request.

To ensure accurate responses, SERRATA uses Retrieval-Augmented Generation (RAG). Instead of relying only on general AI knowledge, it searches project-specific resources such as blueprints, equipment manuals, safety guidelines, and company procedures before generating an answer.

The backend coordinates computer vision, AI reasoning, speech processing, document retrieval, reporting, and cloud synchronization to deliver responses with minimal delay.

Challenges We Faced

One of our biggest challenges was deciding when the AI should speak. Construction sites are already busy and noisy, so constant notifications would quickly become distracting. We wanted SERRATA to provide assistance only when it genuinely improved safety or productivity.

Another challenge was combining multiple AI systems into one seamless experience. Real-time computer vision, speech recognition, document retrieval, and language models all needed to work together quickly enough that conversations felt natural.

Construction environments also present unique difficulties. Changing lighting conditions, dust, moving equipment, and partially obstructed objects make reliable visual understanding much harder than in controlled environments.

Finally, we wanted workers to trust the AI's recommendations. To achieve this, we grounded responses using project-specific documents instead of relying entirely on a language model's general knowledge.

What We Learned

Working on SERRATA taught us that useful AI is not simply about having the most advanced model. The real challenge is designing technology that fits naturally into the way people already work.

Throughout the project, we learned how to combine computer vision, multimodal AI, Retrieval-Augmented Generation, speech interfaces, and cloud services into a system capable of solving practical problems.

More importantly, we realized that the best AI assistants are proactive rather than reactive. Instead of waiting for users to ask questions, they understand context and provide meaningful assistance at exactly the right moment.

Future Scope

SERRATA is designed to grow beyond a construction assistant into a complete AI platform for job sites.

Future improvements include deeper integration with Building Information Modeling (BIM), digital twins, predictive equipment maintenance, worker fatigue detection, drone-assisted inspections, offline edge AI for areas without internet connectivity, and integration with enterprise project management systems.

As these capabilities evolve, SERRATA could become an intelligent partner that supports workers throughout every stage of a construction project—from planning and installation to inspection and maintenance.

Impact

Construction is one of the largest industries in the world, yet many job sites still depend on manual inspections, paper documentation, and delayed communication. These inefficiencies affect safety, productivity, and project costs.

SERRATA demonstrates how wearable AI can change that. By bringing intelligence directly into the worker's field of view, it reduces the need to search for information, improves decision-making, and helps teams work more safely and efficiently.

Our vision is simple: give every construction worker an intelligent teammate that is always present, always aware, and always ready to help.

Built With

Share this project:

Updates

Submission history