FieldWork.AI

Inspiration

We wanted to build better tools for field works that work on maintenance and difficult tasks in the field. Currently mobile apps are the primary way that fieldworkers access complex instruction/procedure. An interactive/immersive would be game-changing.

What it does

Fieldworker.eye lets the field workers complete complex tasks verbally in the field by asking for instruction. They can then view the instruction as well as useful diagrams.

How We Built it

Check out the diagram for details on how we build it.

Challenges

We were planning to use LanceDB but ran out of time.

We are proud of being able to:

1. Allowing the user to provide their question using whisper
2. Integrating the New Relic OpenAI dashboard to be able to monitor cost
3. Using AWS Rekognition object detection to provide labels for the generated images that provide supporting information to the field worker

What we learned

We learned about AWS Rekognition, Sagemaker, APIs, and other tools. We learned about how to integrate whisper with unity.

What's next for FieldWork.AI

What is next is to have an option to check that the worker is wearing the correct PPE. Provide the option to receive the information in smaller chunks (steps).

Built With

• amazon-web-services
• c#
• new-relic
• openai
• python
• rekognition
• unity
• whisper