Inspiration

Looking at the doctors at a hospital or building inspectors in the field I see how much time they spend and how uncomfortable it can be to record and remember their observations and take notes.

Their skills and work are not for scribbling the note, they need to focus their attention on the patient or the building they make.

Let the technology help them by taking care of note taking. Let technology enhance this note taking. Make it safer, faster and more productive to better people lives.

What it does

Umnofon is a mobile app companion for a field professional. It uses voice-to-text and NLP models to process, understand and make available digital notes. With help of NLP and built-in inference logic an app produces a report document which is based on the notes that had been submitted by the user.

We had selected a civil engineering building construction case where an engineer needs to constantly monitor the project's progress by visiting a building site. Without our app an engineer will go through the building and either memorize or write down notes which are then reported to the project supervisor. Such approach can lead to checkpoints being missed (too much to check, limited time), errors (incorrectly captured data) etc.

With our app a civil engineer will dictate notes and they will be automatically recognized using Wit.ai-based algorithm. To avoid excessive repetition an inference algorithm is used to construct complete notes from historical data and incomplete information.

In the MVP app the report lists the resolved notes and their context.

How I built it

The app is split into on-device audio capture and a cloud-based processing. The stack for the app is the following:

  • App: Expo.io SDK37, React-Native-Paper interface,
  • Backend: Google Firebase: Authentication, Firestore, Functions, React-Admin
  • Speech-to-text/NLP: wit.ai

Challenges we ran into

  • wit.ai approach to NLP is different from my existing experience - the model was rebuilt at least twice,
  • mapping a real-world situation (in this case - civil engineering way of working) requires very flexible NLP and complex inference logic

Accomplishments that we are proud of

We had built an MVP full-stack application that runs on a mobile device, collects audio and uses Wit.ai for recognition and analysis in less than 3 days.

What we learned

  • professional settings like civil engineering can certainly benefit from the new AI/ML based technologies,
  • technology can make real and significant impact in traditional fields - reduce time and non-essential effort,
  • improvements in process and technology has real life impact - more patients served, safer buildings built etc.

What's next for Umnofon

There are several areas where Umnofon can develop:

  • security and privacy (encrypted notes)
  • clean and polish of the app,
  • new professional fields with specific terminology and models,
  • new natural languages support,
  • team collaboration when several professionals work on the same project,
  • integration capabilities,
  • compliance (i.e. HIPPA for medical data, GDPR)
Share this project:

Updates