Hermes is a project that is not only ambitious but also socially impactful. Hermes uses a Google Glass that adopts an end-to-end open-source machine learning platform to highlight and label objects in the user's field of view. Hermes places the user in the most conducive environment for learning a new language through reinforced learning by interacting with the environment. Although Hermes is beneficial to everyone interested in learning a new language, it is particularly resourceful for individuals with learning disabilities such as Autism. Cognitive and behavioral research papers have shown that autistic children have the same mental capacity as a child without this condition. Unlike children without this condition, autistic children lack the ability to infer similarities between objects resulting in subpar learning when placed in the real world. Hermes hopes to assist autistic children in overcoming this learning hurdle to maximize the amount of learning that happens during critical development periods.

What It Does:

Hermes is a convenient mobile Android application interfacing with a wearable device, such as Google Glass. Hermes will then provide real-time labeling and highlighting of different objects in the user's field of view. Users can conveniently continue their day and learn at the same time. This software is easily accessible and functional allowing for the smoothest user experience.

How We Built It:

Hermes leverages Azure's robust open-source libraries and augmented reality devices to develop an affordable communication interface. In particular, we used Azure's open-source object-detection API. We also used a large dataset of items to improve the precision and accuracy of item identification. This device will then provide visual and audio cues to assist the user in identifying recurring everyday objects. Hermes will be compatible with Android smartphones by providing real-time analysis of different items.

Challenges We Ran Into:

There were several challenges that Hermes encountered that required innovative solutions. Since object-detection API doesn't take depth into account, Hermes had to develop a solution that highlights the most pertinent objects in the interface when many nonessential identifiable items were present in the user's field of view. Another challenge was ensuring that Hermes conforms to the stringent guidelines set forth by the national data privacy regulators. Since users will use the device in public and private spaces, our device will have several built-in features to ensure user privacy when necessary through encryption, notifications through LEDs, amongst many others.

Accomplishments That We're Proud Of:

It is rewarding to work on a project that is intuitively complex but also socially profound. It is also fulfilling for adopting and realizing the power in Azure's open-source API. We are proud to announce that we have deployed a finished and testable product!

What We Learned:

We learned how essential frequent team communication is when developing a project that has many moving variables. We have also familiarized ourselves with the powerful open-source Azure API.

What's Next For Hermes?:

Hermes will conduct a study using this technology with Emory Autism Center and the Children's Healthcare of Atlanta. The frequent feedback will allow Hermes to revise and upgrade current features to make the device more tailored to the market.


American Psychiatric Association. Diagnostic and statistical manual of mental disorders. Fourth. Washington, DC: Author; 1994.

Anderson DK, Lord C, Risi S, Shulman C, Welch K, DiLavore PS, et al. Patterns of growth in verbal abilities among children with autism spectrum disorder. Journal of Consulting and Clinical Psychology. 2007;75(4):594–604.

Bailey A, Phillips W, Rutter M. Autism: Towards an integration of clinical, genetic, neuropsychological, and neurobiological perspectives. Journal of Child Psychology and Psychiatry. 1996;37:89–126

Baldwin D. Infants' contribution to the achievement of joint reference. Child Development. 1991;62(5):875–890.

Explore OrCam. 2021. Empower the Blind Community. [online] Available at: [Accessed 15 March 2021].

Iliescu, A., 2021. MondlyAR - World's first augmented reality language learning app. [online] Available at: [Accessed 15 March 2021].

P. Bigham, J., 2021. VizWiz – Algorithms to assist people who are blind. [online] Available at: [Accessed 15 March 2021].

S. MCDUFFIE, A., 2021. Labels increase attention to novel objects in children with autism and comprehension-matched children with typical development. [online] Available at: [Accessed 15 March 2021].

Built With

Share this project: