UltraZoom

School is easy

Comment

Inspiration

Inspired by a project we worked together in school, where we understood the difficulties of children who are differently-abled. We also visited our family and learned that it is tough for children to sit in a place and just listen to the class on Zoom and UltraZoom feature is an addition to help that process given the global pandemic. We saw as students faced mental pressure because they couldn’t pay attention throughout and forgot important deadlines and were reminded just away before or even later than the submission date again disrupting their mental state. This might be even worse for children with ADHD or ASD. We saw interactions amongst children and how it is hard for children on the spectrum to sense how people are talking to them and to help in such interactions we added a feature to measure “toxicity” UltraZoom is a way for children, can be used for all children to help them study and focus in class.

What it does

UltraZoom has multiple features that can help children in their classroom 1) Real-time speech to text feature - Since children have trouble keeping up with what the teacher is saying, this feature, can help them read the text written in front of them 2) Save notes on this text - Children have the option of saving texts for future reference or ask questions if it isn't clear. 3) Toxicity Identifier - If the teacher makes remarks against the student, this is recorded in the application and the student also made aware of the emotion and can prepare themselves for this - call a parent, replace behaviors. Further, this application is made available to the parents as well, so they know what is happening in school. This feature is mainly focused on children with ASD as they sometimes cannot understand the emotion. So, the comments of other children can also be recorded. 4) A calendar and notes - To mark important dates and events, such as homework submissions, test dates, school events 5) Chat - When the child gets anxious or needs help, they can contact their family, friends, teachers or anyone using a chat functionality 6) Use transcripts to help students find friends based on similar ideas mentioned in class with voice recognition.

How we built it

The frontend is built using React - we stream the speech from the browser and use the SpeechRecognition hook which under the hood uses the Web Speech API Toxicity is detected using TensorFlow's toxicity classifier We use a backend build in Flask, to detect dates and keep a record of them Finally, we built and integrated a React-based Twilio app which is used to help students chat with family, friends etc.

Challenges we ran into

We were working in different time-zones, one from Gulf Standard Time and another in Eastern Standard Time We worked on multiple components independently as we tried to include as many features as we could, and had trouble integrating it Detecting date was also a challenge, which is why we decided to build an API for it - as we wanted to detect words like "tomorrow", "today", and phrases like "end of the month"

Accomplishments that we're proud of

We believe that we have built an application that will truly help students in the classroom. Provide an equal learning opportunity to everyone Worked on the end-to-end product, with additional APIs

What we learned

Working with various technologies - react, flask, TensorFlow, Twilio Finding a good way to collaborate with each other even though we were in different time-zones. Worked with TensorFlow js to include AI-based solutions for current needs We worked on different components individually and worked on the integration part during the overlap.

What's next for UltraZoom

1) We would like to add events to the google calendar which can help set reminders 2) Help highlight things that the teacher mentions as important in the class, like an auto-save functionality for certain sentences the teacher talks about, in case the student misses it. 3) Automatically detect the event rather than just the date

Built With

Share this project:

Updates