Reader's Ear

The following is my submission for the Hack for Africa: A Microsoft Challenge under the Best Education/Literacy Hack category.

Preface

I joined the Hack for Africa competition late. Unfortunately, a working prototype is all but impossible given my time constraint (< 5 hours) and personal abilities.

None-the-less, I recently stumbled upon these Hackathons, am overjoyed at their existence, and had to start right away!

That being said, I present what I'd consider an executive summary for an application that aims to assist new readers with learning their fundamentals and correcting errors when reciting passages of text.

Inspiration

A while ago I watched one of Google's I/Os where they went over a fairly complex feature. Their "Okay Google" voice-powered assistant was able to generate, using sound, automated messages for things like answering the phone. This technology has existed for companies, who use it to automate parts of service desk jobs.

The bread and butter that interested me was the ability to deconstruct a given sentence, then construct one as a response. Is there a way to gather a person's broken sentence? This is to say, for instance, if someone new to reading English was to butcher a sentence could this technology identify the mistake(s) and provide an alternative?

What it does

With the above stated, I can begin to explain more where my original idea of a "new reader assistant" comes into fruition.

From here on out, a user refers to a person using the application with the intention of improving their reading and pronunciation abilities. An instructor refers to a person offering academic support to their users.

Core idea

The core idea is for an application to deploy a neural network. This network would be trained on the proper pronunciation of pre-chosen passages. When this network is introduced to a new reading of a passage, given by speaking through a device's microphone, it will compare this reading to its trained counterpart.

Any differences the network detects would be signs of the user reading the passage improperly (see Challenges I forsee below for a caveat). The application would keep track of the errors encountered while the user is reading. At the end of the passage, the user will go back through their errors with the application. The application will provide information as to why it considered an error an error. In addition, the application would offer a number of tools for the user to correct their issue. This would chiefly include a correct reading from the application and the ability for the user to correct themselves, by rereading their mistake, while the application affirms this correction.

Extending the platform

The biggest area for an extension I see is letting instructors control certain instances of the application. This would be especially useful for the initial setup and configuration of the app.

For the user, this would eliminate any language-related pre-requisites they may encounter when using the application. The user may only need to press a "Begin" button and start reading. Completing a passage with a certain level of correctness may prompt them to graduate to the next configured passage.

This relationship between a user and an instructor would let the instructor track an individual's, and community's, progression. Letting them identify problematic passages or words and overall make more informed decisions when offering help.

How I built it

N/A

Challenges I forsee

As I said before, a caveat I foresee is the network's responsiveness to differences in diction as a result of geographic and language barriers. In that I mean would the network need to be trained based on someone from a similar area it's deployed in? As to avoid unwanted, false-positive errors.

Another area of concern I have is how the application aims to replace a face-to-face instructor and user interaction. I can't possibly imagine the app overshadowing that kind of relationship. So instead, the focus I had in mind was for areas where individualized help is hard to provide. Or, where the application can act as a tool for instructors to identify a user who requires additional support.

I've elected to leave out more technical challenges to save myself speculating too much on the idea.

Accomplishments that I'm proud of

I'm proud of my initial idea, given I came up with it in about half-an-hour. I feel that for my first participation it is something I'd of enjoyed building.

What I learned

While I wasn't apart of this particular Hackathon long enough to move past the planning step, I have discovered Hackathons in general! I will certainly be participating in more to sharpen my skills and help make an impactful difference in the world.

What's next for Reader's Ear

The next thing I would do is begin planning a proposed technical architecture. Laying out the various features and what will be needed to accomplish them. I'd also go about creating some designs in a prototyping software such as Adobe XD or Figma.