LaTeX Dictation

Inspiration

Imagine you have a submission due in one hour and have to submit a LaTeX document, but you keep forgetting how to write LaTeX functions. This is the tool for you!

All three of us are STEM undergrads so we've had our share of writing latex, and are privy to several problems with LaTeX:

Steep learning curve for beginners.
Really verbose, resulting in excessive and time-wasting typing.
Hard to use for blind people with screen readers.

What it does

Using our LaTeX Dictation tool, you can dictate math formulas straight into your computer and see the equation being built in real-time, producing a valid LaTeX expression.

TL;DR You say stuff and a LaTeX expression appears.

How we built it

We use Microsoft's speech-to-text API to transcribe the text. This text is corrected for common homonyms, such as "eggs" for "x" and "to" for "2".

For the real-time LaTeX conversion, we built our own math parser which handles common equations and expressions. This process is instantaneous.

When the user has finished speaking, the final text transcription is sent to the Wolfram Alpha API, giving us the expression in Mathematica code, which is then converted into LaTeX code.

The final conversion using the Wolfram Alpha API is more accurate than our own math parser, but takes more than one second to complete, which is why we used both methods in our final product.

Challenges we ran into

For converting Mathematica code into LaTeX code, we used the Mathematica program. However, we found a bug in Mathematica that caused a function which works fine in interactive mode to fail when run as standalone code. This led to hours spent trying to debug it properly, finally ending with us just hacking around it: for every call, we emulate a whole fake terminal running Mathematica, and send fake keyboard input while capturing screen output in order to get data in and out of interactive mode.

Accomplishments that we're proud of

It works! Now we can help beginners get over the steep learning curve of LaTeX, and help people write better documents.

What we learned

A bunch about virtualization, at one point we even considered emulating a whole Linux system just to communicate with Mathematica in interactive mode. We ended up emulating a shell to run it.