Inputting complex mathematical symbols into a text editor proves highly time consuming, impractical, and most importantly, frustrating. Research papers and math homework sets frequently use such symbols, and manually inputting them is simply not a good use of time.

What it does

Our project eases this problem by taking in a voice input of a mathematical expression or equation, and seamlessly converting it into LaTeX format. It also inputs it into a calculator to solve or reduce tough math problems. This allows for quick and painless input of complicated symbols, with the added flexibility of doing the calculations for you.

How we built it

We are hosting a Flask webapp that listens for an audio input and converts it to a string. We then parse it using custom algorithms to search for keywords like "integral," "log," or "divided by" to intelligently re-order the sub-expression so that it fits the LaTeX formatting. We also had to convert all human friendly function names into LaTeX specific terminology. For Example, "1 divided by 1" must be converted to "/frac{1}{1}". We then used the Wolfram Alpha API, with the LaTeX formatted string as the input, and can display the calculated output on our webapp.

Challenges we ran into

Parsing the input string None of us had front end experience, so using html and css to call functions on button clicks was much more challenging than we expected. We tried using Google's Cloud Platform but it proved too buggy so we switched to the SpeechRecognition API We also tried making a chrome extension, but we couldn't implement the communication system between the extension and out python file, so instead we used flask to directly run the python code.

Accomplishments that we're proud of

Coming in, with little experience and no project idea, we weren't sure what to expect or whether we would even have a working project to demo. We are extremely proud that we were able to integrate multiple APIs with our custom parsing algorithm to get a working app. We also originally planned to just format in LaTeX, but then were also able to implement an intelligent calculator that performs quite well at calculating square roots, fractions with complex parenthesis, derivatives, integrals, and greek symbols.

What we learned

We learned alot about frontend, implementing APIs, lots of string formatting, and building a flask app.

What's next for TeXTalk

There are many more complex, but rarely used, functions that we can add functionality for. There is lots of linguistic flexibility to be added, in terms of different ways people say the same thing. For example, "divided by," "over," and "fraction" all mean the same thing mathematically, but parsing them is very unique and challenging. We can also make the chrome extension so its easier for the public to access and use. For longer expressions, a live speech to text function would be better than having to say the whole expression, waiting, and then reading the output. If the LaTeX form could be shown as the person talks, that would be much better.

Built With

Share this project: