smtex

Architecture

Try it out!

Text \sqrt{x} to 970-400-1784!, or scan this QR code!

Inspiration

Most students in STEM majors have, at some point, run into the problem of needing to discuss equations over text. Unfortunately, it usually means sending SMS messages that look like y = (int(x^2-4x)/(sqrt(x^2 + 4^2)). Now I don't know about you, but looking at that just makes me want to give up on whatever I was trying to do. And, being someone proficient in LaTeX, I often thought it would be awesome to be able to send LaTeX code to a phone number and have it respond with a picture, so I could quickly discuss equations over SMS or in person. Thus smtex was born.

What it does

When you send a valid LaTeX equation to 970-400-1784, within a few seconds Twilio will respond with a professional-quality rendered image of your equation, along with a link to the image. This turns what would normally be a confusing, incomprehensible text formula into a form that can be easily shared with and understood by friends and colleagues!

How we built it

We started trying to do most of the processing on AWS Lambda in Node JS. When doing this, we ran into a lot of issues with I/O restrictions (see Challenges). Eventually we shifted off of AWS Lambda an onto a t2.Nano ec2 instance. When our Twilio number gets a text, it sends all of the data from the message to an express.js server, which routes the contents of the message and the user's phone number to another API, this time a MathJax server. The MathJax server is the one that does the heavy-lifting of parsing the LaTeX into svg, converting the svg to a jpg, uploading the jpg to a S3 bucket, and calling the Twilio API to return the image to the user.

Challenges we ran into

The underlying architecture of this application went through several heavy revisions before reaching its current form. In fact, the current technologies supporting smtex are almost completely opposite to what we originally thought they would be.

The first iteration of the project was intended to be written using twilio for SMS integration, which would be routed through AWS API Gateway to an AWS Lambda function which would perform the svg-to-png transformation. The lambda function would then handle returning the png to the original sender.

It did not take us very long to find a way to parse the LaTeX format into a .svg image. But .svg files on their own cannot actually be saved by mobile phones, so our next step was to change that .svg file into a .png. It was a seemingly simple process, but it lead to our most major road block. As it turns out, rasterizing (converting) svg files into normal images is not as easy as it might seem. As a result, nearly all third-party libraries for svg to png conversion were far too large to be used on AWS Lambda, which limits the maximum size of a function. We struggled to make Lambda work for quite some time, before we eventually were forced to the conclusion that Lambda would not work for our purposes and switched over to using an Amazon EC2 instance to do the svg-to-png conversion.

However, even then we ran into more problems. Inexplicably, the Twilio API and Amazon's API Gateway could not be made to communicate. We spent hours isolating all the variables and incrementally testing and adding features, but no matter what we did we were unable to make the API Gateway consistently return the images to the original sender. So API Gateway had to be ditched too, and in its place we wrote a customized REST API in Express on our EC2 instance to coordinate twiliio with the svg-conversion backend.

Only then did all the pieces start to fall into place.

Accomplishments that we're proud of

We are really proud of getting the data out of the Twilio API message body. For some reason getting that data to parse was very difficult. We were also proud of not giving up, even though we spent many hours on the frustratingly elusive task of converting one kind of image into another. It was really difficult not to give up when we were on the 10th hour of converting svgs, but the payoff was huge when we got that rendered text back for the first time.

What we learned

For one,don't be so set on using Lambda, especially when dealing with a lot of 3rd party APIs. Sometimes having a server where we can store our temp files can make things go a lot smoother and faster. We also learned 80% of the work comes from 20% of the issues. We thought we were pretty home free when we thought all we had left was the svg response and the API, but that portion of the project ended up taking us nearly half of the time of the competition.

What's next for smtex

There are a couple of priorities going forward with smtex. First, making it open source. Due to some time constraints we weren't able to clean up the code in time to publish it really, but I do really want to make it open source. Also, it would be cool to look into other API's like WhatsApp, Slack, and Telegram, or iMessage integration. Along with that, supporting more equation formats, so you can type in your simple "x=sqrt(x)' and get back what you want. It would also be really cool to try to connect with Voice APIs so you could speak your equations.