Every time we see these very funny memes, we want to know the source, or say the hashtag of the emoticon, and we also want to see other similar memes, but there has been no such function before. So, we made this meme search app iOS.
What it does
This BlossoMeme app is using for images, for example, memes searching. We can use it to search the similar memes, and it can recommend us with daily favorite memes, for example, memes of the day.
How we built it
On the backend side. We deployed the service to AWS. Our AWS API Gateway will accept a meme image url as request and integrate the input with the machine learning model to get the top 5 most relevant tweets. Then we return the response to the front end.
On front end, we built our app on iOS platform using Swift UI to implement a friendly user interface for getting the relevant tweets. And we set this cute image as our icon.
For our image search algorithm, there are two main parts: the feature vector generation, and the nearest neighbour search. For feature vector extraction, we utilize a pretrained model MobileNetV2 available at TF hub based on inverted residuals and linear bottlenecks that is proven to be effective on multiple tasks and benchmarks as well as across a spectrum of different model sizes. We use this model for fast generation of feature vectors of the meme’s we collected and query. After fetching the feature vectors, we perform nearest neighbour search among them to find the memes we are searching for. In particular, we utilize the spotify/annoy packet, usd by Spotify for their music recommendation, to perform nearest neighbour search. Annoy has the advantage of fast speed and small memory usage and is proven to be of high performance in high-dimensional space, a perfect match for our task.
Challenges we ran into
Our machine learning model requires extra space while running. We need to optimize the way we deploy the service so the latency for each request will decrease. We plan to improve that in next step.
Accomplishments that we're proud of
We made a very user friendly interface, and the accuracy is at a very high level.
What we learned
For the image search problem in general, there are various choices of solutions out there focusing on different subtasks, many of which I’ve laid my hands on, for example some for same style search, some for object classification and some for focus based tasks. For our specific goal, we aim to find the exact same pictures out of our database so that our user can learn where the memes come from and what are the similar tweets using the same memes, and we want those found fast, out of potentially millions of pictures stored in the database. For that purpose, we find our approach most suitable. One of our key segments, the Spotify annoy library works extraordinarily well because it achieves fast speed and small memory usage at the same time, perfectly matching our needs. It also has the benefit to use static files as indexes, this means you can share indexes across processes. Annoy also decouples creating indexes from loading them, so you can pass around indexes as files and map them into memory quickly.
What's next for BlossoMeme
First of all, we want to build widgets for our application such that this meme search could be triggered not only from twitter app, but anywhere and anytime while using the phone. In our app, apart from the meme search functionality, we also recommend some trending memes or ‘meme of the day’ for the user. For our meme lovers to more conveniently access it, we also want to add voice control to support it. In some of our experiment cases, we found that the returned tweets from our search might be multiple retweets of the same over and over, and the search result did not prioritize the source. In our next step, we want to also consider ranking according to the tweets’ popularity so to find the original posts using the tweets.