PokeCat

Inspiration

We wanted to create an experience that can evoke the warmth and nostalgia of seeing a familiar cat for the 9374th time at the void deck, or in school.

What it does

A PokeDex, but for stray cats! Collect and immortalise records of all the cats you've ever seen, whether it be from a nearby shelter or a one-off encounter on an overseas trip. View your PokeCat entries anytime, anywhere!

Make cute sticker portraits of your favourite cats Every time you add a new cat to your collection, our application will create a cute pixel art portrait of your cat and add it to your sticker collection on Telegram. Show your stickers to your friends and families!

How we built it

We used the Python Telegram library to build the bot. and define the flow of the application. We used it to define new Command Handlers and States etc. We store all records in a MongoDB database: name, description of meeting details, pictures etc.

For generating pixel art portraits of the cats, we used two state-of-the-art multimodal machine-learning models. Firstly, the photograph of the cat is passed into an open-source model, Salesforce Blip Image-Captioning Base, producing a textual summary of what the cat looks like. Then, this summary is passed into OpenAI's DALL-E 3 model to draw the final pixel art portrait of the cat.

Challenges we ran into

We tried to implement a battle system, but we did not have enough time to do it because it was difficult to manage concurrent turn-based actions from both players.

We also tried to store our records with MySQL at first, but it was difficult as it was unsuitable for storing images. We also spent a significant amount of time coordinating the various image formats, like PIL, Image ByteStreams, .png/jpg, and image dimensions.

In addition, we initially also tried to use a single Image2Image model for generating the pixel art portraits. However, most of them we could find were locked behind a paywall or were just really large and inconvenient.