Inspiration

It's often difficult to find photos on a Mac unless you know them exactly by name. Unlike text files, which can be found using descriptive searches in Spotlight/Finder, pictures do not carry any inherent descriptive data. Thus, we knew that one of the most important advances in photo organization OSX would be the ability to automatically and intelligently tag photos in a way that can be detected by Spotlight. We were also drawn to the power of Microsoft Cognitive Services, which lets one gain a wealth of descriptive data about images through simple API requests. As a result, the idea of Taggr was born: a desktop application that automatically categorizes your photos, making searching and indexing more convenient for the end user.

What it does

Taggr asks users for a photo directory and recursively searches for all pictures of filetype jpg, jpeg, png, bmp, or gif. To save on space and time, these images are scaled and placed in the user's tmpdir using OSX's built-in QuickLook thumbnail manager and sent to Microsoft's Cognitive Services Vision API. Retrieved tags with high confidence assignments are used to tag photos in a system-wide manner, automatically updating Spotlight's search database. The software terminates once all photos from the provided directory are tagged.

How we built it

We initially prototyped API calls using Python-requests in both Python 2 and Python 3. We then created a program to recursively search directories for images, whose output is piped into the API-calling program. After that, we worked on creating a Cocoa/Swift frontend for directory selection but eventually settled in favor of a Python-tk frontend.

Challenges we ran into

We initially tried building a Cocoa/Swift application, but it turns out that shell scripts and external programs are run in a separate environment from the standard one, making new environment variables and causing errors with loading python modules. To cut on debugging time, we ported our project to a Python/Tk frontend during the final few hours. Other challenges included debugging during the initial stages (e.g. API calls) and compatibility testing between Python 2 and Python 3, but these were quickly resolved.

Accomplishments that we're proud of

Despite none of us being computer science majors, we were able to put together an awesome website and an incredibly useful, first-of-its-kind Python application in under 36 hours. We used our enthusiasm for programming to learn the tools for both products, resulting in a final product we are proud to show others.

What we learned

We learned how to use several standard libraries, such as Python's os, Tk, and sys, in addition to learning how to use Python-requests. We also learned some basic shell scripting to create the installation script and initial scripts for tagging. None of us had worked with Swift before the hackathon, so learning to create a partial front end with it was an eye-opening experience.

What's next for Taggr

Using more Microsoft Cognitive APIs, Taggr will grow into a full-fledged multimedia file management system for OSX. For video processing, we plan to obtain highlights from videos using the Video Preview API and process them with the Vision API. For audio processing, we plan to use music recognition APIs to tag songs by name and/or genre. The addition of these two elements would give Spotlight/Finder the ability to search all media files on one's computer (i.e. not only text files), making it a powerful tool for creative content creation. We also hope to migrate to a Cocoa/Swift front end to create an OSX-native application and improve the Python frontend in the interim.

Built With

Share this project:
×

Updates