My home buying process took me 22 weeks and often involved scanning through 40 images on average in over 400 homes for sale. Most of the search portals let you do it by beds, baths, and prices. Finding a dream home takes much more than that. It's manually scanning the images for things like backyard, updated kitchen or outdoor patio to narrow down your search.

What it does

Using Watson's computer vision, we built sherlock to help you search homes by the information in images. "Alexa, Ask Sherlock homes with backyard" and Alexa will announce the homes it finds, load up the web page on your computer, and send a text message (Twilio) with the detailed information including turn-by-turn directions from Mapquest.

How we built it

We started by manually gathering information from and then we trained IBM Watson visual recognition API with 378 images in 4 different categories: backyard, balcony, hardwood floors, and updated kitchen. We built a Java backend which would take newer listings on sale and run it against IBM Watson to identify the class in which the image belongs, using our custom classifier we built earlier. Frontend was developed with ReactJS to showcase all the classified listings in 4 categories. We created an Alexa Skill so you can verbally ask for home info from Sherlock. This was connected to a Node server running locally that would fire up the according frontend webpage and send a text message with the house details plus turn-by-turn directions via Mapquest.

Challenges we ran into

Connecting Alexa to Wifi. Training limitations on IBM Watson Visual Recognition API - daily usage limits on free API. Manually getting all the training images and information.

Accomplishments that we're proud of

Connecting so many different APIs and services together to create a unique, great experience for a home buyer. Ability to classify home images accurately using custom classifier.

What we learned

We need a lot more images to further improve our accuracy, so expanding our training data to a few thousand images.

What's next for Sherlock

Profile home buyer taste with machine learning based on interest (hardwood floors, high ceilings, natural light, etc) in existing properties to create an accurate "View Similar Properties" experience. Build a search portal powered by Sherlock for all of San Francisco that can be used by realtors and home buyers.

Share this project: