Classification recognizing Good View and Bathroom, the view is more interesting
Gallery of all floorplans for a building
Gallery of kitchens
Photos of a Tribeca condo for sale at 166 West 18th Street, all classifications done by AI
Algorithm catches a lazy agent using the wrong photos
We realized brokers send and receive many thousands of photos per day, and aggregation portals receive millions of images per day, almost none of which have proper labels. That's a problem for agents as they prepare and organize listings presentations for their clients, or for VOW and portal designers helping a customer browse available inventory. A consumer's first impression of a listing is the photos presented, and no one has packaged together an autonomous solution for categorizing and personalizing the presentation of residential photos.
What it does
Brokers can upload a series of photos pertaining to their listings inventory, with or without captions and labels, and RealtyHop automatically classifies the photos into dozens of different possible categories, not all mutually exclusive. Furthermore, we can identify key objects in the photo and assist with writing listing descriptions or identifying amenities and features of the property.
As agents present listings to customers, RealtyHop can customize the display based on the preferences of the customer towards certain room types or residential features. For example, an agent may want to showing listings with modern kitchens first or specific floorplan categories based on the needs of the customer.
How I built it
We trained an image classification system using a combination of deep learning and a customized expert system tailored to residential real estate. The deep learning portion trained on over one million distinct residential real estate photos that we received from hundreds of brokerage and MLS partners.
The expert system overlay added more real estate specific features that are often weaknesses of a mutually exclusive classification approach using standard normalized posteriors, because of the overlap between image classes.
In plain English, that means most traditional AI approaches fail for real estate, because brokers and consumers want to know that the kitchen has granite counter tops and stainless steel appliances, with a nice view of the park. Simply saying something is a kitchen is not good enough.
Challenges I ran into
As mentioned above, many features of an image can have high overlaps between two categories. A small studio can look like both a living room, dining room, and kitchen all in one. The window might also show natural light and have a nice view. An out-of-box approach that uses machine learning and deep nets will not perform well, because such techniques often require mutually exclusive categories and little or not hybrid categories.
We had to expand upon a naive "tensorflow and grind" approach and create various correlation analysis techniques to allow for hybrid categories.
Accomplishments that I'm proud of
I am proud we could produce a practical machine learning system and deploy it using real life data, in a useful application domain that is ripe for innovation!
What I learned
Real Estate is very rich with data, and the next generation of innovation will certainly leverage modern techniques in an age-old industry. The intersection between the two fields has plenty of untapped opportunity.
What's next for RealtyHop
We have released the beta photo classification and listings edit tool. Every day we classify thousands of new photos from our MLS and brokerage partners and continue to further enhance the system. However, new innovative features are coming, such as the ability to extract home features/amenities/and details straight from a set of photos for a listing. We plan to package the system as an important component within a larger investment analytics tool.