HavenAI

Haven's homepage interactive picture.
Haven's simple explanation of AI and ML integration.
Haven's additional recommended requirements in the Checklist section.
Haven's property assessment radar that analyzes all of the application features, from confidence level to moisture concerns and more.
One of Haven's Arlington property report titles.
Haven's in-depth AI analysis on potential hazards and safety concerns from a property.
Haven's machine learning and adaptive learning process explained.
Haven's confusion matrix, which displays its prediction accuracies across the 6 hazard categories.
Haven's feature importance chart, which gives a transparent approach to which factors weigh its decisions the most.
A couple of Haven's structural insights.

Inspiration

Our inspiration for this project was how difficult it is in the current housing market to find a high quality home that fits all the tenant's needs. Often, these landlords and real estate agents will hide issues with their housing unit, making it extremely hard for individuals to find a genuine, good place to stay. Not only that, people often have to go through multiple house tours and search through hundreds of reviews to find the best pick. Even our team's close family members have struggled with this, which is why we built HavenAI to make this process much more efficient, authentic, and easy.

What it does

The HavenAI technology converts the phone's camera into an instant inspection device. The user first inputs some basic details about the property (its type, rent value, size), and then walks around inspecting each room one-by-one while taking photos. Each picture is inspected by an AI-powered visual system that identifies potential problems (water damage, cracks, electricity-related issues, etc.) and assigns a severity level along with an explanation why this issue is significant. As more and more photos get uploaded, the app calculates how well the property has been actually covered (based not only on the number of pictures, but rather based on what exactly has been photographed), and gives a single Haven Score that helps compare different properties to one another.

How we built it

Frontend: React and Typescript with Tailwind communicating with the backend (Express) using tRPC, so we did not have to maintain my API schema separately. Data storage: SQLite with help from Drizzle. Photos are simply saved to disk.

AI: Vision-language model run locally using Ollama service rather than cloud-based API, thus saving any need for an account or charges. Each photo is accompanied with JSON schema in order to force the model to output the same structure each time (found issues, severity, confidence level, quality of the image) rather than leaving us to make sense of any free text. That structured output goes into our scoring algorithm: Coverage checklist of all items which are important for the particular room, Haven Score itself and recommendation for next photo to take.

Challenges we ran into

When it came to the AI portion of this project, there were several problems. First of all, since Ollama has no built-in API options to control the amount of context available for its AI models, photos kept giving us "exceeded context size" errors. The solution was rather simple – just use a different Ollama API endpoint. In addition, full-size photos from phones caused a lot of problems, both in terms of performance and reliability, so I decided to resize photos prior to sending them to Ollama API. Another problem worth mentioning is that the application could not connect to Ollama even though it was running. The issue was that Node.js was attempting to connect using IPv6 rather than IPv4.

Accomplishments that we're proud of

Actually just having everything work from start to end without anything being done in the cloud. You take a picture, your local model analyzes it, and it goes directly into the scoring system without an API key or any account being used at all. It feels great to have a fully working AI project that has machine learning training, data, and insights integrated into it that could potentially help people.

What we learned

Just how much different running the model locally is than just hitting an API. Limitations of context, outputting structured responses, even something as simple as networking. It doesn't work in the same way when there isn't a cloud service that handles all of this in the background anymore. Also discovered the importance of crafting your prompt correctly, as this application is making claims about the area where people may be renting or buying a house, so the application cannot simply assume things.

What's next for HavenAI

Two elements that we did not end up including in the final project from our initial idea are the integration of location information such as whether or not the area is safe or expensive, as well as a feature to compare two properties against each other. It seems like both would be feasible to implement on top of the existing system.