Broker AI

Inspiration

"There's not a single database for food halls in the United States" —VP Real Estate Development, ocV!BE

The issue we are addressing today involves real estate, specifically the reliance on static, non-customizable, and expensive databases prevalent in the American market. Currently, real estate developers rely on data-driven decisions for land development, utilizing databases like CoStar, which sells data on apartments and homes to aid in making informed decisions. However, a significant gap exists: there are no current databases for food and market halls in the United States. A decade ago, food halls numbered around 100 locations nationwide. This figure has since surged by 400%, with over 500 food halls nationwide. Yet, real estate developers lack information on these properties due to the absence of a comprehensive tracking system. This information is sparse on the internet, and no active database collects it. Given the rapidly changing nature of food halls and trends, we cannot rely on existing databases, which are often outdated, static, and prone to bias.

Currently, real estate developers have two options: (1) purchase data from a data broker at an exorbitant cost, or (2) conduct the research themselves. Even when opting for the first choice, the data aggregation process by brokers is very manual and time-consuming. This data is often biased because it is derived from past projects, resulting in a non-comprehensive view and inaccurate data.

Our inspiration for this project came from Joshua's experience working with a real estate firm that is developing a $4 billion development project in the SoCal Anaheim area. A major challenge Joshua’s colleagues—with over 20 years of experience—face is the lack of customizable data and an accurate pulse on market trends for food, market halls, and other commercial real estate sectors. Even brokers from Cushman & Wakefield couldn't find any databases on food halls. Our goal with Broker AI is to bridge this gap, moving away from costly manual research to employing smart web crawling tools.

Introducing Broker AI

Our solution is an AI-powered web crawler and scraper that continuously searches the internet data on food halls, recording what characteristics lead to successful food halls. This approach allows us to bypass the traditional methods real estate researchers are forced to choose from: either relying on brokers, which often results in biased, non-customizable, and expensive data or conducting research independently—a time-consuming process that can waste significant human capital. Our smart web crawlers leverage AI and large language models to navigate the web, analyzing and extracting accurate information about food halls. This enables us to feed our databases in real-time, offering the most comprehensive and up-to-date information available, thus providing real estate developers with the best insight into the commercial real estate market for food halls.

This untapped territory, now accessible through our Broker AI, represents a significant opportunity. By creating a database akin to what CoStar offers for homes and apartments, but focused on food halls, we can capitalize on this growing market segment. For context, I am a consultant for a real estate development firm currently working on a $4 billion project in the SoCal Anaheim area. A major challenge we—and many other developers with over 20 years of experience—face is the lack of customizable data and an accurate pulse on market trends for food, market halls, and other commercial real estate sectors. Our goal is to bridge this gap, moving away from costly independent research to employing smart AI crawling tools.

How we built it

At the heart of our innovation are the AI Agents, specifically designed for web crawling the food hall market. These aren't your run-of-the-mill LLM bots; they're equipped with advanced algorithms that enable them to navigate complex website structures intelligently. By chaining a team of agents together for a specific aspect of market research (data entry, navigation, synthesis, retrieval, etc) these agents can achieve impeccable levels of online ability, ensuring a thorough report on each food hall.

To tackle the challenge of web scraping at scale, we've also employed multithreading techniques. We enable multiple agents to operate concurrently on a single report, drastically reducing the time required. It's akin to having several skilled data miners working simultaneously, each extracting valuable insights without stepping on each other's toes. As soon as new data is available, the user is notified immediately.

Once the data is collected, the real magic begins. We use a sophisticated data pipeline to process and analyze the information, preparing it for integration. This is where Convex comes into play. Our choice of Convex as the database solution is strategic; it excels in handling real-time data updates. By leveraging Convex's capabilities, we ensure that the data not only feeds into our database seamlessly but also triggers updates to our React-based frontend in real time. This ensures that users get to see the latest trends in the food hall real estate market without any delay.

The front end, crafted with React, is designed to be as dynamic as the data it displays. It's built to handle real-time updates efficiently, providing a smooth and responsive user experience. This level of interactivity is crucial for engaging users and offering them up-to-the-minute insights.

Challenges we ran into

Building our do-it-all web crawler was no walk in the park. We hit every bump in the road, from sneaky edge cases that tried to block our data scraping to the tricky task of making our crawler smart enough to think on its feet—or wheels? Anyway, we gave it some cool moves like website-hopping or doing a graceful exit when things got tough. Throwing Convex into the mix for our real-time database was like adding a puzzle to our puzzle, especially with all those threads running around. And let's not even start on the front end – we're backend heroes, so you can imagine the adventures (and misadventures) we had there. Lots of "Oops" and "Aha" moments, but hey, we made it through with a bunch of laughs and a whole lot of learning.

Accomplishments that we're proud of

We built the first-ever self-growing food hall database in the country, hands-off from researchers! Tough project, and lots of late nights, but we finished strong and proud.

What we learned

Oh man, this hackathon was a wild ride. We thought our project was going to be pretty straightforward, but boy, were we wrong. Ended up pulling an all-nighter trying to fix this one bug that just wouldn't go away. It was like every time we thought we got it, something else popped up. Learned a lot though, especially about how important it is to be ready for anything and that sometimes, things are way harder than they look. Plus, we got to see how teamwork kicks in when you're all equally confused and tired but still determined to figure it out. Definitely a memorable experience, but next time, I'm bringing more snacks and maybe a pillow.

What's next for Broker AI

Next up for Broker AI? Big plans! We're turning this real estate game-changer into a commercial powerhouse. After getting thumbs-up from some big-shot VPs, we're all in on making research a breeze—not just for properties, but for anything. Want the lowdown on medical breakthroughs? Or which fish tank is trending? Our super scraper's got you. We're here to track all the trends, making sure you're always in the know, no matter the topic. Stay tuned; it's going to be exciting!