Inspiration

Looking for stuff on government websites often means searching for specific and detailed articles or documents that have updated information about the city and laws. But there are two problems:

1) using a typical search engine means results are cluttered behind irrelevant and out-of-date web pages

2) the search bar on sf.gov is limited, using only basic keywords for search.

What it does

You can search all articles and documents on SF.gov. It even works with more complicated search queries, for example, "sf police commission car chase resolution"

How we built it and overcame challenges

Building a web scraper that went to sf.gov and continuously clicked all the links it could find, would not have found all the links. For example, there is not link for the Police Commission on the Departments page. We devised a clever way to use sf.gov's keyword search option to find the hidden pages and scraped those.

With all our articles and pdfs stored in Supabase, we used a BERT-based text embedding language model called jina-embeddings-v2-base-en to make document comparisons. Our client is built with Sveltekit and hosted on Cloudflare.

What's next for Better SF.gov Search

To make information accessible to everyone, we will continue using our web scraping and database technology to organize the unorganized parts of internet, such as legal documents and research papers. Instead of basic keyword search, we will enable people to search based on document context and LLM-based understanding of information.