A Python-only search engine that indexes both live and historical content from Planet Python, the official blog aggregator for the global Python community.
I wanted to try a simpler one to start and decided to go with a Python-specific search engine to make the scope small and interesting (as a Python Dev).
What it does
It is a search engine like Google, but only has Python content indexed.
Python + Google = Pyoogle
It indexes content from featured sites present in the Planet Python feed (https://planetpython.org/titles_only.html). Planet Python is an aggregator that lists high-quality Python blogs. The Pugle worker crawls pages present in the Planet Python feed and uses the Common Crawl dataset to fetch historical posts from listed Python sites.
How we built it
This was built over 3 weeks. I started by brainstorming with Codex on the tech stack and asked Codex to implement everything step by step.
The entire code was written by Codex. Initial implementation was done by GPT 5.5. I moved to 5.6 Sol once it was released.
I also used "/goal" so that Codex can watch the live crawl and historical post ingestion jobs and optimize those when needed.
Challenges we ran into
Planet Python lists URLs of XML feeds. For example, URLs like dev.to/example_user1.xml were included in the feed. The initial implementation fetches all the historical content from the subdomain, so dev.to was included in the crawl list along with blogging sites like blogspot.com. This led to a large number of wasted crawls. This also resulted in delays since we used delays to prevent draining the resources of a single server.
A total of 50k pages had to be cleared (codex wrote commands for that in Pyoogle CLI). I later moved to an allowlist-based implementation and automated allowlisting / blacklisting using dedicated Codex threads.
Accomplishments that we're proud of
Good Python-focused results for test searches
7.6 GB of search artifacts stored in Cloudflare R2
50k indexed pages (crawl ongoing)
What we learned
Building a search engine is fun.
Having proper visibility/observability is really helpful. The admin side stats page helped me identify the blogging domains crawl issue I mentioned in the challenges section.
What's next for Pyoogle
Improving search: Codex already added Manticore as a dedicated search database. I will test it out, and hopefully we get better matches with less delay.
Improve AI overviews: Current AI overviews are expensive. I am thinking of the possibility of connecting to local models (via LM Studio or Ollama) running on users' devices for free AI overviews.
Log in or sign up for Devpost to join the conversation.