PENN

Inspiration

On my way home from work, I got stuck in traffic for over an hour because of a race I had no idea was happening. Around the same time, the streets I usually park on were stripped for repaving and marked for construction, so I had to hunt for another spot. Google Maps and Apple Maps warned me about neither.

Both of these were planned well in advance, and the information was public. It was just split across sources that don't talk to each other and not actively informed:

Source	What it tells you	What it leaves out
City street-permit layer	Which streets have permits, and the dates	Hours, and any permit that has already expired
City press releases	Race and parade closures	Everything else; they're the only place these are announced
Ticketmaster and Visit Philadelphia	What's happening, and where	The streets around the venue

None of them keeps history, so "does this race close the same streets every year?" has no answer anywhere.

I built PENN to bring these sources into one map, with an agent that answers questions about the streets. A wrong answer here sends someone into a parade route, so I designed PENN so that code resolves every place and date, and checks every citation before an answer reaches the user.

What it does

PENN shows street activity across Philadelphia on one map, and adds Ask PENN, an agent that answers questions about it. Ask PENN runs on NVIDIA Nemotron 3 Super through Nebius Token Factory, answers only from PENN's records, and cites each one.

"Can I drive to Old City this weekend?" The answer names what changes the drive, counts the minor permits, and highlights every cited street on the map.

Reading the map

The map shows road closures from construction, block parties and citywide events. It also shows ticketed events at venues, because drop-offs and rideshare pickups can back up the streets around them.

What you see	What it means
Color	The reason: orange for construction, purple for an event closure, green for a block party
Line weight	The extent: thick for a full closure, thin for one lane, dots for sidewalk work
Fading	The timing: solid lines are today, fainter lines are coming up
Flags	Things to go to. An event listed by both Ticketmaster and Visit Philadelphia gets one flag, not two
Top bar	Switches the map between the next 24 hours, 7 days and 30 days

What an answer looks like

Every answer has the same shape, because someone deciding whether to drive cares about a few things:

1. One opening line: is getting there normal or not?
2. Happening there: concerts, games, festivals and other listings.
3. Getting around: the closures that actually change a drive, including ongoing work.
4. One count of everything minor: lane work, loading zones and sidewalk work, with the details left on the map.

Each numbered chip is a citation. Click one, and the exact record opens beside the answer: the City notice, the permit, or the event page. The cited streets light up in yellow.

Rittenhouse Square this weekend: clicking citation 2 opens the City's closure notice for 18th Street.

What you can ask

Ask	What PENN does
Can I drive to Old City this weekend?	A trip brief for a neighborhood
Spruce and 15th next week	Finds the intersection on the City's street centerline
What's happening around the Art Museum?	Finds the landmark, then the closures and events around it
When is the Rocky Run, and what closes?	Finds the event by name, reads its past editions, estimates the next date and maps the route
Does the Mummers Parade close Broad Street every year?	Shows the years PENN has records for, and says it can't vouch for the others
What about the week after?	Keeps the place from the last answer and changes only the time
Is Camden closed tomorrow?	Says it covers Philadelphia only

If PENN has no record of an event, it searches the web with Tavily, and each result becomes a record it can cite.

"When is the Rocky Run?" names an event, not a place. PENN reads the 2024 and 2025 City notices, estimates Saturday, Nov 14, 2026, and draws the route along Kelly Drive.

Under each answer, a Hallucination guard line shows how many citations matched records PENN retrieved, and confirms that the place and dates were worked out in code.

How I built it

City permits, City closure notices and event listings flow into one ClickHouse store. Every draft Nemotron writes passes checks in code before the cited answer reaches the map. Tavily searches the web only when PENN has no record.

The stack

Layer	What I used
Model	NVIDIA Nemotron 3 Super (120B total, 12B active) through Nebius Token Factory's OpenAI-compatible API
Web search	Tavily (owned by Nebius), cached, used only when PENN has no record
Agent and API	Python, FastAPI, the standard openai client, Pydantic
Database	ClickHouse (ReplacingMergeTree, H3 bloom-filter index)
Geo	Uber H3 at resolution 9, the City's street centerline, 112 landmarks, neighborhood outlines
Frontend	Next.js 16, React 19, MapLibre GL 6, OpenFreeMap tiles
Hosting	One Docker image on AWS EC2, HTTPS through Caddy and Let's Encrypt, Amazon S3 for the hand-saved data

1. The data: four sources in one store

Source	What it adds	How it gets in
City Street Lane Closures (OpenDataPhilly, ArcGIS)	4,389 current street permits	Loaded every morning. A GitHub Action has saved a daily snapshot since Oct 5, because the layer drops a permit the day it expires
City closure notices (phila.gov)	2,097 race, parade and ride closures from 2016 to 2026, 1,481 of them placed on the map	Pages saved by hand. A Python parser reads each closure list and matches every street to the City centerline. Lines it can't read are logged, never guessed
Ticketmaster and Visit Philadelphia	648 event listings	Ticketmaster loads every morning. Visit Philadelphia's local picks are saved by hand each week
My event series catalog	38 named events, with aliases like "Rocky Run" and "the Rocky race"	Each edition is backed by a source URL

Every record is placed on H3 cells, each about a block group in size, and matched to the City's street segment IDs. That way, a search at a corner catches the right blocks.

2. One turn of Ask PENN

One turn, following "Can I drive to Old City this weekend?" Nemotron plans and writes. Code resolves places and dates, runs every search, and checks every draft.

1. Context in. Nemotron receives the question, the last three turns, the current time, and the place and time window from the previous answer.
2. Code resolves the place and time. The model calls resolve_place("Old City"). Code turns the name into hundreds of H3 cells, keeps them on the server, and returns a short handle, p1. resolve_window("this weekend") returns w1, labeled "the weekend of Oct 17". Code computes the dates and weekdays. A handle the tools never issued is an error, so the model can only search places and dates that code resolved.
3. Search, then sort. search_events(p1, w1) returns the records, and code sorts them into three tiers: events, getting around, and minor (counted only). Closures that belong together, like one City notice on one day, fold into one group.
4. Rules arrive with the data. Writing rules for permits, events and web pages are attached to the first tool result of that kind, not stored in the system prompt. This cut the system prompt in half.
5. Draft, then check in code. A draft goes back to Nemotron for one rewrite if it cites a record no tool returned, states facts without a citation, leaves out a permit count, says "no record" before trying the web, or was cut off. If the rewrite fails too, PENN says it can't answer.
6. The API adds the rest: fixed notes for ambiguous questions, the same dates in past years, and the street highlights on the map.

The six tools

Tool	What it does
resolve_place	Finds a neighborhood, a corner ("12th and Walnut"), an address, or one of 112 landmarks ("the Art Museum")
resolve_window	Turns "this weekend", "next Tuesday" or "Nov 14" into exact dates
search_events	Returns the records for a place and time window, sorted into tiers
find_event	Finds an event by name across the city: past editions, the estimated next date, and a route summary
get_location_history	Returns every year PENN has records for at that place
search_web	Searches with Tavily and turns each result into a citable web: record

3. How I used NVIDIA Nemotron and Nebius

• Every Ask PENN turn runs on Nemotron 3 Super through Nebius Token Factory, at temperature 0, with up to 8 tool steps.
• Switching models took one line. Token Factory's endpoint works with the standard openai client, tool calls included. So I ran three Nemotron models on the same agent code and the same questions before choosing.
• I measured, then shipped Super. On Oct 28, I ran all three on the current agent, averaged over four runs:

Model	Questions passed (of 52)	Pass rate
Nemotron 3 Super	41	79%
Nemotron 3 Ultra	38.25	74%
Nemotron 3.5 Lightning	22	42%

Super was also faster than Ultra (7.6 s against 8.0 s at the median) and wrote less than half the output tokens.

4. An eval scored by code

PENN is graded by code, not by another LLM. The set has 58 questions:

Question type	Count	Example
Lookups: specific, multi-day, place	16	What street permits are in effect on the blocks at 13th and Locust on Saturday, October 10, 2026?
Negatives ("no record" is the right answer)	7	Is there a block party near 13th and Locust on Saturday, October 10, 2026?
Street history	8	Was Broad Street closed at Snyder Avenue for the Broad Street Run this year?
Traps	21	Is Spruce Street closed to cars at 1327 Spruce Street? A permit is in effect, but that isn't proof the street is closed
Trip briefs	6	Rittenhouse next week

A question passes only if:

1. the agent called the right tools,
2. every required fact appears,
3. every cited ID was really returned by a tool, and
4. no forbidden claim appears.

Every run uses a frozen copy of the city from Oct 17, 2026 at 10:00 a.m. That way, an answer never fails because a permit was filed after the question was written. Every saved answer keeps its full trace, so anyone can re-score my runs with no API key and no database. The six trip briefs are left out of the model comparison for now, because their answer key predates an Oct 21 change in how minor permits are counted.

5. A public demo that's safe to share

The live demo is one Docker image with ClickHouse, both APIs (the live city and the frozen demo city), the map, and a daily refresh. Because a Devpost link is public, it has guardrails:

• per-visitor rate limits,
• a separate allowance for judges,
• an answer cache, and
• token and time limits on each question.

Challenges I ran into

The data

• No dataset of past closures existed. The City's open data covers current permits, not past ones, so I had to build the history myself. Visit Philadelphia's site terms bar scraping, and phila.gov started blocking scripts on Oct 21. So I saved the pages by hand, wrote a Python script to pull the events out of them, and checked the results against what I knew. PENN never contacts either site.
• It was my first time with geospatial data. A closure isn't a single point. It's a shape on the map, made of a sequence of coordinates. It took me a while to understand how GeoJSON works and how to process it.
• The City's data wasn't clean. Some construction records had locations that were off by a margin, so they landed in the wrong place or the wrong neighborhood. The map in production went through several rounds of cleaning. Even the event sources disagree: Ticketmaster and Visit Philadelphia place the same arena about 400 m apart.
• The same event came from more than one source. Overlapping records cluttered the map, so I merged duplicates and combined overlapping points. When sources disagreed, I kept the most official one:

When these disagree	This wins	Why
Approved permit vs. City notice	The permit	It's what the City actually approved
Ticketmaster event page vs. an announcement	The Ticketmaster page	It's the official listing

The model

• Nemotron Super's 【】 citation habit. I ask for citations as [id:…]. Super often writes 【id:…】 instead, even with a tiny prompt. At first, only 52% of its citations came back in the requested format. A prompt rule made it rarer but never stopped it. So I handled it in layers: the prompt names the formats to avoid, a lenient parser reads the 【】 variants, and a check sends back any draft that cites an ID no tool returned. Now 79% of its citations arrive in the requested format, and 99% after parsing.
• Prompt rules leak. The model followed must-do rules only some of the time. Asking it to flag ambiguous questions worked on 0 of 3 eval questions across three runs. So I moved every must-do into code: fixed sentences for ambiguous questions, the "permits list dates, not hours" line, and the permit counts. If a sentence must always appear, code adds it.
• The model couldn't carry a neighborhood. My first version let the model pass places and dates directly. A neighborhood is hundreds of H3 cells, far too many to copy between tool calls, and the model also got weekdays wrong. Short handles (p1, w1) with dates computed in code fixed both.
• Answers depend on time. The same question means something different today than next month. I had to pass the current time into every turn, and make sure the citations matched the dates the user asked about. One eval question even started failing because a new permit was filed after I wrote it. That's why the eval and the demo run on a frozen copy of the city.

The product

• One user question exposed a missing tool. On Oct 14, I asked "When is the Rocky Run, and where are the road closures?" PENN asked them to drop a pin, because the question names an event, not a place. Yet the answer was already in my data: two editions, both on the second Saturday of November, starting at the Art Museum steps and running up Kelly Drive. So I built find_event and added 112 landmarks to the place finder.
• 89 correct bullet points nobody could use. "Rittenhouse next week" returned 89 permits as a list. All of it was correct, and none of it helped anyone decide whether to drive. So code now decides what changes a drive: an announced closure, a full closure on a collector street, or a short lane closure on an arterial. Everything else becomes a count.

Hosting

• I planned to host on Nebius AI Cloud, but it had no CPU-only VM option, which I found out while requesting quota. So the same Docker image runs on AWS EC2, and inference stays on Nebius Token Factory.

A check that catches the model

In a saved eval run from Oct 28, question sp04 asks "Is there construction on the Ben Franklin Parkway near Logan Square on Wednesday, October 21, 2026?" Super's first draft found the right permit, but cited it as event_id:lane_closures:2026-17342, an ID no tool returned that turn. Code didn't guess what the model meant. It sent the draft back with the reason, "cited a record PENN never retrieved". The rewrite cited the real record, lane_closures:2026-17342, and passed. The same thing happened in all three Super runs that day, and the check caught it each time. Across the Oct 28 runs, about a quarter of Super's answers were sent back like this before anyone would have read them.

A real catch from a saved eval run: the first draft cited an ID no tool returned, so code sent it back once with the reason.

Fixing the agent in code beat a bigger model

• Before my fixes, Nemotron 3 Ultra passed more questions than Super (30.25 against 25.0 on average), but took 33.4 s at the 90th percentile against Super's 7.3 s.
• After I moved the wording rules into code, Super passed more than Ultra (79% against 74%) and stayed faster.
• On trap questions, Super's pass rate went from 52% to 74% across the full set.
• On the 12 questions that best separate the models, prompt fixes took Super from 6/12 to 10/12, and moving the last two rules into code took it to 12/12. I made those fixes by studying those same 12 questions, so the full-set trap number above is the fairer measure.

What else is done

• It's live. PENN runs end to end in production, not just as a local demo. - Judges can use judge_code.txt in the attachment for further judging of the application.
• A decade of closure history that didn't exist before, rebuilt from City notices from 2016 to 2026, with every row citing its source.
• All of Philadelphia on one map, with permits, City notices and ticketed events together, readable without a key.
• Cited answers. Every fact links to the record it came from, so a user can check it.
• An eval anyone can re-check: 58 questions scored by code, and saved answers that re-score with no API key and no database.
• Careful wording. PENN says "a permit is in effect", never "the street is closed". It says when a date is an estimate. Tavily receives a short search query, never the user's message.

What I learned

• LLM costs depend on how the agent uses the model, not just the price per token. Running three Nemotron models through Token Factory showed me that the smallest model isn't automatically the cheapest. Lightning made almost twice as many model calls per answer as Super (5.3 against 2.8) and used about 7 times the prompt tokens (148k against 21k). Now I compare tokens per answer, not just the price per token, when I choose a model.
• Scoping is part of the work. I found more event sources than I could use: Eventbrite, Luma, venue calendars, the City's special events calendar and Center City District. Some had closed or paid APIs, and others would have taken longer to clean than I had. Fewer sources that I could clean and verify turned out more useful than many messy ones.
• Geospatial data is shapes, not points. A closure is a line along a street, made of many coordinates. Once I understood GeoJSON, I could build the history of past closures. That history is also what route planning needs: which streets a recurring event closed in past years.
• Know the model's habits, and design around them. Nemotron Super was good at finding the right records. But it wrote citations in the wrong format, followed must-do prompt rules only some of the time, and slipped on exact wording like "the street is closed." Prompting reduced these problems but never removed them. Checks in code did.
• Check the infrastructure early. I found out during the deadline week that Nebius AI Cloud had no CPU-only VMs. Because I had built the app as one Docker image, I could move it to AWS without changing any code.
• A public demo needs cost controls. Anyone with the Devpost link can ask questions, and every question costs inference. Rate limits per visitor, a separate allowance for judges through judge code shared by judge_code.txt, an answer cache, and token limits per question keep one link from running up the bill.
• Color has to carry one meaning. On a map this dense, each visual property can answer only one question. Color shows the reason (construction, event or block party), line weight shows how much of the street is affected, and fading shows how soon it starts. Once each property had one job, the map became readable without a key.

What's next for PENN

Data

• An automatic ingestion pipeline. Right now, many past records are saved by hand. I want PENN to pull from more sources automatically and keep the dataset up to date. This would require getting permissions on automated data scraping from government pages and city tourism web pages, which at the moment have no response in sharing the data.
• Events that don't need a permit. Rallies, markets and gatherings at places like City Hall and Dilworth Park never show up in permit data, so I want to add event calendars that cover them. Neighborhoods have more local calendars so potentially a good way to add additional data.
• Tighter neighborhood edges. A "Fairmount" search can spill about one block past the boundary. The fix is designed: keep a permit only if one of its blocks lies inside the outline, with about 20 m of slack for boundary streets.

The agent

• Close the trap-question gap. Super passes 74% of trap questions, against 85–100% in the other categories. Each wording rule that moves from the prompt into code closes more of it.
• Street history is next. It's the next weakest category, at 66% for both Super and Ultra.
• Update the trip-brief answer key to the new permit counts, and put trip briefs back in the comparison.
• Help models cite street records. The cleaned records could help fine-tune models like Nemotron to cite street information more accurately.

The product

• Alerts, like "tell me if anything closes near my block or the stadium this weekend", built on the same checked records.
• A phone app. PENN only runs as a web app today. A phone app could connect to your car over Bluetooth and show what's ahead on your route automatically.
• Route planning. It would show how much time an upcoming closure adds to a trip, and suggest a way around it. I left this out of the hackathon to focus on getting the data right within Philadelphia.
• More cities. The same approach could be adapted to other large cities with a permit layer and closure notices, for both drivers and people on foot. PENN isn't only for avoiding closures: it also helps visitors find events and plan their time, so it could be useful for tourism too.

Built With

• amazon-ec2
• amazon-web-services
• caddy
• clickhouse
• docker
• esri
• fastapi
• github-actions
• maplibre-gl
• nebius-token-factory
• next.js
• node.js
• nvidia-nemotron
• openai-python
• openfreemap
• openmaptiles
• openstreetmap
• python
• react
• tavily
• ticketmaster-api
• typescript
• uber-h3
• uvicorn