Inspiration

Criminals steal hotel staff logins, usually with infostealer malware, open the hotel's account on the booking platform, and message real guests through the platform's own chat: "your card failed, re-verify within 24 hours or the booking is cancelled", followed by a link. The guest has every reason to trust it. It's the right hotel, the right dates and the real app. I wanted something that stops those messages before a guest pays, without blocking the hotels that send thousands of normal ones every day.

What it does

StayGuard decides on every message: deliver, deliver with a warning, hold, or block. Each decision comes with reason codes and quotes from the message.

  • Booking-aware rules. On a pay-at-property booking nothing is due before arrival, so any payment demand is a red flag. It also catches card-detail requests, off-platform payment (bank transfer, UPI, Pix, crypto, gift cards and many local payment apps), moves to WhatsApp or phone, and text aimed at the AI reviewer.
  • Link checker. Phishing feeds (Phishing.Database, OpenPhish), brand look-alikes, free site builders, open redirectors, and links hidden inside other links.
  • A small language model (TF-IDF + logistic regression) for scam wording the rules don't name.
  • Two LLM judges for unclear cases. A prosecutor and a defender must each quote the message. Quotes that aren't really in it are thrown out, and the judges can never release something a rule held.
  • Account-takeover response. Every extranet session gets a risk score. One click logs the attacker out, rotates guest PINs and warns every guest the session touched. Guests can check any message with their booking PIN.
  • Fails closed. Any internal error holds the message. Personal data is masked before anything is stored or sent to an LLM, and the audit log is hash-chained.

How I built it

Python, FastAPI and SQLite, with scikit-learn for the models and real public data: a hotel booking demand dataset for the bookings, the UCI SMS Spam Collection, two phishing feeds and a top-1M domain list. There are 670 automated tests (96% coverage), with lint and strict type checks in CI. A decision takes 2.72 ms at the median and 5.09 ms at p95 with the offline judge.

Challenges I ran into

The hardest number was the one I didn't want to see. On the messages I built it on, StayGuard caught 60 of 60 scams and held no real hotel. On its first blind test it caught 44 of 66, while its own classifier on its own caught 57 (paired McNemar test, p = 0.00024). My rules had been tuned to my own test set.

So I changed how I test, not just the code:

  • Every new test set is written by a separate author who never sees the code. It is hash-locked and run once.
  • The comparison is fixed before the run, and the run refuses to start from an uncommitted tree.
  • Four rounds of adversarial review (an engineer, a scammer and a skeptical judge) found 34, 25, 23 and 25 confirmed issues. The loop never converged, and the write-up says so.

Accomplishments that I'm proud of

  • On blind set v2, StayGuard caught 66 of 74 scams and held 10 of 74 legitimate hotel messages. Its classifier alone caught the same scams but held 18 legitimate ones (p = 0.03857). That gap is what the rules are for.
  • On blind set v3 (312 messages, 20 languages), I tested my own upgrade (a multilingual encoder, booking context and judge routing) against the current system. The upgrade caught 123 scams to the current system's 129 and held 23 legitimate messages to its 22, so it didn't ship. The test did its job.
  • Every number uses written test messages and an offline stand-in for the LLM judges, and the write-up says so next to each result.

What I learned

  • A score on data you built against can climb while real protection falls. Only a set you have never seen tells you which.
  • Most false holds came from the rules, not the model. The fix for those is product design, like routing payments through the platform, more than machine learning.
  • "This didn't work, p = 0.29" is more useful than a bigger number nobody can check.

What's next

  • Shadow mode on real platform traffic with a labelled feedback queue (built, waiting for data).
  • Real LLM judges measured on blind data (a Gemini run is in progress).
  • Network link checks for domain age and shortener destinations (built, needs real links to measure).
  • A test set written by people outside the project.
Share this project:

Updates