-
-
Bank impersonation: learned SMS score and independent warning cues remain separate.
-
Inspect actual hostnames and concrete credential, urgency and payment cues.
-
Model card: held-out historical SMS metrics, reproducibility and explicit limitations.
-
Practice verification decisions using fictional scenarios and explanations.
Inspiration
A convincing message is not proof of identity. Students and families need a way to pause, understand a suspicious request, and verify it without handing private conversations to another cloud service. A single confident-looking AI score can make this problem worse if people mistake it for a safety guarantee.
What it does
SignalCheck is a browser-local scam-triage prototype for AI + Cybersecurity. Paste an SMS, email, or chat transcript. It separates two independent lenses:
- Learned patterns: a trained TF–IDF + logistic regression model scores resemblance to historical SMS spam.
- Contextual evidence: explicit rules quote credential requests, pressure, payment demands, secrecy, identity claims, unexpected rewards, and instructions aimed at an AI checker.
The URL inspector displays actual hostnames and compares them with an independently known domain—without opening a link. Results end with a concrete out-of-band verification checklist. Users can inspect model feature contributions, try a counterfactual phrase removal, download a summary without the original message, and practice on three fictional scenarios. A low score is never labelled “safe.”
How it was built
The dependency-free HTML/CSS/JavaScript app runs inference entirely on the visitor's device. Python and scikit-learn train the model; exported coefficients and IDF weights are used directly in JavaScript. There is no LLM API, backend database, analytics, account requirement, or paid dependency. Message text is not sent over the network.
The public UCI SMS Spam Collection was normalized and deduplicated before a stratified 70/10/20 train/validation/test split. Vocabulary and IDF are fitted only on training data. The threshold is selected only on validation. The deployed model is the same model evaluated on the untouched test split.
What was measured
On 1,026 held-out historical SMS messages (126 spam): 99.13% spam precision, 90.48% recall, 94.61% F1, with 1 false positive and 12 false negatives. These are not modern fraud-detection accuracy figures. Contextual rules are not covered by those metrics; near-duplicate templates may remain, and there is no prospective user study.
Twelve automated checks cover JavaScript/Python inference agreement on 30 held-out texts, deceptive subdomains, URL userinfo, internationalized hostnames, Chinese cues, embedded instructions, invalid input, and the rule that benign results never become a safety clearance. The main workflows were also exercised in the browser.
Challenges and lessons
The central challenge was avoiding false confidence. In the bank-impersonation demo, the learned SMS score is only 49/100, yet a password request, urgency, and a domain mismatch justify pausing. The explanation also reveals ordinary learned associations: removing “your” lowers the score to 29/100. That illustrates model sensitivity, not causal proof or a way to make the original message legitimate. Keeping these signals separate is more honest and more useful than blending them into an unexplained “fraud probability.”
Accomplishments
A working, zero-API-cost prototype with local inference, visible evidence, safe link inspection, practical verification steps, reproducible evaluation, and an educational practice flow. The application and its supporting artifacts were created during the hackathon on October 10, 2026.
Limitations and next steps
The English SMS model cannot authenticate senders, inspect attachments, detect deepfake voices, establish domain reputation, or reliably recognize every modern scam. Chinese risk cues are rules, not a Chinese ML model. Quoted security advice can trigger warnings. Future work is a consented contemporary scam evaluation, grouping near-duplicate templates, calibration under distribution shift, multilingual evaluation, and testing whether users make better verification decisions.
Attribution and AI assistance
Dataset: Almeida & Hidalgo (2011), UCI SMS Spam Collection, DOI 10.24432/C5CC84, CC BY 4.0. Original code is MIT licensed; dataset-derived assets retain attribution. OpenAI Codex was used extensively to design, implement, test, and document this project. That assistance is separate from the app's own trained ML component. The 3:51 demo uses actual application captures and synthetic narration. Demonstration messages are fictional and use reserved .example / .invalid domains.
Source and reproducibility instructions: https://github.com/RuoyunHe/signalcheck-forgehacks-2026 Live demo: https://signalcheck-forgehacks-ruoyun.bushy-larch-7162.chatgpt.site
Log in or sign up for Devpost to join the conversation.