The CREW

Scammers don't need to hack your account - they convince you to open the vault. THE CREW detects manipulation and steps in before the money leaves.

Comment

Inspiration

Most fraud systems are designed to answer one question: "Is this really the account holder?"

But social engineering creates a different problem. A victim may be using their own device, their own password, and their own bank account — and still be manipulated into sending money to a scammer.

Imagine a grandmother receiving a call from someone claiming to be her grandson. He says he has been arrested, needs $2,500 for bail immediately, and begs her not to tell his parents. Nothing has been hacked. From the bank's perspective, she may look like a completely legitimate customer making a legitimate payment.

The criminal didn't break into the vault. They convinced the person holding the key to open it.

That idea inspired THE CREW — an AI-powered social engineering defense system designed to recognize when the human, rather than the account, is being compromised and intervene before the money leaves.

We built it around one question:

"Should this account holder really be making this payment?"

What it does

Most fraud systems are designed to answer one question: "Is this really the account holder?"

But social engineering creates a different problem. A victim may be using their own device, their own password, and their own bank account — and still be manipulated into sending money to a scammer.

Imagine a grandmother receiving a call from someone claiming to be her grandson. He says he has been arrested, needs $2,500 for bail immediately, and begs her not to tell his parents. Nothing has been hacked. From the bank's perspective, she may look like a completely legitimate customer making a legitimate payment.

The criminal didn't break into the vault. They convinced the person holding the key to open it.

That idea inspired THE CREW — an AI-powered social engineering defense system designed to recognize when the human, rather than the account, is being compromised and intervene before the money leaves.

We built it around one question:

"Should this account holder really be making this payment?"

How we built it

THE CREW uses React, TypeScript, Vite, Tailwind CSS, and Framer Motion for the frontend, with a FastAPI backend.

Google Gemini provides semantic understanding. It analyzes conversations and returns structured manipulation signals, evidence, confidence, claimed identity, and requested actions. However, Gemini does not make the final safety decision.

Our deterministic risk engine filters and combines those signals, applies confidence thresholds and evidence rules, deploys the appropriate crew members, and determines whether intervention conditions have actually been met.

Our philosophy is:

AI interprets. THE CREW decides.

For voice, we integrated ElevenLabs for speech-to-text and text-to-speech. LIVE CALL sends microphone audio through ElevenLabs Scribe, then feeds the resulting speaker-aware conversation into the analysis pipeline. ElevenLabs also gives our Case File and intervention experience a voice.

We use Tiger Data / TimescaleDB as an Operational Evidence Ledger. It records risk changes, agent deployments, interventions, milestones, and short evidence snippets so an operation can be reconstructed without storing full conversations or raw audio.

The production application runs on Vultr Cloud Compute behind Nginx and HTTPS at thecrew.work.

Challenges we ran into

One of our biggest challenges was realizing that detecting suspicious language wasn't enough — who said it matters.

For example, if the victim asks:

"Why do you need $2,500?"

the phrase "$2,500" should not become evidence that the victim is requesting money.

We built speaker-aware LIVE CALL logic so victim speech is retained as conversational context but does not directly create scam signals, increase risk, or deploy scam agents. Only caller-originated evidence can contribute to those decisions.

Another challenge was deciding how much authority to give AI. Letting an LLM directly decide whether someone's payment should be stopped would make the system difficult to reason about and potentially inconsistent. We instead separated semantic interpretation from deterministic decision-making.

We also had to design for failure. Voice services, AI APIs, and databases can fail. The core safety logic therefore does not depend on the Evidence Ledger being available, and we built fallback behavior for parts of the voice experience.

Finally, taking the project from a local prototype to a real HTTPS production deployment required configuring the frontend build, FastAPI service, Nginx proxying, environment secrets, and microphone-compatible HTTPS deployment.

Accomplishments that we're proud of

We're proud that THE CREW became much more than a scripted hackathon demo.

We built a working end-to-end system with:

  • Real Gemini analysis of previously unseen conversations
  • Real microphone transcription through ElevenLabs
  • Speaker-aware two-person LIVE CALL analysis
  • A deterministic risk and intervention engine
  • Dynamic deployment of specialized agents
  • Trusted-contact verification
  • A privacy-conscious Tiger Data Evidence Ledger
  • A complete Case File demonstrating detection through intervention
  • A production deployment available at thecrew.work

We're especially proud of the boundary between AI and safety-critical logic. Gemini is extremely useful for understanding manipulation, but the final decision remains explainable and deterministic.

What we learned

The biggest lesson was that social-engineering defense isn't just a classification problem.

Context matters. Speaker identity matters. Timing matters. A single suspicious sentence may mean very little, while several independent signals accumulating throughout a conversation can tell a very different story.

We also learned that good safety systems need to understand uncertainty. THE CREW's risk score is not presented as a magical probability that a conversation is a scam. It represents accumulated evidence used to determine how much protective friction is justified.

Most importantly, we learned that AI is a tool, but empathy is the strategy. Someone under pressure or afraid for a family member doesn't need a frightening red warning. They need a system that slows the moment down, explains what it noticed, and gives them a safe way to verify what is happening.

What's next for The CREW

Our prototype currently uses explicit caller/user identification in LIVE CALL because we wanted to prevent incorrect speaker attribution from becoming safety evidence.

A future version could move toward continuous passive protection with automatic speaker diarization and stable role identification, allowing someone to put a suspicious call on speaker while THE CREW follows the conversation naturally.

We would also explore integrations directly at the payment decision point with banks, wallets, and payment providers, where behavioral and transaction context could strengthen the existing conversational evidence.

The long-term vision is not to replace the user's judgment or automatically block every unusual payment.

It is to give people something scammers deliberately try to take away from them: time, context, and a trusted second opinion before the money leaves.

Built With

Share this project:

Updates

Submission history