Callismatic

An AI phone and personal secretary that listens to voicemails you would otherwise block, auto-handles or blocks scams via real callbacks, and surfaces only what truly needs you.

Comment

PROJECT NAME Callismatic

TAGLINE A personal and phone secretary AI agent that listens to the voicemails you'd never check, decides what needs you, and quietly handles or blocks the rest.

ELEVATOR PITCH (200 char limit) An AI phone and personal secretary that listens to voicemails you would otherwise block, auto-handles or blocks scams via real callbacks, and surfaces only what truly needs you.

STORY

Inspiration

Scam and spam calls get bad enough that people and small businesses block all unknown numbers outright. That also blocks the client, the delivery driver, the interviewer, and the doctor's office — because those are unknown numbers too until someone actually listens. 80–90% of what gets silently blocked leaves a voicemail nobody ever checks. So instead of a chatbot you have to remember to ask, we built an agent that listens to every voicemail for you and only speaks up — or calls back — when it actually matters.

What it does

Callismatic is a personal and phone secretary AI agent — not just a filter. It triages a folder of voicemail recordings (or an incoming SMS/WhatsApp message) one at a time, then carries that same judgment into the follow-up work a human assistant would normally do. For each message it:

  • Transcribes it with real AssemblyAI speech-to-text.
  • Reasons about it with tools: get_today (to judge staleness), check_past_decisions (so a caller already handled isn't re-flagged), check_number_intel (a transcript-content scam-script scan — deliberately not a Truecaller-style lookup, since no public API for that exists for a third-party project), and check_corrections (a human override always wins over the agent's own judgment for that caller).
  • Returns a forced structured decision: caller category, summary, key facts, whether a human needs to decide, whether it's safe to auto-handle with a callback (with the exact task for the calling agent), and whether the number should be blocked.
  • Acts on that decision deterministically: blocks confirmed scam/spam numbers, places a real callback via CALL-E for simple/automatable requests, or surfaces genuinely important calls to the human with full context — then turns the outcome into meeting notes, a to-do, a synced Google Sheets CRM row, and Google Calendar availability/booking, with reminders delivered back over WhatsApp.

How we built it

Python, the Strands Agents SDK for the agent loop and tool-calling, running on Amazon Bedrock and Amazon Nova as model providers, AssemblyAI for voicemail transcription, the CALL-E SDK for placing real outbound callbacks, and Pydantic for the structured-output schema Strands enforces on every response. Around that core: a Google Sheets CRM sync and Google Calendar availability/booking (both via a shared service-account credential), a Twilio-backed SMS/verify number for a second, independent carrier-intelligence signal, and a WhatsApp Business Cloud API receiver/sender running the same triage pipeline on incoming messages.

Challenges we ran into

Three real ones, not hypotheticals:

  1. Honest scam detection. It would have been easy to fake a Truecaller-style lookup, but Truecaller has no public developer API for third parties. We built a real, defensible signal instead — scanning the transcript itself for concrete scam-script markers, plus an optional second signal from Twilio Lookup's carrier/line-type data — rather than a black-box "trust us" verdict.
  2. Bedrock model access has two separate gates, not one. Getting Claude working on Bedrock needed both the Anthropic use-case form and a valid AWS Marketplace payment method — we hit AccessDeniedException: INVALID_PAYMENT_INSTRUMENT even after the use-case form was approved, and fell back to Amazon Nova (no Marketplace subscription required) to keep testing while that gets resolved on the account side.
  3. "International" doesn't mean "works everywhere." A real CALL-E call to an Indian number genuinely dialed (confirmed by a real provider_call_id and attempt log) but never rang — India is served through CALL-E's "International" tier, and local carrier-level filtering of unrecognized international caller IDs is a known, real phenomenon. That finding shaped our call_router.py design: a per-country provider registry so a regional partner can be plugged in later without touching calling code.

Accomplishments that we're proud of

A real, verified end-to-end run: AssemblyAI transcribed an actual audio file, Strands/Bedrock correctly classified four distinct voicemail types (scam, lead, appointment confirmation, important client matter), and a real CALL-E call was genuinely placed to a real phone number — not a mocked demo, an actual dial attempt with a provider-confirmed call ID.

What's next

Since this draft: a hosted dashboard + API deployed live on AWS Lambda (awpfsufk4dofdncv6cgqsaifwy0kvolm.lambda-url.us-east-1.on.aws), a credential-free demo of real triage output on Hugging Face Spaces, a Google Sheets CRM sync verified against a real spreadsheet, Google Calendar availability/booking verified against a real calendar, and a WhatsApp receiver/sender verified directly against Meta's Cloud API — though Meta actually auto-forwarding real incoming messages to it still needs Business Verification (a separate identity/document review process), not yet done. A Twilio Lookup carrier-intelligence signal is wired and tested but currently blocked by Twilio's own trial-account quota. Still ahead: real inbound telephony (Twilio recording webhooks) instead of a local sample folder, and a folder-watcher/background-service mode instead of a batch run.

Roadmap: AMD + Kubernetes

A Kubernetes-orchestrated fine-tuning pipeline on AMD GPU infrastructure is planned next, to train a proprietary, sandboxed model on Callismatic's own accumulated triage decisions and human corrections — turning the Personal Secretary & Phone Manager algorithm from a prompted agent into a purpose-trained, horizontally scalable one. Design is written up in docs/AMD_ACT3_ARCHITECTURE.md (https://github.com/axxess-triaxis/callismatic/blob/master/docs/AMD_ACT3_ARCHITECTURE.md); as of this writing no AMD compute has been provisioned and no training has run.

BUILT WITH python, strands-agents, amazon-bedrock, amazon-nova, assemblyai, calle, pydantic, boto3, google-sheets-api, google-calendar-api, twilio, whatsapp-business-cloud-api, aws-lambda, docker, gradio, huggingface-spaces

TRACK Professional Agents

AGENTS FOR HUMANS — additional fields Try it out (repo): https://github.com/axxess-triaxis/callismatic License: MIT AWS Builder ID: [email protected]

Built With

Share this project:

Updates

Submission history