PetWhisper

• 

Inspiration Over 85 million households in the United States care for companion animals, yet non-verbal communication barriers lead to unrecognized distress, untreated separation anxiety, and preventable medical escalations. Most "smart pet" apps suffer from a fundamental design flaw: they demand constant manual input, notification management, and screen interaction from already busy pet parents.

We asked: What if an AI companion could operate autonomously in the background—perceiving acoustic and postural cues in real time, delivering immediate bio-acoustic de-escalation, coordinating with veterinary clinics, and only interrupting the human when a true decision is required?

This inspired PetWhisperer: an agentic, cross-species guardian built on the Strands Agents SDK, Gemini multimodal vision, and ambient bio-acoustic synthesis.

What It Does PetWhisperer operates as an autonomous, end-to-end taskmaster and background agent across three distinct tracks:

Autonomous 5-Stage Taskmaster Pipeline:

Stage 1 (Passive Telemetry Stream): Continuously samples ambient decibel levels (detecting sudden spikes above 85 dB) and camera frames without recording private data.

Stage 2 (Cognitive Ethology Triage): Employs the Canine Facial Action Coding System (DogFACS) and audio FFT spectrogram classification (Separation Distress, Territorial Alarm, Pain/Lethargy, Play) to isolate trigger vectors and compute a real-time Cortisol Index.

Stage 3 (Autonomous Intervention): Employs a browser-native Web Audio API synthesizer that dynamically generates calibrated 432 Hz and 528 Hz Solfeggio soothing acoustic waves, de-escalating distress within seconds.

Stage 4 (Enterprise Telemetry Sync): Syncs structured time-series metrics (heart rate variability, de-escalation latency, acoustic decibels) to high-speed analytics storage.

Stage 5 (On-Chain Verification & Tokenized Rewards): Computes a SHA-256 state hash and anchors tamper-evident proofs to Solana Devnet, rewarding the household with $TREATS utility tokens.

Strands Multi-Track Agent Mesh:

Everyday Agents (Track 1): Runs quietly at >95% autonomy, silencing low-frequency appliance hums, adjusting circadian smart lights (5000K to 2700K warm), and ordering food supplies automatically.

Professional Agents (Track 2): Listens during clinical visits to synthesize structured, exportable veterinary SOAP notes (Subjective, Objective, Assessment, Plan) with drug interaction audits for Idexx and Cornerstone EHRs.

Good Neighbor Agents (Track 3): Coordinates decentralized neighborhood lost-pet search grids, shelter emergency foster matching, and municipal microchip intake relays.

Strict vb_call Escalation Governance:

The agent handles repetitive micro-decisions silently. When human intervention is required (e.g., medical symptoms, expenses exceeding thresholds), it issues a concise, 60-word voice-actionable card (Decision, Situation, Numbered Choices, and Stakes) designed for eyes-free resolution.

How We Built It Agent Orchestration: Strands Agents SDK architecture paired with AWS AgentCore design patterns for decoupled task loops, persistent memory vectors, and autonomous background execution.

Multimodal Intelligence: Google Gemini 3.7 Flash and Gemini 2.5 Flash for sub-second vision inference, DogFACS action unit extraction (ear pinna posture, commissure elongation, sclera visibility), and real-time ethology reasoning.

Audio & Signal Processing: Native Web Audio API and Fast Fourier Transform (FFT) spectrogram analyzers running client-side for immediate bio-acoustic harmonic generation (432 Hz alpha resonance, 528 Hz restorative sine waves).

Backend & Full-Stack Core: Node.js/Express with TypeScript and Vite, featuring proxy endpoints for Gemini vision inference, ethology routing, and prompt guardrails.

Enterprise Storage & Guardrails: Cloud-backed persistent state, Model Armor safety filters for emergency toxicology detection, and cryptographic SHA-256 verification.

Challenges We Ran Into Cross-Species Multi-Modal Latency: Canine distress signals require rapid intervention before physiological escalation occurs. We addressed this by running local FFT feature extraction on the client to trigger immediate acoustic dampening while asynchronous Gemini vision inference classifies the root cause in parallel.

Minimizing User Interruption (The 95% Quiet Rule): Balancing high autonomy with safety required rigorous calibration. We implemented an escalation threshold matrix based on the vb_call policy: actions involving minor acoustic soothing or sensor adjustments execute silently, while actions involving medication dosage or expenses require explicit numbered voice choices.

Client-Side Synthesis Without Heavy Assets: Rather than streaming large pre-recorded audio files, we generated all therapeutic tones dynamically using Web Audio oscillator nodes and gain modulation envelopes.

Accomplishments That We're Proud Of True Hands-Free Background Autonomy: An intelligent system that performs end-to-end work in the background rather than acting as another passive dashboard.

Zero-Latency Bio-Acoustic Feedback Loop: Synthesizing targeted soothing harmonic waves within four seconds of an ambient noise spike or vocalization trigger.

Seamless Three-Track Strands Architecture: A unified agent architecture supporting everyday home comfort, clinical veterinary workflow automation, and neighborhood-wide lost-pet community networks.

Eyes-Free Voice Protocol: Clear 60-word decision cards that work seamlessly on mobile, audio devices, or smart displays.

What We Learned Ethology-Informed AI Design: Subtle canine facial indicators (such as lateral ear flattening or orbital tightening) are more predictive of early anxiety than vocalizations alone.

Agent Calibration: High-trust autonomous agents must be conservative with interruptions. Background logs and silent execution metrics build greater user confidence than frequent push notifications.

Modular Agent Composition: Separating real-time sensory perception from deliberate clinical reasoning allows small, focused models to handle immediate reactions while larger models handle multi-modal diagnostics.

What's Next for PetWhisperer Wearable IoT Biosensor Integration: Ingesting continuous PPG optical heart rate and accelerometer data directly from smart collars.

Cross-Species Model Fine-Tuning: Expanding the ethology engine from canines to felines, equines, and companion birds.

Direct EHR Clinic Connectors: Automated two-way HL7/FHIR integrations with veterinary clinic software for longitudinal health charting.

Edge Deployment: Compiling the audio FFT and DogFACS vision models for on-device edge execution on low-power smart home hubs and smart pet cameras.

Built With

• all