🧠 NeuroSync — AI-Powered Stress Intelligence

NeuroSync is an iOS application that fuses real-time physiological data from Apple HealthKit with AI-driven social sentiment analysis to give you a holistic picture of your stress levels. It doesn't just track how your body feels — it also monitors your digital communications (Discord, email) to understand what's stressing you out and takes action when you need it most.

📱 What it does

NeuroSync is split into four major layers that work together:

1. Physiological Stress Monitoring (iOS + HealthKit)

The iOS app reads seven health metrics from Apple HealthKit:

Heart Rate — resting and current
Heart Rate Variability (HRV) — the gold standard for stress detection
Sleep Hours — how much rest you actually got
Step Count — daily activity baseline
Exercise Minutes — active energy expenditure
Mindful Minutes — meditation / breath work
Respiratory Rate — breathing patterns

These metrics are sent to NVIDIA's Nemotron-3 Ultra 550B model (via the NIM API) for analysis. The LLM returns a structured assessment:

Stress Level (low / moderate / high)
Confidence Score
Clinical Reasoning (why it thinks what it thinks)
Actionable Suggestion (what you can do about it)

If stress levels are high, NeuroSync automatically creates an iOS Reminder with the AI's suggestion — so you get a push notification like "Try a 5-minute breathing exercise: inhale for 4 seconds, hold for 4, exhale for 6."

2. Social Sentiment Monitoring (Server + OpenClaw)

The backend server runs an OpenClaw orchestrator that polls Discord and Gmail every 15 seconds, feeding every message through an LLM stress analyser. It detects:

Urgent messages and crisis events in real-time
Sentiment trends across channels and senders
Volume anomalies — sudden spikes in message traffic
Cross-channel stress aggregation — is everyone stressed or just one channel?

The iOS app polls this server and displays a real-time social sentiment dashboard alongside your health data. If the server detects a social crisis, the app creates a reminder and correlates social stress with physiological stress (Health: high / Social: critical).

3. AI-Powered Breathing Exercises with 🎙️ Siri Integration {#siri-integration}

When NeuroSync detects moderate or high stress (from either health data or social sentiment), it doesn't just warn you — it actively intervenes with a personalized, LLM-generated guided breathing exercise. Here's how it works:

Voice-Guided Breathing (LLM-Generated)

The NVIDIA NIM API generates a custom StressExercisePlan with:

Personalized timing — inhale/hold/exhale ratios tuned to your current heart rate and HRV
Thematic instructions — wave visualizations for social stress, mountain grounding for health stress
Voice prompts for each phase, spoken aloud by the built-in VoiceGuidanceService (AVSpeechSynthesizer)

Each plan includes:

name — e.g., "Ocean Breaths" or "The Wave Reset"
phases — an array of inhale / hold / exhale / rest phases with per-phase durations, written instructions, and voice prompts
spokenSummary — a natural-language description Siri reads aloud (e.g., "Your heart rate is elevated at 92. I've designed a 3-minute exercise called 'Ocean Breaths' for you.")

Fallback plan: If the LLM is unavailable (no API key, network failure), the app falls back to a hardcoded 4-7-8 breathing plan — so the exercise always works.

The Full-Screen Breathing Experience

The BreathingExerciseView provides a full-screen, distraction-free environment:

Element	Description
Animated breathing circle	Expands on inhale, contracts on exhale — smooth, color-shifting transitions
Phase instructions	Large text showing what to do ("Breathe in slowly through your nose")
Countdown timer	Large numeric display of seconds remaining in each phase
Progress bar	Shows overall exercise progress (e.g., "23s / 120s")
Pause / Resume / Skip	Full state machine: `idle → active → paused → completed`
Auto-dismiss	On completion, shows a checkmark + "Great job!" and auto-closes after 3 seconds

🎙️ Siri Integration (iOS 17+ App Intents)

NeuroSync registers App Shortcuts via the AppShortcutsProvider protocol, allowing users to trigger guided breathing exercises hands-free with Siri. The following phrases are automatically registered when the app is installed:

Phrase	Action
"Hey Siri, start a NeuroSync stress exercise"	Opens app & begins breathing exercise
"Hey Siri, start a breathing exercise with NeuroSync"	Opens app & begins breathing exercise
"Hey Siri, begin a stress relief exercise on NeuroSync"	Opens app & begins breathing exercise
"Hey Siri, help me de-stress with NeuroSync"	Opens app & begins breathing exercise
"Hey Siri, do a breathing exercise with NeuroSync"	Opens app & begins breathing exercise

How the Siri flow works:

Siri voice command triggers StressExerciseIntent (an AppIntent with openAppWhenRun = true)
The Intent itself runs the full HealthKit → NIM pipeline inside the app's own process:
- Fetches live health metrics via HealthKitService
- Sends them to NVIDIA NIM for stress analysis + exercise generation
- If the LLM generates a plan, it's encoded to JSON and saved to UserDefaults under last_exercise_plan
- If health data is unavailable or stress is low, the fallback 4-7-8 breathing plan is persisted instead
The Intent returns a dialog response (e.g., "I've created a personalized exercise for you") and the app opens
The App entry point (NeuroSyncv2App.swift) detects the persisted plan via checkForPendingExercise() — called on onAppear and every time UIApplication.didBecomeActiveNotification fires (covering cold launch, foreground-from-background, and Siri dismissal)
checkForPendingExercise() reads the plan from UserDefaults, clears it, and sets dashboardVM.showExerciseSheet = true
The BreathingExerciseView opens as a full-screen cover with voice guidance

Background Task Integration

The background task (BGTaskScheduler) also generates exercise plans autonomously:

In the background, the system fetches metrics and runs stress analysis via NIM
If stress is high, it generates an exercise plan and saves it to UserDefaults (last_exercise_plan)
When the user opens the app, checkForPendingExercise() detects the pre-generated plan and presents it immediately — no loading, no waiting

VoiceGuidanceService

The VoiceGuidanceService wraps AVSpeechSynthesizer with a calming voice configuration:

Speech rate: 0.45 (slower than default for calmness)
Pitch multiplier: 1.1 (slightly warmer tone)
Audio session: .playback mode with .spokenAudio and .duckOthers — plays even when the device is in silent mode
Pause / Resume / Stop — full voice lifecycle management tied to the exercise state machine

4. Simulation & Demo Mode

Because not everyone has a Discord bot token and a Gmail OAuth setup handy, NeuroSync includes a standalone simulator server that generates 1000+ realistic messages across multiple stress scenarios:

🟢 normal_day — moderate traffic, occasional spikes
🔴 incident_escalation — starts calm, builds to crisis, resolves
🟢 weekend_calm — very low traffic, positive vibes
🔴 crisis_spike — sudden massive burst, then decline
🟡 gradual_buildup — slow burn over time
🔴 rollercoaster — alternating high/low stress
🔴 sustained_pressure — consistent high volume & stress

You can switch scenarios live via API to demo different stress patterns.

🏗️ How we built it

Frontend: iOS (SwiftUI)

Component	Technology
UI Framework	SwiftUI with NavigationStack, TabView
State Management	`@MainActor` ViewModels, `@Published`, `@EnvironmentObject`
Health Data	`HealthKit` (HKHealthStore, HKObserverQuery)
Reminders	`EventKit` (EKReminder, EKAlarm)
Secure Storage	iOS Keychain (API key storage)
Background Tasks	`BGTaskScheduler` for periodic stress checks
Auto-refresh	Task-based polling loop (5 min health, 30s social)
Siri Integration	`AppIntents` framework — `AppShortcutsProvider`, `AppIntent` with `openAppWhenRun`
Voice Guidance	`AVSpeechSynthesizer` + `AVAudioSession` (plays in silent mode)
Breathing Exercise Engine	State machine (`StressExerciseManager`) with timer-driven phase transitions

The iOS app has four tabs:

Dashboard — stress indicator ring, AI insights card, live health metrics grid
Sentiment — social sentiment overview, channel breakdowns, message feed
History — searchable, filterable stress event log with detail view
Settings — NVIDIA API key management, permissions, data management

Backend: Python + OpenClaw + FastAPI

Component	Technology
Orchestrator	Async Python with `asyncio`
Message Polling	OpenClaw CLI + SDK (Discord, Gmail)
LLM Analysis	OpenClaw agent → any LLM (GPT-4o-mini, Claude, etc.)
Fallback Engine	Keyword-based heuristic analyser
Storage	RollingBuffer + JSONL persistence
REST API	FastAPI with Pydantic models
Simulation	Standalone FastAPI server with scenario engine
HTTP Poller	`httpx` AsyncClient for simulated polling

AI Stack

Model	Provider	Use Case
Nemotron-3 550B (NVIDIA NIM)	NVIDIA API	Physiological stress analysis (iOS → cloud)
GPT-4o-mini (or configurable)	OpenClaw Gateway	Social message sentiment analysis (server)

Architecture Diagram

┌─────────────────────────────────────────────────────────────┐
│                     iOS App (SwiftUI)                        │
│  ┌──────────────┐  ┌─────────────────┐  ┌───────────────┐  │
│  │   HealthKit   │  │  SocialMonitor   │  │   Reminders   │  │
│  │  (HR, HRV,   │  │  (polls server   │  │  (EventKit)   │  │
│  │   sleep, …)  │  │   every 30s)     │  │   auto-create │  │
│  └──────┬───────┘  └────────┬────────┘  └───────┬───────┘  │
│         │                   │                    │          │
│         ▼                   ▼                    │          │
│  ┌──────────────────────────────────────────────┐│          │
│  │        NVIDIA NIM (Nemotron-3 550B)          ││          │
│  │   "Given HR=88, HRV=24, sleep=5.2h →        ││          │
│  │    stress=moderate, confidence=0.78"         ││          │
│  └──────────────────────────────────────────────┘│          │
└──────────────────────┬───────────────────────────┘          │
                       │ HTTP (REST)                          │
                       ▼                                      │
┌─────────────────────────────────────────────────────────────────┐
│             NeuroSync Server (FastAPI + OpenClaw)               │
│                                                                 │
│  ┌──────────────────┐   ┌──────────────┐   ┌────────────────┐ │
│  │   Discord Poller  │   │  Gmail Poller│   │ Simulated      │ │
│  │   (every 15s)    │   │  (every 15s)  │   │ HTTP Poller*   │ │
│  └────────┬─────────┘   └──────┬───────┘   └───────┬────────┘ │
│           └─────────┬──────────┘         ┌─────────┘           │
│                     ▼                     ▼                    │
│           ┌──────────────────────────────────────┐             │
│           │          MessageStore                 │            │
│           │  (Rolling buffer + JSONL persistence) │            │
│           └──────────────────┬───────────────────┘             │
│                              ▼                                 │
│           ┌──────────────────────────────────────┐             │
│           │          StressAnalyser               │            │
│           │   (LLM agent OR keyword fallback)     │            │
│           └──────────────────┬───────────────────┘             │
│                              ▼                                 │
│           ┌──────────────────────────────────────┐             │
│           │      Aggregated Summary              │            │
│           │  → REST API → iOS App               │            │
│           └──────────────────────────────────────┘             │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │  OpenClaw Gateway (ws://127.0.0.1:18789)                │   │
│  │  Manages channels + routes LLM inference                │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │  Simulator Server (standalone, port 8081)                │   │
│  │  • 7 stress scenarios                                   │   │
│  │  • 1000+ realistic messages per seed                     │   │
│  │  • Live scenario switching via API                       │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

*Simulated pollers used when NEUROSYNC_SIM_MODE=true

🚧 Challenges we ran into

1. HealthKit authorization is a multi-step dance. You can't just request access and assume it works. The first time a user opens the app, HealthKit prompts might not fire properly, and background HKObserverQuery callbacks require careful lifecycle management. We added a polling loop that waits up to 15 seconds for authorization before fetching data.

2. Getting reliable HRV data from HealthKit. HRV (Heart Rate Variability) is arguably the most important stress metric, but it's also the most sparse. Not all Apple Watch models record SDNN consistently, and some users may never have HRV data. We made all metrics optional and only run analysis when at least 3 metrics have real values.

3. LLM output parsing without the hallucinations. Getting the Nemotron model to return valid JSON every single time without markdown fences or stray characters required careful prompt engineering AND a defensive parser that strips ```json blocks, handles whitespace, and validates the output before treating it as a real analysis.

4. The OpenClaw gateway is a moving target. During development, the OpenClaw CLI's agent mode kept changing. We built a dual-path analyser: if the LLM agent is available, use it; if not, fall back to a keyword-based engine that scans for crisis signals, stress keywords, and sentiment markers. This made the system resilient to gateway issues.

5. Cross-device networking during the hackathon. The iOS app needs to reach the server running on a MacBook. We hardcoded a local IP (172.18.92.67) but this breaks as soon as you switch networks. The launch_demo.sh script now auto-detects the local IP and prints it for easy configuration.

6. iOS Reminders permissions changed in iOS 17. Apple introduced requestFullAccessToReminders() vs requestWriteOnlyAccessToReminders(), and the old requestAccess(to:) is deprecated. We had to add availability checks for both paths.

📚 What we learned

On the AI side:

Prompt engineering for structured JSON output is an art. You have to be absurdly specific ("Respond ONLY with valid JSON in this exact format — no markdown, no code fences") and even then, you need defensive parsing.
Temperature matters. We used temperature: 0.3 for stress analysis because you want consistent, evidence-based results — not creative interpretations.
Nemotron-3 550B is remarkably capable at medical-style reasoning with structured outputs.

On the iOS side:

SwiftUI's @MainActor ViewModels with Combine publishers and async/await can feel clean when done right. The DashboardViewModel coordinating HealthKit, the NIM API, EventKit, and persistence all from one place was a design pattern that worked well.
Background tasks in iOS are finicky. BGTaskScheduler doesn't guarantee execution timing, and the expiration handler must be handled with extreme care (we used a lock to prevent double-completion).

On the backend side:

asyncio.gather with return_exceptions=True is the right way to poll multiple channels concurrently without one failure bringing down the whole cycle.
A generic keyword fallback analyser is surprisingly effective. It doesn't match the nuance of an LLM, but it catches 80% of real crises with simple pattern matching.
Building a message generator with template banks and filler words produces far more realistic test data than random string generation.

🔮 What's next for NeuroSync

More health metrics: Add blood oxygen (SpO2), blood pressure (where available), and glucose data for richer physiological context
More social channels: Slack, Teams, WhatsApp — the OpenClaw SDK theoretically supports them
On-device AI: Run a distilled model (e.g., via CoreML or Apple's MLX) directly on the iPhone for privacy-preserving stress analysis without sending health data to the cloud
Watch app: An Apple Watch companion that shows live stress levels and haptic feedback when the system detects a spike
Trend analysis: Show stress patterns over days/weeks/months — "Your stress is highest on Tuesday afternoons"
Personalized interventions: Learn which suggestions work best for each user and tailor the AI's advice over time
Shared dashboards: Optionally share anonymized stress data with therapists or coaches
Calendar integration: Correlate stress spikes with calendar events — "Your stress peaks during the weekly engineering standup"

🛠️ Quick Start

iOS App

Open NeuroSyncv2.xcodeproj in Xcode
Update AppConfig.serverBaseURL in Models/AppConfig.swift to your Mac's local IP
Build and run on a physical iPhone (HealthKit doesn't work on simulator)
Get an NVIDIA API key from build.nvidia.com and enter it in Settings

Server (for social sentiment)

cd server

# Install dependencies
pip install -r requirements.txt

# Launch the full demo (simulator server + main server)
bash launch_demo.sh

# Or manually:
python3 simulator_server.py --port 8081 --scenario normal_day
python3 -m uvicorn main:app --host 0.0.0.0 --port 8080

Available Demo Scenarios

# Change scenario live:
curl -X POST "http://localhost:8081/sim/scenario?scenario=crisis_spike"
curl -X POST "http://localhost:8081/sim/scenario?scenario=weekend_calm"
curl -X POST "http://localhost:8081/sim/scenario?scenario=incident_escalation"

Server API Endpoints

Method	Path	Description
`GET`	`/health`	Server health check
`GET`	`/social/dashboard`	Full social sentiment dashboard (what the iOS app polls)
`GET`	`/social/messages`	Recent messages with sentiment
`GET`	`/social/urgent`	Urgent/crisis messages
`POST`	`/social/health-correlation`	Receive health stress from iOS
`POST`	`/social/create-reminder`	Queue a social stress reminder
`GET`	`/social/pending-reminders`	Poll pending reminders for iOS
`GET`	`/analysis/latest`	Latest aggregated stress analysis
`GET`	`/analysis/history`	Historical analyses
`GET`	`/sim/status`	Simulator status and scenario info
`GET`	`/sim/scenarios`	List all available scenarios

📁 Project Structure

NeuroSyncv2/
├── NeuroSyncv2/                  # iOS App
│   ├── NeuroSyncv2App.swift      # App entry point + Siri & background task exercise plan detection
│   ├── ContentView.swift         # Tab navigation & root view
│   ├── Models/
│   │   ├── HealthMetrics.swift   # Health data model
│   │   ├── StressResult.swift    # LLM analysis result
│   │   ├── StressEvent.swift     # Stored stress check
│   │   ├── StressExercise.swift  # Exercise plan & phase models
│   │   ├── SocialMessage.swift   # Social message model
│   │   └── AppConfig.swift       # Global configuration
│   ├── Intents/                   # 🎙️ Siri / App Intents
│   │   ├── NeuroSyncShortcuts.swift  # AppShortcutsProvider (5 Siri phrases)
│   │   └── StressExerciseIntent.swift # AppIntent (openAppWhenRun)
│   ├── Managers/                  # 🧘 Exercise engine
│   │   └── StressExerciseManager.swift  # State machine + timer + voice orchestration
│   ├── ViewModels/
│   │   ├── DashboardViewModel.swift
│   │   ├── SocialSentimentViewModel.swift
│   │   ├── StressHistoryViewModel.swift
│   │   └── SettingsViewModel.swift
│   ├── Views/
│   │   ├── DashboardView.swift
│   │   ├── BreathingExerciseView.swift   # Full-screen animated breathing UI
│   │   ├── StressIndicatorView.swift     # Custom ring gauge
│   │   ├── LlmSuggestionView.swift       # AI insights card
│   │   ├── MetricCardView.swift
│   │   ├── SocialSentimentView.swift
│   │   ├── MessageRowView.swift
│   │   ├── ChannelBreakdownCard.swift
│   │   ├── StressHistoryView.swift
│   │   └── SettingsView.swift
│   └── Services/
│       ├── HealthKitService.swift       # Apple Health integration
│       ├── NIMService.swift             # NVIDIA NIM API client + exercise generation
│       ├── SocialSentimentService.swift # Server API client
│       ├── VoiceGuidanceService.swift   # AVSpeechSynthesizer (calming voice)
│       ├── EventKitService.swift        # Reminders integration
│       ├── BackgroundTaskService.swift  # BGTaskScheduler + exercise plan storage
│       └── KeychainHelper.swift         # Secure API key storage
├── server/                       # Python Backend
│   ├── main.py                   # FastAPI bridge server
│   ├── run_orchestrator.py       # CLI orchestrator runner
│   ├── simulator_server.py       # Standalone simulator
│   ├── launch_demo.sh            # One-command demo launcher
│   ├── orchestrator.yaml         # Example config
│   ├── requirements.txt          # Python dependencies
│   ├── orchestrator/
│   │   ├── orchestrator.py       # Main poll-analyse loop
│   │   ├── poller.py             # Discord + Gmail pollers
│   │   ├── simulated_http_poller.py  # HTTP simulator client
│   │   ├── analyser.py           # LLM + fallback analysis
│   │   ├── storage.py            # Rolling buffer persistence
│   │   ├── social_api.py         # Social sentiment REST API
│   │   └── config.py             # All configuration
│   └── simulator/
│       ├── generator.py          # Realistic message templates
│       ├── poller.py             # Scenario-based burst patterns
│       └── api.py                # Sim control endpoints
├── Info.plist                    # App permissions & capabilities
└── .gitignore

🧪 Tech Stack Summary

Layer	Technology
Mobile	SwiftUI, HealthKit, EventKit, Keychain, BGTaskScheduler
🎙️ Siri / App Intents	`AppIntents` framework — `AppShortcutsProvider`, `AppIntent` with `openAppWhenRun`
🗣️ Voice Guidance	`AVSpeechSynthesizer` + `AVAudioSession` (plays in silent mode)
🧘 Exercise Engine	`Timer`-driven state machine with pause/resume/skip lifecycle
AI (Physiological)	NVIDIA NIM — Nemotron-3 Ultra 550B
AI (Social)	OpenClaw + GPT-4o-mini (configurable)
Backend	Python, FastAPI, asyncio, httpx
Message Platform	OpenClaw SDK / Gateway
Persistence	Rolling buffers, JSONL, UserDefaults
Simulation	Custom scenario engine with 1000+ message pools

Made with 🧠, too much coffee, and an Apple Watch that definitely knows when I'm stressed.

Built With

bash
nvidia
ollama
openclaw
python
swift