NeuroSketch

30 seconds to screen. Clinical-grade Parkinson's screening, powered by a pen stroke and processed entirely in your browser. Zero data ever stored.

Comment

NeuroSketch

Inspiration

NeuroSketch was born from personal experience. Two of our team members watched Parkinson's disease slowly take hold within their families, where early symptoms went unnoticed until the condition had already progressed significantly. These experiences exposed two critical gaps in the healthcare system: the difficulty of catching motor symptoms early, and the complete lack of immediate, accessible guidance once symptoms begin to appear. We built NeuroSketch to close both of those gaps.

What It Does

NeuroSketch brings validated neurological screening methods directly into a standard web browser, eliminating the barriers of high costs, long wait times, and limited access to specialist appointments. The platform is designed as a proactive early indicator, not a replacement for clinical diagnosis. By providing actionable, data-driven results, we help users identify subtle motor irregularities before they become clinically obvious, giving them the clarity to know when it is time to consult a medical professional.

Under the hood, the detection engine utilizes a dual-model weighted ensemble. A Convolutional Neural Network (CNN) built on a MobileNetV2 backbone analyzes drawing geometry, while an XGBoost model tracks movement kinematics like velocity, acceleration, and jerk to detect subtle tremors that are completely invisible to the naked eye. To ensure full transparency, NeuroSketch generates Grad-CAM (Gradient-weighted Class Activation Mapping) heatmaps that highlight the exact regions where the AI detected risk. Results are categorized into five human-friendly risk tiers ranging from Low to High, offering clear next steps for both families and doctors.

How We Built It

NeuroSketch is architected as a high-performance web application that turns a simple drawing into actionable clinical insights.

Frontend

The frontend is built with Next.js 14 and TypeScript, with a focus on an immersive, technically transparent user experience.

  • UI/UX: Implemented using Tailwind CSS and Shadcn UI for a clean, professional aesthetic
  • 3D Engine: Features a custom 3D neural mesh with a first-person navigable camera, built entirely with CSS 3D transforms (zero WebGL overhead), allowing users to visually explore the model's internal architecture
  • Animations: Utilizes Framer Motion 12 for cinematic UI transitions running at 60fps, including proximity-based fact reveals via the Intersection Observer API, spring-physics risk badges, and staggered canvas entrance animations
  • Privacy: Architected for zero-retention. All data is processed in-memory and results are delivered without any permanent storage. Drawings are never uploaded, saved, or transmitted beyond the inference pipeline

Backend

The backend serves as the real-time inference engine, built with FastAPI for high-concurrency performance and TensorFlow for model execution.

The backend serves as the real-time inference engine, built with FastAPI for high-concurrency performance and TensorFlow for model execution.

Architecture: The CNN uses a MobileNetV2 backbone pre-trained on ImageNet (1.4M images, 1000 classes) for efficient feature extraction via depthwise separable convolutions (9× fewer parameters than ResNet50, approximately 3.5M vs 25M). Custom dense layers are stacked on top: GlobalAveragePooling2D → BatchNormalization → Dense(128, ReLU) + Dropout(0.5) → Dense(64, ReLU) + Dropout(0.3) → Dense(1, Sigmoid)

Training Strategy: We used a two-phase training approach. Phase 1 freezes the ImageNet backbone and trains only the custom classification head for 30 epochs at a learning rate of 5 × 10⁻⁴. Phase 2 unfreezes the top 40 layers of MobileNetV2 and fine-tunes at a 25× smaller learning rate (2 × 10⁻⁵) to prevent catastrophic forgetting of pre-trained features

Data Augmentation: We expanded our original dataset of 204 labeled images from Kaggle's Parkinson's Drawings dataset to over 1,000 augmented images (∼ 3.5× expansion) using domain-specific augmentation strategies. Spirals receive 360° rotation (rotationally invariant), while waves receive ± 15° rotation with horizontal flips to preserve tremor polarity. No shear transforms are applied, as they would corrupt tremor frequency signatures

Ensemble Logic: Implements a weighted voting ensemble (0.4 × Pspiral + 0.6 × Pwave) to maximize diagnostic sensitivity. Wave drawings receive higher weight because they exhibit more pronounced tremor patterns in PD patients

Explainability: Grad-CAM targets the out_relu layer (final ReLU activation of MobileNetV2), backpropagating gradients from the sigmoid output to produce spatial heatmaps that highlight tremor artifacts, spiral irregularities, and stroke hesitations

Inference Pipeline: Canvas drawings are captured as Base64-encoded PNGs, decoded to RGB arrays, resized to 224 × 224, and normalized to [0, 1]. An automatic color inversion check (if mean pixel intensity < 0.5) handles dark backgrounds. Both models run in parallel, Grad-CAM heatmaps are generated, and the full result payload is returned as JSON in approximately 2 seconds on CPU

Deployment

  • Backend: Dockerized container deployed on Railway
  • Frontend: Deployed on Vercel with Next.js 14 and global CDN distribution
  • CORS: Restricted to the frontend domain only (no wildcard *)

Challenges We Ran Into

One of our biggest challenges was the size of our dataset. With only 204 original labeled images, our models were heavily prone to overfitting, learning patterns that did not generalize to unseen drawings. To overcome this, we engineered domain-specific augmentation pipelines that respected the clinical properties of each drawing type (rotational invariance for spirals, orientation sensitivity for waves). We also collected additional samples from different sources to ensure the model could handle a wider variety of real-world drawing conditions. Balancing the two-phase training strategy was another significant challenge: too aggressive of a learning rate in Phase 2 caused catastrophic forgetting of the ImageNet features, while too conservative of a rate resulted in underfitting on the medical domain.

Accomplishments That We're Proud Of

We are incredibly proud to have taken a personal pain point and turned it into a functional solution that addresses a real crisis in healthcare accessibility. Successfully combining high-level machine learning (transfer learning, ensemble methods, Grad-CAM explainability) with a human-centered interactive interface that makes complex neurological screening accessible to anyone with a browser is something we are genuinely proud of. Seeing the system go from raw canvas data to a live, actionable risk assessment complete with heatmap explanations and confidence scoring is our greatest achievement.

On the model performance side, our Spiral CNN achieved 86.7% accuracy with an AUC of 0.9511, and our Wave CNN reached 91.3% accuracy with an AUC of 0.9627. For models trained on an augmented dataset originating from just 204 labeled images, hitting AUC scores above 0.95 proved that transfer learning combined with domain-specific augmentation can genuinely compete with models trained on datasets orders of magnitude larger.

Most importantly, we are proud that this project honors our grandparents by helping ensure that future families have the early warning and clarity they need to take control of their health.

What We Learned

We realized that a diagnosis does not always have to start with a massive, expensive machine. Through our research into neurology and motor assessment literature, we discovered that the way a person draws a spiral is not just a picture. It is a data-rich map of their neurological health. By capturing tiny tremors in speed, pressure, and geometry that the human eye misses, software can unlock insights previously reserved for complex medical imaging.

We also learned the importance of domain-aware data engineering. Generic image augmentation would have destroyed the very biomarkers we were trying to detect. Understanding that shear transforms corrupt tremor frequency and vertical flips invert wave polarity forced us to think like clinicians, not just engineers.

What's Next for NeuroSketch

Our journey does not end with this prototype. We are committed to evolving NeuroSketch into an industry-standard screening tool. Our primary goal is to fine-tune our models by partnering with clinical institutions to access larger, more diverse datasets, which will significantly increase accuracy across various demographics and disease stages. We also plan to expand our digitized screening suite to include other motor tests such as voice analysis and gait tracking, providing a more holistic neurological health profile. On the infrastructure side, we aim to optimize inference with TensorFlow Lite for on-device execution, making the platform viable in low-bandwidth environments and on older hardware. Above all, we want to fulfill our mission: bringing early detection to every household, regardless of location or resources.

Built With

Share this project:

Updates