Autism Shinigami Eyes
A wearable assistant that reads the facial expression of the person you are talking to, tells you what it is, and advises you on how to respond. Expression recognition runs entirely on-device, so the camera feed never leaves the hardware.
Inspiration
One member of our team is autistic and has difficulty discerning facial expressions and more broadly interpreting social cues. This project came directly from his personal struggle. In a conversation, a missed or misread expression can change how the interaction goes, and there is rarely a moment to stop and work out what someone's face is saying.
We wanted to build something that does that work in real time: a second read of the expression in front of you, paired with practical guidance on what to do about it.
We had covered the fundamentals of machine learning in our courses, but this was our first time applying them to a practical problem outside the classroom. Rather than reaching straight for an existing solution, we designed our own model first to see how an architecture behaves on real data, and then moved to a proven architecture once we understood the trade-offs.
What it does
Autism Shinigami Eyes takes a camera feed from a pair of glasses (a prototype today, with Meta glasses as the planned target) and produces real-time social guidance:
Glasses camera frames
-> on-device expression classifier (9 classes)
-> expression label + confidence, tracked over time
-> fine-tuned language model
-> what the expression is + what would be socially advisable to do
- Expression recognition. A ResNet18 classifier runs on an Arduino Uno Q and labels each frame with one of nine expressions (angry, confused, disgust, fear, happy, neutral, sad, shy, surprise) and a confidence score.
- Expressions over time. The system does not judge each frame in isolation. It tracks how the person's expression changes across successive readings and factors that change into its advice. A face moving from neutral to frustrated calls for a different response than one that has been frustrated throughout.
- Social advice. A custom fine-tuned Thinking Machines model (trained through Tinker) receives the classifier's output and returns plain-language guidance: how the person likely feels, what you might notice, and what you can do. Advice is worded with appropriate hedging, because the underlying prediction is a probability, not a certainty.
- Privacy. The neural network classifier stays on the device. No camera image is sent off-device. Only the text label, confidence, and timing of recent readings go to the feedback model.
Autism Shinigami Eyes is an assistive prototype. It is not a diagnostic tool and should not be used for safety-critical or high-stakes decisions.
How we built it
The data
We trained on a nine-class dataset of 16,912 labeled, close-up face images. Class sizes range from 1,233 images (shy) to 2,900 (fear). The images are small and vary widely in size: 66.5% have a short side under 128 pixels. This matters later, because enlarging a small image cannot restore detail it never had.
Auditing and cleaning the data
Our first models reported a validation accuracy in the mid-40s, but an audit of the dataset showed those numbers could not be trusted:
- Only 12,477 of the 16,912 images were distinct. 1,331 of the 3,383 validation images (39.3%) had an identical copy in the training set, so validation was partly testing memorization.
- 626 groups of identical images carried conflicting labels, covering 1,854 images. A classifier cannot satisfy two different labels for the same pixels.
We removed all conflicting-label images and kept one copy of every remaining duplicate, which left 11,851 unique images. We then built a fresh stratified split with no identical images shared between sets: 8,295 for training, 1,778 for validation, and 1,778 for testing. The test set stayed untouched until we had chosen our final model. We did not relabel any images ourselves. Every result below uses this cleaned split.
Preprocessing and training setup
Every image is corrected for orientation, converted to RGB, resized to fit 128x128 while preserving its aspect ratio (gray padding fills the rest), and normalized. Training uses random horizontal flips, small rotations, and brightness and contrast changes. The loss is weighted by inverse class frequency with light label smoothing, optimization uses AdamW with a reduced learning rate on plateaus, and we stop early when validation stops improving. We select checkpoints by macro F1 rather than raw accuracy, so that every class counts equally.
Our own CNN first
To learn how a network is constructed, we started with a small CNN of our own. Early on, it scored 39.4% validation accuracy at 64x64. A deeper six-layer version with batch normalization, dropout, and augmentation reached 44.8% at 64x64 (both on the original, uncleaned split). Its final form, at 128x128:
Input: RGB image (3 x 128 x 128)
2 x Conv 3x3, 32 channels + BatchNorm + ReLU -> MaxPool 2x2 + spatial dropout 10%
2 x Conv 3x3, 64 channels + BatchNorm + ReLU -> MaxPool 2x2 + spatial dropout 10%
2 x Conv 3x3, 128 channels + BatchNorm + ReLU -> MaxPool 2x2 + spatial dropout 10%
Adaptive average pooling -> 128 x 4 x 4 -> Flatten (2,048 values)
Fully connected 2,048 -> 128 + ReLU -> Dropout 40%
Fully connected 128 -> 9 class scores
This network has roughly 551,000 trainable parameters and learns entirely from scratch.
Input size. We compared 64x64 and 128x128 on the cleaned split: 43.6% versus 44.2% accuracy, with identical macro F1 (0.371). We chose 128x128 as a balance between image detail and training time. It gave a small edge, but not a dramatic one, which fits the source images being small.
Weak classes. Overall accuracy hid how badly the network handled some labels. Recall was only 11.9% for confused, 11.3% for shy, and 9.7% for sad. Inverse-frequency class weighting with macro-F1 selection raised macro F1 from 0.371 to 0.406 and roughly doubled recall on those classes, but accuracy stayed at 43.9%.
Moving to ResNet18
We then switched to ResNet18, an 18-layer architecture designed for vision tasks. It differs from our network in two ways: it uses residual connections, and it starts from pretrained ImageNet weights. It has 11.18 million parameters, about 20 times as many as our CNN.
| Model (cleaned split) | Validation accuracy | Macro F1 |
|---|---|---|
| Custom CNN, 64x64 | 43.6% | 0.371 |
| Custom CNN, 128x128 | 44.2% | 0.371 |
| Custom CNN, 128x128, inverse class weighting + macro-F1 selection | 43.9% | 0.406 |
| ResNet18, 128x128, pretrained, fully fine-tuned | 58.4% | 0.569 |
| ResNet18, 128x128, only final block trained | 54.3% | 0.531 |
| ResNet18, 128x128, stronger augmentation | 57.6% | 0.566 |
Moving to ResNet18 was a gain of 14.5 percentage points in accuracy and 0.163 in macro F1. Recall on confused rose from 27.1% to 44.1%, on shy from 26.1% to 52.1%, and on sad from 20.0% to 40.0%.
ResNet18 also overfit: it reached essentially 100% accuracy on the training images against 58.4% on validation. We tried two remedies, freezing all but the final block and stronger augmentation. Neither beat plain full fine-tuning, so we kept the original model (selected at epoch 20).
Final evaluation
We evaluated the chosen model once on the untouched test set of 1,778 images: 57.8% accuracy and 0.565 macro F1, compared with about 11.1% for random guessing across nine classes. These closely match the validation results, which supports the validation estimate. Recall is strongest for neutral (78.7%), happy (71.6%), and surprise (64.0%), and weakest for sad (41.8%), confused (45.8%), and fear (47.0%).
Running on the Uno Q
The trained classifier is exported to ONNX and runs on board the Arduino Uno Q, which handles the full pipeline from frame to expression label. We checked that the exported model reproduces the original's outputs. The feedback model runs in the cloud and receives only text.
Challenges we ran into
- Misleading evaluation. Our early validation scores were inflated by duplicate images shared between training and validation, and by identical images with conflicting labels. We only found this by auditing the data. Fixing it meant discarding about 30% of the dataset and re-running our experiments on a clean split.
- Weak classes. Confused, shy, and sad were hard to recognize. In our CNN's errors, many shy images were predicted as happy, and sad was often predicted as neutral. These labels overlap visually, and the dataset's labels for them may be ambiguous.
- Overfitting. The pretrained network memorized the training set. Freezing layers and stronger augmentation did not close the gap.
- Compute limits on the Uno Q. Running everything on board is central to our privacy goal, and it is also our main bottleneck. The size of the model we can run on the Uno Q limits how much further we can improve accuracy.
- Small, close-up images. Most images are small, so higher input sizes mostly enlarge existing detail. A standard Haar cascade face detector found faces in only about 18% of the images, so we skipped detection and score the whole frame. The model works best when the face fills the view.
- Moving from coursework to practice. We knew the fundamentals from our ML courses, but this was our first practical project. Real data, a hardware target, and live camera input raise questions that classroom exercises do not.
- Turning predictions into advice. A classifier outputs a label and a confidence score, which is not guidance a person can act on mid-conversation. Getting hedged, useful advice from those signals, including how an expression is changing, required the fine-tuned feedback model.
- Lack of hardware. We were unable to obtain an Arduino camera to go with our Uno Q so we used our cellphone to emulate the feed we would get from glasses. We were still able to compile our code to run on the arduino and verify that it could run in reasonable time.
Accomplishments that we're proud of
- The project grew out of a real need on our team, and we built it end to end: camera input, on-device classification, and social guidance.
- We found and fixed a flaw in our own evaluation. The audit, the cleaned split, and a test set we did not touch until the end mean our reported results are a fair estimate on unseen images.
- We put our coursework to practical use: we designed and trained a CNN from scratch, diagnosed its limits, and moved to a pretrained ResNet18, raising validation accuracy from 43.9% to 58.4% and macro F1 from 0.406 to 0.569.
- We reached 57.8% accuracy on the held-out test set across nine classes, about five times random guessing, and recall on our hardest classes more than doubled.
- The whole recognition pipeline runs on the Uno Q, so the camera feed never leaves the device.
- The advice accounts for how an expression changes over time instead of treating each frame as unrelated which helps average out the model's uncertainty.
- We built a custom fine-tuned model to turn raw classifier output into guidance that is cautious about its own uncertainty.
What we learned
- Check your evaluation before trusting it. Duplicates and contradictory labels made our early scores look better than they were. Auditing the data mattered as much as any change to the model.
- Accuracy can hide weak classes. Per-class recall and macro F1 exposed failures that overall accuracy did not.
- Pretrained weights matter. Starting from ImageNet weights with residual connections gave us a 14.5-point accuracy gain over our from-scratch network.
- Not every fix helps. Freezing layers and stronger augmentation both failed to beat the baseline. Controlled, one-change-at-a-time experiments told us what actually worked.
- Resolution is a trade-off. 128x128 gave a small edge over 64x64, and larger inputs cannot add detail that small source images lack.
- Theory versus practice. The building blocks from our courses (convolutions, batch normalization, pooling, dropout) are simple on paper. Training on real data showed us how each choice affects accuracy, overfitting, and training time.
- Hardware shapes the design. The target device has to be considered when choosing a model, not after.
- Data defines the deployment. A model trained on close-up faces must be given close-up faces.
What's next for Autism Shinigami Eyes
- Better hardware. We plan to move from our prototype to Meta glasses. A stronger camera and more capable hardware should raise accuracy substantially.
- Better data. The remaining gap points to the data more than the architecture. We plan to review ambiguous labels, especially confused, sad, and shy, and to collect more varied training images.
- Teleprompter-style advice. Advice currently has to be read as text. We want to show it in the wearer's field of view like a teleprompter, so guidance arrives without breaking eye contact or the flow of the conversation.
- Sarcasm detection. Sarcasm is another area where many autistic people struggle. We plan to extend the system to flag it, building on our existing tracking of how cues change over time.
- Continued model improvement. We will keep refining the classifier and the feedback model, using the extra capacity of better hardware.
