About the Project: RetinAI

Inspiration

The project draws inspiration from a relative in India who suffered from an undiagnosed eye ailments. Because of age, inconvenience in leaving home, and the high cost of medical visits, its early detection was missed. I wanted to build a tool that makes eye screening affordable and accessible for anybody just with a smartphone.

What I Learned

I went deeper into the learning path of computer vision, React Native app development, TensorFlow/PyTorch training pipelines, and model deployment on mobile devices. On the medical side, I got to learn how early detection for diseases—say diabetes, glaucoma, cataracts, AMD, hypertension, and myopia—could prevent blindness.

How It Came to Be

App Development: For the mobile app development, I took advantage of React Native + Expo + TensorFlow Lite + TorchScript. The app attaches with a clip-on lens and uses the camera of a phone for retinal images.

Model Training: I went on and did a training for both ResNet50 and Vision Transformer ViT models offline for multi-label classification of 8 eye conditions from the ODIR-5K dataset. The training used custom data loaders, augmentation, and fine-tuning with BCEWithLogitsLoss.

Model Hosting and Conversion:

The ViT was trained with PyTorch.

The trained model was then exported to ONNX.

On the TensorFlow side, the model was converted, then TensorFlow Lite to allow for optimized mobile operation.

The last step would have been to wrap the TFLite model in React Native through expo-tensorflow, including custom native modules.

Deployment: The app performs on-device inference to offer disease predictions and with the aid of the Google Maps API, guides the user to the nearest ophthalmologist.

Challenges

The hardest bit was to get the ViT model working correctly on mobile. Going from PyTorch to ONNX to TensorFlow to TFLite often caused incompatibility issues, and quantization caused drops in performance at times. It took a few rounds to really balance speed versus size versus precision on a mobile device implementation.