Project Frend

Project Frend is a wearable AI assistant built on a Raspberry Pi and a Logitech camera for vision. It bridges live perception from its environment and has persistent contextual memory.

Comment

Project Frend: Summary

Overview

Frend is a smart-glasses assistant for blind and low-vision users, built on a Raspberry Pi 5, that warns about obstacles, describes surroundings, reads text, and remembers where things were seen all in real-time. Safety-critical work runs instantly and offline on the device, while heavy AI work runs in the cloud only when the user asks.

Equipment

A Raspberry Pi 4 runs all the code, a Logitech USB webcam provides both the camera and the microphone, and a MacBook connected over Ethernet currently serves as the speaker. The MacBook also serves as the Raspberry Pi's power source. AirPods were tried as a microphone but dropped because of presentation purposes.

On-Device Technologies

NanoDet-Plus, run through OpenCV, detects 80 everyday object types several times per second for instant offline hazard alerts, with a motion meter, confidence gates, and a wall heuristic cutting out false alarms. Vosk handles offline "Hey Frend" voice recognition, a custom intent parser turns phrases into actions, and PipeWire handles audio input and output.

Cloud Technologies

Claude provides scene descriptions, obstacle checks, and answers to questions, Runpod Serverless runs Qwen2.5-VL-7B on a GPU for detailed scans and text reading, and Cartesia provides a natural, low-latency voice. Elasticsearch Serverless stores a searchable visual memory, so Frend can answer questions like "where is my water bottle?"

Why These Choices

Hazard alerts must never wait on the internet, so detection and voice recognition stay on the Pi, while expensive AI runs only on demand and is billed per use. The cloud side is serverless, so there are no servers to run or maintain and the GPU costs nothing while idle.

How It Connects

The webcam feeds frames to NanoDet for alerts and memory logging, and feeds audio to Vosk, which routes each command to Claude, Runpod, Elasticsearch, or a local action. Everything Frend says passes through one priority speech queue: only "stop" interrupts it, alerts pause while the user is speaking, and audio plays through Cartesia and PipeWire to the speaker.

Difference from Ray-Ban Meta Glasses

Ray-Ban Meta glasses are polished, all-day consumer hardware with a general-purpose assistant that mostly looks only when asked and depends on a phone and Meta's cloud. Frend is a purpose-built mobility aid with continuous automatic hazard alerts, offline operation, searchable object memory, and models of your choosing, though its prototype hardware is not yet as wearable.

Built With

Share this project:

Updates

Submission history