A.U.R.A (Assistive Urban Route Awareness)

A Raspberry Pi car guide for visually impaired users. Ultrasonic & camera sensors use ML models to identify obstacles and announce distances in real time, enabling safe, independent navigation.

Comment

Inspiration

Our team was inspired by the daily navigation challenges faced by visually impaired individuals. We saw an opportunity to combine accessible technology, like the Raspberry Pi and the SunFounder PiCar-X kit, with the power of AI to create a helpful tool. Witnessing the capabilities of service animals and the high cost of some electronic travel aids, we wanted to explore a more affordable, tech-based solution that could potentially enhance safety and independence for users navigating complex environments.

What it does

The "AURA", Assistive Urban Route Awareness, is a robotic assistant designed to help visually impaired users navigate their surroundings more safely. Using its onboard camera, it employs an AI object detection model (YOLO11e) to identify common obstacles such as chairs, tables, curbs, stairs, and people within its path. It estimates the distance to these detected objects (using ultrasonic sensors and through object detection) and provides clear, audible alerts via text-to-speech, such as "Object Ahead". The core function is to act as a preventative warning system against collisions.

How we built it

We built the Smart Service Dog using the following components:

  • Hardware: SunFounder PiCar-X chassis, Raspberry Pi 4B, Raspberry Pi Camera Module, ultrasonic sensor(s), speaker/audio output.
  • Software:
    • Operating System: Raspberry Pi OS
    • Core Language: Python
    • Robotics Control: SunFounder picarx library for controlling motors and servos.
    • Computer Vision: OpenCV for camera feed processing.
    • Object Detection: An AI model (YOLO11e) running through a nearby workstation.
    • Text-to-Speech (TTS): A PiCar-X library 'picar-x.tts' to convert text alerts into spoken audio.
  • Integration: Our Python script orchestrates the system: continuously capturing video frames, feeding them to the object detection model via web sockets, processing the results (object type and bounding box), fusing with the distance sensor data. The bounding and distance data is then feed through our histographical navigation algorithm which forms the basis of our navigation decisions making.

Challenges we ran into

We encountered several challenges during development:

  • Servo Calibration: Getting the steering servo perfectly centered for straight movement required careful software calibration and iterative testing.
  • Object Detection Performance: Running real-time object detection required model optimization and managing processing load. Accuracy varied significantly with lighting conditions and object distance/angles. We solved these problems by paralyzing our computations as much as we could and by using various compression and smoothing algorithms.
  • Distance Measurement: Integrating ultrasonic sensor readings reliably and correlating them accurately with detected objects was tricky due to sensor noise and field-of-view limitations. To solve this we lessened our reliance on this sensor focusing more on the camera based object detection.
  • Real-time Integration: Making all components (camera capture, AI processing, sensor reading, TTS output) work together smoothly without significant lag was a key challenge. We used web sockets to solve the real time communication problems. In conjunction with this we focused on the reduction extra calculations as much as we could. We worked to signifincantly compress each frame as well batch similar frames together for efficient object detection Frame Rate was very slow, 3 frames per second, bounding boxes. We eventually solved this challenge through the ways mentioned above and were able to get 20 fps! Histogram and general frame smoothing: This was a crucial aspect of the car that we had to address. It is the backbone of our navigation algorithm. We encountered several issues where minor discrepancies in our bounding boxes would result in our histograms displaying incorrect data. One of the ways we solved this was by aggregating and calculating histogram averages, creating a histogram from data over time instead of a single point in time. histogram and general frame smoothing

Accomplishments that we're proud of

Despite the challenges, we're proud of several accomplishments:

  • Successfully deploying and running a real-time object detection model on the Raspberry Pi 4B within the PiCar-X platform.
  • Integrating the camera, AI model, web socket communication, histogram algorithm, distance sensing (initial), and text-to-speech into a functional pipeline that provides audible alerts.
  • Achieving basic, calibrated movement control of the PiCar-X.
  • Collaborating effectively as a team to troubleshoot complex hardware and software integration issues.

What we learned

This project has been a significant learning experience. We gained hands-on experience with:

  • Robotics fundamentals (servo/motor control, sensor integration). We all have never worked with robotics before.
  • Applied AI/Machine Learning (specifically computer vision and object detection on edge devices).
  • Python programming for complex, integrated systems.
  • Using libraries like OpenCV, PyTorch, NumPy, and sockets.
  • Debugging challenges unique to embedded systems and robotics (timing, hardware interfaces, resource constraints).
  • The importance of iterative development, testing, and calibration.
  • Thinking critically about designing technology for specific user needs (assistive technology).

What's next for SmartCar / Smart Service Dog

Looking ahead, we have several ideas for improving the Smart Service Dog:

  • Enhanced Perception: Improve object detection accuracy by exploring different models or fine-tuning with more specific data. Add detection for a wider range of relevant obstacles (e.g., wet floor signs, specific hazards). Potentially integrate LiDAR or depth cameras for more accurate 3D spatial awareness and distance measurement.
  • Smarter Navigation: Develop more sophisticated autonomous movement, including path planning and dynamic obstacle avoidance based on sensor input.
  • Improved User Interface: Offer more nuanced voice alerts, possibly including directional cues ("Obstacle slightly to your left"). Explore haptic feedback (vibrations) as an alternative or supplementary alert system.
  • Robustness & Design: Enhance the physical robustness of the prototype and consider designing a more integrated, user-friendly form factor.
  • User Testing: Conduct testing with visually impaired individuals to gather feedback and refine the system based on real-world usability.

Built With

Share this project:

Updates