Synchro

Inspiration

Technology is supposed to make everyday life easier, but for people with physical limitations, even simple interactions can become frustrating.

One of our friends experiences wrist discomfort associated with carpal tunnel syndrome. Seeing how this affects everyday activities made us think about how much we depend on holding devices, navigating screens, and repeatedly interacting with technology just to access basic information.

We also considered people with low vision or reading difficulties, for whom accessing printed instructions, labels, and other visual information can present additional challenges.

This inspired us to ask: What if accessing information about our surroundings didn't always require reaching for a phone?

We created Synchro, a wearable camera prototype exploring a different approach to human-computer interaction.

While smartphones and smart glasses already offer visual-assistance tools, we wanted to experiment with a hand-mounted camera that could be deliberately pointed toward objects and documents.

Our long-term goal is to develop an affordable wearable assistant that makes visual information more accessible through camera technology and intelligent software.

What it does

Synchro is a wearable camera prototype built around an ESP32-CAM and a custom macOS application.

The system is designed to transmit camera images to a computer through Bluetooth Classic, where they can be decoded, displayed, and processed.

Our software includes:

Bluetooth Camera Communication: A custom protocol for requesting camera status and JPEG images over Bluetooth Classic RFCOMM.
Camera Viewer: A native macOS application built to display images received from the ESP32-CAM.
Image Processing: JPEG frame reconstruction, validation, and decoding.
Camera Controls: Start and stop controls, connection monitoring, and image rotation.
Performance Diagnostics: Frame-rate calculations, JPEG size measurements, resolution information, and image-transfer timing.
Image Capture Tools: Support for saving original JPEG images and exporting performance data.
Error Handling: Detection of connection timeouts, malformed responses, and corrupted image data.

The camera application has been compiled and launched successfully, and its communication protocol has undergone automated testing using simulated Bluetooth data.

Synchro is still a prototype. Its current implementation establishes the software infrastructure for a wearable camera, while additional hardware validation and accessibility features remain future goals.

How we built it

Hardware: We developed our prototype around an ESP32-CAM and portable power components, exploring how a compact camera could be incorporated into a wearable device.
Bluetooth Communication: We used Bluetooth Classic RFCOMM to communicate with the camera. Our protocol requests camera status and individual JPEG frames. The application reconstructs complete images from incoming data, even when Bluetooth messages arrive in separate fragments.
macOS Application: We developed a native application using Swift, AppKit, and Apple's IOBluetooth framework. It manages device connections, image reception, connection errors, and basic camera controls.
Image Processing: We used Apple's ImageIO framework to decode JPEG images. We implemented checks for valid image boundaries, dimensions, and packet structure to help prevent corrupted frames from disrupting the application.
Performance Optimization: We improved how incoming Bluetooth data is buffered and moved JPEG decoding onto a background queue to reduce unnecessary work on the main interface thread. We also introduced measurements for frame rate, image size, and transfer latency.
UI/UX: We explored different camera interface designs using Lovable and Cursor, eventually prioritizing a minimal interface focused on the camera preview and essential controls.
Testing: We developed simulated protocol tests for fragmented messages, invalid JPEG data, buffer management, timing calculations, and connection handling. This allowed us to test software components without continuously relying on physical hardware.

Challenges

Hardware and Software Integration: Developing a wearable camera meant working across embedded hardware, Bluetooth communication, and native desktop software.
Transmitting Images Over Bluetooth: JPEG images contain significantly more data than ordinary text commands. Incoming image data can arrive in fragments, requiring careful reconstruction before decoding.
Handling Corrupted Data: Malformed headers, interrupted transmissions, and corrupted images can cause connection failures. We implemented data validation and error handling to make the viewer more reliable.
Camera Performance: Capturing, compressing, transmitting, and decoding images can introduce substantial delays. We developed diagnostic tools to identify potential bottlenecks and evaluate future optimizations.
Testing Without Hardware: The physical camera wasn't always available during development. We used simulated Bluetooth messages to test the communication logic independently of the actual device.
Designing for Accessibility: Our friend's experience with wrist discomfort helped motivate the project. However, we also recognized that a hand-mounted device isn't automatically more accessible, especially for people who experience pain with wrist movement. Evaluating comfort and usefulness remains an important part of future development.
Simplifying the Interface: Our early designs included too many technical controls. We refined the experience to prioritize the camera itself, keeping advanced diagnostics separate from the main interaction.

Accomplishments

Built and launched a native macOS camera application using Swift.
Implemented a custom Bluetooth Classic RFCOMM camera communication protocol.
Developed JPEG packet reconstruction, validation, and decoding.
Improved image-buffer management and background decoding.
Added performance measurements for FPS, image resolution, JPEG size, and transfer latency.
Implemented original JPEG saving and performance-data export.
Developed automated tests for malformed packets, incomplete transmissions, image validation, and timing calculations.
Passed 82 simulated camera-protocol checks in a reported development test run.
Established a foundation for future wearable visual-assistance features.

These tests validate the camera software under simulated conditions. They do not establish the device's real-world frame rate, image quality, or accessibility effectiveness.

What we learned

Embedded Systems: How microcontroller limitations and wireless communication affect the design of wearable devices.
Bluetooth Communication: How binary image data can be transmitted and reconstructed through a custom protocol.
Swift Development: How to create native macOS applications using AppKit and IOBluetooth.
Image Processing: How JPEG compression, decoding, and image dimensions affect performance.
Performance Optimization: Why measuring frame rate and latency is important before making assumptions about bottlenecks.
Software Testing: How simulated hardware responses can help validate software when physical devices aren't readily available.
UI/UX Design: How simplifying an interface can make complex technology easier to understand.
Accessibility Design: Why identifying a genuine user need and validating a device with intended users matters as much as implementing the technology itself.

What's next

AI-Powered Visual Assistance

We want to expand Synchro beyond basic camera capture by connecting it to multimodal AI capabilities.

Our goals include:

Allowing users to ask questions about captured images.
Identifying everyday objects and interpreting printed information.
Providing contextual explanations about photographed objects and documents.
Improving how the system communicates uncertainty when images are unclear.

Synchro Read

We want to build a dedicated reading-assistance feature for people with low vision or reading difficulties.

Planned capabilities include:

Extracting printed text from captured images.
Reading recognized text aloud using text-to-speech.
Adjustable speaking speed and repeat controls.
Larger text display and high-contrast viewing options.

Voice Interaction

Add speech-to-text for spoken questions.
Provide optional spoken responses.
Explore low-effort activation methods that don't require repeated interaction with a touchscreen.

Hardware Improvements

Complete testing with the physical ESP32-CAM.
Improve camera focus, image quality, and wireless responsiveness.
Optimize power consumption and portability.
Develop a smaller, more comfortable wearable enclosure.

Accessibility Research

Gather feedback from people with low vision, reading difficulties, and relevant motor accessibility needs.
Evaluate practical tasks such as reading labels and understanding printed instructions.
Compare the usability of hand-mounted cameras with smartphones and smart glasses.
Refine the design based on real user feedback.