Guide Wing

Overseer for those that cannot see.

Comment

Inspiration

Moving through campus can involve obstacles, uncertainty, and moments when someone needs assistance. For blind, low-vision, and older people who need support, a visual alert alone is not enough. Feedback needs to be accessible, and requesting help should lead to a clear acknowledgment.

We envisioned GuideWing: Awareness ahead. Assistance within reach. A companion drone would provide a moving view of the path ahead, while a phone app and connected headphones deliver spoken guidance and accessible assistance controls.

Responders may also be away from their desks. We wanted them to receive and handle assistance requests through a messaging app they already use.

What it does

GuideWing demonstrates obstacle awareness and a complete human-assistance workflow.

  • Detects a chair within a configured camera region and provides a spoken warning.
  • Pauses a simulated journey when an obstruction is detected.
  • Recognizes a deliberate two-second open-palm hold to request help.
  • Provides Help and Cancel controls as alternative inputs.
  • Offers a ten-second cancellation window before dispatching the request.
  • Gives responders a dashboard to acknowledge and resolve requests.
  • Lets registered responders receive alerts and accept or resolve requests through an iMessage agent powered by Photon Spectrum.
  • Provides spoken confirmation when a responder acknowledges.
  • Displays response analytics, with Gemini-generated summaries of calculated metrics.

The current user interface is a website. A native phone app with headphone guidance is the intended next step. Responders can already interact through iMessage without installing a separate app.

How we built it

Our FastAPI backend manages journeys, simulated route progress, and the assistance-request lifecycle. SQLite persists requests so they survive server restarts.

The vision pipeline uses OpenCV for camera handling, YOLO for chair detection, and MediaPipe for gesture recognition. DroidCam supplies camera frames from a phone over USB.

The user and responder interfaces share backend state. A connected 2D simulation provides a browser-based visualization of the journey.

We built a Node.js and TypeScript responder agent using the Photon Spectrum SDK. It listens for backend events and lets allowlisted responders use commands such as accept, resolved, details, and list through iMessage.

After the cancellation window, registered responders receive an alert with the simulated landmark and request details. The first successful acceptance claims the request, and other responders receive an update. The agent uses the same backend endpoints as the web dashboard, keeping both channels synchronized.

We also built an Unreal Engine scene with a walker, following drone, and obstacle. The integration code remains outside the committed repository; its reproducibility and verification status are documented separately. Drone flight and route location are simulated.

Gemini summarizes responder analytics. Counts and durations are calculated from stored records rather than generated by the language model.

Challenges we ran into

Reliable demonstrations required more than detecting an object or recognizing a gesture. We had to prevent repeated requests from a sustained palm hold, handle cancellation near the dispatch deadline, and keep multiple interfaces consistent.

Camera indices changed between runs, lighting affected detection, and campus network restrictions complicated communication between devices. We used DroidCam over USB and prioritized a local setup.

The iMessage agent introduced additional challenges around reconnection, duplicate notifications, and simultaneous responder actions. An early reply loop taught us to ignore the agent’s own messages and non-text events. We also added notification rate limits.

For the current messaging setup, responders must send an initial message after each agent restart before receiving alerts. The shared demo line can also change.

We learned to separate the long-term drone vision from the capabilities we could demonstrate and verify during the hackathon.

Accomplishments that we're proud of

  • A working camera-to-assistance workflow with spoken feedback.
  • Duplicate-request protection and persistent request state.
  • Cancellation and responder-concurrency handling.
  • Live camera testing with a real chair and deliberate palm gestures.
  • A user/responder website and browser simulation fallback.
  • An iMessage responder agent tested live for alert delivery, listing requests, and acceptance.
  • Fourteen passing agent tests, including backend integration tests covering acceptance, resolution, and cancellation.
  • An Unreal companion-drone scene.
  • Analytics grounded in stored records, with an AI explanation layer.

What we learned

Accessibility affects the entire interaction: how a warning is delivered, how help is requested, and how the user knows someone received it.

We also learned that a reliable assistance workflow matters as much as the perception model. Detection becomes useful when it leads to understandable feedback and an actionable response.

Adding iMessage showed us that responders do not necessarily need another dedicated app. A messaging interface can make assistance requests available away from a desk, while a shared backend preserves consistent state.

What's next for GuideWing

  • Validated real-world location and broader obstacle detection.
  • Controlled drone hardware trials.
  • User studies with blind, low-vision, and older participants.
  • Partnerships with authorized campus assistance services.
  • Research into potential distress cues with user confirmation.

GuideWing is an early prototype. It does not currently recognize crimes, provide validated safe navigation, or connect to GSU Police or emergency services.

Built With

Share this project:

Updates

Submission history