SecondSight

Inspiration

Essential services such as banking, healthcare, MyGov, and Service NSW are increasingly digital. For many older adults, however, completing an unfamiliar online task can still mean handing control of their computer to someone else—or sharing private information over the phone.

We wanted to create a safer alternative: a second pair of eyes that can guide, but never take control.

What it does

SecondSight is a privacy-first remote assistance platform for older adults.

With one large button, an older person can request help from an online volunteer or share a six-digit room code with someone they trust. The volunteer can see a privacy-protected version of the screen, speak with the user, and point to the correct place using circles, arrows, and a live pointer.

The volunteer cannot click, type, move the mouse, or control the computer. Every action remains with the older person.

Before any screen frame is shared, SecondSight detects and covers sensitive content locally, including input fields, passwords, personal details, faces, account information, and QR codes. If the privacy checks cannot complete safely, the app stops publishing new frames and keeps the last protected frame on screen.

Users can also enable optional safety monitoring. SecondSight transcribes both sides of the conversation, detects requests for verification codes, money transfers, remote-control software, and other suspicious behaviour, and warns the user in real time. AI provides advice, but the user always decides whether to continue or pause the call.

If no volunteer is available, the built-in AI guide can examine a redacted screenshot, highlight the next button, and read out a simple one-step instruction.

How we built it

The older-adult application is a native macOS app built with Swift and SwiftUI. It uses:

ScreenCaptureKit for screen sharing
macOS Accessibility APIs for identifying editable and sensitive interface elements
Vision and Core Image for on-device OCR, face, barcode, and redaction processing
AppKit overlays for displaying guidance without capturing it in the shared screen
LiveKit for real-time video, audio, and annotation messages

The volunteer interface is built with React, TypeScript, and Vite. A canvas-based annotation layer converts every circle, arrow, and pointer movement into normalised coordinates so the guidance appears in the correct location on the older person’s screen.

Supabase Edge Functions and Postgres manage sessions, audit events, volunteer presence, safety alerts, and atomic request claiming. This ensures that only the first responding volunteer can enter a session.

For safety monitoring, we combine AssemblyAI streaming transcription, fast on-device rules, and DeepSeek multimodal analysis. High-risk phrases can trigger an immediate local warning, while the AI reviews completed conversation turns and, only when necessary, an already-redacted screenshot.

Challenges we ran into

Our biggest challenge was balancing privacy with usefulness. Covering the whole screen would be safe but unhelpful, while missing even one sensitive field could expose personal information. We therefore combined accessibility data with local computer vision and designed the system to fail safely whenever either check becomes unreliable.

Coordinate conversion was another major challenge. Accessibility elements, Vision observations, Retina displays, captured video frames, and browser canvases all use different coordinate systems. We created a shared message contract and normalised every annotation before transmitting it.

We also had to treat security as a protocol-level requirement rather than a user-interface promise. SecondSight contains no keyboard or mouse control messages, rejects forged safety commands from volunteers, and issues volunteer tokens that cannot publish screen-sharing tracks.

Finally, real-time audio introduced ordering, latency, and cost challenges. We separated partial and final transcripts, processed AI requests serially, and reused screenshots until the screen changed meaningfully.

Accomplishments that we're proud of

We are proud that privacy is built into the architecture rather than added as a warning message.

SecondSight provides live human assistance without surrendering control, protects sensitive information before transmission, supports real-time visual guidance, and adds an independent safety layer to the conversation. It also gives older adults a clear and accessible experience without requiring an account or technical knowledge.

Most importantly, the system preserves the user’s agency: volunteers and AI can advise, point, and warn—but only the older person can act.

What we learned

We learned that trustworthy accessibility tools need more than a friendly interface. Privacy must be enforced before data leaves the device, permissions must match the product’s promises, and every failure state needs a safe outcome.

We also learned that AI works best here as a safety net and guide—not as an autonomous decision-maker. Human help, clear technical boundaries, and user control are what make the experience trustworthy.

What's next for SecondSight

Next, we plan to test SecondSight with older adults and community organisations, replace prototype polling with scalable real-time volunteer discovery, strengthen production authentication and auditing, and expand support to more devices and languages.

Our goal is to make asking for digital help feel as safe and natural as asking someone trusted to look over your shoulder.