BE HEARD

Inspiration and the problem

Digital communication often assumes a single input method and asks people to adapt to the interface. BE HEARD explores a more flexible approach: users can express an intention through voice or personal fingerspelling, review what the system understood, correct it, and remain in control of the final result.

What it does

BE HEARD is an Android accessibility and productivity prototype with two input modes and one shared language pipeline.

Voice Mode

Android speech recognition creates an editable transcript. The user chooses an output format such as an email, message, task, or note. Only the approved transcript and writing preferences are sent through a protected gateway to GPT-5.6, which returns a structured result with intent, confidence, and ambiguity information.

The result remains editable. Saving, copying, sharing, and exporting stay locked until the user explicitly confirms it.

Sign Mode

Sign Mode is a constrained personal fingerspelling prototype. Its Build Week scope supports seven static configurations: A, C, I, M, O, T, and U.

MediaPipe Tasks Vision extracts 21 normalized hand landmarks on the Android device. Personal calibration data stays in app-private storage and can be explicitly audited, exported, and restored through a validated backup.

Privacy Preview starts enabled and hides the raw camera image while displaying the live landmark skeleton. Recognition continues locally, and the user can reveal the camera preview only through an explicit action. Manual correction is always available when a pose is uncertain.

The editable letter sequence enters the same GPT-5.6 language pipeline used by Voice Mode, producing one consistent, reviewable, and confirmable output experience.

How we built it

BE HEARD is built with Expo, React Native, TypeScript, Kotlin, and MediaPipe Tasks Vision. Local history uses SQLCipher with a key protected by Android SecureStore.

A Cloudflare Worker acts as the protected OpenAI gateway. The OpenAI API key remains server-side and is absent from the APK and repository. Release requests use HTTPS, OpenAI storage is disabled with store: false, and cloud processing requires explicit user consent.

The architecture separates perception from language understanding:

Voice or Signs -> specialized input processing -> protected gateway -> GPT-5.6 -> editable result -> user confirmation

How we used Codex and GPT-5.6

Codex was used throughout Build Week to audit the inherited Android codebase, document the event baseline, implement scoped mobile and gateway changes, diagnose physical-device behavior, validate security boundaries, test the signed release, and assemble judge-facing evidence.

GPT-5.6 is the shared language layer. It interprets the user's approved input and produces structured text, intent, confidence, and ambiguity information. Speech recognition and hand perception remain specialized systems rather than being presented as one monolithic model.

Product scope and risk decisions remained human-controlled. These included reusing the existing mobile foundation, choosing the event category, limiting the personal Sign Mode vocabulary, enabling Privacy Preview by default, and rejecting uncertain C/O captures instead of lowering thresholds to produce a misleading success.

Build Week work

BE HEARD evolved an existing Android application rather than starting from a new application root. During Build Week, the project added the BE HEARD identity, the protected GPT-5.6 pipeline, cloud-processing consent, structured outputs, session usage telemetry, the personal Sign Mode prototype, local calibration and backup, Privacy Preview, conservative ambiguity rejection, security validation, and signed Android release packaging.

Challenges and lessons learned

The hardest problem was not drawing a hand skeleton, but deciding when the recognizer should refuse to guess. Personal tests showed that C and O were not reliably separable. The safer design was to reject uncertain captures and keep manual correction visible.

We also learned that a useful accessibility prototype needs transparent boundaries. Privacy controls, editable intermediate states, ambiguity reporting, and explicit confirmation are part of the product, not finishing touches.

Transparent limitations

Sign Mode is not a complete or continuous LIS translator. It does not claim full manual-alphabet coverage, sentence-level spatial grammar, expert linguistic validation, or multi-user recognition accuracy.

The current prototype supports seven personal static configurations. Recorded tests did not reliably separate C from O, so uncertain captures are rejected and manual correction is the supported fallback.

Privacy Preview controls what is displayed, but MediaPipe still analyzes the full camera frame locally to locate the hand. BE HEARD does not save or send that frame to the GPT-5.6 gateway.

The temporary judge gateway is not production authentication. A production launch would require account and session controls, abuse protection, monitored budgets, a representative consented evaluation, formal retention policies, and broader accessibility testing.

What's next

After the event, the next steps would be a consented multi-user evaluation with LIS expertise, broader fingerspelling coverage, color-vision accessibility themes, production authentication, monitored usage limits, and a formal privacy and retention program.

Built With

• android
• cloudflare-workers
• codex
• expo.io
• gpt-5.6
• kotlin
• mediapipe-tasks-vision
• openai-api
• react-native
• sqlcipher
• typescript