Pincher

• 

  Hand tracking is visualized

• 

  braindump agent

• 

  context & phrase guide agent

Inspiration

Project description: Non-native English speakers and individuals who struggle to formulate their thoughts into organized text often struggle with trying to capture complex thoughts, reading or writing unfamiliar idioms, and jargon in English that can slow down comprehension. This often leaves editors to translate or look up definitions interrupting their workflow or even creating a writers block. Additionally, speech to text writing results in messy unstructured brain dumps which require many edits to smooth out the text. How your solution works: Our solution is a project that uses computer vision to help the user edit and understand their documents. An open hand starts the brain dump allowing the user to speak and closing the fist stops it. Pointing makes the mouse follow your finger and pinching is a click and holding a pinch drags and highlights text. After the user has highlighted the text they can send it to the context & phrase guide agent to help them understand the text in their native language. Additionally, opening your hand quickly activates the keyboard shortcut that activates superhuman go. The brain dump agent identifies text that was produced by voice to text or speech to text dictation by getting triggered by the word “brain dump” and then it takes this text and puts it into organized text for the user to implement into their document. Agent Integration: We built two specialized agents inside Superhuman Go that help the user draft and edit their documents. Brain Dump Agent: Captures raw speech-to-text dictation bounded by start and end triggers which recognizes large pieces of texts from voice typing or speech-to-text and provides recommendations to reformat messy text into clean, organized, and structured text. Context & Phrase Guide Agent: Scans documents and recognizes any complex idioms, figurative language, or confusing phrases and provides simple breakdowns and native-language translations without overwriting the author’s english draft. The agent also allows the user to select text and ask the agent to translate in the user’s preferred language.

Built With

• mediapipe
• pynput
• python
• superhuman