Koi_carp-01

Inspiration

Problem: When faced with a new program, people give up or constantly put off learning it because of its difficult interface. Solution: Koi Carp is an AI assistant with step-by-step visual instructions. Always-on-top widget + smart highlighting + Open AI Product. Teaches without distracting from work.

What it does

Koi Carp Assistant is an AI-powered desktop application that provides real-time, step-by-step visual guidance for any software application. When users encounter unfamiliar programs (Word, Photoshop, Excel, etc.), instead of watching lengthy tutorials or reading manuals, they simply:

Click the "AI helper" widget in the corner of their screen Select what they want to do (e.g., "Save file", "Export to PDF") Follow highlighted, step-by-step instructions overlaid directly on their screen

How we built it

We used the OpenAI API with GPT-5.1 to analyze software interface screenshots. For step highlighting, we implemented text detection using the Tesseract OCR (Python wrapper pytesseract), which allows us to identify UI elements by locating their text directly on the screenshot.

Challenges we ran into

Fine-tuning prompts to ensure accurate interpretation of user actions.

Selecting the most suitable library and designing an efficient algorithm for UI step highlighting.

Balancing performance and recognition quality across different software interfaces.

Accomplishments that we're proud of

We are proud of everything we achieved in this project — it’s a big step forward for us in AI learning. But most of all, we’re proud of the well-crafted button highlighting for the tutorial, which turned out especially polished and effective.