Post-it!

Inspiration

“Just automate it” is the most common advice in tech, and the least useful. Everyone has a routine they’d love to hand off, but the moment you open an automation tool, you’re stuck with triggers, webhooks, and docs, which can get confusing really quickly. We asked a simple question: what if the hardest part of automation was drawing a few boxes and arrows?

What it does

Post-it! turns a hand-drawn workflow into a running automation. Sketch your steps on paper, a whiteboard, or any surface, then point a camera at it. In seconds, our vision pipeline reads each step and the arrows between them, and builds a live workflow graph. And then executes it with one click. What you can draw: Start / Stop: where your workflow begins and ends Actions: like “post to Slack: prep started” Waits: “wait 10s” Loops: “×3” repeats a step Ask: pauses and asks a human to confirm before continuing While it runs, you can see exactly what’s happening: The camera feed highlights each step on your drawing as it runs. A clean map view mirrors the workflow, with running, done, and error states updating live. Confidence scores show how sure the system is about each step, and plain-English errors like “Couldn’t read step 3” tell you what to fix instead of failing silently. A “scene settling” indicator waits until your hands are out of the frame before it reads the drawing, and Scan now lets you re-read it any time.

How we built it

Post-it! Has three primary components that we split evenly between our group:

Frontend (React, TypeScript)

Everything starts on screen. The left pane of the screen either shows a live webcam feed or enables the user to upload a photo on that wall. Once the user presses scan, the frontend sends the frame to the backend’s frame endpoint as a JPEG. In addition, keeps a WebSocket connection open to the backend, which is how it hears about everything that happens next.

Vision (Perceptron):

The JPEG goes into our vision pipeline, built on Perceptron. First, a detection call identifies every sticky note and numbers them left to right. Then each note is cropped and processed for text, whether it’s an action or a symbol, and how legible it is. A final call then looks at the photos with each note number drawn on it and reads the arrows between them. The order is determined by the arrows, then the numbers written on the notes, and then the position on the wall. The result is converted to JSON with the type, text, settings, position on photo, and confidence score.

Backend (Python, FastAPI)

The backend processes this shared JSON, first by checking the workflow to fill in any missing details, adjust extreme values, and turn problems into warnings rather than crashes. A comparison step then matches notes between photos by what they say since detection IDs change with every photo, which is what allows you to change the wall of notes in real-time. It then runs the workflow step by step, handling actions like wait, loop, and ask. Each action is a plug-in and connects a sticky note to a real service. We built messaging and email integrations for this demo, but any other service can be added in the same way. Grok handles judgment calls, like filtering and summarizing, and steps pass their results along, so each note builds on the previous one. The backend then sends the workflow over the open Weblock with warnings and what changed and a live event for each step as it starts, finishes, or fails.

Challenges we ran into

We faced a couple of challenges while developing this project. First, this was all of our first times working with the Perceptons API and tools, so it took us some time to fully understand how to work with and implement it. Furthermore, with this project having a lot of moving parts, it took us a lot of debugging to integrate the two different pipelines as well as the frontend. Finally, with live capture, the MacBook webcams became a severe limitation, and with their low quality and resolution, text can be difficult to read.

Accomplishments that we're proud of

We had a couple of key goals for this project that we are very proud of. First was the live reading feature. This was a key goal for us, as we feel one of the best parts of writing on a whiteboard or with sticky notes is how easy it is to move things around and change them, and we wanted our app to reflect that. Second, was our UI design, as three teammates with a more technical background, UI design is not our strength. However, I felt we were able to develop a bright, easy-to-read UI which is very inviting for the user.

What we learned

Through this project, we learned a lot about OCR and computer vision, like how to detect handwritten text on a whiteboard automatically, and how to turn that text into scripts that actually get tasks done. Handwriting turned out to be a lot harder to read than we expected, so we spent a good amount of time making the detection more reliable. We also learned a lot about front-end design and UI/UX, and how to make our code not just functional but presentable, so someone using it for the first time can understand what’s going on.

What's next for Post-it!

Next for Post-it! We plan to roll out support for more apps such as Teams, Word, Excel, etc. Also we have plans to add a feature where you can draw your own symbols, upload them, give them a meaning and be able to use them in your automations, saving you from rewriting common workflows. Finally, we believe all your workflows should be in one place, so we plan to implement features for workflow management such as being able to edit old workflows or being able to copy an old workflow and adapt it to a new task.