Scrawler semantic runtime

This project

My idea is that it's absurd to make the agents take a screenshot, analyze the screenshot and use an api to click on a pixel, only to click on ONE SINGLE button. That's why I made Scrawler Semantic Runtime. The goal is to separate the gui and the logic in order to make my system call the functions on click, on hover, on input, etc. instead of that the functions detect theses events which allows human to interact with the app through a gui and an agent both through a mcp server.

My project

My final goal is to make a new OS called "ScrawlerOS" where human and agent can interact with everything together, and Scrawler Semantic Runtime is a part of this projetc, but as I had only 2 days for this hackaton (yes, I discovered the OpenAI build week a litle bit too late) I "only" made Scrawler Semantical Runtime.

My usage of GPT-5.6

As I missed the start of the competition I couldn't get the codex free credits and I'm in free plan so my usage of codex was really limited. I used it to make the structure and the start of the project and it made strong bases, then I continued with an other llm.

Built With

• rust