Procedurize

Inspiration

I was inspired by this very cool github repo detailing the Wave Function Collapse algortihm ( a Procedural Content Generation algortihm which borrows some ideas from quantum mechanics). I have also been reading this book about how PCG and ML can be married together to achieve new super charged generations. My project is an implementation of Wave Function Collapse which gives AI the ability to run operations.

What it does

Procedurize allows users to import tile sets (either directly from canva or from their local files) and then generate and upload Wave Function Collapse generated resulting images which can be used as unique one off backgrounds for designs. Procedurize – with the help of window.ai – puts an LLM in charge of the content generation allowing the user to spin off multiple image generation jobs with a simple request to the AI agent.

How we built it

I used P5.js to display the Wave Function Collapse process and implemented the algorithm within P5's draw loop. The AI (powered by window.ai) is given very strict prompting on the format of outputs to provide which are then interpreted as client side commands and executed. The whole frontend is stitched together with React and Vite. I have a backend node server handling integrations with Canva's APIs for authorizing, getting folders and uploading images which my front end calls at appropriate times.

Challenges we ran into

1. Setting up the authorization flow was quite involved and required dissecting the example project, it would be more helpful in my opinion to provide a much simpler bare bones example project which just handles auth and nothing else.

2. Integrating the AI to output JSON format consistently with the fields I wanted required a lot of prompt tuning which was very difficult since you cannot really "debug" a prompt like you can code. I found that thinly veiled threats to the AI model worked best to get it to do what I wanted.

3. Figuring out a good way to determine tile adjacency was probably the hardest part of this project (and is still not perfect) normally, adjacency is done with hand authored rules, but since I needed my application to generalize to other tile sets, I needed a more generic approach. I implemented a vector similarity approach to detect if edges were similar enough to "connect" based on an arbitrary hard-coded threshold. It works for most tile sets, but there are artifacts for some tile sets with small connectors.

Accomplishments that we're proud of

I am very proud of my ability to integrate AI and Procedural Content Generation to create useful output images, this was my first time implementing Wave Function Collapse and first time using window.ai.

What we learned

I learned a lot about procedural content generation and its pitfalls and how AI and ML can ameliorate some of its pitfalls. I also learned a lot about OAuth and cookies during my debugging sessions to get my Canva Auth set up.

What's next for Procedurize

Up next, Procedurize will give more power to the AI model, allowing it to pick tile sets itself and expand into different forms of Procedural content generation like noise and grammars. I would also like to expand past 2D and try my hand at Wave Function Collapse for 3D generation.

Built With

• docker
• heroku
• node.js
• p5.js
• react
• vite
• window.ai