Inspiration
I have been inspired by Growing Neural Cellular Automata (Mordvintsev 2020), and have done a lot of independent experiment on NCA. My reason to join the hackathon is to push myself to make a model, train it, write inference code, all within a strict timeframe. The main focus point of this project was to expand NCA into practical places, and as someone who has tried to draw and code regeneration that looks real, I know it is hard.
What it does
This Project uses the growing neural cellular automata paper as the baseline architecture, and I have added Factored Conditioning. The regrowth logic is an seed reinjection method, compared to the original training in batches method, but my implementation is easier to train and run due to the reduced complexity of the model itself.
How I built it
Day 1 - Setting up the development environment and creating the basic project structure. Writing the training code, with a focus being on cleaner image generation. Switch training code to be trained on RGBA instead of simple RGB for better clarity, and use L1 loss instead of MSE loss for better image clarity. Download emoji from emojipedia that will be used as the targets during training Write the code for converting the .png files into RGBA tensors that Start training the model. Overnight run on kaggle Day 2 - Turns out, the overnight run was accidentally using CPU instead of GPU, started retraining the model. Planed out the Ui of the HTML to demo the model Model training completed, 2 out of the 25 images are completely distorted. Designed a new architecture using broadcasting seed with alpha channel used as spark to start the NCA Read Growing Neural Cellular Automata paper again to see his implementation of an alive mask to prevent sudden spontaneous explosion of colors Implement the new architecture and retrain the model. Wrote html code for demo of the model. decoupled inference and demo so for testing the current weights can be used Day 3 Model trained, js onnx created for runtime inference and nca_inference.js updated to match new weight and training architectural changes Scripted and editted video for demo of the model.
Challenges I ran into
First challenge : Overnight training I started on Day 1 was using CPU instead of GPU. Fortunately, I had enough time to start another retraining at the start of Day 2 and I expected it to be done by Day 2 Second Challenge : The second run ran fine, with it taking 10.5 hours on GPU, but my training architecture was supervising just the final state, something I previously thought wouldn't be a problem, but after looking at the results, the problem was clear, the model found out the best way to grow the image is flood fill then slowly shrink back into the actual image. So I re read the original growing Neural cellular automata paper again and realised 1 clever trick Alexander Mordvintsev used for his model, and I did a simpler implementation of the same idea Third Challenge : On the morning of Day 3 I checked kaggle, and the model was trained, but the model only downloaded as a .zip instead of a .pth file, I tried to use Ai to see why that was happening, but it didn't help, and I was on a ticking clock, eventually through reading reddit posts and pytorch documentation, I learnt how pytorch actualy makes the .pth and then wrote the python script needed to turn the zip into a .pth
Accomplishments that I am proud of
I am really proud of myself for getting this done, in all the previous independent research I have done, I never had any timeline or external pressure, so I am also really grateful for stemist to give me the opportunity to participate in this hackathon and push myself beyond my comfort zone Beyond completing the project within the timeframe, I'm proud of independently diagnosing and solving the flood fill training failure by re-reading Mordvintsev's original paper and implementing a simpler version of his alive masking approach — a problem I had to identify, understand, and fix mid-hackathon without external guidance.
What I learned
I learnt a lot through this hackathon, I learnt why Alexander Mordvintsev has made the architectural decisions he did, as I was failing to build the correct training architecture.I deepened my understanding of how PyTorch implements the underlying math, which helped me debug training failures more effectively by reasoning about what was actually happening at the tensor level, not just at the API level.
What's next for Regrowth
The next step for regrowth is expanding on the regeneration aspect of it and make the model universal, where instead of training on specific images, each cell will be given target pixel color, but it still has to organically grow from a seed, this allows me to test the model in smaller game dev communities, and get feedback, while removing the need for game devs to spend hours on GPU runs to train the model.
Log in or sign up for Devpost to join the conversation.