Hackathon Roblox AI Agent

Built by two young brothers, our AI agent turns kids’ ideas into playable Roblox games, helping Future Tech Foundation (Our NPO) empower children to create, not just consume, technology.

Comment

Inspiration

The idea behind Hackathon Roblox AI Agent was born due to the idea of our Future Tech Foundation which is a non-profit making organization run by myself and my younger brother who was then aged 13 and 12 respectively. The idea has been conceived due to the reason that there are quite a few children who have an idea of the game but lack the programming skills or whatever is needed to develop the game. The children just waste a lot of time playing someone else's world; however, when they try developing a world in Roblox studio, nothing happens. With the help of our agent, the child can describe his/her game in natural language, see it develop in Roblox Studio, play it and even ask for some changes in it.

What it does

The Hackathon Roblox AI Agent turns a child’s game idea into a playable Roblox experience. A creator describes what they want, and the agent reads the live Roblox Studio project, understands its structure, creates a plan, and begins building. It can generate environments, mechanics, user interfaces, assets, and Luau scripts. It then tests the result, identifies problems, and improves the game instead of assuming the first attempt worked. The agent follows a ReAct-style cycle: reason, act, observe, and decide what to do next. It also uses a Reflexion-inspired process to study failed attempts and turn them into guidance for the next iteration. The result is more than prompt-to-code generation. The agent translates a child’s intent into a tested experience they can play, evaluate, and continue shaping.

How we built it

The agent was implemented using Codex, a TypeScript and Express backend, and a safe Roblox Studio plugin. Codex implements planning, reasoning, tool selection, and coding choices. The backend manages the coordination of the process and provides 26 typed capabilities for inspecting projects, making instances, modifying properties, scripting in Luau, managing assets, and testing. The Roblox Studio plugin transforms the tool calls made by the agent into safe actions on the live project. It allows the creator to work in the Roblox Studio while enabling the backend to manage the agent. Our system constantly analyzes, plans, constructs, tests, reflects, and iterates. We have also developed 16 automated tests that verify syntax, asset loading, game initialization, movement, user interface, mechanics, performance, and win condition. In order to enhance security and reliability of our system, we have implemented confirmations for high impact actions, timeouts, rollback, quarantine of assets, and processing of animations and Meshy 3D asset pipeline.

Challenges we ran into

However, the most important problem that we have encountered was the problem of reliability. There are many interconnected subsystems in the game created in Roblox: scripts, physics, assets, UI, player interaction, goals and victory conditions. Even though the code is syntactically correct, it may fail because of lack of particular elements, wrong order of construction, mechanisms conflicts and so on. The way to deal with the issue was letting the agent watch the evidence rather than believe in the produced code. With the help of the ReAct cycle, the agent will perform the action, analyze its result and change its strategy accordingly. With the help of Reflexion, the agent will analyze the errors that were made during the process and consider them for the next attempt. Moreover, we had to get the balance between autonomy, safety and simplicity. The agent should be autonomous enough to be able to produce the games. At the same time, it should remain clear to the beginners.

Accomplishments that we're proud of

Wear our proud of our agent does not stop with the creation of the code. The prototype can perform analysis of the project in Roblox Studio, its understanding in terms of its composition, changes required, creation of the environment and game logic, generation of Luau scripts, testing the experience, detection of errors, fixing of the error, and validation of the results. Our technology employs 26 typed utilities, 16 automated tests, runtime verification, reflexion-based re-executions, rollbacks, timeouts, confirmation controls, asset quarantine, animation capability, and 3D asset management. The greatest success of our work in this project is the realization of our concept until playing experience level. Even a beginner developer can move from the idea to the game without having any experience in programming. For the perspective of Future Tech Foundation, the prototype demonstrates how powerful AI can be the creative tool available to kids with great ideas.As far as the Future Tech Foundation is concerned, our prototype shows that the AI can become a helpful tool to implement kids' ideas in their creative process.

What we learned

The truth was revealed that an effective AI developer does not need only to program. It has to be able to analyze the situation, develop a strategy, act purposefully, evaluate its result, realize the failure, and transform itself in order to apply a new strategy if the old one fails. The experience with ReAct helped us see how the process of game development can be split into several interrelated stages interacting with each other through the feedback received from Roblox Studio. The Reflexion made us understand that a failure can be a valuable source of data for the next stage. We got one more interesting piece of knowledge from this practice. We learned that access of children to the Future Tech Foundation is not only a matter of giving kids appropriate software. It must be safe, understandable, tolerant, and developed in a beginner language. An AI agent cannot replace the creativity of children or cover the process of learning. On the contrary, it has to be transparent, assist kids in decision-making, and encourage their desire to learn and create.

What's next for Hackathon Roblox AI Agent

The second objective is connected to the idea of developing a visual and creative educational agent. It is supposed to improve the ability to analyze screenshots, evaluate UI, observe the behavior of NPCs, create environment, produce higher quality animations, and work with complicated 3D models. In addition, the agent has to back up its propositions and changes in terms of language that would be clear for young creators. Children must remain active participants making decisions and not passive users of the automatic system. Within the framework of the Future Tech Foundation, we are going to use this instrument in workshops, receive the feedback of young creators, make the connection between the students and their mentors, equip them with necessary tools and access to competitions. The Foundation is going to supply us with the human factor that we cannot get from the software.Therefore, the final objective of our project will not be limited by acceleration of the process of Roblox creation. The long-term objective of our project will also include the help of children in becoming game creators themselves rather than players of other people's games.

Built With

  • agent
  • codex
  • gpt
  • hackathon
  • openai
  • roblox
Share this project:

Updates