Inspiration ✨

We have long been able to manipulate humanoid models inside of computer programs but having the models move themselves has never worked quite right, or at least not autonomous enough to represent human behavior. This novel network loop shifts the paradigm and allow different modes to move themselves based on input from the scene, and when extended out to real world problems such as moving a prosthetic limb, this project can provide benefits beyond the video game realm.

What it does 🔍

Ks-Net takes input from the world and extends that into Nemotron, using data from the scene in a spacetime database, and then outputs a goal post for unity to generator torques to interpret smooth motion and an overall realistic movement sequence for a model. This combined with a processing loop or an Nemotron agent allows the model to think for itself and mimic human behavior empowered by the torque interpretation system.

How we built it 🛠️

First plotting out the diagram of how the model communicates with the other components and the environment, we planned using Argon, and the Unity scene, and explaining to Antigravity how to build it providing constant feedback, stepping in when necessary to ensure the project remained in our vision and not AI slop.

Challenges we ran into 🧩

Nemotron was not trained on matching poses to frame data, so we had to manipulate its responses to get the pose data in the format we needed with the proper reasoning behind it, a specifically trained model that has been given pose datal with labels combined with Nemotron could work much better, but the time allotted was not enough to create the dataset and train the model on it.

Accomplishments that we're proud of 🏆

It's a success, the model can successfully generate position, apply animation principle to figure out on between poses and interpret the torque needed to get form pose A to pose B. And also the model can reason what the next action should be in the environment based on the data its given and work in a continuous loop.

What we learned 💡

  • Animation principles
  • Physics dampening equations
  • Rigid body physics
  • Integrative biomechanics design

What's next for Ks-net 🚀

  • Running a smaller Nemotron model locally
  • Increasing physics realism
  • Custom models
  • Training model on poses with labels

Built With

  • antigravity
  • nemotron
  • spacetime
  • unity
Share this project:

Updates

Submission history