-
-
Calibration Hell when I was at lowes getting a baseboard
-
I have been doing some customized home assistant and embedded work in my personal life, and I could use something like this. able to clean and remember and actually understand what is going on. in the environment. I was always fascinated at the principles of 3d scanners and wanted to firmly believe that AI are able to improve the tools they can use to do work. Just like how we create scripts to automate deployment, or more. I decided to give an LLM the ability to iterate on itself by being able to control a robotic arm, use custom fine tuned CV models with synthetic data and real life data that can bridge the gap. ADA is able to scan objects close up with her arm to inspect items in her field. Then it gets processed by gather data and doing research, which gathers data from know parts and 3d model libraries, and is actually able to generate them via all sorts of connections. On her first try, she was able to identify a 8 year old arduino board, and build and compile a kicad schematic and export it perfectly!
Scanning items and finding them is one thing, but the most important part about any robot is its ability to see and manipulate. In order to see, she is able to use a synthetic dataset replication system that uses ray tracing and other details to bring the most realistic data as possible. She is able to spin up RGB, Depth, Pose, and theoretically reflections via ray tracing to plug into CV training models to grow her own 'vocabulary'. The CV Pipeline uses 6 different models, ranging from on device yolo, Dinov2, and some cutting edge open source pose predictions that allows Ada to even showcase a 'ghost' obj and check for overlap and accuracy. This then feeds into other ML algorithm pipelines.
For manipulation, using the generated 3d render a grasp physics simulation is attempted to try and find the best angle and interactivity for grasping the the object and building a database of optimal grasping strategies for manipulation.
CUDA Accelerated, distributed processing, and a heck of a lot of compute towards getting this jetson to be one of the best processors to run anything this town can see.
It was fun doing this one, and i wish I had more time to flush out the scanning algorithms and rendering parts, but I ran into multiple issues with doing the distributed networking, China ping for AMD compute, and most painful of all, real to sim sensor and arm calibration that makes it difficult to work in practice with cheap servos that are the absolute bane of my existence.
Log in or sign up for Devpost to join the conversation.