Inspiration
We were inspired by the demo General Motors showed us of the Robotorque application in real life.
What it does
Our solution first gets the robot translation and rotation from a depth image using pointclouds. We find the plane which represents the face of the head of the bolt and use the normal vector to that plane to calculate the angle which the bolt is oriented. We then use the points which lie on the plane's equation which we have found to find a center point which we use to get the bolts translation from the origin.
After we have the translation and rotation of the bolt, we focus on moving the robot arm to track the bolt head over time. Here, we use the last locations of the bolt to create a function of best fit, in our case going up to a two degree polynomial using least squares regression. We estimate where the bolt will be by extrapolating this function, and move our robot arm based on this predicted position.
Through these two steps we are able to track the position of a bolt as it changes using our robot arm and depth information from a LIDAR camera.
How we built it
We built our application in python, making special use of open3d and numpy.
From open3d, we used the PointCloud object to make finding a representative plane for the bolt efficient. Using this PointCloud object, we were able to sample thousands of test planes on the bolt and see which test plane contained the most points using a method of random sampling. Since the bolt head is the flattest part of the bolt with the most points lying on the plane which is flush with it, the ouput of this algorithm will be a plane which contains all the points on the head of the bolt.
We also used numpy to make our regression analysis of the bolt translation data more accurate. The polynomial fit tools included in numpy allowed us to effectively create a model for the bolt location over time and use that model to predict future locations of the bolt which follow a physics based swinging motion.
Challenges we ran into
One of the challenges we ran into was finding an accurate modeling function to predict the future location of the bolt. We used many different methods, starting with a simple formula of taking the first location of the bolt, taking the last location of the bolt, and drawing a line through those points to predict the next location of the bolt. While this was pretty accurate for situations in which the bolts translation was generally either decreasing or increasing, in situations where the general motion of the bolt oscillated it did not perform as well.
We also tried using the last four locations of the bolt to create a linear regression model for where the next location would end up. While this modeled general function behavior well, it did not seem to be able to get the peaks and troughs of the bolts oscillation well enough for the reward function.
We then tried numpy polynomial regression with a degree of 3. We found both this regression model to work the best for most of the cases which arise in this problem. The model takes into account all the previous points, and is able to capture the motion of the bolt well in most situations, though we still think there may be some trouble in cases where the general behavior of the function includes many local extrema, like if the bolt follows some quartic or quintic function which cannot be modeled in full detail by our third degree polynomial.
We also tried using the scipy package spline interpolation tool, which had the potential to be very good at modeling the oscillating short-term pattern of the bolt while also keeping up with long-term function behavior. Unfortunately, the short-term oscillations of our model overshot the actual oscillations of the bolt head, and so our values were less accurate than with the numpy method.
Accomplishments that we're proud of
We are proud of the method we used to find the translation and rotation of the bolt for the first segement of this problem. The solution is very elegant once we find the plane which represents the face of the head of the bolt. We first create a PointCloud from the depth image of the bolt. Then we find the representative plane by trying thousands of planes which intersect the points in the PointCloud, selecting the plane which contains the most points. We can then find a normal vector to this plane, which will point in the same direction as the bolt. After this, we just have to do some vector operations to get the angle between that normal vector and the one representing the board the bolt is on. To find the translation of the bolt, we again use our PointCloud and can average the x, y, and z values of every point which lies in the cloud, giving us the center of our bolt head.
What we learned
We learned a lot about matrix and vector math, especially as it used to model translations and rotations; the process of grabbing features from 3d images (like 3d depth images) using the PointCloud tool in open3d; and different regression techniques to model real-life movements of objects (like the oscillating motion of the bolt).
What's next for Robotorque
We would like to improve our modeling function techniqe, perhaps adding a lower and upper bounding function for the location of the bolt over time. We could then alternate between favoring the upper and lower bound function to simulate the type of oscillating behavior which we found hard to capture using a single regression function.
Log in or sign up for Devpost to join the conversation.