Newby

Inspiration

we wanted to figure out some way of better using internal resources at a company because it is really inefficient to keep hiring hundreds of employees rather than using the ones they have. this system of hiring internally is just generally great because it gives mobility to the company and gives an extra reason for current employees to stay at their job because they will feel their skills are actually properly being put to use. this is why we wanted to create an application that can make hiring internally have less friction for human data pipelines

What it does

the apps we created are a combo where the first one is meant to track data on the tasks that the employees are doing, and use this to understand their skill sets so for example if an engineer is able extraordinarily quickly create prototypes in python, but is slow in java, then that is taken into account. then once we find the skillsets of people, you can then search through an internal system that manages which people are most experienced in what so that it is easier to find candidates for new positions on new teams. then if there are some people who have short coming in some area, we created an app that lets you take in markdown files, and use grok to generate interactive courses that create questions relavent to the course at hand. this combo lowers the friction of hiring internally exponentially because it first make it easy to find internal talent, and second makes onboarding potential talent streamline.

How we built it

well we first started with taking snapshots of the computer screen, which we then feed through a series of filtering steps to find relevant checkpoints which generate descriptions of the actions accomplished so that we can summarize the work they accomplished by using grok's vision model, and fast thinking model to generate what they are experienced in. then from there we put the data into a database where you can access it from the website we created. then once you use that website to find potential candidates, you can then use our other website to create courses by feeding markdown files with the relevant info on the topic that you want to engineers to learn. then you're off to the races on your new project with your interally hired team.

Challenges we ran into

finding a way to filter relevant images for the checkpoints. the reason this is hard is because basically, there are a lot of different ways that you interact with your computer and a lot of them are very difficult to find the best place to put a checkpoint. this is also paired with the problem of overcontextualizing the actions taken to the point that it is like "THE LETTER A WAS PRESSED." this is the reason why we created the checkpoint system (also to save tokens/compute).

Accomplishments that we're proud of

we're probably most proud of the fact that we not only could get the data collector done which required a lot of maneuvering around complex issue, but also managed to do the course creator app which can have extraordinarily broad data inputted and it can still make a pretty good course on the content given. (also personally I am a little proud of the check pointing system myself because it uses some really interesting computer vision tricks :) ).

What we learned

The main things we learned was definitely to create scalable infrastructure that can cut down large compute into smaller chunks because that is probably the most important part of our project is fixing the issue of using too many tokens. the other things we learned was how to think from a perspective of trying to optimize the human data pipeline so that people aren't going to waste when their talents could be used elsewhere rather thn being terminated

What's next for Newby

The next steps for this project is simple. create the least intrusive version of this possible because we don't necessarily need all of the data from the image, and a lot of people may see this as a privacy concern, so the next steps we can take is creating an ide level extention that collects just the data in the text editor that the person uses.