Instaboard

Instaboard lets workers record themselves doing day to day tasks, turns it into a step-by-step guide for new hires, and warns you when the guide goes out of date.

Comment

Inspiration

My inspiration for this project came from my own experience in starting a new job. I was responsible for managing sensitive employee data and I recall being nervous whenever I was assigned work to do as I felt that I would probably destroy whatever I was given responsibility for. Unfortunately, busy employees probably don't have the time to walk someone new through every single task. As a result, millions of employees each year suffer through the same uncertainty that I felt.

I had never used DataHub before this hackathon but it seemed like a really interesting product and I had wanted to learn more about it before I heard about the hackathon. Over the course of this project I gained a depth of understanding of the software that surprised even myself. During the last 3 weeks I ended up filing 8 contributions to the repo, including one fix to the MCP server that was merged.

What it does

Instaboard lets employees record how they do a task and it turns the recording into a step-by-step guide for future workers in the same role.

Instaboard comes in two modes, a dashboard and a chrome extension. Employees who are working on DataHub can simply download the chrome extension and hit record while completing a task that they want to save. Every page you navigate to becomes a step and you can then write a short note on each one saying why it exists. The agent fills in other information such as the real owners, the saved SQL and the history of the data from the live catalog.

When a new hire starts in a similar role they are granted access to the guide. They can replay it step by step next to real DataHub pages, tick off progress, and ask questions in chat. Answers are generated from the live catalog. They also have real table names and a URN that can be pasted into DataHub to confirm.

What differentiates DataHub from normal documentation is that it keeps re-checking each step against the catalog. When a column the guide relies on is changed or someone who was supposed to be pinged as part of the workflow leaves their job a warning appears. A proposed correction will appear that a human can either approve or deny.

How I built it

I mostly coded the entire project in TypeScript as it was the language I was most capable in. The app is Next.JS and React with plain CSS. The API routes, agent loops, decay engine and the benchmarks are all TypeScript libraries. I also used DataHub's GraphQL API directly for write-back and document read-back where the MCP server has no tools yet.

Instaboard has three interfaces on one loop. The Chrome extension can be used to record a task while you complete it in datahub, and an agent loop running on the official MCP server provides each step with its owners, SQL data and lineage before writing the finished guide into the catalog. Each step is broken into claims and every claim is pinned to the content of the catalog aspect that backed it. When a claim breaks, the finding becomes a catalog state and a warning is raised.

I also coded my own algorithm for calculating if a renamed column counts as a difference based on how similar the two strings are.

$$\text{score} = 0.75 \cdot \text{tokenOverlap}(a, b) + 0.25 \cdot \text{editSimilarity}(a, b)$$

It proposes above $0.55$ and refuses when the top two candidates sit within $0.1$ of each other.

To support my claim that grounding matters, I ran three passes against a live catalog for 20 questions that seemed common for new hires to ask. The results were $$18.0 \pm 1.7$$ out of 20 with DataHub, $$8.7 \pm 0.6$$ with only warehouse introspection and $$3.0 \pm 1.0$$ with no tools. The command npm run prove goes through the full loop and the CI I have established on github reboots a fresh DataHub on each push and tests the statistics.

Challenges I ran into

One thing I found very difficult was tweaking the renaming rule to be correct. I ended up having to use a lot of trial and error to get it right.

I also spent almost a week on a bug where the agent could not read its own writes. Documents that were saved through the MCP came back as just a URN. It turned out that the same server returns the full body over GraphQL, and the MCP layer was stripping the field. I ended up making a pull request to fix this on the official DataHub repo and it ended up getting merged.

Another minor quirk that I noticed while developing and caused issues was that some of DataHub's frontend routes didn't match their entity type names. I also pushed a fix for this to the repo.

Accomplishments that I'm proud of

One thing I am particularly proud of is the proof loop that I created to demonstrate how effective the system is.

I am also proud of how much I have learned about DataHub and software development as a whole from this experience. Before this, I had never used DataHub or contributed to open source, but by the end of the project I had managed to file eight contributions to the repo in response to bugs I found along the way.

What I learned

I learned a lot about how difficult it can be to debug agentic setups because of how non-deterministic they are. A problem can easily be fixes one minute only to reappear again due to the LLM responding in a different way.

I also learned a lot about budgeting LLM API calls. In the beginning I burned through a lot of tokens while testing the app and, once I switched to free providers I ran into a lot of rate limiting. It took a lot of time and patience to optimise the app to not use as much tokens.

What's next for Instaboard

First of all, given the positive experience I had making Instaboard I think I will probably transfer a lot of the data for the startup I am currently working on over to the software. One of my main reasons for doing this is because I feel that Instaboard could be really helpful for me. Even though we are only 3 co-founders, it is very common for one of us to spend a large amount of time on something like a data pipeline only to have to reteach it when someone else needs to take over. It would also help me to find any more gaps in the tool

Another idea I'm most excited about is creating an MCP server for Instaboard to be able to connect to other agents. This would allow any agent inside a company to answer employee questions based on the guide book. I think this is a part of software that is definitely growing so I would love to implement it.

Built With

Share this project:

Updates