Inspiration

Slowdowns in data centers frequently trace back to SSDs, but pinpointing the cause is difficult. Heavy write workloads and failing drives produce nearly identical symptoms, such as elevated latency and reduced throughput, yet require very different responses. This ambiguity slows down engineers’ response times and prolongs degraded service for users. Our project addresses this by building a system that diagnoses the root cause of SSD slowdowns, a dashboard for engineers to monitor node health, and an optional automated remediation mechanism that applies the appropriate fix once a cause is identified.

What it does

This project collects live measurements from Linux drives or a simulator, then uses a trained random forest to distinguish healthy operations, heavy writes, and suspected drive faults. SpacetimeDB streams those assessments to a React dashboard where operators can follow fleet trends, inspect individual drives, and see what to check first. The simulator can replay up to 10,000 nodes for demos and load testing.

How we built it

We used the React.js framework with the Spacetime platform to create the web-app dashboard and store data from the AI model. For the AI model, we used various Python libraries (FastAPI, scikit-learn, Numpy, joblib) to train the model and produce results. To migrate the solution to a mobile web-app, we used a Cloudflare CLI utility called cloudflared to expose our product to the internet. Our general dataflow path for this project was to take simulated data and pass it to the FastAPI Random Forest Model, and then send the classified output of that model to the SpacetimeDB database, where we then use ReactiveX to parse the data into a dashboard for the end-user.

Challenges we ran into

The main challenge we faced was finding a specific issue that data centers face without a concise solution. We started with the idea of temperature monitoring which as we did more research we came to realize has already been implemented through products like hyperview. We found sources from people who work in data centers and read what issues they face and how it is dealt with. We went through what we found and cut it down to what problems didn’t have wide spread solutions. We attended the NMC^2 workshop to hear more first hand perspectives and used the opportunity to bounce our ideas off of the speakers and see if their company faces these issues and how they are dealt with.

Accomplishments that we're proud of

We're proud of being able to successfully create a web-app that multiple users can use at the same time to check on the SSD slowdown issues in data centers. We are also proud of researching more about unsolved issues that data centers encounter, that being SSD slowdowns, and contributing to a solution for this problem.

What we learned

We learned more about how data centers work as a part of researching for this project, as well as some of the issues that are commonly encountered in data centers, whether this is physical issues such as temperature control or software issues such as what we tackled, SSD slowdowns. Beyond data centers, we also learned a lot about the tech that we used for this project. For all of us, this was our first time using SpacetimeDB and setting up an AI model. For a few of our group members, this was also their first hackathon project experience which allowed them to learn more about project production from end to end.

What's next for WriteOrWrong

Next steps for this project would start with optimization of the dataflow and code. Many parts of the project could benefit from refactoring to make the code/names more representative of the data/functions they store. As well, improving in-code documentation for future maintenance is a crucial next step, especially if this were put into actual production. This project is currently running on mock telemetry data, so a major step would be to connect this to actual data center telemetry data. This would allow us to test and refine the flow of the program based on the requirements from actual data centers and receive feedback from data center engineers to see what is working as intended and what could be improved. Because the corrections are also simulated, they do not truly reflect the actual solutions for SSD slowdowns. A next step might be performing more research on the implementations of these solutions from a programming perspective (such as doing research into network protocols and how they handle flood control) and basing our architecture on similar design principles.

Built With

  • react
  • reactivex
  • spacetimedb
Share this project:

Updates

Submission history