FaultLine

FaultLine is a coding agent that triages sick GPU nodes: Nemotron on Nebius writes and runs analysis code on kernel logs and telemetry, then recommends reset, drain, or RMA with evidence.

Comment

Inspiration

I work on a platform that monitors and repairs GPU clusters. Its job is to spot a failing GPU node, find the root cause, and pick the right repair: reset, reboot, return the card, or hand the problem back to the job owner. Triaging faults with conventional software code is hard. Every new log format, driver version or failure pattern needs new rules, and the rules keep breaking. I believe a coding agent can do this more reliably. It writes code to analyze the logs, finds the root cause from evidence, and recommends the repair. This hackathon was my chance to build that.

What it does

FaultLine takes an alert for a failing GPU node, plus its logs and sensor readings. It finds the root cause and recommends a repair: reset, reboot, return the card, check cooling, or hand the problem back to the job owner. The agent writes its own code to read the logs, so no one has to write a reader for each new format. It checks that code before using it, saves it for reuse, and repairs it when a format changes. It only recommends actions and doesn’t carry them out.

How we built it

Nemotron on Nebius Token Factory is the brain. It has tools to write log readers, run analysis code in a sandbox, look up error codes, and submit a report. Each new reader is checked against the raw file, reviewed by a second model call, and then saved to a library for reuse. Every number in the final report must come from code the agent ran. A small web app shows each step live. Since I had no real fleet data, I built simulated incidents: 3 log formats and 7 fault types.

Challenges we ran into

The agent’s code often looked right but misread the data, so I added strict automatic checks. Some faults print no error at all, so the agent must now review sensor readings before answering. When I asked what happens if NVIDIA changes its log wording, I found my checks were tied to today’s format, so I rebuilt them. Testing this also exposed a reader that silently returned zero for a key sensor, which a new check now catches.

Accomplishments that we're proud of

On 42 simulated incidents, the agent was right 40 to 42 times across three full runs. When I changed the error wording, it repaired its own reader and answered correctly in 7 of 8 first tries, and the old format kept working. Reusing a saved reader costs about four times fewer tokens than learning a new one. It also learned to read three real error lines from public GPU documentation.

What we learned

Don’t trust code a model writes until something automatic checks it against the real data. A second model reviewing it helps, but it can be wrong too. Reusing checked code makes the agent faster, cheaper and more consistent. A failing GPU doesn’t always print an error, so the agent must read sensor data too. Results vary slightly between runs, so I report ranges.

What's next for FaultLine

Test on real fleet logs with labeled incidents. Extend the same approach to other hardware logs. Add fleet-wide pattern detection, an approval step before repairs, and stronger isolation for the code the agent runs.

Built With

  • fastapi
  • html/css
  • javascript
  • nebiustokenfactory
  • nemotron
  • openai
  • openaisdk
  • pydantic
  • python
  • server-sent-events
  • uvicorn
Share this project:

Updates

Submission history