About the Project

Inspiration

Modern AI datacenters contain thousands of interconnected components: GPUs, CPUs, memory modules, network switches, power supplies and cooling systems. Yet much of this infrastructure is monitored and optimised through separate software systems, each focusing on a relatively narrow part of the datacenter.

We wondered: What if an entire datacenter could operate as one intelligent, self-optimising system?

Rather than simply monitoring GPU utilisation or detecting hardware failures, we wanted to build a platform that understands the relationships between physical infrastructure, computational workloads, energy consumption and operational efficiency.

Our vision was to create an autonomous operating system for AI datacenters, where specialised agents monitor different components, collaborate to identify inefficiencies and continuously improve resource allocation.

What We Built

We developed the foundations of an AI-native datacenter management platform, designed to unify infrastructure monitoring, anomaly detection and intelligent decision-making.

The platform is built around several key concepts:

Unified infrastructure model: A structured representation of datacenter components, including GPUs, CPUs, memory, network connections, power systems and cooling infrastructure.
Infrastructure dependency graph: Relationships between components allow the system to understand how problems in one part of a datacenter might affect another.
Hierarchical AI agents: Specialised agents can monitor individual components or racks, while higher-level agents coordinate information and make broader optimisation decisions.
Telemetry analysis: Hardware measurements can be processed to identify anomalies and reduce the volume of information requiring AI interpretation.
Fine-tuned AI models: We explored using Adaption to develop specialised models for infrastructure diagnostics and operational reasoning.

The long-term objective is to move beyond reactive monitoring towards autonomous optimisation of computational throughput, energy consumption and infrastructure reliability.

How We Built It

We began by designing a universal data schema capable of representing the diverse hardware and software systems within an AI datacenter.

Rather than treating every GPU, switch or power supply as an isolated object, we designed the system around entities and relationships. This creates the foundation for a digital twin capable of connecting physical hardware with the workloads running on it.

We then explored a hierarchical multi-agent architecture in which local agents analyse specific infrastructure components and communicate relevant findings to higher-level management agents.

A central design decision was to separate deterministic processing from AI reasoning. Routine telemetry processing, validation and anomaly detection should not require expensive language-model inference. Instead, AI agents should focus on interpreting unusual behaviour, investigating complex dependencies and recommending interventions.

We also experimented with fine-tuning smaller language models through Adaption. Our initial work focused on GPU telemetry and root-cause diagnosis, before exploring real-world system-log datasets as a more reliable source of labelled training examples.

Challenges We Faced

1. The complexity of datacenter infrastructure

Different hardware manufacturers expose different measurements, interfaces and control capabilities. Creating a universal representation required us to distinguish between what can be observed, what can be controlled and what must be estimated.

2. Making AI models genuinely better through fine-tuning

One of our biggest lessons came from an unsuccessful fine-tuning experiment.

Our initial dataset contained synthetic GPU telemetry scenarios intended to teach a smaller model to identify hardware problems. However, the resulting model underperformed our expectations.

We realised that generating more examples does not necessarily produce a better model. Training-data quality, reliable labels and appropriate evaluation matter far more than simply increasing dataset size.

This encouraged us to investigate real infrastructure datasets, including recorded GPU incidents and labelled distributed-system logs.

3. Generalising across different hardware

A GPU operating at a particular temperature or power level might be perfectly healthy under one workload but abnormal under another.

We recognised that models trained on absolute telemetry values could easily learn misleading relationships.

This led us towards approaches based on relative performance, hardware-aware baselines, dependency graphs and verifiable operational data.

4. Balancing autonomy with reliability

Giving AI agents control over physical infrastructure introduces serious operational risks.

We therefore designed the architecture around deterministic safety checks, simulation and controlled execution rather than allowing language models to make unrestricted hardware changes.

What We Learned

The most important lesson was that the value of AI in infrastructure management comes from making better decisions, not merely processing more data.

A conventional monitoring system can report that a GPU is underutilised. An intelligent infrastructure platform should investigate whether that underutilisation is caused by memory constraints, network congestion, workload scheduling or another component.

We also learned that fine-tuning requires rigorous benchmarking. A specialised model is only valuable if it demonstrates measurable improvement over its original base model and existing deterministic alternatives.

Finally, we gained a deeper appreciation for how computational performance depends on the interaction between hardware, software, networking, power and cooling.

What's Next?

Our ultimate goal is to build a self-improving datacenter operating system.

We want to expand the platform to integrate real hardware telemetry, simulate infrastructure changes and develop agents capable of optimising workload allocation across entire GPU clusters.

Future work includes creating better infrastructure-specific training datasets, developing workload-aware scheduling models, and evaluating autonomous optimisation against conventional resource-management algorithms.

Eventually, we envision a platform that can manage heterogeneous datacenters across different hardware manufacturers, continuously learning from operational outcomes.

The ambition is simple: make every unit of compute, every watt of electricity and every piece of datacenter infrastructure work as efficiently as possible.