ICP Splunk AI Observability Agent : Splunk Enterprise

• 

  Spunk AI Observability Solution Poster - ICP Curtin

• 

  Observability App Screenshot

• 

  Splunk Environmental Dashboard

• 

  Splunk Network Monitoring Dashboard

• 

  Meraki MT sensor dasboard

• 

  Openrouter LLM Avticity Monitoring

Inspiration

At Innovation Central Perth (ICP), part of Curtin University, Perth, Western Australia - a Cisco NIIN partner, we been running a Splunk Enterprise instance on-premise to monitor our lab's environment and network operations (Meraki MS355 switch) and some MT sensors and MV12W cameras. This hackathon presents an opportunity to further explore, develop and skill-up to leverage more of Splunk's tools, especially AI and ML tools to automate observability of our assets.

What it does

We built a closed-loop AI observability platform for ICP at Curtin University using live Cisco Meraki hardware integrated with Splunk Enterprise and multiple LLMs via OpenRouter.

The AI Loop:

1. Collect — Cisco Meraki MV cameras (people counting), MT10 environmental sensor (temp/humidity), and MS355 switch (traffic, PoE) stream real data into Splunk Enterprise via HEC every 2 minutes
   1. Detect — Splunk MLTK AI (anomalydetection + predict/LLP5) runs statistical outlier detection and temperature forecasting on ingested time-series data
2. Interpret — LLMs (GPT-4o mini, Llama 3.3 70B, Gemini Flash 2.0) generate contextual incident reports that explicitly cite Splunk AI findings
3. Act — The Automated Response Engine creates Splunk saved alerts via REST API and logs icp:automated_response events back to Splunk — closing the loop

How we built it

We started with sandboxing our solution using spare equipment - once we were pretty confident of our scripts, we migrated over to our live network and environmental monitoring kit. Quite a bit of project frameworking and code development were with assistance of Claude Code (Sonnet 4.6) - we "almost" had Fable 5 before it got pulled :(

Very excite that we finally have live deployment: Running on real hardware at Innovation Central Perth, Curtin University, Perth WA.

Challenges we ran into

As the Splunk and Meraki API documentation were well developed. Most of the issues we had we were figuring out the ID and addresses of specific Meraki devices to ingress into our app. Also in preparation for resilient (e.g. fallback model) and cost effective operations, we decided on using OpenRouter to manage some of the cost and up-time requirements.

Accomplishments that we're proud of

Key Innovation: Splunk is not just a log sink — it's the AI brain. Every LLM response is grounded in Splunk MLTK analysis. Every anomaly automatically triggers an LLM-written incident report AND creates a persistent Splunk saved alert. The entire response cycle is logged back to Splunk for audit and feedback.

What we learned

Lots! Quite often at ICP, we work with open source hardware and software e.g. for indstrial application POCs with final year engineering/CS undergards (or Masters students) - integrating with our "Enterprise" equipment for a hackathon/POC required navigating licenses etc ... was a good experience for us.

What's next for ICP

Keep this AI powered Observability solution running and keep iterating to improve user and management support functions.

Built With

• gemini
• llama
• meraki
• openai
• openrouter
• python
• splunk
• ubuntu