Sentinel

meowl

Inspiration

When tasked with the challenge by NextEra Energy, our team took initiative in translating raw nuclear inspection telemetry into actionable insights. NextEra deploys fleets of SPOT, robots capable of taking and relying automated inspection data.

The fact that data is often messy, difficult to parse, and impractically voluminous (for humans) supplies us with a nascent vantage point as engineers: architecting systems that combine deterministic computation with agentic machine learning. In doing so, we aspire not only to transform vast streams of unstructured telemetry into structured, evidence-backed insight, but leverage artificial intelligence to amplify human analytical capacity and extract actionable insight from information that would otherwise be impossible to interpret at scale.

So naturally, transforming nuclear inspection data from Boston Dynamic's SPOT robots into actionable, event-driven histories is the perfect flagship for such an endeavor.

What SENTINEL Does

Sentinel takes raw data from mission statuses, task results, sensor readings, flagged anomalies, and event logs relayed by Boston Dynamics' SPOT robot, processes it through our API layer, persists in our database, and relays to the frontend in order to help the operator understand the data in depth.

It becomes easier to understand missions when events are mapped to their implications, like:

Recurring Error Patterns associated with the same machinery, same reading failures, and same mission durations,
Conversational Agents to run investigations, summarize the data for you, and explain interrelated trends,
Interactive Maps and Timelines that represent the mission moment-to-moment.

How we built it

The idea behind SENTINEL was to take the initial data csv's we were given as the data model while building a scalable architecture centered around it. At the core of the project is a separately hosted FastAPI layer, which ingests event payloads and maps them to data transfer objects before persisting them in our Databricks Lakebase (PostgreSQL). This is modeled after the Orbit Webhook subscriptions and API documentation, as companies deploying SPOT fleets will have a real Orbit instance. On ingestion, events-- actions that SPOT completes in the course of a mission, go through deterministic logic gates to optimize a priority queue. For example, multiple errors on the same asset may indicate performance degradation; this surfaces it to the administrator faster. The controller service API architecture allows us to take in a lot of other data shapes and payloads, like from the SPOT SDK while staying true to the original data schema, and acts as a perfect broker for a multi-tenant frontend (Next.js and Vercel for Web and Expo and React for iOS).

Challenges we ran into

The main challenge was optimizing a core deterministic ruleset, followed by orchestrating agentic intelligence. For example, we had initially mapped anomalies based on readings of the same type, independent of asset type: a crucial oversight. Instead, establishing baseline equipment ranges needed to be based on that specific asset's historical IDs and the reading type. The frontend also went through many design reworks, with initial markups in Figma, a few UI library switches and theme updates, and lots of trial and error. Finally, working with LangGraph to deploy an agent via the Databricks backend was very interesting; architecting state transfers opened the door to a lot of meaningful conversational navigation and data investigation.

Accomplishments that we're proud of

Being able to have a fully working product end-to-end that we feel really solves the user's issue was a massive accomplishment. Taking something that perhaps felt so complex and really pointing out what needed to be added, removed, or fixed to make such a smooth and rich user experience.

We are incredibly proud of our development of our full-stack app, integrating APIs for our database and agents into our application, and fleshing out our front-end was a massive ordeal that became worth the trouble.

What we learned

Full-Stack Architecture: We challenged ourselves with the implementation of databases, Agents, and a mobile and web version of our app. Sequence flows needed to be perfectly understood and APIs needed to be vetted, but if that was done right the work flowed expeditiously.
UI/UX: We redesigned our entire interface and features to fit the needs of the user, such as the timeline feature or the addition of a conversational agent to the dashboard which significantly improved usability.
Quality Data: The shape of our data sparked many discussions and changes to our project, this lead to investigations of Exploratory Data Analysis, Synthetic data generation to strengthen our database and interactions at a user and agent level. -Orchestration With LangGraph: This certainly beats manually defining rules to store and compress conversation history, retrieve database context, and sent elaborate prompts on every API call. We can't wait to see what we can explore with this going forwards.

What's next for Sentinel

Our long-term vision is to evolve beyond conventional inspection monitoring into an intelligent, agentic platform capable of autonomously coordinating robotic fleets, interpreting complex telemetry, and extrapolating meaningful relationships from otherwise fragmented operational data.

Future development will incorporate agentic fleet management and dynamic route optimization, leveraging scheduling algorithms and operational constraints to intelligently prioritize and coordinate inspection missions. Alongside this, we aim to integrate autoencoders for unsupervised anomaly detection, XGBoost for predictive risk classification, and statistical methods for identifying temporal correlations and relationships across equipment and sensor histories.

Perhaps most significantly, we intend to develop structured agentic investigations: orchestrating deterministic analytical systems with the interpretive capabilities of large language models to formulate hypotheses, interrogate historical evidence, identify potentially causal relationships, and synthesize substantiated conclusions. This is something we scraped on in our v 0.01 scope.

Ultimately, Sentinel aims to transcend simply reporting validated anomalies, instead developing a more comprehensive operational intelligence that can contextualize what happened, investigate why it happened, anticipate emerging risks, and autonomously optimize the processes through which those risks are investigated.