Inspiration

Most computer-vision systems answer a simple question: "What is this person doing?"

We wanted HumanOS to answer a more useful question:

"What is happening to this person's state, why does the system believe it, and what should happen next?"

This inspired HumanOS Sentinel, a privacy-first human-state intelligence system focused initially on safety. Instead of relying on raw video or isolated action labels, HumanOS converts movement into structured skeletal signals and an evolving human state.

The DataHub Agent Hackathon gave us an opportunity to take this further. An AI agent can generate an explanation, but an explanation is much more useful when the agent can verify the data, model, lineage, ownership, and context behind a prediction.

That became the core idea of HumanOS Sentinel:

Detect → Investigate → Trace → Explain → Recommend.

What it does

HumanOS Sentinel monitors human movement locally and estimates an evolving risk state from skeletal landmarks.

The system tracks signals such as:

Body posture
Torso angle
Knee angles
Movement velocity
Stability
Risk trend over time

When the system detects elevated or increasing risk, a local AI agent investigates the prediction using DataHub.

Instead of simply saying:

"High risk detected."

the agent can answer:

"Why was this prediction made?"

It can trace the prediction through:

pose_landmarks
      ↓
motion_features
      ↓
humanos-risk-v1
      ↓
human_motion_events

The agent uses this context to explain the evidence, identify the relevant model and ownership information, highlight potential limitations, and recommend an appropriate next action.

The system is designed to be privacy-first: raw camera frames are processed locally for pose extraction, while downstream intelligence works with skeletal and structured state information.

How we built it

We deliberately kept the architecture lightweight and local.

HumanOS perception layer

We use MediaPipe Pose to extract skeletal landmarks from the webcam in the browser.

These landmarks are transformed into motion features such as posture, joint angles, velocity, stability, and temporal trends.

Risk engine

Instead of asking the language model to perform numerical risk calculations, HumanOS uses deterministic feature processing to produce a structured state.

For example:

{
  "posture": "unstable",
  "stability": 0.31,
  "risk": 0.78,
  "trend": "increasing"
}

This keeps the core signal predictable and gives the AI agent a clear boundary.

Local AI agent

We run Qwen 2.5 7B through Ollama locally.

The LLM is responsible for reasoning, investigation, tool selection, explanation, and recommendations—not computer vision or numerical risk calculation.

DataHub context

DataHub runs locally through Docker.

We created HumanOS metadata entities representing datasets, models, events, and their relationships. The agent accesses this context through DataHub MCP.

This allows the agent to investigate questions such as:

Which model produced this prediction?
What data does the model depend on?
What is the upstream lineage?
Who owns the model or dataset?
What context should be considered before acting?

The result is an agent that reasons over both the live human state and the data context behind that state.

Architecture

Webcam
   ↓
MediaPipe Pose
   ↓
Skeletal landmarks
   ↓
HumanOS State Engine
   ↓
Risk Signal
   ↓
Local AI Agent
   ↓
DataHub MCP
   ↓
Metadata + Lineage + Ownership
   ↓
Explanation + Recommendation

The entire stack is designed to run locally without requiring a paid cloud AI API.

Challenges we ran into

Making DataHub meaningful

Our biggest challenge was avoiding a simple chatbot with a DataHub integration.

We wanted DataHub to be part of the actual reasoning process. The agent therefore uses DataHub to investigate the origin and context of predictions instead of simply displaying metadata.

Keeping the AI grounded

A language model can produce a convincing explanation even when it has incomplete information.

We addressed this by separating responsibilities:

HumanOS handles motion and risk computation.
DataHub provides authoritative metadata and lineage.
The LLM handles investigation and natural-language reasoning.

This makes the system easier to understand and reduces the risk of the model inventing technical context.

Running everything locally

We wanted the project to be privacy-first and offline-capable.

That meant integrating:

MediaPipe
+
HumanOS
+
FastAPI
+
Ollama / Qwen
+
DataHub
+
MCP

without relying on external AI APIs.

Getting these components to communicate reliably while keeping the architecture simple was one of the main engineering challenges.

Building a meaningful prototype quickly

HumanOS has a much larger long-term vision involving more advanced temporal modeling, prediction, reasoning, and multiple application domains.

For this hackathon, we focused on one complete workflow rather than trying to implement the entire platform:

Observe → Detect → Trace → Explain → Recommend.

Accomplishments that we're proud of

We built a genuinely agentic DataHub workflow

The agent doesn't simply answer questions about metadata.

It uses DataHub context to investigate a real prediction and construct an explanation around it.

We kept the AI stack local

HumanOS Sentinel can run with:

Open-source/local AI
Ollama
Qwen 2.5 7B
Local DataHub
DataHub MCP
Local computer vision

No paid LLM API is required.

Privacy is part of the architecture

Raw video does not need to become the system's persistent data layer.

The perception boundary is:

Camera
   ↓
Pose extraction
   ↓
Skeletal representation
   ↓
Downstream intelligence

This allows the system to reason about movement while minimizing exposure of raw visual data.

We connected physical-world signals to data intelligence

One of the things we're most proud of is connecting two traditionally separate worlds:

human motion intelligence and data/metadata intelligence.

HumanOS produces the signal.

DataHub provides the context.

The agent connects the two.

What we learned

The biggest lesson was that agentic AI is not just about generating better answers—it is about giving agents the right context to make those answers meaningful.

A model can tell you that a risk is high. DataHub can help an agent understand where that prediction came from.

We also learned that a strong agent architecture does not require the LLM to control everything.

Keeping perception and numerical risk estimation deterministic while giving the LLM responsibility for investigation and reasoning produced a much clearer architecture.

Most importantly, we learned that lineage can become an input to reasoning, not just something developers look at in a metadata UI.