-
-
The Technical Architecture Diagram (The Brain)
-
The Operational Workflow Diagram (The Job)
-
MaryOS Local Execution
Inspiration & Vision
Facility management has historically been a reactive, human-dependent chore—stuck chasing critical asset failures after they happen. The inspiration behind Mary Catalyst (MaryOS) was to transform this paradigm completely by building an autonomous, local-first Agent Operating System (Agent OS). Rather than acting as a simple Q&A chatbot, Mary sits in the agent orchestration layer to read context, plan multi-step tasks, query local knowledge databases, and execute real-time operational actions (such as drafting work orders, triaging telemetry, and verifying vendor SLAs) with total data sovereignty and zero cloud dependency.
What it does
MaryOS (Mary Catalyst) serves as an autonomous, local-first Agent Operating System (Agent OS) designed to replace high-friction administrative and operational workflows in enterprise facility management. Rather than acting as a simple Q&A chatbot, Mary operates directly within the agent orchestration layer to perform real-time, multi-step tasks:
Real-Time MQTT Telemetry & Topic Subscriptions: Integrates an Eclipse Mosquitto MQTT broker pipeline to handle lightweight, publish-subscribe messaging. Mary continuously listens to live sensor feeds and device streams across real-time facility topics to track asset health instantly.
Autonomous Telemetry & Anomaly Triage: Automatically runs batch telemetry checks for critical assets (such as evaluating live Wh readings against micro-hourly load curve baselines) and triggers immediate preventive isolation directives when electrical anomalies breach safety thresholds (e.g., ThingsBoard RAG asset profiles crossing 50% deviation limits).
Multi-Layer Knowledge Retrieval & Reasoning: Queries local vector databases, site-specific metadata, and historical event logs to instantly pull up standard operating procedures (SOPs), equipment warranties, and past failure histories.
Operational Action Execution: Automatically plans and executes operational outcomes—such as drafting precise work orders, pre-filling vendor dispatch emails, and setting strict SLA response timers.
Seamless Multi-Modal Communication: Supports flexible multi-modal interaction channels pairing local speech-to-text (Whisper tiny) and text-to-speech (Kokoro TTS) so users can talk or type depending on their environment.
Hands-On On-Site Operational Control: Empowers facility managers with direct, hands-free command capabilities over critical building infrastructure and receives real-time audio status updates directly on the plant floor
How We Built It & Architecture
The development journey of Mary Catalyst was an intensive exercise in local-first AI optimization: MaryOS was engineered from the ground up as a Type 3 Vertical AI Agent, balancing deep industrial domain intelligence with lightweight local-first execution:
The Core Brain & Hermes Architecture: Built around fine-tuned Hermes models leveraging a massive 131k token context window (131,072 tokens) to ingest multi-page building blueprints, complex vendor contracts, and extended operational dialogs in a single pass.
The Four-Layer Memory Architecture: fmmc.vectors.db (Local Vector Database) for fast, offline semantic search across Facility Management Master Content (FMMC).
facility-profile.json (Site State & Metadata) mapping equipment serial numbers, vendor rosters, and lease boundaries.
Historical Data Logs for pattern recognition and predictive maintenance.
Raw FMMC Documentation for deep context extraction.
Fine-Tuning & Quantization Pipeline: Used the Unsloth framework to fine-tune our base model into Mary Catalyst (mary-catalyst-q4_k_m.gguf), optimized via 4-bit quantization to run efficiently on standard 8GB–16GB RAM client hardware without OOM errors.
Systems Engineering Protocol (SEP.md): Established a master systems engineering blueprint to govern Mary’s response structures, locking down strict behavior guidelines for zero-latency operational decision-making.
Local Edge Audio Pipeline: Integrated Whisper tiny (int8 format) for instant, offline voice command transcription and Kokoro TTS for high-fidelity audio synthesis.
Challenges Faced
Hardware & Resource Constraints: Operating a high-performance 8B model locally within tight memory constraints meant balancing quantization levels (GGUF Q4_K_M) against generation speed and system headroom.
Precision vs. Latency Tradeoffs: Facility management requires instantaneous reaction times. Ensuring that Mary could parse complex batch telemetry queries (like evaluating hourly power anomalies) without sacrificing speed or precision required careful prompt structuring and deep framework tuning.
Accomplishments that we're proud of
Successful 8B Edge Optimization: Successfully fine-tuned and deployed an 8B GGUF-quantized model (mary-catalyst-q4_k_m) that runs efficiently on consumer hardware while delivering precise industrial actions.
Rigorous Benchmark Execution: Implemented a robust local testing and profiling pipeline, establishing stable execution times and strict behavioral constraints through our custom master SEP.md protocol.
Bridging AI with Real Infrastructure: Successfully translated deep domain expertise in facility management into structured, functional AI prompts and RAG workflows that solve actual operational bottlenecks.
What we learned
The Power of Structure over Size: We learned that specialized system protocols (SEP.md) and precise domain fine-tuning via Unsloth can dramatically sharpen an 8B model's output quality, making it behave like a much larger, highly specialized enterprise asset.
Balancing Latency and Precision: Facility management demands speed. We gained deep insights into tuning quantization parameters (Q4_K_M) to achieve the optimal balance between high inference throughput and absolute operational accuracy.
What's next for MARY
Full Production Field Deployment: Scaling MaryOS from local simulation environments into live commercial properties for continuous, automated building telemetry monitoring.
Advanced Tool-Calling Execution: Expanding automated workflows for direct one-click vendor dispatching, automated purchase order generation, and dynamic SLA response timers.
Predictive Maintenance Loops: Deepening historical anomaly tracking so Mary can autonomously predict equipment failure and issue preventative work orders before faults occur.
Log in or sign up for Devpost to join the conversation.