LIBERO

Because no one should get lost. Not even the caregiver.

Inspiration

My grandmother experienced mental confusion before developing dementia.

I saw how dementia affected not only her, but also my family. I watched my mother navigate the fear, exhaustion, and sleeplessness that can come with caring for someone you love while trying to keep them safe. I began to understand that caregiving is not just a collection of tasks. It is an ongoing responsibility that can reshape an entire family's life, often in ways that remain invisible to everyone outside it.

Then, a tragedy in my city made the risks of that responsibility impossible to ignore.

A man living with Alzheimer's left home while his caregiver had stepped away for a brief errand nearby. He became lost. Twenty-nine days later, he was found in a wooded area, and he had died.

The caregiver had not intended to leave him alone for long. Ordinary life had demanded attention for a few minutes, and those minutes became part of a tragedy.

I cannot know whether technology could have changed that outcome, and I will never claim that Libero would have saved him. But the case forced me to confront a question that had already been growing in my mind: How can families be expected to provide constant care in a world where even a brief moment of inattention can carry serious consequences?

That question became the starting point for Libero.

I also wanted to challenge the assumption that supporting someone with dementia means simply reminding them of facts, correcting their confusion, or tracking their daily tasks.

Imagine someone living with dementia saying, "I need to go home."

A system might answer that they are already home. But what if the words are expressing fear, disorientation, or a search for familiarity? Correcting the statement does not necessarily address the feeling behind it.

I wanted to explore a different approach: technology that considers conversational context, preserves meaningful personal stories, and helps families revisit moments that might otherwise disappear into an endless sequence of daily responsibilities.

As I worked on Libero, I sometimes felt a painful sense of guilt. I found myself wondering why I had not built something like this earlier, when my grandmother needed care. I wished I had known then what I am learning now. I wished I had already possessed the tools and understanding I am developing through this project.

I also had to confront a difficult truth: I cannot go back, and I cannot know whether Libero would have changed my grandmother's experience. Turning that wish into a promise about what technology could have done would not be fair to her, to my family, or to the families I hope to support.

What I can do is learn from what we experienced and use that understanding to build something that may be useful to others.

That realization changed the meaning of this project for me. Libero became more than a technical challenge. It became an attempt to approach a deeply human problem with both engineering discipline and humility.

Because no one should get lost. Not even the caregiver.

What it does

Libero is a functional research prototype designed to explore how AI can support dementia care at home. It brings together contextual conversation, personal memory, experimental safety analysis, and a browser-based interface for reviewing interactions and care-related information.

Its central idea is simple: a reminder tracks a task; Libero explores how to preserve the human context around a conversation.

1. Context-aware conversational support

Libero can process conversational input and generate supportive responses through its orchestration layer. Conversation history and saved memories can provide context for subsequent interactions.

For example, when someone says, "I need to go home," the goal is not merely to correct the statement. Libero explores a more context-aware response inspired by validation and reminiscence principles.

These principles inform the design of the system; they do not establish clinical effectiveness. Libero has not been clinically validated.

2. Personal memories that can be revisited

Libero includes a memory component that can identify and preserve meaningful personal stories.

Consider someone sharing:

"When I was young, I danced with my husband at the club."

In the current prototype, a personal story can be saved to local storage and reviewed through the Memories interface. The conversation can therefore produce a record that remains available beyond the immediate interaction.

The purpose is not to replace family memories or the relationships that give them meaning. It is to explore whether technology can help preserve details that families may want to revisit.

3. Experimental safety analysis

Libero also includes an experimental safety pipeline for selected events and situations related to falls, wandering, agitation, and possible emergencies.

The implementation includes contextual text analysis, audio-related processing, and vision-related components. A four-level decision model organizes the system's internal response into:

Observation
Conversation
Intervention
Emergency

The decision logic considers contextual factors such as negation, past events, and references to other people. A risk-related word should not automatically be treated as evidence of an immediate emergency.

For example, "I fell yesterday," "my father fell years ago," and "I am not in pain" should not be interpreted as identical situations simply because they contain words associated with risk.

These mechanisms are experimental heuristics, not a clinically calibrated safety model. They can generate false positives, miss important events, or misinterpret ambiguous input.

4. A family-facing interface

Libero connects its backend to a browser-based interface for interacting with the system and reviewing conversations, saved memories, and care-related records.

For the hackathon, the main demonstration focuses on a real software flow: a message is submitted through the web interface, processed through Libero's MCP-connected backend, and returned through the application. A personal story can then be saved and reviewed through the memory interface.

This is a simulated Alexa+ experience in a web application, not a deployed integration with a physical Alexa+ device.

5. Privacy-conscious design choices

The prototype includes authentication roles for family, medical, and administrative users, along with access-control mechanisms. Its optional camera pathway requires explicit consent and is disabled by default.

Libero is intended to complement human care, not replace it. It does not guarantee 24/7 monitoring, automatically contact emergency services, or replace caregivers and healthcare professionals. Its current capabilities must be understood within the limits of a research prototype.

How we built it

I built Libero around a modular architecture that separates local input processing, backend orchestration, specialized agents, memory handling, decision logic, and the browser interface.

Rather than placing every responsibility inside a single prompt or monolithic application, I separated the system into components with distinct purposes. This makes the implementation easier to inspect, test, and extend.

Architecture and technology stack

Core backend: Python and FastAPI.

Communication layer: Model Context Protocol (MCP), using JSON-RPC over Streamable HTTP.

Audio processing: Voice-activity and audio-energy processing, Groq Whisper when configured, and a local Whisper fallback when the required dependencies are available.

Vision-related processing: OpenCV and MediaPipe, with camera capture optional and dependent on explicit consent.

Language-model providers: Groq, an optional Amazon Bedrock integration, and a limited offline fallback. The provider used by a particular execution depends on configuration and available credentials.

Voice output: An optional Amazon Polly integration, with a local speech-output alternative.

Persistence: Local JSON-based storage for the prototype's main conversational and care-related data.

Interface: A browser-based application connected to Libero's backend.

Testing: Automated tests covering selected safety rules, authentication, memory behavior, and MCP tool operations.

1. Edge processing

The edge layer coordinates supported local input and output components. It includes microphone processing, speech transcription, optional camera-related processing, and voice output.

The audio pipeline uses voice-activity processing to help determine when speech is present before transcription. Groq Whisper can be used when configured, with a local Whisper fallback when available.

The vision-related path uses OpenCV and MediaPipe for experimental processing. Camera capture is controlled through an explicit consent configuration rather than being enabled by default.

This separation allows the prototype to distinguish between an available component and one that is actually active. A missing microphone, disabled camera, or unavailable provider should not be presented as though the corresponding functionality is operating normally.

2. A real MCP-connected conversation flow

The backend exposes an MCP endpoint through FastAPI, using JSON-RPC over Streamable HTTP.

The browser conversation exercises an actual protocol flow:

Initialize the MCP connection.
Complete the initialization notification.
Discover available tools.
Invoke a tool through tools/call.
Process the result through the application.

The current repository exposes 13 MCP tools for the prototype's supported operations.

This distinction matters: the hackathon demonstration is not solely a static UI or a prerecorded concept. The browser interface communicates with Libero's backend and exercises the MCP tool flow.

The server uses Streamable HTTP with JSON responses. It does not provide an SSE stream through the same endpoint, and the implementation should not be represented as having interoperability that has not been tested with an independent client.

3. Orchestration and specialized agents

The orchestration layer coordinates separate components responsible for conversation, memory, safety analysis, care-related logic, family-facing actions, health-related correlation, and governance.

The companion component generates conversational responses. The memory component identifies and retrieves saved stories. The safety component evaluates selected risk-related events, while the decision layer assigns an internal response level.

This separation helps prevent the system's responsibilities from collapsing into one undifferentiated AI interaction. It also provides clearer boundaries for testing and future improvements.

The existence of a component does not mean all related external actions are already active. For example, proposed calls to emergency services are recorded as proposed actions and are not executed by the current prototype.

4. Context, memory, and model providers

Libero can use recent conversation history and saved memories when constructing context for subsequent responses. Its current data flow relies primarily on local persistence rather than a deployed production database.

The project includes optional integrations with external model and voice providers. These integrations require configuration and, where applicable, credentials. The demonstration must identify the provider actually used by a given execution rather than implying that every available integration is always active.

This approach keeps the implementation testable in a local environment while making the boundaries of optional services explicit.

5. Authentication and access control

The backend includes roles for family, medical, and administrative users. The authentication implementation includes password hashing with scrypt, session-cookie controls, login rate limiting, and authorization checks for access to records.

The project also distinguishes development configuration from production requirements.

These mechanisms are useful foundations for a system that handles sensitive care-related information. They do not constitute formal healthcare or privacy certification. Production deployment would require additional security review, operational controls, and a more comprehensive privacy framework.

Challenges we ran into

One of the most difficult challenges was contextual ambiguity.

In a care-support system, the same word can refer to a current event, an old memory, another person's experience, or a situation that has already been resolved. A simple keyword-based approach can confuse these cases.

I implemented contextual rules to account for negation, past events, and references to other people, then connected those rules to a four-level decision model.

This was an engineering attempt to make the system more context-sensitive, not a claim that the ambiguity problem has been solved. Heuristic rules are inherently limited, particularly when the system cannot reliably establish what is happening in the physical world.

A second challenge was integration.

Having individual modules is not the same as having a coherent end-to-end product. The interface, backend, memory component, model provider, and MCP flow all need to work together for the user to see a meaningful result.

I therefore focused on connecting the browser conversation to the MCP server and making the main conversation-and-memory experience demonstrable.

A third challenge was handling dependencies and unreliable environments. Microphone access, camera availability, transcription latency, external credentials, and model-provider configuration can all affect behavior. I had to distinguish between features that exist in the code, features that work under particular configurations, and capabilities that are not yet implemented as complete workflows.

Finally, I had to confront the difference between a technical signal and a trustworthy safety decision.

A fall-related signal or a concerning sentence is not the same as a confirmed emergency. False positives can create unnecessary alarm, while false negatives can create a dangerous sense of reassurance. Software tests can help verify defined behaviors, but they cannot establish that the system is safe for real-world care.

These challenges influenced both the architecture and the way I present the project: the implementation should be demonstrable, and its limitations should be equally clear.

Accomplishments that we're proud of

I am proud that Libero has moved beyond a concept into a functional software prototype with connected components and demonstrable flows.

A working browser-to-MCP interaction

The web interface connects to Libero's MCP server and exercises initialization, tool discovery, and tool invocation. This gives the demonstration a concrete integration path rather than relying entirely on a visual mock-up.

A demonstrable conversational memory flow

A personal story can be submitted, saved in the prototype's local storage, and reviewed through the Memories interface. This is a concrete expression of the project's central idea: a conversation can preserve context that may be useful later.

A modular system rather than a single chatbot

The repository separates conversation, memory, safety analysis, decision logic, input processing, and backend operations into distinct components. This provides a foundation for inspecting and testing individual responsibilities rather than hiding all behavior inside one prompt.

Multiple input and output paths

The code includes an audio transcription pathway, optional vision-related processing, experimental multimodal fall analysis, and speech-output components. These paths have different runtime requirements and limitations, but they demonstrate the breadth of the implementation.

Explicit decision levels and contextual rules

The four-level decision model and contextual handling of negation, past events, and references to other people represent concrete attempts to address ambiguity instead of treating every risk-related word as an emergency.

Authentication and testing

The prototype includes role-based access controls and an automated test suite. The repository documents a run in which 112 tests passed in an offline environment.

That result is evidence about the software paths covered by those tests under the conditions in which they were run. It is not evidence of clinical accuracy, physical-device reliability, or safety in real homes.

A focused hackathon demonstration

Libero existed before the hackathon. Rather than claiming the entire project was built from scratch during the event, I focused on making a coherent part of the existing system easier to evaluate: connecting the browser conversation to MCP, refining the language and contextual conversation flow, demonstrating saved memories, and preparing a clearer end-to-end presentation.

I am proud not only of the architecture, but of the decision to show what actually works and identify what still needs to be built.

What we learned

The first lesson was that care cannot be reduced to task completion.

A person may need reassurance rather than correction. A personal story can matter even when it does not correspond to an immediate practical task. A caregiver may need context that cannot be captured by a checklist alone.

These observations shaped Libero's focus on conversational support and personal memory.

The second lesson was that contextual understanding must be treated as an engineering problem, not just a writing problem. Prompts and language models can produce compassionate-sounding responses, but the system still needs clear boundaries, testable components, and careful handling of ambiguous input.

The third lesson was that a working prototype and a trustworthy care product are not the same thing.

Passing tests does not prove clinical effectiveness. Implementing a notification component does not mean every event is automatically delivered. Supporting an optional provider does not mean it was active during a demonstration. Building a safety heuristic does not establish reliable emergency detection.

I learned to treat these distinctions as part of the design itself, not as disclaimers added at the end.

I also learned that privacy and consent cannot be postponed until a product is ready for deployment. Audio, video, conversation history, and care-related records are sensitive. A responsible system needs to consider access, retention, security, transparency, and the conditions under which its components are allowed to operate.

Finally, building Libero made the relationship between personal motivation and technical responsibility much clearer to me.

My family's experience explains why I began this project. It does not prove that my solution will work for other families. That requires evaluation, feedback, and a willingness to change the implementation when evidence shows that something does not work as intended.

The most meaningful progress is not simply adding features. It is learning to build with a clearer understanding of who might depend on them and what could happen if they fail.

What's next for Libero

The next stage is to move from a demonstrable research prototype toward a system whose usefulness and limitations can be evaluated more rigorously.

1. Evaluate the conversational experience

I want to study whether the conversation and memory flow is genuinely useful to caregivers and relevant care professionals. This should happen with appropriate consent, safeguards, and feedback—not through assumptions about what families need.

The goal is to assess the usefulness, clarity, and appropriateness of responses, rather than assuming that a supportive-sounding answer is automatically helpful.

2. Evaluate safety behavior

The experimental safety pipeline needs systematic testing across realistic scenarios, including ambiguous language, negation, references to past events, and situations that could be mistaken for emergencies.

Future work should measure false positives and false negatives, document failure modes, and establish what evidence would be necessary before any safety-related claim could be justified.

3. Complete and validate event-delivery pathways

The current prototype records edge-detected events locally, but automatic delivery of every edge event to the dashboard's real-time overlay and remote family notification pathways is not fully implemented.

I want to develop and test these workflows with explicit attention to authorization, connectivity failures, delivery status, and what happens when a component becomes unavailable.

External emergency actions must remain separate from simulated or proposed actions until their safety, reliability, and legal requirements have been properly addressed.

4. Strengthen privacy and deployment readiness

The prototype's local persistence is useful for development, but production use would require a more robust deployment model, reviewed transport security, stronger operational monitoring, clear data-retention rules, and a comprehensive privacy and security assessment.

The project also needs to address applicable legal and regulatory requirements before being used with real care situations. It is not currently a certified medical system or a fully compliant production healthcare product.

5. Build around evidence, not feature count

There are possible future extensions involving more reliable notifications, production storage, additional devices, and broader care workflows. But the priority is not to add integrations simply because they are technically possible.

The priority is to understand which capabilities make a meaningful difference, which introduce unacceptable risks, and what evidence is needed before they can responsibly be used.

Libero began with a question shaped by my family's experience: how can technology support the people living with dementia and the people who care for them?

I cannot promise that an application will prevent every tragedy, understand every emotion, or remove the difficulties of caregiving. I do believe that these problems deserve more than a generic chatbot or an ambitious idea presented without working software.

I built Libero to explore a specific possibility: connecting contextual conversation, personal memory, and experimental safety analysis in a system that families could eventually evaluate and help shape.

The current prototype is an early step. The work ahead requires evidence, collaboration, safeguards, and honest evaluation.

A reminder tracks a task. Libero explores how to preserve the human context around a conversation.