Project CareBot

Simulated robot camera image stored in S3 and analyzed by the ImageProcessingAgent with Amazon Nova Lite.
Sample ASI:One conversation showing CareBot identifying the target, coordinating specialist agents, and returning recommended robot actions.
Medicine Demo — Camera View Simulated camera image containing the medicine bottle used to test CareBot’s object-finding workflow.
Sample ASI:One conversation showing CareBot locating the medicine bottle and generating the steps the robot should take to retrieve it.

Inspiration

CareBot was inspired by assistive technologies like Be My Eyes and by the broader challenge of helping elderly and disabled users interact with their surroundings more independently.

We wanted to explore what it would look like if a conversational AI assistant could do more than answer questions. Instead, it could understand a user's request, gather relevant context, visually inspect the environment, and coordinate specialized agents to determine what should happen next.

Our goal was to combine conversational AI, multimodal vision, multi-agent orchestration, and simulated robotic action planning into one accessible workflow.

What it does

CareBot is a Fetch.ai multi-agent prototype for assistive object finding, health-context monitoring, and simulated robot action planning.

A user interacts with CareBot through ASI:One using the Agent Chat Protocol (ACP). For example, a user might ask:

"Can you help me find my medicine?"

CareBot acts as the central coordinator. It uses ASI:One for reasoning about the user's request and decides whether to call one of its specialist agents:

MonitoringAgent evaluates the latest synthetic health snapshot and returns relevant monitoring information.
ImageProcessingAgent analyzes a simulated robot camera image stored in Amazon S3 using Amazon Nova Lite through Amazon Bedrock.
ActionAgent converts the vision result into an ordered simulated movement plan such as TURN_RIGHT, MOVE_FORWARD, and PICK_UP.

The ImageProcessingAgent reports whether the target object is visible, its approximate left/center/right location, and a short description of where it appears.

Because we did not use physical robot hardware in this prototype, an image stored in S3 acts as the robot's camera frame, and the ActionAgent returns the steps the robot should take rather than executing physical movement.

How we built it

We built CareBot using Fetch.ai's uAgents framework, Agentverse, ASI:One, Amazon S3, Amazon Bedrock, and Amazon Nova Lite.

CareBot is the main workflow coordinator. It exposes an ACP-compatible chat interface so that users can discover and interact with it through ASI:One. CareBot also calls the ASI:One API for orchestration reasoning, allowing it to interpret the request, decide which specialist agent should be called, and reason about the results returned by those agents.

The system is divided into several uAgents:

CareBot — ACP chat interface and workflow coordinator
MonitoringAgent — evaluates the latest synthetic health snapshot
ImageProcessingAgent — performs object detection and scene understanding
ActionAgent — generates an ordered simulated movement plan

The agents communicate through structured uAgents protocols rather than calling each other's Python functions directly.

For vision, the ImageProcessingAgent retrieves an image from Amazon S3 using Boto3 and sends the image bytes and target object to Amazon Nova Lite through the Bedrock Converse API. Nova returns structured JSON containing:

{
  "target_visible": true,
  "direction": "left",
  "description": "medicine bottle on the back counter"
}

That structured result returns to CareBot, which can then decide to invoke the ActionAgent. The ActionAgent turns the result into a simulated plan such as:

TURN_LEFT
MOVE_FORWARD
PICK_UP

Our AWS-dependent ImageProcessingAgent and MonitoringAgent run locally and remain connected to the rest of the system through Agentverse Mailbox. CareBot and ActionAgent can run as hosted Agentverse agents or locally. This let us combine Fetch.ai's lightweight agent infrastructure with external AWS services that require Boto3.

The MonitoringAgent uses a synthetic health snapshot stored in S3. It applies demo-specific rules to that stored data and can optionally use Bedrock for additional reasoning. The current prototype bases its monitoring assessment on the stored snapshot rather than treating free-form symptom text as medical input.

Challenges we ran into

One major challenge was deciding how responsibilities should be divided across the agents. Early versions mixed orchestration, vision, and action logic together. We refined the architecture so that CareBot acts as the coordinator while specialist agents perform narrower tasks and return structured results.

We also ran into deployment constraints when integrating AWS services with Fetch.ai. Our ImageProcessingAgent and MonitoringAgent require Boto3 to communicate with Amazon S3 and Bedrock, while the lightweight hosted agent environment is more restricted. We solved this by running the AWS-dependent agents locally and connecting them to the hosted parts of the system through Agentverse Mailbox.

Another challenge was visual localization. Nova Lite could recognize target objects in our images, but determining whether an object was on the left, center, or right of a scene was not always consistent. We refined the vision prompt and response structure to reduce ambiguity while keeping the information useful for the ActionAgent.

We also had to test robotic perception without physical robot hardware. To solve this, we abstracted the camera by storing static test images in S3. This allowed us to build and test the entire perception and action-planning pipeline without requiring a physical robot.

Finally, coordinating independently developed agents was a challenge in itself. Different parts of our team worked on separate specialist workflows, so we had to define compatible message models, agent addresses, and protocols before merging them into CareBot's orchestration flow.

Accomplishments that we're proud of

We're proud that CareBot became a working distributed multi-agent system, rather than a single chatbot or isolated AI demo.

Our agents can communicate through Agentverse even when they are running on different machines. For example, CareBot can send a request through Agentverse to an ImageProcessingAgent running on another teammate's computer, which can then retrieve an S3 image, invoke Bedrock, and return the structured result back to CareBot.

We also successfully connected several different technologies into one workflow:

Natural-language interaction through ASI:One
ACP-compatible agent communication
Fetch.ai uAgents and Agentverse
Multi-agent orchestration
Amazon S3 for simulated camera frames and monitoring data
Amazon Bedrock and Nova Lite for multimodal image understanding
Boto3 for AWS integration
Simulated robotic movement planning

We are especially proud that each specialist has a clearly defined role and can be expanded independently without requiring the entire system to be redesigned.

What we learned

We learned that building a multi-agent application involves much more than simply creating several agents. The most important part is orchestration: deciding which specialist should be called, defining what information should be passed between agents, and reasoning about each result before deciding what happens next.

We also learned how Fetch.ai's ecosystem fits together. uAgents provides the agent framework and structured protocols, Agentverse provides registration, discovery, hosting, and communication, and ASI:One provides the user-facing conversational experience and reasoning layer.

On the AWS side, we gained experience working with Amazon S3, Amazon Bedrock, Amazon Nova Lite, IAM permissions, and Boto3. We also learned how authentication and permissions affect distributed systems when different agents run in different environments.

Working with multimodal AI taught us that identifying an object and reasoning about its spatial position are different problems. Carefully constraining the output format and defining what information the model should return made the vision result much more useful to downstream agents.

We also learned how useful structured agent-to-agent communication can be when different team members are developing different parts of a system independently.

What's next for Project CareBot

The biggest next step is moving from our current single-image prototype to a continuous perception-action loop:

SEE → REASON → ACT → SEE AGAIN

Instead of analyzing one static S3 image and generating a movement plan, a future version could capture a new camera frame after each movement, analyze the updated environment, and continuously adjust its next action.

We would also like to connect CareBot to physical robotic hardware so that the ActionAgent's movement plan can control a real robot rather than only returning simulated commands.

The monitoring workflow could also be expanded to incorporate richer contextual data and more sophisticated reasoning, while maintaining clear boundaries around medical safety.

Longer term, CareBot could incorporate persistent spatial memory, additional sensors, personalized assistance, and real-time perception to become a more capable assistive system for everyday environments.

Link to Github: https://github.com/Xiaoyuun/CareBot