AgentCore AI for S.M.A.R.T. Goal Generation

AgentCore: S.M.A.R.T. Goal Generation Powered by LLM-as-Judge — Built on AWS

Comment

Inspiration

At SIPPA Solutions (a CUNY-based startup), we believe everyone should be able to use their data for better health.

In our clinical study, clinicians spend precious clinical hours manually reviewing patient records to craft compliance-heavy S.M.A.R.T. goals (Specific, Measurable, Actionable, Relevant, and Time-bounded). This crucial process is tedious, slow, and prone to human error.

We asked ourselves: Can AI do this smarter—and faster—while staying compliant?

What it does

Our project tests the power of Amazon AgentCore to automate clinical goal generation using a powerful, specialized AI architecture:

  • Goal Generator Agent — Analyzes patient data to create personalized S.M.A.R.T. goals.
  • LLM-as-Judge Agent — Independently reviews and scores each proposed goal for quality, relevance, and clinical compliance.

Together, they act like a “coach and referee” for goal setting, creating a validated output ready for clinician approval.

How we built it

Leveraging best practices from AWS workshops, we built a complete, production-grade Amazon AgentCore runtime inside SageMaker.
This required mastering complex architectural challenges, including:

  • Multi-agent orchestration across kernel sessions
  • Secure Lambda and API Gateway integration for essential MCP tool access
  • Robust OAuth2 authentication via Cognito
  • Deployment of the BedrockAgentCoreApp in Docker and hosting on ECR

Special thanks to William Sia of Amazon, who helped us uncover the right function call to run agents across different kernels — a true game-changer!

Challenges we ran into

  • 🌀 Distinguishing between Bedrock Agents and AgentCore Agents
  • ⚙️ Avoiding recursive agent dependencies within a single session
  • 🔐 Setting up complex IAM roles and trust policies
  • 🧾 Handling malformed JSON outputs from LLMs
  • 🌐 Securing a Streamlit + VPC architecture for front-end access
  • ⏳ Overcoming SageMaker session timeouts despite custom settings

Accomplishments that we're proud of

  • Built a working multi-agent system using AgentCore
  • Solved major architectural hurdles (e.g., IAM roles and agent dependency management)
  • Implemented partial fixes for JSON validation and front-end integration

We’re proud to be among the first teams to demonstrate LLM-as-Judge evaluation using Amazon AgentCore in a regulated health context.

What we learned

Building production-ready AI agents is about more than just prompt engineering — it’s about runtime orchestration, security, and compliance-aware design.
We gained hands-on experience integrating multi-agent systems into real-world healthcare workflows using AWS infrastructure.

What's next for SMART Agent for Goal Generation with LLM-as-Judge Evaluator

We’re engaging hospital partners and AWS to explore pilot deployment of this system in our ongoing clinical investigation.
Our goal is to make AI-assisted goal generation a trusted, compliant, and time-saving tool for clinicians — starting with diabetes care and expanding to other chronic conditions.

Built With

  • agentcore
  • api-gateway
  • bedrock
  • cognito
  • ecr
  • lambda
  • mcp
  • python
Share this project:

Updates