Intelligent Image Annotation

• 

  The architectural diagram

• 

  Detecting houses' vulnerability to flood

• 

  Detecting skin diseases

• 

  Brainstorming with AI on diseases' visual descriptions.

Inspiration

Specialized AI needs specialized data. But doctors, agronomists, and researchers can spend hours manually labeling images because generic annotation tools don't understand their domain knowledge.

We asked: what if experts could simply explain what to look for, and an AI agent did the tracing?

What it does

Intelligent Image Annotation uses AI to turn domain knowledge into image annotations.

Users provide a detection taxonomy and natural-language instructions. The AI agent analyzes images according to those rules and generates detections for human review.

It can also improve an taxonomy using an existing taxonomy, a reference webpage, and a human prompt.

How we built it

We built a React/Vite + Node.js/Express application with Amazon Bedrock at the center of the AI workflow.

A multimodal agent receives the image, taxonomy, and instructions, reasons about what should be detected, and returns structured annotations.

We also designed the system to be model-flexible, allowing experimentation with Amazon Bedrock models such as Amazon Nova alongside other vision models.

Challenges we ran into

The hard part isn't just seeing objects—it's understanding what an expert means by "this is what matters."

Scientific annotation requires more than simple labels. The agent must interpret detailed rules, context, and visual characteristics.

We also had to make this workflow accessible without requiring researchers to write custom computer-vision code.

Accomplishments that we're proud of

We built a working AWS-powered, agentic image-annotation pipeline from natural-language knowledge to visual detections.

Instead of building a new detection model for every domain, the same workflow can adapt its behavior through the taxonomy and instructions.

Experts define the rules. AI does the repetitive work.

What we learned

We learned that image annotation is as much a knowledge problem as a vision problem.

A powerful multimodal model becomes significantly more useful when we give it structured knowledge about what the expert actually wants to detect.

We also learned how Amazon Bedrock enables this workflow without requiring us to train and deploy a specialized model for every new task.

What's next for Intelligent Image Annotation

Next, we want to close the human-in-the-loop loop.

Experts will correct AI annotations, and the agent will use that feedback to improve the taxonomy and future annotations.

We also want to ingest scientific papers, documentation, and webpages directly to automatically build detection taxonomies.

The long-term vision: turn expert knowledge into scalable AI-generated datasets.

Built With

• amazon-bedrock-agentcore
• amazon-web-services
• natural-language-processing
• node.js
• react
• strands-agents
• vision
• vue