VerbaTask: Multi-Agent AI for Meeting Action Tracking

VerbaTask uses a multi-agent AI pipeline to extract and structure trackable action items directly from meeting transcripts, helping teams capture commitments and reduce manual follow-ups.

Comment

## Inspiration

Most meetings end the same way: someone takes notes, the action items get buried and nothing gets tracked. Project management tools are great for tracking tasks, but someone still needs to manually create every ticket, assign owners, and set deadlines after the meeting. I wanted to build something that could turn a raw meeting transcript into a fully tracked, queryable set of action items, automatically, without anyone having to manually create a single ticket. Elasticsearch's Agent Builder, with its native MCP support and tool orchestration, made that possible.

## What it does

VerbaTask is a multi-agent AI system that processes meeting transcripts and extracts structured action items with zero manual effort. You paste or upload a transcript, hit a button, and four specialized agents run sequentially: extracting tasks, summarizing the discussion, validating deadlines, and making everything queryable through natural language.

The agents run on Elasticsearch Agent Builder, with a Python backend, a FastMCP server exposing custom tools to the agents, and a Streamlit dashboard for real-time task tracking.

## How I built it

I started by designing the data model, four Elasticsearch indices to store transcript messages, action items, summaries, and validation reports. Then I built the MCP server in Python using FastMCP, exposing two custom tools: create_action_item and update_action_item. I also built a search_meeting_messages tool directly in Elastic Agent Builder to let agents fetch transcript messages by meeting ID. These tools are what give the agents the ability to read and write data in Elasticsearch.

Each agent was set up in Kibana's Agent Builder with custom instructions and the right tools enabled.

  • The Action Item Extraction Agent reads the transcript and extracts tasks.
  • The Meeting Summary Agent generates a concise summary.
  • The Overdue Item Validator Agent flags action items past due.
  • The Insights Agent answers natural language questions using Elasticsearch's native platform.core.search tool.

The frontend is a Streamlit dashboard with real-time checkboxes, date and risk picker and a natural language insights page.

## Challenges I ran into

  • The biggest challenge I faced was getting the agents to resolve relative dates correctly. When a transcript says "next Monday," the agent needs to know what today's date is, and what day of the week it is to calculate the right absolute date. I solved this by including the full day and date in every agent prompt.

  • Another challenge was connecting the local MCP server to Kibana. Since the server runs locally, I used ngrok to expose it publicly and registered it as an MCP connector in Kibana.

  • Finally, parsing WEBVTT transcripts required a lot of regex work to handle blank lines, timestamps, message numbers, and speaker labels.

## Accomplishments that I'm proud of

  • Getting the full pipeline working end to end, from raw transcript to structured tracked action items in Elasticsearch
  • Four agents running sequentially with each one passing context to the next
  • Natural language insights working across all meetings using ES|QL

## What I learned

Building VerbaTask taught me how to design a multi-agent pipeline where each agent has a focused responsibility and clear tool boundaries. It also taught me how Elasticsearch's Agent Builder works in practice especially figuring out how agents, tools, and connectors fit together. I learned how MCP works as a protocol connecting agents to external tools, and got hands-on experience with ES|QL and the update_by_query API.

## What's next for VerbaTask: Multi-Agent AI for Meeting Action Tracking

  • Supporting more transcript formats beyond WEBVTT
  • Letting users assign and reassign action items directly from the dashboard
  • Fully automated deployment so the MCP server does not need to run locally
  • Add email or slack notifications for overdue items

Built With

Share this project:

Updates