Ancora AI

• 

  Ancora AI

Inspiration

Meetings generate important decisions, commitments, and follow-ups, but that information often gets buried in recordings and scattered notes. When the same topic comes up weeks later, teams have to search through old meetings to understand what happened.

We wanted to build something beyond a meeting summarizer. Ancora AI was inspired by the idea of giving meetings a persistent memory — one that remembers commitments, connects follow-ups across meetings, and makes a project's history easy to query.

What it does

Ancora AI turns recorded meetings into a structured, queryable memory.

It processes both speech and visual information from a meeting. Audio is transcribed using Whisper, while on-screen information is extracted through frame sampling, scene detection, and OCR. These are combined into a timestamped multimodal timeline.

An LLM extracts structured action items including the task, assignee, deadline, and supporting evidence.

Ancora then connects new action items to previous meetings. Embeddings are used to find potentially related threads, while an LLM determines whether an item continues an existing commitment or represents something new.

Users can ask natural-language questions about a project's history. Ancora retrieves the relevant thread history and generates a grounded answer with citations pointing back to specific meetings and action items.

How we built it

Ancora AI is a full-stack multimodal AI system built around three main stages:

1. Meeting understanding — extract speech and visual information and create a timestamped timeline.
2. Persistent memory — extract action items and connect them into evolving cross-meeting threads.
3. Grounded retrieval — retrieve relevant histories and answer questions using the accumulated project memory.

The system uses Whisper for transcription, OpenCV for frame processing and scene detection, Azure Vision for OCR, and Azure OpenAI for embeddings and LLM reasoning.

The backend uses FastAPI, SQLAlchemy, and SQLite, while the frontend is built with React and TypeScript. Google OAuth and JWT provide authentication.

A major design decision was using hybrid thread linking. Cosine similarity quickly identifies candidate threads, while the LLM makes the final decision using contextual information such as assignees, deadlines, and commitment continuity.

Challenges we ran into

The biggest challenge was making the system understand whether an action item was actually connected to something discussed in an earlier meeting.

Simple semantic similarity was not enough. Two tasks can use similar language while representing completely different commitments. We therefore combined vector retrieval with LLM-based reasoning.

We also had to handle multimodal information arriving at different timestamps, incremental database updates, long-term thread history, and failures during the processing pipeline.

Another challenge was ensuring that generated answers remained grounded. We added citation validation that checks every cited action-item ID against the context provided to the model, preventing unsupported citations from reaching the user.

Accomplishments that we're proud of

We built a complete working pipeline rather than just a meeting summarization prototype.

Ancora can take a meeting video, understand both its audio and visual content, extract structured commitments, persist them, connect them to previous meetings, and answer questions about their evolution.

We are especially proud of the cross-meeting memory system. Instead of treating every meeting as an isolated event, Ancora creates persistent threads that evolve as new meetings are processed.

We also implemented automated video processing, live pipeline status tracking, project organization, authentication, persistent memory, grounded Q&A, and citation validation.

What we learned

We learned that building AI with memory is not simply about storing embeddings.

A useful memory system needs to understand what information matters, how separate pieces of information relate to each other, and how historical context should be reconstructed when answering a question.

We also learned the value of combining deterministic methods with LLM reasoning. Embeddings provide efficient retrieval, while the LLM handles contextual decisions that similarity alone cannot capture.

Most importantly, we learned that combining what was said with what was shown creates a much richer representation of a meeting than relying on transcripts alone.

What's next for Ancora AI

We want to take Ancora beyond meeting recordings and turn it into a broader project intelligence and memory layer.

Future versions could connect meetings with documents, decisions, tasks, and project milestones, allowing teams to understand not just individual conversations but how an entire project has evolved over time.

We also plan to improve temporal reasoning, thread resolution, scalability, and support for additional sources of project context.

Our long-term vision is for Ancora to become a persistent memory for teams — one that remembers what happened, connects it across time, and makes that knowledge accessible whenever it is needed.

Built With

• azure
• cv
• open
• python
• whisper