AuditVision AI – Autonomous Visual Audit Intelligence

OpenCV detects six visual annotation types in audit PDFs; Gemma via Amazon Bedrock adds asynchronous AI validation, while AWS S3 stores structured results for auditor review.

Comment

Inspiration

Auditors often work with financial statements, control reports, and supporting documents containing numerous visual annotations. Ticks, crosses, circles, arrows, highlights, and underlines can carry important context during an audit review, but locating and organizing these marks across multiple PDF pages can be time-consuming.

I wanted to explore how computer vision could make this process more efficient while keeping auditors in control of the final review.

That idea led to AuditVision AI – Autonomous Visual Audit Intelligence, a platform that combines OpenCV-based visual detection with generative AI to help transform annotated audit documents into structured, reviewable findings.

My goal was not to replace professional audit judgement, but to build a practical system that helps auditors find relevant visual evidence faster and investigate it more consistently.

What it does

AuditVision AI allows users to upload audit PDFs and automatically identify six types of visual annotations:

  • Ticks: check or tick marks.
  • Crosses: cross-shaped annotations.
  • Circles: circular marks around figures or content.
  • Arrows: directional annotations pointing to content.
  • Highlights: visually emphasized regions.
  • Underlines: emphasized text or values.

The application presents the detected findings through a dashboard, helping reviewers explore annotations and their associated document information.

Beyond computer vision, AuditVision AI includes an optional AI validation stage. After OpenCV processing publishes its results, selected findings are sent for additional analysis using Google Gemma through Amazon Bedrock Mantle. This validation runs asynchronously, allowing the initial document-processing workflow to return without waiting for the AI response.

The application saves OpenCV findings and AI validation results separately as structured JSON, making the outputs easier to retrieve and review.

Visual detection is only the first step: a detected tick does not prove that a control passed, and AI-generated validation does not establish the correctness of financial statements. Important findings still require human review.

How we built it

I built AuditVision AI using a React and TypeScript frontend with a Python FastAPI backend.

Computer vision: OpenCV processes the uploaded PDFs and identifies supported visual annotation patterns. The backend organizes the resulting detections into structured data that the application can display.

Generative AI: An asynchronous validation service sends selected OpenCV findings to Google Gemma through Amazon Bedrock Mantle. Its results are handled separately from the initial detection output, so AI latency does not block document processing.

Cloud infrastructure: I containerized the frontend and backend and published their Docker images to Amazon Elastic Container Registry (ECR). The application is deployed through Amazon ECS Express Mode, which provides a managed container deployment and HTTPS endpoint.

Persistent storage: Amazon S3 stores the demo dashboard JSON and is configured as the storage backend for document metadata, OpenCV results, and AI validation results. This separates persistent application data from the lifecycle of individual containers.

Security and operations: AWS IAM provides access control, while AWS Secrets Manager is used to supply the Bedrock API key to the deployed backend without embedding it in the Docker image.

The architecture keeps the main responsibilities separate: React handles the user experience, FastAPI coordinates processing, OpenCV detects visual annotations, Gemma provides supplementary validation, and S3 stores structured outputs.

Challenges we ran into

One of the main challenges was designing the workflow so that AI validation could run in the background without delaying the initial document-processing response. This required separating OpenCV processing from the subsequent AI validation stage and persisting their results independently.

Another challenge was making the dashboard data durable. Hardcoded demo data might make an early prototype look populated, but it does not provide a reliable storage model. I introduced a JSON storage abstraction with local and S3 backends and moved the two default demo entries into S3.

Cloud deployment brought a different set of challenges. Container images, task definitions, IAM roles, secret injection, and application configuration must all work together. Publishing a new image to ECR is not the same as deploying it, and a running container needs the correct permissions to access S3 and retrieve secrets.

I also worked through local Docker and Windows build-environment issues, while keeping the existing application architecture and deployment separate from development changes.

These challenges reinforced the importance of validating individual components first and then testing the complete workflow.

Accomplishments that we're proud of

  • Built a working application that identifies six categories of visual audit annotations using OpenCV.
  • Added asynchronous AI validation after computer-vision processing.
  • Separated OpenCV findings from AI-generated validation results.
  • Implemented an S3-backed JSON storage approach and moved the default demo dashboard data out of hardcoded runtime arrays.
  • Created a user guide explaining the detection categories, application workflow, and responsible use.
  • Containerized the frontend and backend and successfully published versioned images to Amazon ECR.
  • Deployed the updated task definition through the existing ECS Express service, with an application task role and a Secrets Manager reference for the Bedrock API key.

Most importantly, I established an architecture that brings computer vision and generative AI together while keeping the original visual detections distinct from AI-assisted analysis.

What we learned

This project taught me that building a useful AI application involves much more than calling a model.

Computer vision and generative AI solve different problems. OpenCV identifies visual patterns, while an AI model can provide additional analysis of selected findings. Keeping these responsibilities separate makes the system easier to reason about and helps prevent AI output from being mistaken for verified evidence.

I also learned the importance of asynchronous processing, durable storage, container deployment, IAM least privilege, and secret management.

Another important lesson was the need to distinguish a successful deployment from a fully validated application. A healthy container is an important milestone, but the complete workflow must also be tested, including document processing, cloud storage, AI responses, and retrieval of the saved results.

Finally, I was reminded that human review remains essential when dealing with financial and audit-related information.

What's next for AuditVision AI – Autonomous Visual Audit Intelligence

The next phase is to strengthen the link between each detected annotation and the relevant text, figure, or document region.

I plan to explore:

  • Improving annotation-to-text association and page-level evidence traceability.
  • Evaluating detection performance across different PDF qualities, layouts, and annotation styles.
  • Making AI validation results easier to inspect alongside the original OpenCV findings.
  • Expanding test coverage for asynchronous processing, storage failures, and model errors.
  • Adding clearer review workflows so auditors can confirm, reject, or investigate detected findings.
  • Exploring additional audit-document checks while keeping each capability clearly defined and independently testable.

The long-term vision is to make AuditVision AI a dependable assistant for visual audit review: one that helps auditors navigate complex documents, organize evidence, and focus their attention where it matters most, while keeping professional judgement at the center of the process.

Built With

Share this project:

Updates

Submission history