ScreenTalk

ScreenTalk is a Chrome extension that enables instant AI-powered analysis of screenshots using Chrome's Built-in AI APIs. Capture any part of your screen and immediately ask questions about it

Comment

Inspiration

The frustration of constantly copy-pasting screenshots to ChatGPT sparked ScreenTalk. We noticed developers, designers, and researchers spending countless hours switching between screenshot tools and AI chat interfaces. Why not eliminate that friction entirely? One hotkey to capture, analyze, and act - that's the vision that inspired us.

What it does

ScreenTalk is a Chrome extension that revolutionizes how you interact with AI for visual content:

  • Instant Capture: Screenshot with a single hotkey (configurable, default Cmd+Shift+S)
  • Smart AI Integration: Automatically sends screenshots to Claude/ChatGPT with customizable prompts
  • Context-Aware Analysis: Supports OCR, code extraction, design feedback, and accessibility reviews
  • Seamless Workflow: Results appear instantly in your AI chat interface - no manual uploads

How we built it

  • Frontend: Chrome Extension Manifest V3 with vanilla JavaScript for minimal overhead
  • Architecture: Service worker for background processing, content scripts for page interaction
  • AI Integration: Direct API connections to Anthropic Claude and OpenAI ChatGPT
  • Storage: Chrome Storage API for settings persistence and hotkey customization
  • UI/UX: Minimalist popup interface with native Chrome design patterns

Challenges we ran into

  1. Chrome Extension Permissions: Balancing activeTab permissions with screenshot capture capabilities across all domains
  2. Image Processing: Optimizing screenshot encoding to base64 while maintaining quality and managing API payload limits
  3. Hotkey Conflicts: Handling global keyboard shortcuts that work reliably across different OS and browser states
  4. API Rate Limiting: Implementing intelligent queuing for rapid screenshot bursts
  5. Cross-Platform Consistency: Ensuring hotkeys work identically on Mac, Windows, and Linux

Accomplishments that we're proud of

  • 10x Faster Workflow: Reduced screenshot-to-AI analysis from ~30 seconds to <3 seconds
  • Zero Context Switching: Users stay in their workflow without opening new tabs
  • Universal Compatibility: Works seamlessly across all websites and Chrome-based browsers
  • Smart Defaults: Pre-configured prompts for common use cases (code review, accessibility, design)

What we learned

  • Chrome Extension Manifest V3 service workers require careful lifecycle management
  • Base64 image encoding performance varies significantly - WebP offers better compression
  • Users want customization but value smart defaults more
  • Screenshot context matters - adding page URL and title improves AI response quality

What's next for ScreenTalk

  • Multi-Modal Support: Video snippet capture with audio transcription
  • Annotation Tools: Draw, highlight, and add notes before sending to AI
  • Team Collaboration: Share screenshot-AI conversations with team members
  • Advanced OCR: Local text extraction before AI processing for faster responses
  • Custom AI Models: Support for local LLMs and custom fine-tuned models
  • Browser Expansion: Firefox and Safari extensionsß

Built With

Share this project:

Updates