AutoLap

Making Alexa+ work autonomously with the customer's laptop to enhance their productivity

Comment

Inspiration

We built what voice assistants currently lack in everyday life: true desktop productivity. today, smart speakers are great at setting kitchen timers, booking groceries, or playing music, yet they remain completely blind and hands-off when it comes to the device where students, developers, and knowledge workers spend eight hours a day—their laptop. Almost every customer purchasing an Alexa+ device already owns a personal computer, and that single realization inspired AutoLap.

We asked ourselves: What if Alexa+ could act as an autonomous, ambient co-pilot for your laptop—capable of seeing your screen, fixing broken code, researching live market numbers, running statistical data analysis, and organizing messy folders via voice?

However, giving an LLM raw control over a user's mouse, keyboard, and filesystem is normally a massive security hazard. If an AI hallucinates a terminal command, falls for visual prompt injection on a webpage, or deletes important personal files, trust is broken instantly. We set out to build AutoLap: a self-hosted Alexa+ MCP Server and multimodal Echo Show Web Simulation where every calculation and safety guardrail is enforced deterministically in Python code rather than left to LLM "vibes".

What it does

AutoLap turns Alexa+ into a voice-first, multimodal laptop productivity hub powered by 10 specialized Model Context Protocol (MCP) tools served over Streamable HTTP:

  1. Screen Vision & Visual Chaining (ask_about_screen): Captures the live primary display via mss, automatically masks out the assistant's own UI window so it never reads or clicks itself, and answers visual questions or transcribes on-screen charts and metrics directly into our data analysis engine.

  2. Surgical On-Disk Code Fixing & Guarded GUI Control (interact_with_screen & approve_screen_plan): Controls the laptop's mouse and keyboard safely or fixes code errors visible on screen. Instead of blindly typing into an editor, AutoLap reads OS window titles to locate the exact source file on disk, verifies there are no unsaved editor buffers, parses the proposed patch with ast.parse() or json.loads(), presents a unified diff (difflib.unified_diff) for human-in-the-loop voice approval, and atomically swaps the patched file on disk.

  3. Evidence-First Data Analysis (analyze_data): Processes spreadsheets (.csv, .xlsx, .xls, .json), text files, or numbers transcribed straight off the screen. To eliminate LLM math hallucinations, Pandas and NumPy compute all statistics first—stripping currency/percentage symbols (₹, $, €, %), excluding incomplete edge date periods (_partial_periods) so they do not skew trends, and calculating linear regression $R^2$ trend strength, IQR outliers, segment shares, and Pearson correlations. The LLM only interprets verified figures, returning a 4–5 sentence spoken summary alongside a formatted PDF report.

  4. Multi-Angle Web Research to Dual PDFs (research_topic): Plans 3 to 8 targeted DuckDuckGo news and web queries (specifically hunting for hard metrics and downside risks), compiles a raw Research Dossier PDF, extracts stated numbers into a structured entity x metric pivot CSV, and automatically chains into analyze_data to generate a second Analytical Overview PDF.

  5. Zero-Risk File Automation with Dry-Run Rehearsal (write_script & run_script): Generates custom Python scripts to organize folders, rename files, or clean data. Before touching a single real file, AutoLap rehearses the script on a throwaway copy inside a restricted sandbox, computes an exact diff of what will be moved, created, or modified, and waits for explicit user confirmation.

  6. Cryptographic Multi-Step Undo (undo_last_script): Maintains an undo manifest (manifest.json) with SHA-1 file digests for every script execution and code fix. Saying "Alexa, undo that" reverses file moves, restores modified backups, and reverts code patches—while checking SHA-1 hashes first so it never overwrites a file the user edited manually afterward.

  7. Instant Safe Local Search & Disambiguation (open_item): Performs a bounded breadth-first search across the computer to open documents, images, folders, URLs, or direct YouTube videos by name without touching executable binaries or system directories.

  8. extra_productivity folder: This gives even more and it incredibly productivity. this is switched off by default to reduce the set-up time. if an user ever wants to automate tasks, there are tools like; git_bootstraper, git_sync, notifier, etc. Go through Readme and requirements under this folder(not root), to switch these on.

How we built it

We architected AutoLap into two decoupled repositories to prove universal MCP compatibility:

  1. Standalone FastMCP Server (mcp_alexa):

(a). Built with Python and FastMCP running over the modern MCP Streamable HTTP transport (http://127.0.0.1:8000/mcp).

(b). Zero-Code Client Onboarding via SERVER_INSTRUCTIONS: Our MCP server embeds all multi-tool chaining rules inside the MCP instructions handshake field (instructions.py). When any MCP-compliant client (Alexa+, LangGraph, Claude, or MCP Inspector) initializes a session, it dynamically discovers all 10 tools and their chaining workflows without requiring custom client-side prompts.

(c). Data, Vision & PDF Pipeline: Built with pandas, numpy, ddgs (DuckDuckGo Search), mss, Pillow, pyautogui, pyperclip, fpdf2, and gemini-2.5-flash.

  1. 6-Layer Deterministic Safety Architecture:

(a). Layer 1 — Zero-Byte Rehearsal (_rehearse & _placeholder_copy): When organizing external user folders, write_script constructs a temporary mirror of zero-byte placeholder files with matching timestamps so rehearsal never reads private file contents and never risks real data.

(b). Layer 2 — Kernel-Level Python Audit Hook (sandbox_runner.py): In addition to static AST validation (static_check) that blocks eval, exec, import, and hardcoded absolute paths, our sandbox attaches a runtime sys.addaudithook that blocks all network sockets (socket.connect), subprocess spawns (subprocess.Popen, os.system), and access to sensitive directories (.ssh, .aws, .env, credentials, browser cookies). On user folders outside AlexaWorkspace, SANDBOX_NONDESTRUCTIVE=1 strictly forbids deleting or overwriting any existing file.

(c). Layer 3 — Visual Prompt-Injection & Pixel-Drift Guards (screen_control.py): Screen text is treated strictly as untrusted data. Before executing any click, _region_changed() compares the live pixel region against the planning screenshot using ImageChops.difference and aborts if a window or popup shifted under the cursor. _refuse_unsafe_window() inspects active Win32 window titles (ctypes.windll.user32) and refuses to send keystrokes to PowerShell, cmd.exe, Terminal, WSL, or Windows Security prompts.

  1. Alexa+ Multimodal Web Simulation (Alexa-Web-Simulation):

(a). Backend (Simulated_backend): A FastAPI server running a LangGraph agent (agent.py & main.py) that connects to the MCP server at runtime via MultiServerMCPClient (langchain-mcp-adapters) over streamable_http and mcp.ClientSession.

(b). Frontend (Simulated_frontend): A split-screen React + Vite + Tailwind CSS Echo Show Multimodal Interface featuring a reactive Live Voice Orb (Web Speech API STT/TTS), real-time MCP tool execution trace logs, interactive PDF report viewers, unified code diff approval cards, and smart path-to-speech sanitization so Alexa speaks natural folder names aloud instead of raw file paths.

Challenges we ran into

  1. Preventing the Vision Model from Clicking Its Own UI: When testing ask_about_screen and interact_with_screen, the agent could see its own simulated web app on the monitor and occasionally tried to click its own chat buttons or read its own transcript as instructions. We solved this at the OS level using Win32 APIs (ctypes.windll.user32.EnumWindows) to detect the bounding rectangle of the assistant's window (_assistant_rects) and paint solid black masks (_mask_regions) over it before sending any screenshot to the vision model.

  2. Eliminating Statistical Hallucinations on Messy Datasets: Early tests showed that passing raw CSVs or web snippets directly to an LLM resulted in invented percentages and skewed trends caused by partial first/last months in time-series data. We re-engineered analyze_data.py so Python code coerces currencies/dates, automatically detects and drops partial edge periods (< 85% coverage), and computes exact $R^2$ linear fits and IQR outliers before the LLM ever sees the evidence.

  3. Making File Automation Truly Reversible on User Folders: Copying gigabytes of files just to preview or undo a folder organization task in Downloads would be too slow. We solved this by creating zero-byte placeholder trees (_placeholder_copy) for instant dry-run rehearsals and using fast size:mtime_ns digest matching (_diff) paired with SANDBOX_NONDESTRUCTIVE=1 so undo_last_script can move every file back to its exact original path instantaneously without duplicating large files.

  4. I also researched about the connection between Alexa and customer's laptop.

Accomplishments that we're proud of

  1. True Zero-Client-Code Portability: Our LangGraph agent (agent.py) has zero hardcoded tool schemas and zero hardcoded chaining instructions—it learns how to chain ask_about_screen $\rightarrow$ analyze_data and write_script $\rightarrow$ run_script purely from the MCP server's initialize() handshake.

  2. Surgical Code Fixing with AST Validation & SHA-1 Undo: Watching AutoLap spot a bug on a live VS Code screen, resolve the file path from the OS window title bar, verify the syntax of its fix via ast.parse(), display a clean unified diff, patch the file on disk upon voice approval, and revert it byte-for-byte on "Undo" feels like magic.

  3. Defense-in-Depth Sandbox: Combining AST static analysis, zero-byte rehearsal copies, runtime sys.addaudithook kernel event interception, pixel-drift detection, and SHA-1 undo manifests into a cohesive MCP server that makes desktop AI genuinely safe for everyday consumers.

What we learned

  1. Determinism + LLMs Beat Pure LLMs Every Time: Using LLMs for planning and natural-language synthesis while delegating math (pandas/numpy), syntax verification (ast), and security enforcement (sys.addaudithook, ImageChops) to deterministic code produces dramatically higher reliability.

  2. Designing for Voice + Screen Multimodality: A great Alexa+ experience requires separating what is spoken from what is displayed. Returning a concise 4–5 sentence spoken_summary alongside rich visual artifacts (PDF dossiers, markdown tables, and approval diffs) prevents voice fatigue while maximizing utility.

What's next for AutoLap

  1. Cloud-to-Local Companion Tunnel for Physical Echo Devices: In a production consumer rollout, users will install AutoLap as a lightweight background system-tray daemon on Windows, macOS, or Linux. Upon signing in with their Amazon account, the daemon establishes an authenticated, outbound TLS/WebSocket relay tunnel to the Alexa+ Cloud. When a user speaks to a physical Echo Show or Echo Dot on their desk, the cloud Alexa+ reasoning engine routes MCP Streamable HTTP tool calls down that encrypted tunnel to the local laptop and streams the multimodal artifacts back to the Echo Show display.

  2. Cross-Platform OS Accessibility Trees: Expanding screen_control.py beyond Windows user32 hooks to integrate native macOS Accessibility APIs and Linux AT-SPI for even faster element targeting alongside vision coordinates.

  3. Scheduled Ambient Workflows: Allowing users to set recurring voice routines—such as "Alexa, every Friday at 5 PM, tidy my Downloads folder and generate a PDF summary of my weekly sales CSV".

Built With

Share this project:

Updates

Submission history