AutoLap

Making Alexa+ work autonomously with the customer's laptop to enhance their productivity

Comment

Inspiration

We built what voice assistants currently lack in everyday life: true desktop productivity. today, smart speakers are great at setting kitchen timers, booking groceries, or playing music, yet they remain completely blind and hands-off when it comes to the device where students, developers, and knowledge workers spend eight hours a day—their laptop. Almost every customer purchasing an Alexa+ device already owns a personal computer, and that single realization inspired AutoLap.

We asked ourselves: What if Alexa+ could act as an autonomous, ambient co-pilot for your laptop—capable of seeing your screen, fixing broken code, researching live market numbers, running statistical data analysis, and organizing messy folders via voice?

However, giving an LLM raw control over a user's mouse, keyboard, and filesystem is normally a massive security hazard. If an AI hallucinates a terminal command, falls for visual prompt injection on a webpage, or deletes important personal files, trust is broken instantly. We set out to build AutoLap: a self-hosted Alexa+ MCP Server and multimodal Echo Show Web Simulation where every calculation and safety guardrail is enforced deterministically in Python code rather than left to LLM "vibes".

What it does

AutoLap turns Alexa+ into a voice-first, multimodal laptop productivity hub powered by 10 specialized Model Context Protocol (MCP) tools served over Streamable HTTP:

  1. Screen Vision & Visual Chaining (ask_about_screen): Captures the live primary display via mss, automatically masks out the assistant's own UI window so it never reads or clicks itself, and answers visual questions or transcribes on-screen charts and metrics directly into our data analysis engine.

  2. Surgical On-Disk Code Fixing & Guarded GUI Control (interact_with_screen & approve_screen_plan): Controls the laptop's mouse and keyboard safely or fixes code errors visible on screen. Instead of blindly typing into an editor, AutoLap reads OS window titles to locate the exact source file on disk, verifies there are no unsaved editor buffers, parses the proposed patch with ast.parse() or json.loads(), presents a unified diff (difflib.unified_diff) for human-in-the-loop voice approval, and atomically swaps the patched file on disk.

  3. Evidence-First Data Analysis (analyze_data): Processes spreadsheets (.csv, .xlsx, .xls, .json), text files, or numbers transcribed straight off the screen. To eliminate LLM math hallucinations, Pandas and NumPy compute all statistics first—stripping currency/percentage symbols (₹, $, €, %), excluding incomplete edge date periods (_partial_periods) so they do not skew trends, and calculating linear regression $R^2$ trend strength, IQR outliers, segment shares, and Pearson correlations. The LLM only interprets verified figures, returning a 4–5 sentence spoken summary alongside a formatted PDF report.

  4. Multi-Angle Web Research to Dual PDFs (research_topic): Plans 3 to 8 targeted DuckDuckGo news and web queries (specifically hunting for hard metrics and downside risks), compiles a raw Research Dossier PDF, extracts stated numbers into a structured entity x metric pivot CSV, and automatically chains into analyze_data to generate a second Analytical Overview PDF.

  5. Zero-Risk File Automation with Dry-Run Rehearsal (write_script & run_script): Generates custom Python scripts to organize folders, rename files, or clean data. Before touching a single real file, AutoLap rehearses the script on a throwaway copy inside a restricted sandbox, computes an exact diff of what will be moved, created, or modified, and waits for explicit user confirmation.

  6. Cryptographic Multi-Step Undo (undo_last_script): Maintains an undo manifest (manifest.json) with SHA-1 file digests for every script execution and code fix. Saying "Alexa, undo that" reverses file moves, restores modified backups, and reverts code patches—while checking SHA-1 hashes first so it never overwrites a file the user edited manually afterward.

  7. Instant Safe Local Search & Disambiguation (open_item): Performs a bounded breadth-first search across the computer to open documents, images, folders, URLs, or direct YouTube videos by name without touching executable binaries or system directories.

  8. extra_productivity folder: This gives even more and it incredibly productivity. this is switched off by default to reduce the set-up time. if an user ever wants to automate tasks, there are tools like; git_bootstraper, git_sync, notifier, etc. Go through Readme and requirements under this folder(not root), to switch these on.

How we built it

We architected AutoLap into two decoupled repositories to prove universal MCP compatibility:

  1. Standalone FastMCP Server (mcp_alexa):

(a). Built with Python and FastMCP running over the modern MCP Streamable HTTP transport (http://127.0.0.1:8000/mcp).

(b). Zero-Code Client Onboarding via SERVER_INSTRUCTIONS: Our MCP server embeds all multi-tool chaining rules inside the MCP instructions handshake field (instructions.py). When any MCP-compliant client (Alexa+, LangGraph, Claude, or MCP Inspector) initializes a session, it dynamically discovers all 10 tools and their chaining workflows without requiring custom client-side prompts.

(c). Data, Vision & PDF Pipeline: Built with pandas, numpy, ddgs (DuckDuckGo Search), mss, Pillow, pyautogui, pyperclip, fpdf2, and gemini-2.5-flash.

  1. 6-Layer Deterministic Safety Architecture:

(a). Layer 1 — Zero-Byte Rehearsal (_rehearse & _placeholder_copy): When organizing external user folders, write_script constructs a temporary mirror of zero-byte placeholder files with matching timestamps so rehearsal never reads private file contents and never risks real data.

(b). Layer 2 — Kernel-Level Python Audit Hook (sandbox_runner.py): In addition to static AST validation (static_check) that blocks eval, exec, import, and hardcoded absolute paths, our sandbox attaches a runtime sys.addaudithook that blocks all network sockets (socket.connect), subprocess spawns (subprocess.Popen, os.system), and access to sensitive directories (.ssh, .aws, .env, credentials, browser cookies). On user folders outside AlexaWorkspace, SANDBOX_NONDESTRUCTIVE=1 strictly forbids deleting or overwriting any existing file.

(c). Layer 3 — Visual Prompt-Injection & Pixel-Drift Guards (screen_control.py): Screen text is treated strictly as untrusted data. Before executing any click, _region_changed() compares the live pixel region against the planning screenshot using ImageChops.difference and aborts if a window or popup shifted under the cursor. _refuse_unsafe_window() inspects active Win32 window titles (ctypes.windll.user32) and refuses to send keystrokes to PowerShell, cmd.exe, Terminal, WSL, or Windows Security prompts.

(d). Layer 4 — Code-Decided Risk Tiers & Human Approval (screen_control.py RISK, approve_screen_plan; script_writer.py run_script): Risk is decided by a fixed table in Python, never by the model. Opening links, YouTube, documents and scrolling run at once. Clicks, typing, shortcuts and code edits return a plain-language plan (and a code diff) that runs only after the user says yes. Plans expire after 3 minutes and run once. Shortcuts come from an allow-list (Alt+F4 and Win+R are rejected), and keyboard actions must follow a click. Scripts also need a clean rehearsal and a yes, run once, and expire after 10 minutes.

(e). Layer 5 — Path & File Policy (fs_tools.py is_blocked, screen_control.py SAFE_FILE_EXT, sandbox_runner.py): Drive roots, the whole user profile, Windows/Program Files/AppData, credential folders (.ssh, .aws) and the assistant's own program folder are off limits for organizing. By name, only documents, images and folders can be opened, never .exe, .bat, .ps1, .py, .js or .lnk. In user folders the sandbox allows only move, rename and create, so nothing can be deleted, overwritten or moved as a folder. Files outside the workspace are analyzed read-only, up to 25 MB, and never from protected folders.

(f). Layer 6 — Integrity-Checked Undo & Audit Trail (script_writer.py manifests, record_code_fix, screen_actions.jsonl): A backup is saved before a code fix is written, and every script run records an undo manifest with content fingerprints (SHA-1 in the workspace, size and modified time in large user folders). "Undo" restores moved, deleted and edited files, and refuses to overwrite anything the user changed afterwards. A code fix is also refused if the file changed after it was read. Every executed screen action is logged with time, goal and result in AlexaWorkspace/logs/screen_actions.jsonl.

  1. Alexa+ Multimodal Web Simulation (Alexa-Web-Simulation):

(a). Backend (Simulated_backend): A FastAPI server running a LangGraph agent (agent.py & main.py) that connects to the MCP server at runtime via MultiServerMCPClient (langchain-mcp-adapters) over streamable_http and mcp.ClientSession.

(b). Frontend (Simulated_frontend): A split-screen React + Vite + Tailwind CSS Echo Show Multimodal Interface featuring a reactive Live Voice Orb (Web Speech API STT/TTS), real-time MCP tool execution trace logs, interactive PDF report viewers, unified code diff approval cards, and smart path-to-speech sanitization so Alexa speaks natural folder names aloud instead of raw file paths.

How it connects to a real Alexa+ device (design)

→Alexa+ runs in Amazon's cloud, and a laptop sits behind a home router, so Alexa can't reach it directly. Our design:

  1. The user installs a small companion app on the laptop and signs in with their Amazon account.
  2. The companion opens an outbound, authenticated TLS connection to a relay in the cloud, so no ports are opened on the laptop.
  3. Our MCP server is registered as an Alexa+ add-on pointing at the relay's HTTPS MCP endpoint, protected by OAuth account linking.
  4. When the user says "Alexa, fix the error on my screen", Alexa+'s own model picks our tool and calls it over MCP Streamable HTTP. The relay forwards the call to the local server, which runs the tool and returns the result.
  5. Alexa speaks the 4–5 sentence spoken_summary. PDFs, code diffs, and approval cards appear on an Echo Show or the laptop, and risky actions still need a spoken yes.

→Built today: the MCP server, which is the part that doesn't change, and a simulated Alexa+ client. →Not built: the companion relay, add-on registration, and OAuth, which need Alexa developer access and a cloud service. Amazon's add-on guidance also expects fast responses (about 500 ms), so slower tools like screen vision and research would need a quick spoken acknowledgement while the result is prepared.

Challenges we ran into

  1. Preventing the Vision Model from Clicking Its Own UI: When testing ask_about_screen and interact_with_screen, the agent could see its own simulated web app on the monitor and occasionally tried to click its own chat buttons or read its own transcript as instructions. We solved this at the OS level using Win32 APIs (ctypes.windll.user32.EnumWindows) to detect the bounding rectangle of the assistant's window (_assistant_rects) and paint solid black masks (_mask_regions) over it before sending any screenshot to the vision model.

  2. Eliminating Statistical Hallucinations on Messy Datasets: Early tests showed that passing raw CSVs or web snippets directly to an LLM resulted in invented percentages and skewed trends caused by partial first/last months in time-series data. We re-engineered analyze_data.py so Python code coerces currencies/dates, automatically detects and drops partial edge periods (< 85% coverage), and computes exact $R^2$ linear fits and IQR outliers before the LLM ever sees the evidence.

  3. Making File Automation Truly Reversible on User Folders: Copying gigabytes of files just to preview or undo a folder organization task in Downloads would be too slow. We solved this by creating zero-byte placeholder trees (_placeholder_copy) for instant dry-run rehearsals and using fast size:mtime_ns digest matching (_diff) paired with SANDBOX_NONDESTRUCTIVE=1 so undo_last_script can move every file back to its exact original path instantaneously without duplicating large files.

  4. Reaching a laptop from the cloud: Alexa+ can't connect inward to a laptop behind a router, so we designed an outbound companion relay (see How it connects to a real Alexa+ device).

Accomplishments that we're proud of

  1. True Zero-Client-Code Portability: Our LangGraph agent (agent.py) has zero hardcoded tool schemas and zero hardcoded chaining instructions—it learns how to chain ask_about_screen $\rightarrow$ analyze_data and write_script $\rightarrow$ run_script purely from the MCP server's initialize() handshake.

  2. Surgical Code Fixing with AST Validation & SHA-1 Undo: Watching AutoLap spot a bug on a live VS Code screen, resolve the file path from the OS window title bar, verify the syntax of its fix via ast.parse(), display a clean unified diff, patch the file on disk upon voice approval, and revert it byte-for-byte on "Undo" feels like magic.

  3. Defense-in-Depth Sandbox: Combining AST static analysis, zero-byte rehearsal copies, runtime sys.addaudithook kernel event interception, pixel-drift detection, and SHA-1 undo manifests into a cohesive MCP server that makes desktop AI genuinely safe for everyday consumers.

What we learned

  1. Determinism + LLMs Beat Pure LLMs Every Time: Using LLMs for planning and natural-language synthesis while delegating math (pandas/numpy), syntax verification (ast), and security enforcement (sys.addaudithook, ImageChops) to deterministic code produces dramatically higher reliability.

  2. Designing for Voice + Screen Multimodality: A great Alexa+ experience requires separating what is spoken from what is displayed. Returning a concise 4–5 sentence spoken_summary alongside rich visual artifacts (PDF dossiers, markdown tables, and approval diffs) prevents voice fatigue while maximizing utility.

What's next for AutoLap

  1. Cloud-to-Local Companion Tunnel for Physical Echo Devices: In a production consumer rollout, users will install AutoLap as a lightweight background system-tray daemon on Windows, macOS, or Linux. Upon signing in with their Amazon account, the daemon establishes an authenticated, outbound TLS/WebSocket relay tunnel to the Alexa+ Cloud. When a user speaks to a physical Echo Show or Echo Dot on their desk, the cloud Alexa+ reasoning engine routes MCP Streamable HTTP tool calls down that encrypted tunnel to the local laptop and streams the multimodal artifacts back to the Echo Show display.

  2. Cross-Platform OS Accessibility Trees: Expanding screen_control.py beyond Windows user32 hooks to integrate native macOS Accessibility APIs and Linux AT-SPI for even faster element targeting alongside vision coordinates.

  3. Scheduled Ambient Workflows: Allowing users to set recurring voice routines—such as "Alexa, every Friday at 5 PM, tidy my Downloads folder and generate a PDF summary of my weekly sales CSV".

How to test (free, about 5 minutes)

  1. Repo: https://github.com/mohan-c0des/MCP_Alexa , Public, license shown in About.
  2. You need Python 3.11+, a free Gemini API key from https://aistudio.google.com/apikey (no billing needed), and Node 18+ only if the client is web UI.
  3. Follow the README: start python server.py (MCP server at http://127.0.0.1:8000/mcp), then use MCP Inspector, the terminal agent, or the web UI.
  4. No key yet? MCP Inspector still lists all tools, and the tools that don't call Gemini (for example open_item search) run without one.

Built With

Share this project:

Updates

Submission history