DeskPilot — Autonomous Multi-Agent Desktop

Autonomous multi-agent desktop workspace powered by Bedrock Nova Pro. Organizes files, generates verified Excel & Word reports, and sweeps clutter with a 3-tier Human-in-the-Loop safety engine.

Comment

🚀 DeskPilot — Autonomous Multi-Agent Desktop Workspace

"Meet your autonomous desktop co-pilots — they manage your files, draft executive reports, build verified financial spreadsheets, perform deep research, and keep your workspace clean, interrupting you only when a genuine human decision is required."

Built specifically for the AWS "Agents for Humans" Hackathon (Professional Agents Track), DeskPilot is a production-grade multi-agent autonomous desktop workspace. Powered by Amazon Bedrock Nova Pro (amazon.nova-pro-v1:0) and the Strands Agents SDK, DeskPilot bridges cutting-edge cloud agentic reasoning with local Windows desktop automation, verifiable document generation, and an unbypassable 3-tier Human-in-the-Loop (HITL) safety architecture.

💡 Inspiration

Every professional faces the same daily digital friction:

  1. Desktop Clutter & File Chaos: Dozens of loose downloads, screenshots, invoices, and draft spreadsheets scattered across the Desktop.
  2. Repetitive Office Document Work: Manually extracting numbers from PDF invoices, building Excel spreadsheets with formulas, or formatting multi-page Word reports.
  3. The "Chatbot Disconnect": Traditional AI chatbots live in a browser tab. They can generate text snippets, but they cannot organize your desktop, verify files on disk, or safely interact with your local operating system.
  4. The Autonomy vs. Safety Dilemma: Full agent autonomy is dangerous (risk of deleting important files), while asking for permission on every single file move causes decision fatigue.

We built DeskPilot to solve this: an autonomous co-pilot that acts directly on your desktop with privileged local tools, yet remains strictly governed by a 3-tier safety gate that interrupts the user only when a genuine human decision is required.

🌟 What DeskPilot Does

DeskPilot transforms how knowledge workers interact with their operating system through six core pillars:

1. 💬 Modern Multi-Turn Conversational Workspace

  • Real-Time Streaming: Token-by-token streaming with live tool execution step badges (e.g. 🔧 organize_files (running)...).
  • Collapsible Reasoning Accordions: Captures Bedrock Nova Pro <thinking>...</thinking> reasoning tokens in sleek disclosure blocks so internal thought processes don't clutter executive answers.
  • Deliverable Action Cards: Generated .docx, .xlsx, and .pdf files appear as interactive badges with 1-click "Open File" and "Show in Folder" buttons.

2. 📁 1-Click Autonomous Desktop Organization (@tool organize_files)

  • Scans loose desktop clutter and categorizes items into dedicated subfolders:
    • 📄 Documents (.docx, .txt, .md, .rtf)
    • 📊 Spreadsheets (.xlsx, .csv)
    • 📑 PDFs (.pdf)
    • 🖼️ Images (.png, .jpg, .svg)
    • 💻 Code (.py, .js, .json, .sql)
    • 📦 Archives (.zip, .rar, .7z)
  • Single Batch Approval: Eliminates confirmation spam by asking for approval once for the entire batch rather than prompting for every individual file.
  • Safe Execution: Preserves system files (desktop.ini), handles file locks with retries, and avoids naming collisions.

3. 📝 Verified Executive Document & Spreadsheet Generation

  • Microsoft Excel Workbooks (create_excel_workbook): Creates formatted .xlsx files with real numeric values (int/float), multi-currency formatting (PKR, EUR, USD, INR), and working native formulas (e.g., =SUM(B2:B10)).
  • Microsoft Word Reports (create_word_report): Generates formatted .docx reports with hierarchical headings, bullet points, callout boxes, and styled tables.
  • Automated Self-Verification: Immediately validates generated files on disk via verify_word_document and verify_excel_workbook before presenting them to the user.

4. 🛡️ 3-Tier Human-in-the-Loop Trust Engine

Enforces strict autonomy boundaries across all 37 tools:

  • 🟢 Green (Autonomous): Read-only sweeps, research, and new document generation run silently with zero interruption.
  • 🟡 Yellow (Confirmation Required): Moving, renaming, or batch organizing files. Features "Approve All" session caching for uninterrupted batch workflows.
  • 🔴 Red (Unbypassable Approval Gate): Permanent file deletions (safely routed to the Windows Recycle Bin via send2trash) and software installations (strictly validated against an explicit WINGET_ALLOWLIST).

5. 👁️ Proactive Ambient Watcher Engine

A background daemon thread that periodically sweeps the desktop workspace for loose clutter, %TEMP% junk accumulation, and incoming PDF invoices. When an issue is detected, it sends native Windows Toast Notifications and in-app modal cards offering 1-click autonomous resolution.

6. 🪄 Natural-Language Custom Agent Builder & Dynamic Forms

  • Describe any desired workflow in natural language (e.g., "Student Research Assistant extracting citations from papers"), and Bedrock synthesizes the persona and selects an authorized tool subset.
  • When an agent lacks onboarding information (e.g., budget numbers or currency), it calls ask_user_form to display an interactive popup form, saving structured answers to persistent memory (user_profiles.json).

🛠️ How We Built It

DeskPilot is architected into five decoupled layers:

  1. AI Brain & Reasoning: Amazon Bedrock Nova Pro (amazon.nova-pro-v1:0) invoked through the Bedrock Converse API for streaming reasoning and strict tool adherence.
  2. Multi-Agent Runtime: Strands Agents SDK orchestrating dynamic agent lifecycles, persona injection, and tool whitelisting.
  3. Privileged Tool Execution Suite: 37 custom Python tools spanning:
    • Document Intelligence: python-docx, openpyxl, pdfplumber, PyMuPDF
    • Safe OS Operations: send2trash, psutil, Windows Package Manager (winget)
    • Web Intelligence: DuckDuckGo Search API & BeautifulSoup4
  4. Desktop Shell & Presentation: Python 3.12 with Microsoft Edge WebView2 (pywebview) and a responsive Glassmorphism SPA (Vanilla HTML5, CSS Grid/Flexbox, ES6 JavaScript).
  5. Universal Portability: A local Flask REST & SSE server allowing DeskPilot to run in headless Docker containers, cloud environments (AWS App Runner / ECS), or as a standalone Windows .exe via PyInstaller.

⚡ Challenges We Ran Into

  • Preventing AI Hallucinations in File Paths & Math: Early on, LLMs tended to hallucinate dollar signs ($) or invent file paths. We engineered a universal DocumentResolver that automatically searches across Desktop, Documents, and output folders, coupled with strict guardrails requiring real numeric formulas (=SUM()) and profile-based currency detection.
  • Approval Fatigue vs. Operating System Safety: Asking confirmation for every single file moved during a desktop clean-up created unbearable friction. We solved this by designing a Batch Approval & Session Caching Engine: batching all move operations into a single structured summary and caching approvals within the turn.
  • Synchronous Human-in-the-Loop on Async Agent Threads: Strands agents run on background threads while the UI operates asynchronously. We built a thread-safe synchronization bridge with Python threading.Event() to cleanly pause agent reasoning while the user fills dynamic forms (ask_user_form) or reviews Yellow/Red tier modals.

🏆 Accomplishments We're Proud Of

  • True Desktop Autonomy with Human Trust: DeskPilot isn't just a chatbot that writes code snippets; it actually organizes your physical workspace and installs approved software safely.
  • Zero Placeholder Culture: Every generated deliverable (.xlsx, .docx, .pdf) contains real formatted data and is self-verified on disk before the user even sees the link.
  • Native Windows & Cloud Parity: Works seamlessly as a native desktop application with Windows Toast notifications, while simultaneously running containerized in Docker with sub-second responsiveness.
  • Comprehensive Test Suite: 42+ unit and integration tests covering document verification, trust tier classification, packaging, and ambient sweep logic.

📚 What We Learned

  • Amazon Bedrock Nova Pro delivers an outstanding balance of speed, cost efficiency, and structured tool-calling reliability compared to generic LLM endpoints.
  • Building autonomous agents for humans requires designing for graceful degradation and explainability—collapsible <thinking> accordions and live tool badges build immense user confidence in autonomous decisions.

🔮 What's Next for DeskPilot

  • Voice-Activated Co-Pilot: Hands-free voice commands via Amazon Transcribe and Polly.
  • Local Semantic Desktop Search: Integrating Amazon Titan embeddings with ChromaDB to allow semantic cross-document search across all local PDFs and Word documents.
  • Multi-Monitor Window Management: Expanding system tools to organize open application windows and workspace layouts.

Built With

Share this project:

Updates

Submission history