Steward: Consent-Gated Alexa+ Browser Co-Pilot

Voice-first AI browser agent for Alexa+ & Web with a Personal Context Vector RAG vault, "No Safe Path" screen co-pilot, and a Cedar / Amazon Verified Permissions gate for payments & submissions.

Comment

Inspiration

Every browser agent shipping today assumes a sighted user sitting at a desktop monitor with a hand on the mouse, ready to catch mistakes in real time. When a user delegates a web task through Alexa+, speaks in a local Indian language (ta-IN, hi-IN, en-IN), or relies on screen-reader narration, three failure modes of conventional browser agents become catastrophic:

  1. Hallucinated Claims & Context-Oblivious Searches: Agents invent field values, dump raw conversational prompts into search bars instead of synthesizing queries from the user's uploaded resume or documents, or claim an application was submitted when the page actually failed.
  2. Silent Stops When No Safe Automatic Path Exists: When an agent lands on a multi-option search results page or an ambiguous checkout screen, it either stops with no explanation or clicks a destructive button without asking.
  3. Unapproved Financial Payments & Dark-Pattern Opt-Ins: Portals pre-tick paid add-ons (such as optional ₹499/year insurance) or execute irreversible Submit and Pay actions without cryptographic human consent.

We built Steward so anyone can delegate real-world web workflows—applying for jobs using their resume, shopping and checking out with saved shipping preferences, or registering for government schemes like PM-KISAN—through Alexa+ (via Streamable HTTP MCP 2025-11-25) and our 4-Page Web/Mobile PWA Console, while keeping every sensitive submission and payment under strict human and Amazon Verified Permissions (Cedar) control.

What It Does

Steward is a human-controlled, cross-platform AI browser agent organized into a 4-Page Application Shell (#/workspace, #/vault, #/security, #/audit) and a self-hosted Streamable HTTP Model Context Protocol (POST /mcp, spec 2025-11-25) server for Alexa+:

  • Personal Context & Vector RAG Vault (#/vault): Users upload PDF (pdf-parse), DOCX (mammoth), TXT, MD, CSV, JSON, or HTML documents—or paste resume/shipping notes directly. Steward chunks documents, embeds them into normalized vectors using Amazon Bedrock Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) with a deterministic 256-dim offline fallback, extracts canonical facts with line-level provenance, refuses credentials (Password, OTP, CVV), and masks secret identifiers ([redacted: secret]).
  • Context-Grounded Search & Form Auto-Fill (#/workspace): When a user says "Based on the details from my resume, find me jobs which will suit me", Steward retrieves their authorized employment and education facts (role, skills, qualification, location) and synthesizes a high-signal search query (e.g., "Software Development Intern TypeScript Python Java C++ jobs Chennai") rather than typing raw conversational filler. On job portals (/jobs), Steward dynamically ranks open roles against the user's resume (99% Resume Match) and auto-fills application forms.
  • Dark-Pattern Radar: Automatically detects and unticks pre-selected paid add-on checkboxes (data-steward-preselected / defaultChecked) before reaching the consent boundary.
  • "No Safe Path" Interactive Screen Briefing & Guided Co-Pilot: Whenever Steward reaches a page where no single automatic safe action exists (such as a multi-product search results page), it transitions to no_safe_path, dictates a grounded summary of visible prices, fields, and options, and presents clickable Recommended Next Steps. Users can command reversible steps ("Add Nike Revolution 7 to cart") or sensitive financial steps ("Buy this and do the payment").
  • Cryptographic Consent & Cedar / Amazon Verified Permissions Gate (#/security): Every irreversible action (Steward::Action::"Submit", Steward::Action::"Pay", Steward::Action::"Cancel") pauses at a Consent Required modal bound to length-prefixed SHA-256 digests (actionHash, parameterDigest, previewHash) and is evaluated against 7 Cedar policies in Amazon Verified Permissions (ap-south-1).
  • First-Class Human Handoff & Reconstructable Audit (#/audit): Steward never guesses at CAPTCHAs or OTPs—it freezes partial progress and hands control to the human. Every step is recorded in a value-free, append-only audit log that reconstructs the entire run into plain speech.

How We Built It

Steward is architected as a strict TypeScript monorepo (packages/core, packages/cedar, packages/context, packages/adapters, apps/web, apps/landing, apps/cli):

  1. Alexa+ Streamable HTTP MCP Server (apps/web/src/mcp.ts & mcp.json): Implements JSON-RPC 2.0 over Streamable HTTP (POST /mcp, protocol version 2025-11-25), exposing 12 consent-gated tools (steward_start_task, steward_step, steward_direct_action, steward_approve_consent, steward_deny_consent, context_query_rag, etc.) plus a live interactive Alexa+ MCP Inspector in the web console.
  2. PII-Free Amazon Bedrock Planner (packages/core/src/model-planner.ts & packages/adapters/src/model/bedrock.ts): Uses the Amazon Bedrock Converse API (apac.amazon.nova-lite-v1:0) with a single forced tool schema (emit_plan). Personal field values never enter the LLM prompt—the model receives only hasProfileValue: boolean and control roles, while trusted code supplies the citizen's values at execution time.
  3. Injective Digest Binding (packages/core/src/digest.ts): To prevent parameter-concatenation collisions (("a", "bc") vs ("ab", "c")), every consent record binds to length-prefixed SHA-256 digests over sorted keys:

$$ \text{parameterDigest}(P) = \text{SHA256}\left( \bigoplus_{k \in \text{sorted}(P)} |k| : k \,|\, |P[k]| : P[k] \right) $$

$$ \text{actionHash}(A) = \text{SHA256}\left( |\text{portal}|:\text{portal} \,|\, |\text{type}|:\text{type} \,|\, |\text{targetRef}|:\text{targetRef} \right) $$

  1. Semantic Vector RAG Ranking (packages/context/src/embedding.ts): Document chunks and user queries are embedded into unit-normalized vectors $\mathbf{q}, \mathbf{c} \in \mathbb{R}^d$ and ranked by cosine similarity in-process ($<5\text{ ms}$, $\$0$ idle cost):

$$ \text{sim}(\mathbf{q}, \mathbf{c}) = \frac{\mathbf{q} \cdot \mathbf{c}}{|\mathbf{q}|_2 \, |\mathbf{c}|_2} $$

  1. Amazon Verified Permissions & Single-Table DynamoDB (packages/cedar/src/avp.ts, packages/adapters/src/store/dynamo.ts): Evaluates 7 Cedar policies live in ap-south-1 (Policy Store NSFzNHyeXp5DtXDfokxk2X) and enforces single-use consent consumption via DynamoDB ConditionExpression (attribute_not_exists(usedAt) AND #used = :false).

Challenges We Ran Into

  • Real-World Resume Layouts & Context-Oblivious Search Queries: Standard PDF resumes often combine contact details with pipe delimiters (email | +91... | Chennai, India), omit https:// on LinkedIn/GitHub URLs, and split skills across multiple sub-headings (Languages:, Frontend:, Databases & Cloud:). Early on, asking Steward to "find jobs based on my resume" on Google caused it to treat Google's search box (q) and hidden telemetry (csi) as missing profile fields and type the raw prompt into Google. We solved this through a 12-suite Test-Driven Development (TDD) loop: upgrading LabelledFactExtractor to parse multi-segment PDF headers and consolidate skill sections, filtering search/telemetry inputs out of contextFieldNames, and building synthesizeSearchQuery to construct skill- and location-grounded search queries directly from the user's resume.
  • Live Browser Search Overlay Interception: On sites like google.com, typing into q opens an autocomplete overlay that covers the first hidden btnK ("Google Search") input. We upgraded observePage and actReversibleOnPage in page-session.ts to filter hidden DOM elements, prefer :visible locators, and automatically fall back to pressing Enter on the search input when an overlay blocks pointer clicks.
  • Parity Between Local cedar-wasm and AWS Verified Permissions: AVP rejects empty Cedar schema namespaces (""), whereas cedar-wasm accepts them. We unified both evaluators under the Steward:: namespace (Steward::Citizen, Steward::Action, Steward::PortalAction) and verified 38/38 live parity.

Accomplishments That We're Proud Of

  • 12/12 End-to-End TDD Specification Suites Passing: Every core guarantee—multi-format upload, vector RAG retrieval, dark-pattern removal, resume-grounded Google job search, dynamic /jobs role matching, "No Safe Path" screen briefing, user-directed payment authorization (Steward::Action::"Pay"), and Alexa+ MCP tool execution—is verified in tests/tdd-suite.ts.
  • Live AWS Production Deployment in ap-south-1: Deployed on AWS App Runner (https://vppap4me86.ap-south-1.awsapprunner.com) wired to live Amazon Verified Permissions (NSFzNHyeXp5DtXDfokxk2X), with atomic consent consumption proven under 25-way concurrent load against live DynamoDB.
  • Zero-PII Prompt Architecture: Even when planning with Amazon Bedrock Nova Lite, raw personal values (Aadhaar numbers, phone numbers, addresses) never enter the LLM context window or the permanent audit log.

What We Learned

  • Security Boundaries Must Live Outside the Prompt: Prompt instructions alone cannot stop an agent from clicking "Pay Now" or leaking sensitive fields. Enforcing injective SHA-256 preview digests, a PII-free planner payload, and externalized Cedar policies at the tool boundary makes safety structural and testable.
  • Co-Piloting Beats Blind Autonomy on Ambiguous Screens: Pausing to dictate a structured Screen Briefing and offering clickable/spoken next steps when no safe automatic path exists turns dead ends into collaborative human-AI decisions.

What's Next for Steward

  • Multi-Device Alexa+ Session Continuity: Start a resume-matched job search or shopping task by voice on an Echo Show via Alexa+ MCP, and receive the cryptographic consent card on mobile to approve the final submission or UPI payment.
  • Expanded Multilingual Voice Synthesis: Adding regional Indian voice packs (te-IN, kn-IN, ml-IN, bn-IN) for rural Common Service Centre (CSC) kiosks.

Built With

Share this project:

Updates

Submission history