SpatialDesk

• 

  Her request, in Spanish, heard by Amazon Transcribe: Nova 2 Lite answers with comfort changes, each with its reason.

• 

  Her companion asks how it feels, and the conversation keeps the context. Nothing moves until she applies it.

• 

  She saves the setup that works: five comfort settings only, kept in this browser, restored with review on another visit.

• 

  One switch runs every control: the buttons light up in turn, a thick amber frame on the one lit, and one press chooses.

• 

  Fit to my reach: one pinch where her hand is comfortable measures distance, height and side. Review, then Apply.

• 

  A session at her pace: she chooses her time and her energy, and the agent proposes one task and one short block.

• 

  A return point when she has to stop: her own next step, saved locally, ready when she comes back.

• 

  The AI log: every step the agent took, with its reason and its own Undo, even after later changes.

• 

  In her room, in passthrough: the board stands on the real desk and the other boards hang on the walls, anchored.

• 

  In VR, a calm dusk studio: a walnut desk, a focus plant that grows with every block, and soft spatial sound.

Inspiration

"Hoy es un mal día. Acércalo todo." Today is a bad day. Bring everything closer.

Someone I love, who lives with a disability, said that from her sofa. On a bad day her neck hurts, her arms tire and her aim shakes; spatial apps expect you to turn, reach and pinch the air. Accessibility usually means a menu of toggles, changed on the day you have least to spare. In a headset the room can move, so the room should adapt.

What it does

SpatialDesk is a seated, hands-first workspace: task boards, focus timers and an assistant, arranged around the body you have today.

• It fits you. A one-minute setup asks where you sit and has you pinch where your hand rests; that sets every panel's distance, height and side.
• You say what you need. Amazon Transcribe hears you in English or Spanish; Amazon Nova proposes changes with reasons. Nothing moves until you tap Apply; panels glide, and Undo always works.
• Every way in. Hands (poke, pinch), gaze dwell (eyes where tracked, else head), voice, steady aim for tremor, one-switch scanning. Rest your head on a card and say "move this to done".
• Your real room. In passthrough, Fit room reads the detected planes: your board stands on your real desk, the others on your walls, and a persistent anchor restores them tomorrow. Tested in all 13 rooms of Meta's Synthetic Environment Module.
• Any field of view. The layout reads each eye's projection and keeps essential panels in the comfortable view, down to Meta VR Glasses' 58° × 54°.
• Help from a phone. After you press Allow, a carer's phone shows your Today board and one large button, Help / Recalibrate, that opens the setup in your headset. They ask; you decide.
• A reason to come back. A focus plant grows a leaf with every focus block.

How we built it

Meta's Immersive Web SDK, in TypeScript over WebXR. One Kotlin Multiplatform engine holds the layout and the rules: the headset runs it as JavaScript; the Spring Boot server runs it on the JVM to validate every Nova 2 Lite plan (Bedrock Converse, typed tool) before you see it. Guardrails enforce intent ("closer" must end nearer), and a built-in planner answers when the AI cannot: SpatialDesk works without it. Audio streams from the browser straight to Transcribe.

199 Playwright scenarios drive the production build in Meta's IWER emulator; a CI gate holds frame time (p95 24.6 ms, CPU throttled 3x) under Meta's IWSDK starter scene (29.4 ms), at 78 draw calls.

Challenges we ran into

Turning comfort into geometry: reach, head-turn limit, tilt, side and text size conflict, so one shared engine solves them, and every comfort rule is a test that fails the build.

What's next for SpatialDesk

Target launch date: March 2027, on the Meta Horizon Store as an installable web app, free at launch. Before then: testing on Quest 3 and 3S hardware, sessions with disabled users and carers, and comfort profiles for good and bad days.

Built With

• amazon-bedrock
• amazon-ecs
• amazon-nova
• amazon-transcribe
• amazon-web-services
• aws-cdk
• aws-fargate
• cloudfront
• elastic-load-balancing
• kotlin
• kotlin-multiplatform
• meta-immersive-web-sdk
• meta-iwer
• playwright
• spring-boot
• three.js
• typescript
• vite
• web-audio
• webxr