Inspiration

Some days I would close my laptop in the evening and realize I hadn't spoken to anyone all day.

I run a development company in South Korea. As I got better at building things in Unity, more clients came to me, and the company grew enough to hire a team. Then AI changed the market for the work we did. Projects became scarce, and eventually I was running the company on my own again.

One thing I missed was being able to turn to someone beside me, point at my screen, and ask, “What do you think we should do with this?” That small gesture carries a lot. The other person can see what you mean, try an idea with you, and help you finish the work. I entered this competition to see how much of that experience I could build. Yonder is the project. Nilo is the teammate I wanted beside me.

What it does

I can hold up my coffee mug and say, “Hey, Nilo, make me one of these.” Nilo uses the Quest camera to identify the mug and make a textured 3D version. I can place it on my desk in mixed reality or ask him to bring it into my computer, where I can open the same model in a 3D editor.

Crossing the screen is useful because each side gives us a different way to work. In the room, I can show Nilo an object, point to a detail, or compare designs at their actual size. On the computer, we can use those results as files and tools. If I point at an icon on my monitor and ask for alternatives, we can look at them together in MR, then bring the PNG I choose back to my desktop.

I can also describe a tool I wish I had. In the demo, I ask for a Meta stock chart, watch it take shape in the work queue, and open it on my computer. I reach toward a button from outside the screen, and the chart responds to my hand. I can publish the finished module to the marketplace so another person can add that capability to their own Nilo.

How we built it

I built the mixed reality experience in Unity with OpenXR and hand tracking. The headset locates the physical monitor, then maps the spot I'm pointing at to a position on the desktop. When Nilo walks into the screen, the headset and desktop companion share the crossing time and position. Each draws the part of his body on its side of the glass. That's how one foot can already be on the desktop while the other is still in my room.

For real-time creation, I chose Lua. A Lua script is small enough to generate and load during a session. Unity already provides the rendering and interactions, so a new module only needs to describe its own behavior. That keeps the build lightweight enough for me to ask for a tool and use it while the task is still in front of me, without reinstalling Yonder.

Challenges we ran into

A monitor knows how many pixels it has. A headset knows where my hand is in the room. Getting them to agree on exactly where my fingertip was pointing was one of the hardest parts. Tennis became a practical test. I could watch the ball approach from inside the monitor and try to return it as it crossed the glass. A small error in alignment or timing was something I could feel in a missed swing.

Object recognition posed a related problem. When I ask Nilo to copy my mug, the camera also sees my fingers, desk, and monitor. I had to isolate the object I meant before sending it for 3D generation. Otherwise, a detailed result could still be a detailed model of the wrong thing.

Accomplishments that we're proud of

The moment I'm proudest of is reaching toward the stock chart and watching it change. A few moments earlier, that tool was just a request I'd spoken aloud. By the time I touched it, it was running on my computer and responding to my hand from the room. That one interaction brings together the three things I set out to build: understanding what I mean, making what I need, and letting me use it across the screen.

What we learned

Working alone taught me how much of collaboration happens before anyone gives an instruction. When I had teammates beside me, pointing at a troublesome icon was often enough to start working on it together. Building Nilo taught me to pay as much attention to that first gesture as to the finished output. Recognizing which icon, keeping track of the version I chose, and putting the file where I need it are what make the exchange feel like help.

What's next for Yonder

I want to make the marketplace useful for the small, specific tools people build for themselves. A solo developer who makes a module to save ten minutes every morning should be able to share it with someone who has the same problem.

I also want to extend the crossing from computers and connected TVs to phones. If I leave my desk halfway through a task, I'd like to pick it up on another display and have Nilo there with the object we were making and the decision we were about to make. That's the future I hope to build for people who work alone.

Built With

  • .net-8
  • anthropic-claude-api
  • ar-foundation
  • c#
  • lua
  • meta-quest-3
  • moonsharp
  • openai-api
  • openxr
  • tripo-api
  • unity
  • unity-xr-hands
  • websocket
  • windows
Share this project:

Updates

Submission history