JARVIS x FIBI | QNX x XREAL Glasses

What you see, vs what I see.
W Photo
XREAL and QNX Hardware
Inside the Glasses

Inspiration

We were inspired by the contrast between two very different computing models: the flexible, probabilistic reasoning of large language models and the predictable, dependable architecture of QNX. LLMs are increasingly capable of understanding what people want, but they usually remain confined to chat windows. We wanted to explore what happens when an LLM becomes part of the interface itself.

QNX introduced themselves as a reliable OS that many of us rely on without even knowing it. Their task was simple, Build something using their OS. And in today's day and age of AI, what better project would be had, than an AI project.

The XREAL Air 2 Pro gave us the ideal environment for testing this idea. Rather than limiting the user to a physical laptop screen, we imagined a portable workspace where virtual displays surround the user and intelligence is available wherever they look. The project began as a small IMU experiment displaying a placeholder image. It evolved into a three monitor desktop that combines head tracking, voice interaction, screen understanding, a local language model, ElevenLabs speech, and a Raspberry Pi 5 running QNX.

Our goal became more than creating virtual monitors. We wanted to investigate what the interface of an operating system might look like when natural language, spatial context, and dependable edge computing are designed together from the beginning.

Giving birth to FIBI, The Focal Intelligence Boundless Interface.

How we built it

We first had to setup the raspberry pi, and configure it to our needs of holding an LLM that worked on QNX, and was able to actively share information with the glasses. Using the glasses ability to track 3 degrees of freedom, I was able to build an interface that you interact with via Voice commands.

By integrating a feedback loop, I'm able to share data from both the glasses and the QNX raspberry pi, allowing for the LLM to see the current screens, where you're looking, and as well as helping you interact and edit the monitors available on the glasses.

Challenges we ran into

Coming from a Hardware-only background, It was difficult to work my head around the more software oriented challenges. Usually as a Mechanical Engineer, I'm used to solving my problems just by simply looking at issues in a physical space. But as I worked with QNX and Windows operating systems, I found it hard to look at text and realize what exactly Is causing my issues on every iteration of change.

Accomplishments that we're proud of

Xreal glasses are unfortunately pretty strict about their software, and don't share their given secrets on how to exactly find said software. Being able to find the limited amount of Data provided by the display glasses, and being able to repeatedly use it in my project was a huge success, and showed me that the issue was less to do with what Xreal withholds, and more to do with how effectively you're able to use the data on your own.