• 

From Understanding Your Commands to Understanding You.

Inspiration

AI has fundamentally changed how we interact with computers.

We can now speak to an AI, show it our screen, ask it questions, and even let it operate applications on our behalf.

But something was still missing.

Most AI agents are remarkably capable at understanding what is happening and what needs to be done. Yet they often lack something that makes human intelligence so useful: an understanding of culture, personal interests, and taste.

They can search for a movie, but do they understand what makes someone love a particular kind of cinema?

They can identify a brand, but do they understand the cultural associations behind it?

They can recommend a restaurant based on ratings, but can they connect someone's preferences across cuisines, experiences, and interests?

We realized that building a better AI assistant wasn't just about giving an LLM more tools or making it execute more actions.

We needed to give it a way to understand the world beyond the immediate conversation.

That idea became the foundation of our Qloo integration into keyboard.wtf.

The Foundation: keyboard.wtf

keyboard.wtf is an AI assistant designed to live directly on your computer.

Instead of opening another chatbot every time you need help, you summon Jarvis using a global keyboard shortcut and interact with it naturally through voice or text.

Jarvis can understand your instructions, analyze your screen, interact with applications, navigate the browser, perform Windows tasks, and execute workflows.

Imagine saying:

"Open Chrome, Spotify, and my project folder."

Or highlighting a paragraph and asking:

"Make this sound more professional."

Or looking at an error on your screen and asking:

"What's wrong here?"

The objective was to make the computer understand intent instead of requiring users to translate every thought into a sequence of clicks.

But while building keyboard.wtf, we encountered another limitation.

An AI that can interact with your computer doesn't necessarily understand the cultural context of what it sees.

It can recognize a movie poster, identify a brand, or describe an image. But understanding the relationships, interests, and preferences behind those entities requires something more.

That is where Qloo changed the direction of the project.

The Game Changer: Qloo's Cultural Intelligence

Qloo offers something fundamentally different from another LLM or conventional search engine.

Its Taste Graph connects more than 250 million entities across cultural domains including music, movies, dining, fashion, and more.

We saw an opportunity to bring that intelligence directly into an AI assistant that already understands the user's computer and context.

Instead of creating a separate recommendation application, we integrated Qloo as a cultural intelligence layer within Jarvis.

The distinction is important:

Gemini understands and reasons. Qloo provides cultural intelligence. Jarvis connects that intelligence to the user's computer and actual workflows.

When a request involves cultural preferences, the agent can identify relevant entities, retrieve the user's applicable preferences, translate the context into supported Qloo signals, and use the returned insights to ground its reasoning.

Qloo doesn't replace the language model. It gives the language model access to a different kind of information that generic reasoning alone cannot reliably reproduce.

This transforms Qloo from an API that simply returns recommendations into an intelligence capability available throughout the desktop experience.

What We Built

1. A Cultural Intelligence Layer Inside the Desktop

We introduced Qloo as a dedicated connector within the existing Jarvis architecture.

Rather than building another chatbot around Qloo, we made it available as a capability that the agent can invoke when cultural intelligence is relevant.

The assistant can work with entities and preferences across different domains without requiring a separate application for each category.

The intention is simple:

Qloo should be available wherever cultural understanding can make an AI interaction better.

2. Connecting the User's Digital Context to the Cultural World

One of the most interesting aspects of keyboard.wtf is that Jarvis already operates within the user's computer environment.

It can understand what the user is looking at, what they are asking about, and which applications or workflows are involved.

Qloo adds another dimension to that context.

Consider a user exploring fashion products, discovering music, researching films, or browsing restaurants.

Jarvis can interpret the relevant information, identify cultural entities, and use Qloo to discover meaningful relationships that would otherwise be missing from a conventional AI response.

The important innovation is not simply retrieving an answer.

It is connecting what the user is doing right now with a much larger network of cultural knowledge.

3. An Agent That Progressively Understands Its User

We also wanted personalization to go beyond asking someone to select five favourite movies during onboarding.

Our approach introduces a persistent preference-memory layer that can distinguish between explicitly stated preferences, inferred interests, contextual choices, and negative feedback.

For example, repeated interactions can help Jarvis recognize patterns in the user's interests and use relevant preferences in subsequent interactions.

Combined with Qloo's cultural intelligence, this creates the foundation for an assistant that can become increasingly relevant as it learns more about its user.

The goal is not to store every conversation.

It is to understand which pieces of information actually matter when the user needs help.

4. Intelligence That Leads to Action

Unlike standalone recommendation applications, keyboard.wtf already has the ability to interact with the computer.

This means cultural intelligence can become part of an actual workflow.

Jarvis can discover relevant content, help explore alternatives, navigate applications, and take supported actions based on the user's request.

The experience goes beyond receiving a list of recommendations.

The agent can connect understanding, cultural intelligence, reasoning, and computer interaction in one continuous workflow.

5. An Optional Intelligence Layer

We deliberately preserved the original Jarvis experience.

Qloo is integrated as an optional connector.

With Qloo disabled, Jarvis retains its original AI-assistant behaviour.

With Qloo enabled, the same assistant gains access to cultural intelligence when it is relevant.

This also gives us a meaningful way to demonstrate the difference between conventional LLM reasoning and Qloo-grounded intelligence.

How We Built It

The architecture combines three complementary components:

Gemini — The reasoning engine

Handles natural-language understanding, screen interpretation, planning, agent orchestration, and final responses.

Qloo — The cultural intelligence engine

Provides access to taste-grounded insights and relationships across supported cultural domains.

keyboard.wtf — The execution layer

Connects these capabilities to the user's computer through voice interaction, application control, browser automation, and existing agent tools.

The integration follows a deliberate flow:

Understand → Identify → Retrieve → Reason → Act → Learn

The system first understands the user's request and relevant context, determines whether Qloo can provide useful information, retrieves the appropriate cultural insights, and combines them with its reasoning and the user's preferences.

We also designed the integration around efficient API usage, caching, request optimization, and graceful fallback behaviour.

Importantly, Qloo is not called for every request. Its value comes from knowing when cultural intelligence is useful and when ordinary reasoning is sufficient.

The Challenges and What We Learned

The most interesting engineering challenge was not simply connecting to an API.

It was figuring out how to translate messy, unstructured human context into meaningful signals that a cultural intelligence system could actually use.

A screenshot, a conversation, a product, and a user's previous choices are very different sources of information.

The agent needs to identify relevant entities, understand which signals are supported, retrieve useful insights, and combine them without confusing cultural affinity with factual certainty.

We also had to think carefully about preference memory, API efficiency, privacy, caching, fallback behaviour, and the division of responsibilities between an LLM and a specialized intelligence system.

One important realization was that cultural intelligence and reasoning are not competing technologies. They solve different problems and become considerably more useful when combined.

An LLM can reason about information. Qloo can provide a structured cultural perspective that the LLM might otherwise lack.

The interesting engineering lies in connecting the two properly.

What We Believe Comes Next

We believe the future of AI assistants will involve more than larger language models and increasingly sophisticated computer-use capabilities.

An assistant that operates your computer should eventually understand more than the applications you use and the instructions you give it.

It should understand the interests, preferences, and cultural context that make its assistance relevant to you.

Qloo gives us a way to move toward that vision.

With keyboard.wtf, we are exploring what happens when an AI agent that already lives on your computer gains access to cultural intelligence that can inform its understanding, recommendations, and actions.

We started with a simple idea:

What if you could talk to your computer instead of operating it one click at a time?

Qloo helped us take that idea further:

What if your computer didn't just understand what you wanted it to do, but progressively understood the world and the preferences behind why you wanted it?

That's the direction we're building toward.

keyboard.wtf — Jarvis with Taste

An AI that doesn't just understand your commands. It understands the context, culture, and preferences that make them yours.

Built With

• adaptivelearning
• agenticai
• agents
• ai
• api
• artificialintelligence
• automation
• computeruse
• culturalintelligence
• decisionmaking
• desktopai
• gemini
• llm
• machine-learning
• memory
• multimodal
• personalization
• productivity
• qloo
• recommendations
• taste
• vision
• voice
• windows