Jarvis

Inspiration

As AI assistants become more capable, they are moving beyond answering questions and starting to perform actions on behalf of users. They can interact with files, execute commands, and communicate with external services.

However, giving an AI agent the ability to act also introduces risks. Users may not always understand what an agent intends to do, what information it might modify, or when it might interact with an external service.

We wanted to explore a simple question: How can we make AI automation safer while keeping users in control?

This inspired us to develop Jarvis, a personal AI assistant designed around transparency, risk-based permissions, and human oversight.

What it does

Jarvis is a personal AI assistant that combines conversational AI, tool execution, memory, and scheduled automation with a human-in-the-loop approval system.

Its key features include:

• Telegram interaction: Users can communicate with Jarvis through a Telegram bot using natural language.
• Web dashboard: A centralized interface for monitoring conversations, sessions, scheduled jobs, approvals, and agent activity.
• Risk-based permissions: Tools are classified into four categories: read, write, outbound, and exec.
• Human approval: Outbound operations and command execution require explicit approval before execution.
• Memory management: Stores useful information that can be retrieved across interactions.
• Audit logging: Records tool activity, sources, risk classifications, timestamps, and execution statuses.
• Scheduled automation: Supports recurring tasks through configurable scheduled jobs.

The central idea is to allow AI to assist with digital tasks while ensuring users retain control over potentially sensitive actions.

How we built it

Jarvis is built on an existing open-source AI agent foundation and uses a modular architecture connecting an AI model, a local gateway, tools, and user interfaces.

The implementation uses:

• Node.js and TypeScript/JavaScript for the agent runtime and backend components.
• pnpm for package management.
• OpenAI-compatible model integration for connecting the AI provider.
• Telegram Bot API for conversational interaction.
• Web dashboard for monitoring and controlling agent activity.
• Local gateway for coordinating requests and tool execution.
• Cron-based scheduling for recurring automation.
• Persistent memory tools for storing useful context.

A key architectural component is the risk-based permission system. Each tool operation is assigned a risk category. Read and write operations can execute automatically, whereas outbound and execution operations require human approval.

We configured the assistant, connected the Telegram interface, paired the web dashboard with the local gateway, and tested memory, file operations, approval requests, and audit visibility.

Challenges we ran into

One of our main challenges was configuring communication between the AI model provider, Telegram bot, local gateway, and web dashboard.

We also encountered compatibility issues with model tool-calling responses while experimenting with different AI providers.

Another challenge was testing external operations. Although Jarvis successfully generated an approval request for an email draft, completing the operation required Google OAuth authentication, which was not configured in our demo environment.

We also had to distinguish between an approval mechanism and complete execution security. Human approval adds an important control layer, but it does not replace operating-system isolation or other security protections.

These challenges helped us understand the practical limitations of building AI agents that interact with real tools.

Accomplishments that we're proud of

• Successfully configured a working Telegram interface connected to Jarvis.
• Established communication between the local gateway and web dashboard.
• Demonstrated memory storage through natural-language interactions.
• Demonstrated tool requests for file operations.
• Tested the risk-based approval workflow using an outbound email draft request.
• Verified that approval requests and tool activity can be monitored through the dashboard.
• Configured scheduled automation jobs.
• Brought together AI interaction, memory, automation, and human oversight in one prototype.

We are particularly proud of demonstrating how a potentially sensitive operation can be held for human review instead of being executed immediately.

What we learned

This project taught us that building a useful AI agent involves more than connecting a language model to a set of tools.

We learned about:

• Integrating AI models with tool-calling workflows.
• Managing communication between a local gateway and multiple interfaces.
• Designing risk classifications for different types of operations.
• Implementing human approval as part of an agent workflow.
• Maintaining visibility through audit logs.
• Managing API credentials and external authentication.
• Understanding the difference between permission controls and actual execution isolation.

Most importantly, we learned that trust and transparency should be considered fundamental parts of AI automation rather than features added later.

What's next for Jarvis

We want to evolve Jarvis into a more robust and safety-focused personal AI agent.

Our future plans include:

• Stronger isolation for shell and command execution.
• More granular, user-configurable permission policies.
• Clear explanations of why an action is classified as risky.
• Action previews before file modifications or external operations.
• Stronger authentication and authorization mechanisms.
• Improved monitoring and security-focused audit reports.
• Better integration testing for external services.
• More accessible approval workflows across devices.

Our long-term vision is to build an AI assistant that can perform useful tasks while remaining understandable, accountable, and under the user's control.

Jarvis — AI automation with human control.

Built With

• approval
• cron-scheduling
• human-in-the-loop
• javascript
• local-gateway
• node.js
• openai-compatible-llm-integration
• openrouter-api
• persistent-memory
• pnpm
• risk-based-permissions
• telegram-bot-api
• typescript
• web-dashboard