Inspiration
The inspiration behind Autonomous AI Agent came from the idea of having a personal AI assistant that does more than simply answer questions. Instead of switching between multiple applications and manually performing repetitive tasks, we wanted to build an assistant that could understand natural-language commands, plan actions, and help users complete tasks directly on their computers. We envisioned an assistant that could interact with desktop applications and educational platforms, such as an LMS, while keeping the user in control of important actions.
What it does
It is a voice-enabled, permission-aware autonomous AI agent designed to make computer interaction more natural and efficient.
The system combines natural-language understanding, AI-based task planning, and computer interaction. Users can communicate with JARVIS through voice or text, describe what they want to accomplish, and allow the agent to determine the appropriate steps.
Some of our intended capabilities include:
Voice-based interaction and command understanding.
AI-powered task planning and tool selection.
Interaction with desktop applications.
LMS assistance, including checking academic information and pending assignments.
Document and PowerPoint creation assistance.
A permission layer that keeps users informed and in control of sensitive actions.
How we built it
We developed it using a modular architecture that separates the user interface, backend services, AI processing, and computer interaction.
The frontend provides the interface for voice and text-based communication. The backend handles incoming requests and coordinates the different components. We integrated a locally running language model through Ollama to support AI-powered responses and task understanding.
A key design principle was to make the agent permission-aware. Rather than allowing unrestricted computer access, the system is designed to involve the user before carrying out sensitive operations.
Challenges we ran into
One of our biggest challenges was connecting the AI model with the backend and ensuring that requests were processed reliably. We also encountered difficulties with local model response times, backend communication, microphone permissions, and desktop application integration.
Another important challenge was designing an agent that could move beyond predefined commands. We wanted it to understand user intent and determine suitable actions instead of depending entirely on rigid, rule-based instructions.
We approached these challenges by testing individual components, debugging API communication, and improving the separation between AI reasoning and task execution.
Accomplishments that we're proud of
We are proud of building JARVIS as more than a traditional chatbot. We successfully brought together voice and text interaction, a locally running AI model, backend integration, and a permission-aware agent architecture. We are especially proud of exploring how an AI agent can understand user intent, plan tasks, and interact with desktop applications while keeping the user in control. This project helped us turn our vision of a personal AI assistant into a working prototype and gave us valuable hands-on experience in developing AI-powered autonomous systems.
What we learned
Building this helped us understand how AI agents work beyond simple chatbot interactions. We gained practical experience with local language models, API integration, backend development, voice interaction, task planning, and permission-aware computer automation.
Most importantly, we learned that building an autonomous AI assistant is not just about making AI understand a command. It is about connecting reasoning with real-world actions while maintaining reliability, transparency, and user control.
What's next for Permission-Aware Autonomous AI Agent
Our vision is to evolve JARVIS into a dependable personal AI assistant that reduces repetitive digital work, supports students in their academic activities, and makes everyday computer interaction simpler through natural conversation.
Built With
- fastapi
- javascript
- llama
- ollama
- python
- react
- vite
