TheCommander

A personal AI assistant that remembers how I work, handles scheduled tasks, and turns research into GitHub issues I review and approve.

Comment

Inspiration

I started TheCommander with a simple idea: an AI assistant should do more than answer questions. It should remember, understand the work, and actually help get things done.

While working on engineering projects, I kept running into the same limitation. AI could help me research something, write code, or plan a task, but the continuity was missing. Once the conversation ended, I often had to explain the context again, reconnect tools, and manually finish the work.

I wanted to build something different. Not another chatbot with a collection of prompts, but a personal AI system that could maintain context, work with tools, handle scheduled tasks, and let me stay in control of what it actually executes.

TheCommander existed before this hackathon as an experimental AI workflow platform. I used this opportunity to rethink its architecture and rebuild it into a more practical personal engineering assistant.

My goal was straightforward: an assistant that remembers, takes permitted actions, keeps working when I close the browser, and shows me what actually happened.

What it does

TheCommander is an owner-controlled personal AI assistant designed to connect conversations with real engineering workflows.

It combines persistent memory, reusable skills, external integrations, and background execution into one application.

Persistent memory: TheCommander can store preferences and useful context in PostgreSQL, retrieve them in future conversations, and let me review, correct, or delete what it remembers. I don't have to start from zero every time.

AI reasoning with NVIDIA Nemotron: The assistant uses NVIDIA Nemotron through Nebius Token Factory for reasoning, planning, and proposing tool operations. The model helps decide what to do, but the application controls what it is allowed to execute.

Reusable skills: I built workflows for engineering briefings, research-to-action tasks, and scheduled follow-through. The goal is to turn common multi-step tasks into something repeatable rather than manually rebuilding the workflow every time.

Controlled tool access: TheCommander supports scoped integrations for Tavily research, workspace notes, and GitHub. Actions are checked against configured permissions. External writes require an explicit approval of the exact operation.

Background execution: A separate worker processes durable tasks and schedules. Work does not depend on keeping the browser open, and the application records execution status and results.

Activity and traceability: Instead of simply saying a task is finished, the system records what ran, what failed, and what evidence or receipt is available.

The bigger idea is to connect memory, reasoning, execution, and verification without giving an AI model unrestricted access to the user's environment.

How we built it

I originally built TheCommander around Python and Flask as an experimental AI orchestration platform. For this hackathon, I moved beyond that initial implementation and rebuilt the supported application architecture.

The backend now uses Python, FastAPI, and PostgreSQL. PostgreSQL stores persistent memory, conversations, schedules, tool approvals, execution records, and application settings.

For the interface, I used React, TypeScript, and Vite to build a responsive application with dedicated sections for Assistant, Memory, Skills, Settings, and Activity.

The AI layer connects to Nebius Token Factory, using the NVIDIA Nemotron 3 Super 120B A12B model through its inference API.

I designed the execution flow so the model can propose structured tool calls, but a separate policy layer validates the requested operation before anything executes. This keeps model reasoning separate from application permissions.

For integrations, I implemented Tavily-based research, controlled workspace operations, and a GitHub adapter with explicit approval and confirmation handling for issue creation.

I also developed a separate Python worker for scheduled tasks, execution recovery, and continued processing independently of the frontend.

The application is packaged with Docker Compose, includes database migrations, and has automated backend, integration, and browser tests.

One important design decision was keeping the application data under the owner's control while being transparent about external services. Memory and execution history live in the configured PostgreSQL database; selected model context is sent to Nebius, and external integrations receive only the data necessary for their permitted operations.

Challenges we ran into

One of the biggest challenges was turning an experimental AI application into a system that could execute real tasks without relying on assumptions.

Memory was the first major problem. Keeping information in a running process was not enough. I needed memory and task state that would survive new conversations, application restarts, and worker recovery.

Safe tool execution was another challenge. Letting an AI propose an action is relatively easy. Making sure it cannot exceed its permissions, execute an outdated approval, or repeat an external write after a timeout is much harder.

I spent considerable time working through permission checks, approval boundaries, cancellation, uncertain execution results, and recovery behavior.

Model reliability was also something I had to handle carefully. During live Nebius testing, I encountered a malformed model continuation. The application rejected it rather than treating the output as valid. A later diagnostic run succeeded, but the original failure reinforced why validation and visible error handling matter.

Testing the full system was more involved than testing individual API endpoints. I needed to verify how the frontend, database, model, tools, and background worker behaved together.

The GitHub integration is implemented and exercised through controlled tests, but completing the live end-to-end write and confirmation against an authorized disposable repository remains an outstanding verification step.

Accomplishments that we're proud of

What I'm most proud of is how far TheCommander has moved from its original architecture.

It is no longer just an experimental interface for sending requests to AI models. It has a real application backend, a responsive frontend, persistent state, reusable workflows, a background worker, and permission-controlled execution.

Some of the milestones that matter most to me are:

  • Rebuilding the application from its earlier Flask-based structure into FastAPI, React, and PostgreSQL.
  • Implementing cross-session memory with owner-controlled correction and deletion.
  • Connecting real NVIDIA Nemotron inference through Nebius Token Factory.
  • Building typed tool execution with explicit approval and traceable results.
  • Adding durable schedules, worker recovery, and protection against blindly repeating uncertain external writes.
  • Validating 30 out of 30 cases in a bounded live acceptance run on October 27, covering memory recall, planning, local follow-through, tool proposals, cancellation, and disconnected integrations.

That evaluation was a focused acceptance test, not a claim of perfect model reliability. But it gave me evidence that the important parts of the system could work together beyond mocked responses.

I'm also proud that I kept the project open source. I wanted the implementation, architecture, and limitations to be inspectable rather than hidden behind a demo.

What we learned

This project changed how I think about building AI agents.

I learned that generating a good response is only one part of building a useful assistant. The harder engineering work is managing state, permissions, failures, and execution over time.

Persistent memory needs clear rules for what is saved, retrieved, corrected, and deleted. Background tasks need durable state. External actions need explicit authorization and a way to confirm their outcomes.

I also learned the importance of separating a model's proposed actions from the application's authority to execute them.

Working with NVIDIA Nemotron through Nebius Token Factory gave me practical experience building and evaluating structured model-tool interactions. It also showed me why model integration needs validation and failure handling, not just a successful API response.

The biggest lesson for me was simple: an AI assistant becomes genuinely useful when its actions are as understandable and controllable as its answers.

What's next for TheCommander

I see TheCommander as a foundation that I can continue building rather than a one-time hackathon project.

My immediate priorities are completing live GitHub action verification, providing a secure hosted environment for evaluation, and strengthening end-to-end reliability testing.

After that, I want to expand its capabilities with more useful integrations, stronger memory retrieval, richer task planning, and improved long-running workflow management.

I also want to make installation and configuration simpler so people can run their own instance without needing to understand every internal component.

Longer term, my vision is an always-available personal AI that can understand ongoing projects, remember relevant context, coordinate permitted work, and provide verifiable results while keeping the owner in control.

I don't want TheCommander to be another AI that simply tells me what I could do. I want it to help me actually do it.

Source code: https://github.com/RafalW3bCraft/TheCommander

Hackathon track: Personal AI

Core technologies: Nebius Token Factory, NVIDIA Nemotron, Python, FastAPI, React, TypeScript, PostgreSQL, Tavily, GitHub API, Docker Compose.

Built With

Share this project:

Updates

Submission history