Jarvis

Jarvis is an agentic AI desktop app that boosts productivity and accessibility by automating emails, docs, calendars, and pipeline risk analysis using ML driven insights from real inspection data.

Comment

Inspiration

We were inspired by two problems that are usually tackled separately but shouldn’t be: fragmented productivity tools and limited accessibility. At the same time, industries like energy depend on complex inspection data that is hard to translate into timely decisions. We wanted to prove that one intelligent agent could simplify daily digital life while also delivering high impact, data driven insights for real world operations.

What it does

Jarvis is an agentic AI desktop platform that unifies productivity, accessibility, and advanced analytics into one seamless experience. Users interact with Jarvis through voice or text to send and read emails, create and edit Google Docs, manage calendar events, open links, and shop online. Jarvis can also send and receive phone calls and text messages, allowing users to access the assistant from anywhere, not just their desktop. This makes Jarvis especially impactful as both a productivity tool and a disability aide, enabling hands free and remote interaction across devices. For the RCP track, Jarvis analyzes pipeline inline inspection data by ingesting CSV files, detecting anomalies, and predicting future integrity risks using machine learning. The platform transforms raw inspection data into actionable insights that improve pipeline monitoring, risk identification, and operational decision making.

How we built it

Jarvis is built as a desktop application using Flutter to ensure a clean, responsive cross platform experience. We leveraged Google ADK to implement agentic behavior, enabling Jarvis to reason, plan, and execute multi step actions across external services. Featherless.ai powers the LLM layer, allowing for fast and natural conversational interaction. For pipeline analytics, we built a data processing and modeling pipeline using Pandas for structured analysis and Random Forest models for anomaly detection and risk prediction. Jarvis learns from historical inspection data to flag high risk pipeline segments and forecast potential issues, all accessible through conversational commands or automated workflows.

Challenges we ran into

One major challenge was working with Google ADK, a framework we had no prior experience with and limited documentation. We had to rapidly experiment and iterate to design reliable agent workflows, manage state, and coordinate tool execution. This forced us to deeply understand agentic system design and ultimately enabled Jarvis to perform complex real world actions. Another challenge was integrating real phone calls and text messaging for the first time. We had to manage asynchronous communication, real time transcription, and reliable message delivery while keeping the system responsive. Solving this was critical to making Jarvis truly accessible from anywhere.

Accomplishments that we're proud of

We built a fully functional agentic AI system that performs real actions across productivity tools, phone calls, and messaging. Jarvis is accessible beyond the desktop, supports voice first interaction, and delivers meaningful machine learning insights for a real industrial use case. Most importantly, it goes beyond a demo and demonstrates real end to end usability.

What we learned

We learned how to design and deploy agentic AI systems that interact with the real world. Along the way, we gained experience with unfamiliar frameworks, cross platform development, accessibility focused design, and applied machine learning for infrastructure risk analysis. We also learned that combining domain knowledge with strong user experience design significantly increases real world impact.

What's next for Jarvis

Next, we plan to expand Jarvis’s desktop capabilities by enabling deeper disk and system access so it can interact with more native applications beyond the Google ecosystem. This would allow Jarvis to automate workflows involving local files, PDFs, spreadsheets, and common desktop software through the same interface. We also aim to integrate an LLM with a larger context window to support longer conversations, more complex workflows, and deeper analysis without frequent resets. These improvements would make Jarvis more powerful while keeping it practical, responsive, and ready for real world deployment.

Share this project:

Updates