Mayllo

Its my multimodal asistent

Comment

Inspiration

MAYLLO started as a simple idea: creating a personal AI assistant that could interact with a computer instead of only answering questions.

What began as a basic Python assistant gradually became a much more ambitious project. We wanted MAYLLO to understand natural language, interact through voice, use computer tools, and feel more like an actual desktop assistant rather than a program based only on predefined commands.

As the project evolved, we also became interested in understanding what happens behind the AI itself. Instead of only using an existing model, we wanted to explore whether we could adapt a model specifically for MAYLLO and its Spanish-speaking use case.

What it does

MAYLLO is a desktop AI assistant built primarily with Python. It allows the user to interact with an AI through voice and receive responses through text and speech.

MAYLLO can:

  • Receive voice input.
  • Convert speech into text.
  • Process natural-language requests using an AI language model.
  • Generate responses through an AI model.
  • Convert responses into speech.
  • Interact with computer tools.
  • Open applications and files.
  • Provide an animated desktop interface.
  • Keep the interaction more natural than a traditional command-based assistant.

The main goal is to make computer interaction feel more natural by allowing the user to communicate with the assistant instead of memorizing specific commands.

How we built it

MAYLLO was developed primarily with Python and built incrementally as we learned new technologies.

The application combines several components, including speech recognition, text-to-speech, an AI language model, computer tools, and a graphical interface. The interface was developed using CustomTkinter and PIL, while threading helps keep the application responsive while MAYLLO processes requests or performs actions.

For the AI component, we experimented with different approaches to connect MAYLLO with language models. We also explored the possibility of adapting a model specifically for MAYLLO instead of relying only on an existing model through an API.

We created a custom Spanish dialogue dataset for MAYLLO and experimented with Adaptation Labs, including augmented data and AutoScientist. We then explored a more direct model fine-tuning workflow using Hugging Face, Google Colab, PyTorch, Transformers, PEFT, Accelerate, and BitsAndBytes with Llama 3.1 8B Instruct.

This part of the project took us into an area we had not worked with before and required us to learn how datasets, model adaptation, training environments, and computational resources interact.

Challenges we ran into

One of our biggest challenges was combining many different systems into one application. Speech recognition, the AI model, text-to-speech, the graphical interface, and computer tools all need to work together while keeping the application responsive.

Another major challenge was exploring AI model customization. We had never worked directly with the process of adapting or fine-tuning a large language model, so we had to learn many concepts and tools from scratch.

We created our own Spanish dialogue dataset and experimented with Adaptation Labs, augmented data, and AutoScientist. We also tried to move toward a more traditional fine-tuning workflow using Google Colab and Hugging Face.

However, we eventually encountered a practical limitation: the available computing resources were not enough for the iterative fine-tuning process we wanted to perform with an 8B-parameter model. This prevented us from completing the training process as originally planned.

Although this was frustrating, it also became one of the most valuable parts of the project. We learned that developing an AI system is not only about writing code. Data preparation, model architecture, training methods, memory requirements, hardware limitations, APIs, and deployment all play an important role.

Accomplishments that we're proud of

We are proud of how far MAYLLO evolved from its original idea.

What started as a simple Python assistant became a desktop AI application capable of voice interaction, AI-generated responses, text-to-speech, computer tools, and an animated interface.

We are also proud that we went beyond simply consuming an AI API. We created a custom Spanish dialogue dataset and explored the process of adapting AI models through Adaptation Labs, AutoScientist, Hugging Face, Google Colab, and open-source model tooling.

Even though we were unable to complete the fine-tuning process, we consider the experiment an important accomplishment because it allowed us to explore a part of AI development that was completely new to us.

Most importantly, we learned to treat failed experiments as part of the development process rather than as wasted work.

What we learned

The biggest lesson from MAYLLO was that building an AI assistant involves much more than connecting an application to a language model.

Throughout the project, we learned about Python application architecture, APIs, speech recognition, text-to-speech, threading, graphical interfaces, computer automation, error handling, and integrating external tools.

We also stepped into an area we had never worked with before: AI model customization.

We learned how to create and structure a dataset for a specific use case, how augmented data can be used in experimentation, how tools such as AutoScientist can evaluate or explore model improvements, and how platforms such as Hugging Face and Google Colab fit into the machine-learning workflow.

Working with Llama 3.1 8B Instruct also showed us the difference between using a pretrained model and actually trying to adapt one. We learned that fine-tuning requires much more than having the correct code: computational resources, memory, model size, training configuration, datasets, and the right hardware all become important factors.

We did not achieve the fine-tuning result we originally wanted, but the process gave us practical knowledge about what is required to make it possible in the future.

This experience also taught us that entering an unfamiliar technical field does not always mean immediately achieving the final goal. Sometimes the most valuable result is understanding the process, discovering the limitations, and knowing what needs to change the next time.

What's next for MAYLLO

The next step for MAYLLO is to make the assistant more natural, capable, and adaptive.

We want to improve its context and memory, expand its computer tools, improve its interaction and animations, and make its behavior more adaptive to the user.

We also want to return to model customization in the future with better computational resources and a more developed dataset.

Our long-term goal is to continue transforming MAYLLO from a desktop assistant into a more capable personal AI system that can understand context, learn from structured data, and perform useful tasks across the computer.

Built With

Share this project:

Updates

Submission history