ERNIE-Arduino-FT

HuggingFace Model Card
Training loss log

Inspiration

We all know that arduino is a widely used platform in robotics, and having a specialized, offline-capable assistant for Arduino developers can significantly improve productivity and accessibility. The primary motivation behind this project was to explore whether small language models (SLMs) can be fine-tuned effectively for real-world, domain-specific tasks while remaining lightweight enough to run on edge devices. Large language models often require significant compute and are impractical for on-device deployment in robotics and embedded systems.

What it does

ERNIE-Arduino-FT is a domain-specialized question-answering model focused exclusively on Arduino. It provides accurate and contextual answers related to Arduino programming, hardware APIs, libraries, pin configurations, and usage patterns. Since the model is trained on extensive official documentation, it can handle both conceptual explanations and practical implementation questions with consistent reliability.

How we built it

The system was built using an end-to-end fine-tuning pipeline. I first performed large-scale web scraping of official Arduino documentation and converted the raw content into structured instruction–response pairs. In total, 17,945 high-quality samples were generated after cleaning and filtering. I have a paid Google Gemini API so I used it to remove noise and clean the data.

For model training, I fine-tuned the 0.3B parameters ERNIE-4.5 model from HuggingFace using Unsloth for efficient training. To minimize memory usage and speed up convergence, I applied LoRA adapters to freeze the majority of model parameters and used 4-bit quantization. This allowed stable training on limited GPU resources while maintaining strong performance.

Challenges we ran into

One of the major challenges was model compatibility with Unsloth. Many larger and more capable models were not supported, limiting available options. I also experimented with a 21B parameter model, but it was not feasible to fine-tune on a T4 (16GB) GPU. Even when using an L4 (24GB) GPU, training times extended to several days, making it impractical. Balancing model size, training stability, and hardware constraints lead me to choose the small model.

Accomplishments that we're proud of

I successfully built a complete data-to-model pipeline, starting from raw documentation to a fine-tuned domain-specific model. Achieving stable training with a smooth, decreasing loss curve confirmed the effectiveness of the dataset and hyperparameter choices. The final model demonstrated reliable inference behavior and produced accurate Arduino-specific responses, validating the overall approach.

What we learned

This project provided hands-on experience in large-scale data scraping and curation, including noise removal and dataset refinement using Gemini API calls. I also learned to use Unsloth for parameter-efficient fine-tuning, which proved significantly faster and more resource-efficient than my previous workflows based on Hugging Face PEFT. Additionally, I gained deeper insight into practical constraints involved in fine-tuning models on limited hardware.

What's next for ERNIE-Arduino-FT

As future work, the project can be extended by fine-tuning a larger ERNIE model to further improve reasoning and response quality. Another direction is building a multimodal Arduino assistant that can accept images as input, allowing users to ask questions about physical boards, components, or wiring diagrams. This would make the system even more practical for real-world robotics and embedded development scenarios.