ERNIE-4.5 0.3B Distilled Gemini/Claude

Inspiration

Need for a small model that can run on phones with minimal hallucinations and good responses

What it does

Provides chat bot experience at ultra light 300 million parameter size

How I built it

Unsloth, datasets from Hugging Face user TeichAI and ERNIE-4.5 Base Model

Challenges I ran into

The model began over fitting with high loss, after many unsuccessful changes, I decided to perform full fine tuning which not only solved the issue but made the fine tune even better.

Accomplishments that I am proud of

A relatively good chatting experience with the model is delivered with such minimal fine tuning

What I learned

How Fine Tuning works, LoRA/QLoRA, SFT, DPO, Unsloth Library, Dataset formatting

What's next for ERNIE-4.5 0.6B Distilled Gemini/Claude

Fine tuning with bigger datasets will help it perform even better

Built With

• python
• scikit-learn
• unsloth