BizInsight

Inspiration

Small and medium enterprises (SMEs) across Africa manage hundreds of customer support tickets daily, but face high cloud API costs, unstable internet connectivity, and data privacy concerns. This inspired the creation of an AI assistant to provide enterprise grade customer support insights to any African retail business.

What it does

BizInsight is an offline customer support assistant designed for SME store owners, operations managers, and support teams. It analyzes and generates summarizes of user supplied tickets, evaluates them for customer sentiment, and enables the drafting of customer communication.

How we built it

Base Model: We selected unsloth/Qwen3.5-2B (Qwen 3.5 2B Instruct architecture with ~1.94B active parameters) for its strong instruction adherence, structured output reliability, and native long-context support. Fine-Tuning: The model was fine-tuned on the bitext/Bitext-customer-support-llm-chatbot-training-dataset (26,872 customer service instruction-response pairs) using Unsloth and Hugging Face TRL's SFTTrainer on an NVIDIA RTX Pro 6000 (Blackwell). We configured LoRA ( r=16, α=16, dropout = 0) across all attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj). Activation-Aware Quantization (imatrix): To maximize inference throughput while preserving output quality, we replaced the baseline Q8_0 (2.08 GB) model with an importance matrix calibrated Q4_K_M GGUF model (qwen3.5-2B-bitext-imatrix-Q4_K_M.gguf). Quantizing with imatrix preserves high precision for critical attention weights while aggressively compressing less sensitive parameters, yielding a 37% file size reduction (down to 1.31 GB). Offline Runtime: Deployed with llama.cpp for pure CPU execution on everyday 8 GB RAM laptops with integrated graphics.

Challenges we ran into

Issues were faced with finding the appropriate base model. In addition, the initial plan was to build an automotive industry support assistant. However, due to a lack of quality public datasets, a change was made to general SME customer support. nitial testing of extreme quantization like IQ4_XS (~1.23 GB) caused a 2% accuracy degradation. Conversely, the initial Q8_0 baseline was memory-heavy. We solved this by generating an activation importance matrix (imatrix) for Q4_K_M, matching Q8_0 quality and sentiment nuance while shrinking the memory footprint.

Accomplishments that we're proud of

1. Engineering a fine-tuned 2B parameter LLM that runs smoothly on standard consumer hardware with a generation speed of 6.36 tokens/second while consuming only about 2.10 GB RAM at peak usage leaving about 5.9 GB for other business applications that might run on the system.
2. Successfully delivering a zero-cost privacy preserving AI solution for African retail SMEs.
3. Peak RAM consumption stays at 2,002.45 MB (~2.0 GB) (with 1,927.3 MB steady-state RSS) and 70.3% P99 CPU utilization with zero thermal throttling, leaving ~5.9 GB of free RAM for inventory, POS, and accounting software.
4. Achieved an inference generation speed of 8.94 tokens/second (up from earlier 6.36 t/s benchmarks) with a time-to-first-token of ~15.6s on a standard participant laptop (HP EliteBook 840 G3, Intel Core i7, 7.9 GB RAM, integrated graphics).

What we learned

1. The importance of utilizing lightweight base models as a foundation for creating one's custom LLM.
2. The various forms of quantization and the tradeoff between balancing model performance and inference efficiency.

What's next for BizInsight

The next step is to integrate BizInsight directly into local African invoicing, inventory and accounting software solutions as well as exploring edge deployment optimizations for mobile devices.

Built With

• hugging-face-transformers
• llama.cpp
• python
• pytorch