ComplaintLens

• 

  training data distribution

• 

  complete 01 training

• 

  final per category comparison

• 

  batch classification and export

• 

  email review

• 

  cover current live demo

Inspiration

Debt-collection complaints arrive as free-text narratives, and each one has to be routed to the right issue category before anyone can act on it. Doing this by hand is slow, but handing it to an automatic classifier is risky: a wrong label can send a real person's complaint to the wrong place.

We wanted to explore the middle path: let the model suggest, and let a person decide.

What it does

Complaint Intelligence Workbench is a local, English-language assistant for classifying debt-collection complaints into seven CFPB Issue categories.

• Single-narrative and CSV classification (UTF-8 CSV, up to 5 MB / 500 rows; invalid rows stay visible instead of being silently dropped)
• One suggested category per complaint, plus an ambiguity hint and an alternative category when the top-two score gap is below 0.10
• Review tools: narrative inspection, batch filtering, category counts, and export of unconfirmed suggestions
• An English learning guide that explains the project and its limits
• A separate local progress monitor for our fine-tuning experiment

It is a research prototype, not an autonomous complaint-processing system. A reviewer must read the narrative and confirm every decision.

How we built it

• Application: Python and Streamlit, running entirely on 127.0.0.1. No external AI API, no cloud deployment, no automatic online learning.
• Active model: a balanced word/bigram TF-IDF + logistic regression classifier trained on historical CFPB complaint narratives.
• Experiments: we compared data scaling (DATA50, DATA100), unigram features, frozen MiniLM embeddings, a soft-vote ensemble, and a task-specific DistilBERT fine-tuning run (FT01, one fixed CPU run, capped at three epochs and 8 hours / 6 GB RAM).
• Engineering safeguards: unit tests on synthetic fixtures, input size limits, a fine-tuning script that refuses to overwrite existing runs, and a repository that contains code and aggregate documentation only, with no complaint text, weights, or credentials.

Results

All numbers come from the same 1,428-record development validation set.

Adding more historical training data raised Macro-F1 from 0.3931 to 0.4409. We use Macro-F1 because the categories are imbalanced and small ones matter:

$$ \text{Macro-F1} = \frac{1}{K}\sum_{k=1}^{K} F1_k $$

Challenges we ran into

• Weak small categories. Overall accuracy hides poor performance on rare issue types, which is why we track Macro-F1.
• Evaluation honesty. Our validation set has been viewed repeatedly and our historical holdout has already been seen, so we cannot claim a fresh independent final test. We report everything as development evidence.
• Experiments that did not help. The ensemble corrected 68 errors but broke 91 previously correct predictions, so we did not adopt it. We kept these failures in our research summary.
• Template similarity and label ambiguity in the source data limit how far conclusions can be pushed.
• Uncalibrated scores. A high score, or the absence of a warning, never authorizes automatic adoption, which is why human confirmation is central to the design.

What we learned

• More data helped more than more complex models in our setting.
• Reporting failed experiments and limits openly makes the results more trustworthy.
• For sensitive complaint handling, a clear human-in-the-loop workflow matters as much as the model.

What's next

• Read the FT01 fine-tuning results and decide whether to integrate DATA100 or a fine-tuned model
• Implement storage for human confirmations
• Build a genuinely independent evaluation set
• Improve performance on small categories

Built With

• distilbert
• joblib
• logistic-regression
• minilm
• natural-language-processing
• python
• pytorch
• scikit-learn
• streamlit
• text-classification
• tf-idf