SherlockML

Inspiration Production ML systems often fail silently. APIs stay online, dashboards look calm, but model quality can collapse because of data drift, pipeline bugs, or bad model releases. SherlockML was inspired by the idea of an AI engineer that can join the ML on-call rotation and investigate reliability incidents before humans lose hours debugging. What it does SherlockML simulates an insurance fraud detection model in production. When the model degrades, it detects the incident, gathers evidence, investigates root causes, debates competing hypotheses, proposes a bounded repair, retrains or rolls back a candidate model, validates the improvement, and generates an engineering report for human approval. How we built it We built SherlockML with a Streamlit command-room UI, FastAPI backend, deterministic synthetic fraud data, scikit-learn/XGBoost models, MLflow-style experiment tracking, and a LangGraph-style multi-agent workflow. Agents act as a detective, statistician, infrastructure reviewer, engineer, moderator, and model doctor. Challenges we ran into The hardest part was making autonomy feel trustworthy. The system needed to do real work, not just narrate a diagnosis. We had to connect evidence, metrics, code/config changes, retraining, validation gates, and reports into one coherent workflow while keeping the final decision safely human-reviewed. Accomplishments that we're proud of We are proud that SherlockML demonstrates a full ML incident lifecycle: silent failure, detection, evidence gathering, root-cause analysis, repair, validation, and approval reporting. The demo feels like an AI reliability engineer actually investigated and recovered a model. What we learned We learned that autonomous AI systems need receipts. Metrics, artifacts, diffs, experiment records, and clear boundaries matter as much as the agent reasoning itself. Trust comes from showing what happened and why. What's next for SherlockML Next, SherlockML could connect to real model monitoring tools, CI/CD pipelines, feature stores, and alerting systems. The long-term vision is an AI reliability engineer that safely maintains production ML systems alongside human teams.