3D digital twin of an EV factory seen from above: robot weld cells on an assembly line, stations S01 to S12 labelled, live KPI panels.
KPI dashboard: OEE, availability, downtime cost tiles, output and throughput charts, and a repair-now versus ignore-it what-if comparison.
Close-up of weld cell B in the 3D twin: a robot arm beside a conveyor with car bodies and a monitor showing the robot’s vibration trend.
System flow diagram: synthetic data feeds a simulation engine that drives the 3D twin, dashboard and what-ifs, with a chat assistant on top
Anomaly detector results: charts comparing healthy and faulty machine data, how early faults were caught, and the signals behind alerts.

Inspiration

When a car plant's line stops, it costs roughly $38,000 a minute (Siemens' 2024 estimate for a large automotive plant). The warning signs usually exist hours earlier: a bearing starts to vibrate, a temperature creeps up. But the person who sees them still has to work out what it means, who to call, which part to fetch and whether it's safe to keep running. We wanted an AI copilot that answers those questions for a plant manager in plain words, backed by a live digital twin they can see.

What it does

"Talk to Your Plant" is a live 3D digital twin of a fictional EV factory (Voltline EV Works) that you can talk to.

You watch the plant run: 12 stations, a paced assembly line, people, AGVs and cars being built and painted as they move.
A machine-learning detector watches every machine and flags the weld robot's bearing wear about 16 minutes after it starts. That is hours before a "2x vibration" rule would fire. It also says which signals drove the alert and how many minutes of life are left.
You ask questions in a chat: "Why is M05 red?", "Can we keep running it 2 more hours?", "Who do I page?", "Which part do we need and is it in stock?". Answers come back as short action cards.
"What if" runs the real simulation forward: keeping the robot running 2 more hours costs about $9k, while 4 hours means a seized robot and about $2.8M.
Click any station for a briefing. Break any machine to watch the line stop, starve and block, and a technician walk to fix it.
A dashboard shows KPIs, the anomaly detector, and history from the same engine.
Volty, a small animated robot in the chat, turns red with a siren when an answer is critical.

How we built it

Simulation (Python, SimPy): a strict-pitch paced line with buffers, mid-job machine failures, a technician and work orders, and replayable random streams. It was calibrated to within 8% of our generated scenario data. A FastAPI server streams the state over WebSocket and exposes tool endpoints for the chat. What-if replays the shift from now to the end and compares it with doing nothing.
Anomaly detection (scikit-learn): an Isolation Forest trained only on healthy data, on load-normalised features with a thermal model, with exact Shapley explanations and conformal remaining-life intervals. On 40 held-out faults it caught 40 of 40 before failure with no false alerts on healthy runs.
3D twin (React, TypeScript, react-three-fiber): a procedural plant built from the stations' reference images, people playing a generated schedule, and a HUD kept clear of the scene.
Synthetic data: layout, 24 employees, sensor streams, a 30-day history and 12 months of maintenance estimates, all generated by seeded scripts.
The brain (teammate): a knowledge graph of the plant, retrieval over SOPs, manuals, work orders and failure records, and a rule-based router that builds answer cards from live engine data. A local LLM is optional and sits behind a grounding guard.
Tests: engine tests (belt spacing, what-if, API, detector), brain tests and a 19-question live demo check.

Challenges we ran into

Getting a believable line: our first versions were too forgiving, then too strict, before the pacing, buffers and lost-pitch accounting agreed with the data.
Making the detector robust: the engine's temperature behaviour differed from the training data and caused false alerts, which we fixed with a thermal-lag model.
Cars in the 3D view overlapped, vanished at the inspection gate and were never coloured, and a leftover scripted shuttle collided with the live cars.
A what-if that took a minute with browsers connected; we moved the simulation branches to worker processes.
A "playing" button that wasn't playing: the twin and engine disagreed about play state, so the clock snapped back and looked frozen.
Retrieval once cited the battery-fire SOP for a weld robot fault. We now cite only procedures that apply to the machine.

Accomplishments that we're proud of

A working loop from sensor to alert to explanation to what-if to work order to repair, all inside one simulation.
A detector that explains itself and gives a remaining-life range, not just a score.
Answers a manager can act on, with cost in dollars.
A twin that stays readable: the scene first, one panel at a time.

What we learned

Deterministic answers go a long way; a language model is optional polish, not the foundation, and a plausible wrong answer is worse than none.
Synthetic data is only as honest as its caveats. We state clearly that the fault signatures are ours, so the results show the method works, not how it will do on a real plant.
Most "AI" bugs in a demo are really state-sync and timing bugs.

What's next for TalkToYourPlant

More failure types (motor, gearbox, hydraulic), each with its own sensor signature, so the detector and chat handle more than one story.
Real maintenance schedules and parts inventory behind the answers.
Voice and photo input ("what is this part?").
Connecting to real PLC or historian data instead of synthetic streams.
A modelled, rigged 3D look for the hero machines and people.