Sidekick

An AI investigation workbench that uses NVIDIA Nemotron to propose sensor-fault experiments, reproduce warning-model failures, and compare remedies with measurable evidence.

Comment

Inspiration

Predictive-maintenance models depend on sensors that can disconnect, freeze or gradually drift. A model can perform well on clean data while losing useful warnings when its inputs become unreliable. Sidekick grew from our earlier ABB Accelerator project, which compared equipment-warning models under sensor faults. We wanted to extend those fixed tests into an investigation workflow: an engineer describes a concern, AI proposes bounded experiments, and the system measures what actually happens. Our motivation is to help engineers move from an impressive model score to a defensible decision about which model, warning threshold and sensor-failure cases deserve further testing.

What it does

Sidekick is an investigation workbench for reliability engineers and data engineers evaluating equipment-warning models. An engineer selects a fitted model, reviews its clean performance, chooses an equipment profile and sets experiment and AI-call allowances. NVIDIA Nemotron proposes sensor-fault scenarios involving missing readings, frozen sensors and gradual drift, including combinations across multiple sensors. Sidekick validates each proposal against the selected policy and runs accepted experiments against fixed development models. It compares clean and corrupted readings on matched equipment histories, measuring useful warnings, missed or late warnings, alarm burden and coverage. When a scenario meets the engineer’s failure criterion, Sidekick can reproduce it, simplify it through recorded reruns, export it as a regression case and compare a remedy against the original model. Results remain available through a browser workbench, HTTP API and CLI. Every conclusion distinguishes model-generated hypotheses from measured evidence.

Who it is for and why it matters

The intended users are reliability and data engineers selecting an equipment-warning model for a supervised trial. The decision Sidekick supports is specific: which model and warning rule should be tested next, and which sensor failures must the team plan for? Its intended value is to organize experiments, expose consequential weaknesses and preserve the evidence behind a recommendation. We have demonstrated the software workflow on synthetic data; we have not established maintenance savings, reduced downtime or real-equipment deployment readiness.

How we built it

The frontend uses React, TypeScript and Vite. A Python/FastAPI backend manages model evaluation, investigation operations and evidence exports. SQLite stores investigation records, proposals, execution attempts, budgets and controller activity. The evaluation pipeline uses NumPy, pandas, SciPy, scikit-learn and XGBoost. During an investigation, fitted models, preprocessing, thresholds, fault policy and data identity remain fixed.

NVIDIA Nemotron runs through Nebius Token Factory’s OpenAI-compatible Chat Completions API. We use the OpenAI Python SDK as the transport client; the investigation model is Nemotron. The controller sends an approved objective, aliased channel descriptions, permitted fault combinations, timing coverage and aggregate results. Nemotron returns a structured proposal and hypothesis, or a stop reason. Provider-facing schemas express fault-specific requirements, while local validators enforce the authoritative domain rules.

Accepted experiments run through a durable runner with saved limits, duplicate detection, cancellation and lease checks. Deterministic code calculates metrics and classifies findings. A reducer tests simpler scenarios, and remedy comparisons preserve the original results while evaluating explicit changes. Containerized workers support finite experiment jobs with checksummed inputs and results. Replay requires a compatible numerical runtime.

How we used NVIDIA and Nebius

We used nvidia/nemotron-3-super-120b-a12b, served through Nebius Token Factory, to propose sensor-fault experiments and respond to previously measured results. Token Factory provided hosted Nemotron inference without requiring us to operate the model-serving infrastructure. Its compatible API let us integrate inference into an existing Python application while retaining local validation, experiment execution and evidence storage. Our integration uses structured JSON output, explicit reasoning settings, per-call completion limits and persistent cumulative allowances. We record request outcomes, reported token usage and latency. Token Factory is the Nebius service used in the verified inference workflow. We are not claiming deployment on Nebius Serverless Jobs, Serverless Endpoints or AI Cloud.

What makes the project different

Sidekick combines AI experiment selection with an executable evaluation workflow. The model’s explanation does not determine whether a failure exists. A finding must satisfy a saved criterion using matched measurements and sufficient coverage. Invalid proposals, execution errors and scenarios that affect no eligible readings remain separate outcomes. The workflow also continues beyond finding a weak case. It reproduces the result, tests simplifications, compares remedies and exports the evidence. Scenario origin is recorded, so a hand-authored demonstration cannot be presented as an AI discovery.

Verified results

These results belong to specific recorded synthetic contexts.

  • Live Nemotron investigation: Three accepted proposals, each evaluated on all 22 development histories. Later proposals referenced earlier results. No qualifying failure was found.
  • Provider usage for that investigation: Three calls, 1,083 reported completion tokens; recorded request times were approximately 2.8–4.1 seconds.
  • Seeded failure demonstration: Simultaneously stuck vibration and oil-particle channels reduced useful warnings from 22/22 to 8/22: six histories became missed warnings and eight became late warnings.
  • Reduction: The failure replayed exactly and was reduced to later fault onsets. Minimality claims cover tested component removals and the specified timing grid.
  • Development remedy comparison: Changing the warning threshold from 0.95 to 0.5 resolved the reduced case, degraded none of the previously passing tested cases, and left the original stronger case failing.
  • Fresh synthetic assessment: On 12 fresh histories, the reduced case retained useful warnings for 8/12 histories before the remedy and 12/12 afterward. The stronger original case improved from 0/12 to 5/12, remaining a failure.

The failure demonstration uses a deliberately fragile 0.95 threshold to make the investigation workflow visible. Its failing case was supplied by us, not discovered by Nemotron. The recorded live benchmark does not establish that Nemotron outperforms fixed, random or hill-climbing search.

Challenges we ran into

One challenge was producing domain-valid experiments rather than merely valid JSON. Initial responses combined incompatible parameters or omitted required values. We introduced provider-facing alternatives for fault kinds, duration modes and onset modes, while keeping local validation authoritative.

Another challenge was applicability. A proposal could fit the declared policy yet choose an onset outside every available history. The controller now receives timing-coverage information, and unavailable coverage remains explicit.

Reproducibility also required attention. Windows and Linux numerical execution can produce different prediction digests despite similar warning metrics. We retained strict replay verification and standardized coordinator/worker checks within the same runtime.

Finally, we had to prevent a useful-looking demonstration from overstating its evidence. We now record who selected each scenario, distinguish development findings from fresh validation and report remedies that leave failures unresolved.

Accomplishments we are proud of

We built a workflow connecting live Nemotron proposals to validated experiments, persistent results and reproducible evidence. The project also supports compound sensor faults, deterministic finding criteria, component and timing reduction, remedy comparisons, regression exports, recorded-search benchmarking and a browser investigation workbench. A particularly useful outcome was demonstrating a remedy’s limits: it resolved a reduced failure but did not eliminate the stronger original failure. Sidekick preserves that distinction rather than presenting a blanket success.

What we learned

Structured output still needs domain validation. Experiment timing, coverage and sensor constraints matter as much as whether a response parses. We also learned that discovering a weakness and choosing a remedy use the same development evidence, so both need a separate assessment on fresh histories. AI-assisted search must be compared fairly against simpler methods. Our current results demonstrate functioning adaptive proposals, not superior discovery performance.

What changed from the earlier project

The earlier Sidekick project trained and compared equipment-warning models under fixed, single-sensor fault protocols. The hackathon adaptation added:

  • NVIDIA Nemotron inference through Nebius Token Factory.
  • Typed compound-fault scenarios and versioned sensor profiles.
  • A durable bounded experiment runner and adaptive controller.
  • Fault-specific provider schemas, timing coverage and corrective feedback.
  • Deterministic finding classification and recorded hypothesis lineage.
  • Reproduction, scenario reduction and regression-case exports.
  • Remedy comparisons with costs and unresolved failures.
  • Recorded-search benchmarking.
  • Investigation API routes and a React workbench.
  • Containerized workers and public evidence exports.
  • Frozen-procedure assessment on fresh histories with exposure tracking.

This extends Sidekick from fixed robustness testing into an AI-assisted experimental investigation system.

Feedback on Nebius Token Factory and NVIDIA Nemotron

Token Factory’s compatible API made integration with our existing Python stack straightforward. Reported usage and request metadata helped us preserve an audit trail for bounded investigations. In an early integration, schema-valid responses could still violate cross-field domain requirements. Explicit alternative schemas and timing-coverage context improved the integration. Clear documentation and examples covering nested schema alternatives, reasoning-token allowances and structured-output failure modes would help builders developing tool-driven agents. In one recorded three-call investigation, all proposals were accepted and evaluated across the full development set. Responses took approximately 2.8–4.1 seconds and reported 1,083 completion tokens in total. These are small-run observations, not service-wide performance claims.

What's next for Sidekick

We want to evaluate more contexts and repeatable search workloads, measure when adaptive selection helps relative to simpler strategies, and validate sensor-profile assumptions with engineers. Further work includes representative equipment data, reviewer usability studies, dependable hosted access and carefully bounded parallel execution where the workload justifies it.

Built With

Share this project:

Updates

Submission history