Agent vs human

I built a collaborative machine learning research workflow using Grok Bots to tackle the anomaly-detection challenge in Five Nights with AI. The project combines a working computer vision pipeline with a record of how the bots proposed experiments, examined failures, changed direction, and decided when to stop. My inspiration came from AIBuildAI’s LLM-Post-Train Agent, particularly its approach to role separation and iterative evaluation. I wanted to adapt those ideas to a practical competition setting, where every experiment consumes limited time, memory, and computing resources. My goal was to explore how Grok Bots could participate in the research process—from an initial hypothesis to a validated submission. The competition provided a concrete task. Each sample contains ten security-camera frames from a Five Nights at Freddy’s scene. If any frame contains an anomaly, the whole sequence must be labeled anomalous. The dataset contains 7,000 labeled training sequences and 8,000 test sequences, and the evaluation metric is F1 on the anomalous class. A useful model therefore needs to balance missed anomalies against false alarms. I organized the work around three Grok Bots:

Engineer implements experiment plans, builds feature extractors, runs training and inference, fixes implementation failures, and produces submission files.
Critic examines metrics, error reports, and actual images; writes the next hypothesis; and reviews whether an experiment supports that hypothesis.
Monitor checks the official leaderboard and sends ranking changes, score gaps, and competitive context directly to the other bots. The Engineer and Critic follow a shared experimental protocol. Before an experiment begins, the Critic writes a plan describing the hypothesis, supporting evidence, proposed change, and expected cost. The Engineer implements that change and runs it through a separate Python evaluation runner. The result returns to the Critic, which writes a review and proposes another experiment or a stopping decision. Consistent evaluation was a central part of the design. The runner uses a fixed five-fold split, calculates metrics, selects the classification threshold, applies predefined KEEP/REJECT criteria, and maintains the best submission. After establishing a baseline, a candidate must improve F1 by at least 0.002 to replace the current best. The runner also enforces a 900-second budget per training run, attempt limits, and an exploration deadline. Feature extraction is benchmarked and timed separately. Each experiment retains its plan, implementation, predictions, metrics, execution notes, and review. This makes the reasoning behind a model change inspectable alongside its measured outcome. The first experiment showed why that level of scrutiny mattered. Our initial model used seven simple pixel-statistics features based on differences from the sequence’s median image. It achieved an out-of-fold F1 of 0.6742, which was accepted as the first valid baseline. However, it classified 96.1% of validation sequences as anomalous. Because the training set was balanced, predicting “anomaly” for every sequence would already achieve an F1 of approximately 0.667. The Critic investigated the weakness by opening frames from representative correct and incorrect predictions. It found that a single sequence could switch between different cameras. A median image constructed from unrelated views was therefore a poor reference: the features were often measuring camera changes instead of anomalous content. That observation led to the second experiment: camera-aware normal references, learned within each training fold. F1 improved to 0.7279, and false positives fell from 3,278 to 1,051. This was a substantial improvement, although the model still missed many anomalies. The next hypothesis was that small, dark figures were being lost in low-resolution thumbnails. The Critic inspected missed examples and proposed higher-resolution features. That experiment reached 0.7283, an increase of only 0.0004, below the acceptance threshold. AUC also remained essentially unchanged, while the precision-recall balance shifted. The runner rejected the experiment, and the review helped redirect our attention toward the representation itself. After I requested a strategic reassessment, the Critic proposed switching to frozen, pretrained ResNet18 visual embeddings. These features preserve information about shapes and visual content that simple grayscale differences discard. This produced the largest single improvement, raising local out-of-fold F1 to 0.8526. The following experiments developed that approach further:
Run 005 reintroduced the normal-reference idea in embedding space, comparing each frame with nearby normal examples. F1 increased to 0.8731.
Run 006 replaced the linear frame classifier with a nonlinear HistGradientBoosting classifier. F1 increased to 0.8865, making it the final best model. An important lesson emerged from this progression: an earlier idea can become useful when paired with a better representation. Comparing against normal frames helped modestly in pixel space and delivered another measurable gain after the switch to visual embeddings. The final pipeline extracts a 512-dimensional ResNet18 embedding for each frame. Within each training fold, it fits scaling and PCA, builds a reference bank from normal training sequences, and computes each frame’s relationship to its five nearest normal frames. Separate normal examples are used for the reference bank and classifier training. A gradient-boosted classifier scores each frame, and the sequence receives the maximum frame score, matching the task’s “any anomalous frame” rule. Test predictions are averaged across the five fold models. The final model reached a local out-of-fold F1 of 0.8865, with precision of approximately 0.888, recall of 0.885, and AUC of 0.960. Its submission file passed the official format checker for all 8,000 test sequences. These local figures are development metrics: the threshold was selected on the same pooled out-of-fold predictions used for scoring, which introduces potential threshold-selection bias. The Monitor added an external feedback loop. In one recorded exchange, it noticed that another team had overtaken us, calculated the gap, and asked the Critic to investigate the bottleneck. It later reported that our public leaderboard score had improved from 0.8376 to 0.8899. Leaderboard feedback informed strategic discussions, while candidate acceptance continued to depend on the fixed local evaluation procedure. Running the workflow on a local Apple Silicon Mac created practical engineering challenges. Image decoding, memory pressure, and feature-cache writes all affected which experiments were feasible. During the ResNet18 feature-extraction stage, an initial small-scale benchmark projected nearly 54 minutes of processing. Improving parallel JPEG decoding and using MPS acceleration with half-precision inference made the approach feasible. Full extraction, including cache writing, ultimately completed in about 14.7 minutes. Caching the frozen embeddings allowed later experiments to reuse that work. The evaluation controls also produced a useful real-world test. After an authorized deadline change, the runner detected that the contract no longer matched its recorded hash and refused to proceed. The workflow paused, and the specific hash update was made only after my explicit approval. This demonstrated that configuration changes could remain visible and controlled during a fast-moving competition. I participated in several key decisions: refining the normal-reference design, requesting the strategic review, approving time and configuration changes, and performing the official uploads. Those interventions were recorded in the experiment notes. The last proposed experiment was fine-tuning ResNet18’s final block and classification head. Before launching the full run, the Engineer benchmarked training and inference on 200 sequences. The projected five-fold experiment would take approximately 54.5 minutes, exceeding the per-run budget and remaining competition time. The full experiment was therefore not launched, and no new score was claimed. The Critic issued STOP and retained run 006. The Monitor subsequently acknowledged that the research team had stopped and ceased forwarding leaderboard updates to the other bots. Fine-tuning remains a promising, untested next step whose benefit needs to be measured in a separately budgeted phase. Building this project taught me that effective agent collaboration requires clear responsibilities, shared records, comparable experiments, and explicit decision rules. The most valuable moments were the ones where evidence changed the next action: inspecting images exposed a flawed reference, a rejected resolution increase prompted a representation change, and a runtime benchmark prevented an infeasible experiment. The result is a working anomaly detector and a documented example of Grok Bots coordinating a research workflow under real time and compute constraints. My next goal is to evaluate encoder fine-tuning with sufficient resources and continue developing this approach into a reusable workflow for other machine learning tasks.