Inspiration
SortingFactory began with a question:
- Could a simulated factory generate useful, traceable data for future SO-101 robot training?
I wanted to build more than a visually impressive demo. The goal was to create a system that could see objects, make time-sensitive decisions, perform actions, recover from real failures, and explain why each episode succeeded or failed.
I was especially interested in the experience of working with a long-running coding agent. After defining the architecture and acceptance criteria, I could let GPT‑5.6 sol execute a stage while I made coffee or went for a walk, then return to review a concrete, testable result. The surprising part was not simply the amount of code it produced—it was how consistently it remembered why earlier design decisions existed.
What it does
SortingFactory is a decentralized, multi-robot conveyor-sorting simulation built in Unity.
Three robotic arms are positioned along a moving conveyor. Each arm has its own camera, workspace, target-selection logic, and drop zone. There is no central queue assigning objects to robots.
The system follows this loop:
Observe → Detect → Track → Evaluate → Pick or Skip → Record → Recover
Unity streams camera frames to a Python FastAPI service over WebSocket. YOLO detects supported objects, while ByteTrack maintains persistent identities across frames. The service returns each object’s class, confidence, bounding box, tracking ID, and tracking state.
Inside Unity, each arm evaluates whether enough time remains to complete a grasp before the object leaves its workspace:
Remaining pickable time
≥
Required grasp time + Safety margin
If the condition is satisfied, the arm locks the target and begins the grasp cycle. If the object arrives too late, the arm skips it. A downstream arm must then detect the object again using fresh local vision rather than receiving an outdated position from the upstream workstation.
A localhost Control Room displays:
- Three live camera feeds
- Camera and vision connectivity
- Current targets and decisions
- Robot workflow and motion states
- Attempts, successes, failures, and skips
- Success rate, utilization, throughput, and cycle time
- The most recent failure reason for each arm
- Independent controls for sessions, cameras, arms, and conveyor speed
During a recording session, each arm writes SO-101-aligned telemetry at 10 Hz. The logs include six joint channels, perception results, tracking information, timing decisions, outcomes, failure reasons, and episode durations. Each session also produces metadata and a final summary.
How we built it
I started by asking GPT‑5.6 sol to create a phased, verifiable plan. We kept the complete project inside one continuous Codex session and implemented it milestone by milestone:
Scene → Camera Pipeline → Vision → Tracking → Decision Logic
→ Grasping → Failure Recovery → SO-101 Telemetry
→ Control Room → Session Management
I defined the core product and engineering constraints:
- Each arm must operate independently.
- Workspaces must not overlap.
- Downstream arms must use fresh local vision.
- Detection alone is insufficient; the system must consider physical timing.
- Real failures must be retained and explained.
- The joint hierarchy should align with the SO-101 topology.
- Monitoring must remain observational and must not become a central task allocator.
GPT‑5.6 sol helped translate those constraints into working implementations across several technologies:
- Unity and C# for the factory, conveyor, cameras, robot rigs, decision logic, IK motion, and data recording
- Python and FastAPI for WebSocket frame reception, HTTP endpoints, and Control Room state
- YOLO26n and ByteTrack for object detection and per-camera tracking
- HTML, CSS, and JavaScript for the local Control Room
- CSV session records for frame-level telemetry and episode summaries
The Unity-to-Python protocol includes arm, camera, frame, timestamp, resolution, image format, and workspace ROI metadata. Each camera has an independent tracker to prevent IDs from leaking between workstations.
The simulated arm follows this six-channel hierarchy:
shoulder_pan
→ shoulder_lift
→ elbow_flex
→ wrist_flex
→ wrist_roll
→ gripper
We repeatedly validated the implementation through C# compilation, Python unit tests, real JPEG inference, WebSocket round trips, tracking continuity checks, and Play Mode recording sessions.
Challenges we ran into
One major challenge was connecting computer-vision detections to physical Unity objects. A valid bounding box does not guarantee that the detected object still exists, remains reachable, or has a movable parent that can be attached to the gripper.
Real development sessions produced failures such as:
- A detected image having no movable parent object
- A selected physical object no longer matching the detected class
- An object leaving the conveyor before attachment
- No pickable physical object matching a detection
- IK being unable to reach the grasp position
- An object failing to remain attached to the gripper
Instead of hiding these cases or injecting random failures, we turned them into explicit episode outcomes.
Tracking was another challenge. Brief occlusion or a missed detection could cause unstable decisions. The Python service therefore preserves predicted tracks for a short grace period, while Unity maintains confirmed, coasting, lost, and skipped target states.
We also had to prevent two arms from manipulating the same physical object without introducing a central scheduler. A lightweight physical claim protects the object during manipulation, while target detection and decision-making remain decentralized.
Finally, we learned that SO-101-aligned telemetry is not automatically a training-ready LeRobot dataset. The current prototype does not yet contain synchronized training video, hardware calibration, or independently recorded goal actions. We therefore describe it as a bridge toward LeRobot conversion—not a drop-in dataset for real-hardware training.
Accomplishments that we're proud of
The largest accomplishment is completing an end-to-end robotics pipeline across Unity, Python, computer vision, web monitoring, and structured data recording.
During development, SortingFactory recorded:
- 14 prototype sessions
- 82 completed episodes
- 63 successful episodes
- 19 failures with specific diagnostic reasons
One completed validation session recorded 10 successful grasps from 10 attempts. Another recorded four successes and one real failure, producing an 80% success rate while preserving the exact failure reason for later analysis.
We are also proud of several less visible engineering decisions:
- The Latest Pick Line represents a physical timing constraint rather than an arbitrary UI marker.
- Each arm maintains its own camera and ByteTrack state.
- Downstream arms never inherit stale object positions.
- Session stopping waits for active arms to return safely to Idle.
- Frame-level telemetry and episode summaries are recorded separately.
- Failures release target locks and physical claims so the system can recover.
Most importantly, the entire architecture evolved inside one continuous GPT‑5.6 sol session. The model maintained constraints across C#, Python, protocols, vision, UI, tests, and later failure fixes without requiring the project to be repeatedly re-explained.
What we learned
We learned that long-context coding agents are most valuable at the system level. Generating an individual function is useful, but preserving architectural intent across many components is transformative.
We also learned that robotics requires more than perception. A detection must be connected to timing, reachability, physical state, concurrency, recovery, and measurable outcomes.
Observability should be designed from the beginning. Success counts alone are not enough; failure stages, timing, environmental conditions, and recovery outcomes are what make the data useful for improving a robot.
Finally, simulation telemetry and real-world training data are different products. Building an honest bridge between them requires synchronized images, independent observation and action semantics, stable timing, calibration, dataset conversion, and real-hardware validation.
What's next for SortingFactory
The next phase is to turn the prototype telemetry pipeline into a complete robot-learning dataset workflow.
Planned improvements include:
- Replacing the placeholder rig with a dimensionally accurate SO-101 digital twin
- Recording independent commanded actions and observed joint states
- Adding synchronized RGB video to every episode
- Converting sessions into the LeRobot v3 Parquet and MP4 structure
- Adding calibration, unit conversion, joint limits, and hardware feedback
- Connecting the system to a physical SO-101 follower arm
- Improving contact physics, grasp verification, and placement accuracy
- Training a custom detector for realistic factory objects
- Adding domain randomization for lighting, camera position, and object appearance
- Building real historical dashboards from recorded sessions
- Comparing success rates by arm, object class, conveyor speed, model version, and failure category
- Measuring whether downstream arms successfully recover objects missed upstream
The long-term goal is for SortingFactory to become a closed learning loop:
Simulate → Observe → Diagnose → Train → Deploy → Measure → Improve
Log in or sign up for Devpost to join the conversation.