Inspiration
Silex began with a question: how can an AI assistant move beyond a screen and communicate through a physical body?
Most AI assistants interact through text or audio. I wanted to create a robot capable of listening, speaking, observing its environment, remembering useful information, and expressing its responses through coordinated facial and body movement.
My goal is not simply to connect ChatGPT to a group of servos. Silex is intended to become an embodied social robot whose voice, gestures, perception, memory, and physical presence work together as one coherent system.
Silex existed as a functioning robot before OpenAI Build Week. During the event, I worked with Codex to extend it with a measured inverse-kinematics architecture and connect GPT-5.6 safely to its physical movement system.
What it does
Silex is a physical social robot controlled through voice, browser, and mobile interfaces.
A user can speak or type an instruction. GPT-5.6 interprets it and returns a structured response containing:
- the text Silex should say;
- the detected intent;
- an emotion;
- an intensity;
- a sequence of movements.
Silex speaks through multilingual Azure neural TTS while coordinating its mouth, cheeks, eyelids, eyebrows, head, arms, hands, and fingers.
Its current capabilities include:
- configurable wake-word voice control;
- multilingual neural speech;
- facial expressions and speech articulation;
- expressive gestures and physical choreographies;
- inverse-kinematics movements;
- person and face tracking;
- recognition of poses, hands, gestures, objects, and environmental sounds;
- stereo sound-direction estimation;
- local SQLite memory for people, preferences, conversations, and tasks;
- WhatsApp messaging, incoming-message announcements, and camera snapshots;
- Jitsi meeting invitations;
- desktop and mobile control panels;
- private remote access through Tailscale;
- CPU, memory, perception-latency, and motor-delay monitoring;
- automatic perception-frequency reduction during computer overload.
Silex processes many perception tasks locally with MediaPipe, EfficientDet, and YAMNet. It does not need to send every camera or audio frame to an external AI service.
GPT-5.6 coordinates the high-level response, but it cannot access the serial ports directly. Mechanical limits, movement priorities, cancellations, and emergency stopping remain under deterministic local control.
How we built it
Silex combines Node.js, ESP32 motor controllers, Azure TTS, the OpenAI Responses API, MediaPipe, SQLite, FFmpeg, WhatsApp Web, Jitsi, and browser-based control interfaces.
The main interaction pipeline is:
Voice or text -> GPT-5.6 -> structured response -> Azure TTS + motion scheduler -> physical response
Before Build Week, Silex already had voice interaction, expressive servo choreographies, local perception, WhatsApp communication, memory, camera streaming, and remote control.
During Build Week, I focused on replacing manually estimated servo poses with a measured inverse-kinematics foundation.
I measured Silex's body and created a model containing:
- the torso coordinate system;
- neck and eye positions;
- shoulder locations;
- upper-arm and forearm lengths;
- physical workspaces;
- mechanical limits;
- torso collision-safety distances.
I then worked with Codex to develop:
- analytical arm-reach and head-look solvers;
- servo-to-joint calibration;
- measured JSON robot definitions;
- a hardware-free IK simulator;
- frontal and lateral visualizations;
- complete choreography playback without activating motors;
- continuous Cartesian trajectories;
- smooth acceleration and deceleration;
- unreachable-target rejection;
- local physical safety clamps;
- an adapter connecting IK to the existing motion scheduler;
- reduced-speed and reduced-amplitude hardware tests;
- automated tests for calibration, simulation, safety, and physical execution.
The IK movement library currently includes greeting, pointing, presenting, and thinking as physically validated unilateral movements. It also includes explaining, offering, welcoming, and celebrating movements that remain simulation-only until the left arm is completely revalidated.
Codex helped inspect the existing repository, propose the incremental architecture, implement modules, diagnose failures, build calibration and simulation tools, connect IK to the scheduler, and create automated tests.
I made the physical and behavioral decisions: robot dimensions, servo directions, rest positions, mechanical restrictions, desired movements, and final approval after observing Silex move in the real world.
Challenges we ran into
One of the main challenges was translating mathematical joint movement into real servo movement.
One commanded servo degree does not always equal one physical joint degree. Some motors are inverted, and others move their joints through mechanical linkages with different ratios. We created a calibration workflow that records commanded servo travel and measured physical joint travel.
Another challenge was protecting the hardware. The B17 elbow has a physical limit of 90 degrees. Exceeding it could damage the mechanism or burn the motor. This restriction is now enforced locally, and the thinking gesture uses B19 to compensate without forcing B17 beyond its stop.
The B20 motor failed during development. Its current calibration is explicitly marked as provisional, and bilateral physical IK movements are automatically disabled until the replacement motor is installed and calibrated.
Natural motion was also difficult. Early movements looked abrupt, moved through disconnected poses, or had less physical amplitude than the simulator. We changed the movement system to use continuous Cartesian paths with smooth acceleration and deceleration.
Coordinating perception was another challenge. Person tracking, facial imitation, hand recognition, sound reactions, speech, and explicit commands can compete for the same motors. A perception coordinator and motion scheduler now prioritize intentional actions and suppress unnecessary simultaneous movements.
Finally, Silex runs on a Windows computer with integrated graphics. Camera streaming, MediaPipe, speech, GPT communication, and motor control must share limited resources. The system monitors performance and automatically reduces lower-priority perception frequencies when sustained overload is detected.
Accomplishments that we're proud of
I am proud that Silex now connects GPT-5.6 intent to measured physical movement while preserving deterministic local safety.
Instead of defining every gesture only as fixed servo angles, the new IK system can describe where a hand should move in Cartesian space and calculate a safe physical trajectory.
The visual simulator can reproduce the measured proportions of Silex, compare legacy choreographies with IK intentions, and preview complete movements without sending commands to the hardware.
The new architecture preserves compatibility with Silex's existing TTS workflow, control panels, prompts, and movement names. This allowed the migration to happen incrementally without losing the robot's previous capabilities.
The project currently passes 216 automated tests with zero failures. These tests cover GPT model selection, authentication, memory, microphone lifecycle, perception, object and sound detection, motion scheduling, mouth articulation, servo calibration, inverse kinematics, safety limits, simulation, and reduced physical movement.
The most important accomplishment is that the language model provides intelligence without receiving unrestricted authority over the robot's motors.
What we learned
I learned that building embodied AI requires a clear separation between intelligence and physical authority.
GPT-5.6 is effective at understanding language, selecting an emotional response, and coordinating a high-level reaction. However, final control over motors must remain in deterministic local software.
I also learned that inverse kinematics is only as reliable as the measurements and calibration behind it. A mathematically valid result can still be mechanically unsafe when the real direction, transmission ratio, or physical stop is incorrect.
Natural behavior does not come from activating more motors at the same time. It comes from timing, prioritization, smooth trajectories, and coordination between speech, gaze, expression, posture, and environmental perception.
Working with Codex also demonstrated the value of combining software analysis with human physical validation. Codex could inspect, implement, simulate, and test the system, while I measured the real robot and decided whether each movement was safe and expressive.
What's next for Silex: Embodied AI for Everyday Life
The immediate next step is to replace and calibrate the B20 motor. Once that is complete, I can safely validate bilateral IK movements on the physical robot.
I also plan to install and calibrate an RGB-D camera. Depth information will allow Silex to transform detected objects into robot coordinates and eventually reach toward them safely.
Future development includes:
- collision-aware arm planning;
- supervised object manipulation;
- gripper feedback;
- improved mechanical head balance;
- power, temperature, and battery monitoring;
- consent-based person recognition linked to local memory;
- Home Assistant integration for allowlisted lights, sensors, outlets, and routines;
- improved real-time conversational turn-taking;
- more organic coordination between perception and movement.
The long-term goal is for Silex to navigate and walk. That stage will require a lower-body mechanical design, balance sensing, center-of-mass control, foot-contact detection, fall protection, a stronger power system, and a dedicated locomotion safety controller.
Silex is an evolving platform for exploring how conversational intelligence, local perception, persistent memory, expressive movement, and physical safety can operate together in everyday life.
Built With
- azure-ai-speech
- computer-vision
- efficientdet
- embodied-ai
- esp32
- express.js
- ffmpeg
- html5
- inverse-kinematics
- javascript
- jitsi-meet
- mediapipe
- node.js
- openai-codex
- openai-gpt-5.6
- openai-responses-api
- progressive-web-app
- robotics
- serialport
- sqlite
- tailscale
- tensorflow-lite
- web-speech-api
- whatsapp-web.js
- yamnet
Log in or sign up for Devpost to join the conversation.