Inspiration

We wanted to explore how an ordinary object could feel more alive through human-robot interaction. Instead of treating LeLamp as just a controllable light, we wanted it to behave more like an expressive desk companion that can see, listen, react, and communicate through movement, light, and sound.

Our goal was to make interactions intuitive enough that someone could walk up to the lamp and understand how to interact with it without needing instructions.

What it does

Our project turns LeLamp into an interactive robotic companion using computer vision, voice interaction, expressive movement, and lighting.

LeLamp can recognize and respond to several different interactions:

When it detects a face, it greets the user with “Hello there,” flashes its light, and nods. When it detects a hand, it responds with a playful side-to-side wiggle. When it recognizes a peace sign, it performs a more dramatic dance and plays a short disco-style sound. When it detects a phone, it dramatically shakes, flashes its light, and tells the user: “Stop scrolling, please lock in twin.” It recognizes basic happy and sad facial expressions and responds with different movements and lighting behaviors. It accepts simple voice commands, including turning the lamp on and off. When a user points in a direction, the lamp follows the direction of the pointing gesture with its head. If there is no new interaction for 10 seconds, LeLamp enters an idle state, gradually dimming its light and lowering its head as if it is falling asleep. When someone interacts with it again, it wakes up and resumes responding.

The system also uses behavioral priorities so that overlapping detections do not cause conflicting actions. For example, if a face, peace sign, and phone are detected simultaneously, the phone reaction takes priority.

How we built it

We developed and tested the project using the LeLamp model in MuJoCo, allowing us to prototype the robot's movements and lighting behavior in simulation.

The project was primarily built in Python. We used computer vision to process a live webcam feed and recognize faces, hands, gestures, facial expressions, and objects.

We combined several components:

MuJoCo for LeLamp simulation and actuator control OpenCV for webcam input and image processing MediaPipe for hand tracking, gesture recognition, and facial detection YOLO for object detection, including cell-phone recognition Facial-expression recognition for basic happy and sad expressions Speech recognition for voice commands Text-to-speech for LeLamp's voice responses A behavior/state system to coordinate movement, lighting, audio, and detection priorities

Rather than directly triggering arbitrary motor positions, we created reusable behaviors such as nodding, wiggling, dancing, looking in a direction, sleeping, and waking up.

Challenges we ran into

One of the biggest challenges was combining several real-time systems without making the interaction feel slow or unresponsive. Computer vision, object detection, speech recognition, MuJoCo simulation, movement, lighting, and audio all need to operate at the same time.

We also had to prevent detections from triggering continuously every frame. A face visible for several seconds should not make the lamp repeatedly say hello, so we implemented confirmation frames, cooldowns, and state-based triggering.

Another challenge was handling overlapping detections. A person holding a phone may simultaneously produce a face, hand, and gesture detection, so we created a priority system to determine which behavior LeLamp should perform.

Finally, making the movements feel expressive rather than robotic required tuning the speed, amplitude, timing, and combination of different joints.

Accomplishments that we're proud of

We are especially proud that LeLamp does more than respond to individual commands — its different sensing and behavior systems work together to create a recognizable personality.

Some of our favorite accomplishments include:

Building real-time interaction between computer vision and the simulated robot Making LeLamp respond differently to faces, gestures, expressions, and objects Implementing directional pointing so the lamp can physically follow where the user points Combining movement, lighting, speech, and music into coordinated reactions Creating an idle/wake system that makes the lamp appear to fall asleep when ignored Building behavior priorities so multiple detections can be handled naturally Turning a simple robotic lamp into something that feels responsive and expressive What we learned

We learned that making a robot feel interactive is about much more than simply detecting an object or moving a motor. Timing, movement speed, pauses, lighting, sound, and context all influence how users interpret the robot's behavior.

We also gained experience integrating computer vision, object detection, gesture recognition, speech systems, and robotics simulation into one real-time application.

Most importantly, we learned how small physical behaviors — such as looking toward someone, lowering the head when idle, or reacting differently to a gesture — can make a robot feel significantly more engaging.

What's next for TBD

Next, we would like to move beyond predefined reactions and give LeLamp a more adaptive personality.

Future improvements could include:

More hand gestures and facial expressions Continuous face and gaze tracking More natural voice conversations Longer-term memory of previous interactions Dynamic moods such as curiosity, excitement, or boredom Generative AI for context-aware responses Additional expressive movement and lighting patterns Transferring our simulation behaviors to the physical LeLamp hardware

Ultimately, we want LeLamp to move from being a robot that simply reacts to inputs into a companion whose behavior changes naturally based on the person and environment around it.