CLIP-E

CLIP-E turns your hairstyle into a personalized robotic haircut using computer vision, live camera feedback, and intelligent robotic control.

Comment

CLIP-E — A Robotic Haircut Assistant

Problem & Solution

People struggle to accurately see, assess, and maintain hard-to-reach areas of their hair, making consistent self-grooming difficult and often inaccessible.

CLIP-E combines four-view image capture, MediaPipe geometry tracking, Gemini-powered hair and face analysis, and a 100-style matching system to build a personalized profile and recommend the user's top hairstyle options.

The platform then translates that profile into visual cut guidance, AR mapping, and robotic-arm assistance designed to support controlled trimming in areas such as the back, sides, crown, and neckline.

By connecting computer vision, AI analysis, personalized recommendations, and physical automation in one workflow, CLIP-E explores how intelligent grooming tools can make haircut planning and maintenance more precise, understandable, and accessible.

Inspiration

Getting a haircut is more than just choosing a style. It can be difficult to communicate exactly what you want, visualize how a hairstyle will look, and maintain precision—especially when working on areas that are hard to see, such as the back of the head.

We wanted to explore how computer vision, AI, robotics, and an interactive web application could work together to make the haircut process more personalized, precise, and accessible.

This led us to create CLIP-E, a robotic haircut assistant that connects hairstyle recommendations with computer vision, AI-powered analysis, robotic movement, and live camera feedback.

What it does

CLIP-E combines a web application, computer vision, generative AI, a robotic arm, and a camera into one interactive system.

The workflow is:

  1. Capture — Four-view images provide different perspectives of the user's head and hair.
  2. Analyze — MediaPipe geometry tracking and Gemini-powered analysis help understand facial and hair characteristics.
  3. Match — CLIP-E compares the user's profile against a library of 100 hairstyle options.
  4. Recommend — The system identifies personalized hairstyle options based on the user's profile.
  5. Map — The selected hairstyle is translated into visual cut guidance and AR-style mapping.
  6. Assist — Robotic-arm assistance is designed to support controlled trimming of areas such as the back, sides, crown, and neckline.
  7. Feedback — Camera and visual feedback help the user understand areas that are difficult to see directly.

The goal is to connect the idea of a hairstyle with a personalized, mapped, and guided physical haircut process.

How we built it

We built CLIP-E by combining web development, computer vision, generative AI, embedded systems, robotics, and camera hardware.

Web Application

  • React
  • Vite
  • JavaScript / JSX
  • HTML
  • CSS
  • MediaPipe Tasks Vision
  • Supabase

The web application provides the interface for capturing information, exploring hairstyle options, viewing recommendations, and interacting with the CLIP-E system.

AI & Computer Vision

We integrated Google Gemini APIs for AI-powered hair and face analysis and used a 100-style matching system to generate personalized hairstyle recommendations.

We also use MediaPipe geometry tracking to work with facial and head geometry, providing information that supports the personalized profile and visual mapping process.

Robotics

The robotic system uses:

  • Arduino / ESP32
  • C++
  • Servo motors
  • Robotic arm
  • Control and safety logic

The robotic system is designed to provide controlled assistance based on the haircut mapping generated by the software.

Camera System

We built a camera system using:

  • ESP32-S3
  • ArduCAM Mini 5MP Plus
  • OV5642 camera
  • Wi-Fi
  • HTTP live JPEG streaming

The camera system provides visual feedback and supports views of areas that are difficult for the user to see.

Overall Architecture

Four-View Image Capture
          ↓
MediaPipe Geometry Tracking
          ↓
Gemini Hair & Face Analysis
          ↓
Personalized User Profile
          ↓
100-Style Matching System
          ↓
Hairstyle Recommendation
          ↓
Visual / AR Cut Mapping
          ↓
Robotic Arm Assistance
          ↓
Camera + Live Feedback

Challenges we ran into

One of our biggest challenges was getting the physical and digital systems to work together reliably.

  • Motors burning out: We had to troubleshoot our robotic arm hardware after motors burned out during testing, forcing us to rethink how we controlled and powered the motors.
  • Phone-to-robot motion: We worked on matching the movement of the phone with the movement of the robotic arm so that the user's inputs could translate into predictable robot motion.
  • API integration: Connecting the robotic arm and camera APIs to the website was challenging because we needed multiple systems to communicate reliably and exchange data in real time.
  • Designing the CLIP-E mount: We also had to design and iterate on a physical mount for CLIP-E that could securely hold the components while still allowing the system to move and operate effectively.

These challenges pushed us to work across both hardware and software and taught us how important communication between physical components and digital systems is when building a robotics project.

Accomplishments that we're proud of

We are proud of bringing together several different technologies into one cohesive concept.

CLIP-E combines:

  • Four-view image capture
  • Computer vision and geometry tracking
  • Gemini-powered hair and face analysis
  • A 100-style matching system
  • Personalized hairstyle recommendations
  • AR and visual haircut mapping
  • Robotic-arm assistance
  • Embedded camera hardware
  • Live visual feedback

We were especially excited to connect AI analysis and personalized recommendations with a physical robotic system, creating a workflow that moves from understanding the user's appearance to providing actionable haircut guidance.

We are also proud of building a system that bridges the gap between a digital hairstyle recommendation and physical robotic assistance.

What we learned

We learned that building a robotics project is as much about integration as it is about individual components.

A web application, AI model, computer vision system, microcontroller, robotic arm, and camera can each work independently, but connecting them into one reliable system introduces many additional challenges.

We also learned how difficult it can be to translate visual and geometric information into physical actions. AI can help understand and recommend a hairstyle, but turning that information into useful visual guidance and controlled robotic movement requires careful planning.

Working through issues like burned-out motors, motion synchronization, API communication, and physical mounting also taught us how important iterative testing is when combining hardware and software.

Finally, we learned the importance of multiple views and live feedback when designing human-robot interactions. Giving the user a better understanding of areas they cannot easily see is a central part of CLIP-E's approach.

What's next for CLIP-E

Our next step is to make the system more precise, adaptive, and reliable.

We want to improve:

  • Hair and face analysis
  • Head and hair geometry tracking
  • Hairstyle matching
  • Personalized recommendations
  • AR haircut mapping
  • Robotic movement precision
  • Adaptive movement based on head position
  • Live camera feedback
  • User interaction and controls
  • End-to-end integration

Ultimately, we want CLIP-E to evolve from a hackathon prototype into a more complete intelligent grooming assistant that can understand a user's appearance, recommend a personalized hairstyle, visualize the desired cut, and safely guide the physical haircut process.

Built With

Share this project:

Updates

Submission history