Gemo

• 

  RC car with raspberry pi and battery

Inspiration

Can a large language model actually drive a car?

With multimodal AI rapidly advancing, I wanted to test the limits of general-purpose models by connecting one directly to the physical world. That question became Gemo.

What it does

Gemo is a vision-only autonomous RC car powered by a Gemini-family model.

• Uses a single front-facing camera
• No LiDAR, no maps, no extra sensors
• The AI interprets images and decides how to drive in real time

How I built it

• RC car platform controlled by a Raspberry Pi
• Camera input sent to a Gemini multimodal model
• Prompt engineering + function calling to translate AI decisions into motor commands
• Iterative real-world testing and tuning

Challenges I ran into

• Unstable control requiring tiny, incremental movements
• Frequent crashes due to small perception errors
• Latency and inconsistency from model outputs
• Power and motor control limitations on hardware
• Live models can be unstable or intermittently fail depending on model version, rollout, or API conditions.

Accomplishments that I'm proud of

• Successfully connected a large language model to real-world driving
• Proved that LLM-driven autonomous control is possible, even if imperfect
• Built a complete end-to-end system from AI reasoning to physical motion

What I learned

Just as humans don’t consciously think in language while driving,
autonomous driving may require models specifically designed for driving,
not just general-purpose transformer-based language models.

This experiment highlighted the gap between language reasoning and real-time control.

What's next for Gemo

• Test driving-specialized models
• Fix some malfunctions of Gemini Live Model
• Improve control stability and feedback
• Combine classical control with AI reasoning
• Push further into real-world autonomy experiments

Built With

• gemini
• python
• raspberry-pi