SENSE

SENSE lets you control hardware with plain English. It detects connected devices, builds the logic, and deploys it instantly—turning ideas into working physical systems.

Comment

SENSE

A physical-computing sandbox where you describe what you want real hardware to do in natural language.

Inspiration

Robotics is incredibly rewarding, but getting from an idea to a working physical prototype can be frustrating.

Even a simple interaction can require:

  • wiring components
  • figuring out pin mappings
  • installing libraries
  • writing embedded code
  • compiling
  • flashing firmware
  • debugging everything before you can even test whether the idea is good

I wanted to make that first step feel different.

Instead of starting with code, I wanted to start with intent:

"Make the LEDs flash faster as I squeeze harder."

From there, the system should figure out how the connected hardware can satisfy that request.

That idea became SENSE, a physical-computing sandbox where users can describe hardware behavior in natural language and immediately experiment with real sensors and actuators.

For beginners, SENSE is a way to explore robotics concepts without being blocked by syntax and boilerplate.

For more experienced builders, it acts as a rapid-prototyping environment where physical interactions can be tested before committing them to permanent embedded firmware.

What it does

SENSE connects real sensors and actuators to an ESP32 and Raspberry Pi, then exposes those devices as capabilities rather than hard-coded application behavior.

A user can enter requests such as:

"Flash all four LEDs faster as I squeeze harder."

"Make the servo follow the potentiometer smoothly."

"Use the four buttons as a password. If the password is correct, turn on all the lights. Otherwise, wave the servo and buzz."

SENSE sends the user's request to Gemini, along with a live description of the hardware that is currently available.

Gemini then compiles the request into a small Python-like language called SenseScript.

The generated program is:

  1. parsed
  2. validated
  3. repaired if necessary
  4. executed through a custom interpreter
  5. connected to the real hardware through a controlled API

The key idea is that the ESP32 firmware does not need to be rewritten every time the behavior changes.

The firmware exposes hardware capabilities once; SENSE defines behavior live.

How I built it

SENSE is divided into several layers.

Gemini Semantic Compiler

Gemini acts as the semantic compiler.

It receives:

  • the user's natural-language request
  • the currently available hardware
  • the capabilities of each device
  • recent sensor values
  • telemetry quality information
  • the SenseScript language rules

Instead of simply generating Arduino code, Gemini produces a program specifically for the SENSE runtime.

SenseScript

SenseScript is a constrained Python-like language for physical behavior.

For example:

async def behavior(hw):
    while True:
        delay = clamp(
            map_value(
                hw.pressure.percent,
                0,
                100,
                1.0,
                0.08
            ),
            0.08,
            1.0,
        )

        await hw.leds.all(True)
        await sleep(delay)

        await hw.leds.all(False)
        await sleep(delay)

The important part is that generated code is not executed using exec() or eval().

Instead, SENSE:

  1. parses the generated program into a Python AST
  2. validates it against an allowed subset of syntax
  3. rejects unsupported or unsafe constructs
  4. asks Gemini to repair invalid programs when possible
  5. manually interprets the approved AST

Unsafe capabilities such as these are blocked:

  • imports
  • filesystem access
  • subprocesses
  • arbitrary networking
  • private attributes
  • unrestricted Python objects
  • unsupported syntax

This became especially important when I tested more complicated prompts.

For example, one security-system request caused Gemini to generate list comprehensions and is / is not expressions that were outside the supported SenseScript subset.

Rather than running the program anyway, SENSE rejected it and offered to generate a simpler implementation.

The AI is not trusted blindly.

Hardware Runtime

SenseScript programs interact with the real world through a controlled hardware API.

Examples of reading sensors include:

hw.pressure.percent
hw.distance_front.cm
hw.button1.pressed
hw.oak.person_visible

Examples of controlling actuators include:

await hw.led1.on()
await hw.servo.angle(90)
await hw.buzzer.tone(1200)

There is also a generic capability interface:

value = hw.read("device_id", "capability")

await hw.write(
    "device_id",
    "action",
    value
)

This makes the runtime less dependent on any specific sensor or actuator.

Physical Hardware

An ESP32 handles the physical sensors and outputs.

The prototype includes:

  • 2 ultrasonic distance sensors
  • 4 push buttons
  • 4 LEDs
  • pressure sensor
  • potentiometer
  • photoresistor
  • PIR motion sensor
  • DHT11 temperature/humidity sensor
  • passive buzzer
  • servo motor
  • Luxonis OAK-1 camera

The ESP32 communicates with SENSE over a serial protocol.

Output commands use:

  • acknowledgements
  • sequence IDs
  • retry handling
  • duplicate suppression

This prevents a retried command from accidentally applying the same physical action multiple times.

Multiple output changes can also be batched together so actions happen more synchronously.

Computer Vision

A Luxonis OAK-1 camera is connected to a Raspberry Pi.

The Pi captures frames from the camera and performs person detection using YOLOv5n.

The resulting information is published into the same SENSE hardware registry as every other sensor.

For example:

hw.oak.person_visible
hw.oak.person_x
hw.oak.person_y
hw.oak.person_count

This means computer vision and physical sensors can be combined in the same behavior.

For example:

"Track the person with the servo and increase the buzzer pitch as they get closer."

Real-World Reliability

One of the biggest differences between simulation and physical hardware is that real sensor data is messy.

Sensors can:

  • stop responding
  • produce impossible values
  • become stale
  • temporarily disconnect
  • fluctuate because of noise

SENSE validates incoming telemetry before using it.

For each sensor stream, it tracks information such as:

  • accepted samples
  • rejected samples
  • last received time
  • last valid sample
  • last known-good value
  • current health state

The runtime can distinguish states such as:

healthy
stale
fault
missing

A generated program can explicitly check whether data is trustworthy:

if hw.valid(
    "distance_front",
    "distance_cm"
):
    distance = hw.read(
        "distance_front",
        "distance_cm"
    )
else:
    await hw.safe_state()

If a critical sensor stops providing valid data, the system can stop relying on the old value and move into a safe state.

I also added fault injection, allowing sensor failures to be simulated intentionally during testing.

Live Semantic Context

Another challenge was realizing that a sensor's theoretical range is not always the range that matters during an interaction.

For example, an ultrasonic sensor may support approximately:

[ 2\text{ cm} \rightarrow 400\text{ cm} ]

but during a tabletop interaction, the actual observed range might be closer to:

[ 20\text{ cm} \rightarrow 120\text{ cm} ]

If Gemini maps behavior across the entire theoretical range, the response can feel almost static.

SENSE therefore tracks recent telemetry and provides Gemini with:

  • the current value
  • the recent observed range
  • a suggested interaction range

This makes generated behavior much more responsive to how the hardware is actually being used.

Challenges I Faced

Building a Safe Execution Environment

The hardest part of SENSE was not getting an LLM to generate code.

The hard part was safely putting an AI system between a human and the physical world.

Generated Python cannot simply be executed directly when it controls real hardware.

I had to design:

  • the SenseScript language
  • an AST validator
  • operation limits
  • an allowlisted syntax subset
  • a controlled hardware API
  • a manual interpreter

I also had to handle cases where generated programs were syntactically valid but attempted operations the interpreter did not support.

That led to hardening the validator and runtime so unsupported constructs are caught before the physical program is installed.

Handling Unreliable Hardware

I encountered issues with:

  • unstable serial communication
  • ultrasonic sensor dropouts
  • USB power limitations
  • OAK-1 camera disconnects
  • sensor calibration
  • stale telemetry
  • physical power and wiring problems

At one point, the OAK camera repeatedly disconnected because the Raspberry Pi 5 was not configured for its higher USB peripheral current limit.

Fixing the Pi USB power configuration made the camera stable.

This reinforced one of the biggest lessons of the project:

Physical systems require much more defensive programming than purely digital applications.

Making Generated Behavior Perceptible

Using the full theoretical range of a sensor sometimes produced behavior that technically worked but did not feel responsive.

For example, mapping an ultrasonic sensor across its entire 2-400 cm range could make normal hand movement barely affect the output.

Tracking recent real-world sensor ranges and exposing that context to Gemini significantly improved the quality of generated interactions.

Balancing Flexibility and Safety

The more Python-like features SenseScript supports, the more useful it becomes.

However, every additional feature also expands the surface area of the sandbox.

I had to balance allowing useful features such as:

  • state
  • lists
  • loops
  • timing
  • arithmetic
  • randomness
  • sensor fusion

while still blocking unsafe or unpredictable operations.

What I Learned

The biggest lesson from SENSE was that natural-language hardware control is not really just a code-generation problem.

The interesting problem is building the runtime between the AI and the physical world.

That runtime has to understand:

  • what hardware exists
  • what each device can do
  • what data is trustworthy
  • what syntax is allowed
  • what happens when something fails
  • how physical outputs should be applied safely

I also learned that exposing generated SenseScript is useful educationally.

Instead of hiding everything behind an AI interface, a user can see how their natural-language request becomes:

sensor
  ↓
calculation
  ↓
decision
  ↓
actuator

For someone learning robotics, this creates a path from understanding behavior to understanding code.

What I'm Proud Of

The part I am most proud of is that SENSE does not rely on a collection of predefined demos.

The same hardware can become:

  • a password lock
  • a proximity alarm
  • a reaction game
  • a pressure-controlled lighting system
  • a person tracker
  • a sensor-fusion experiment

without changing the ESP32 firmware.

A user can describe a new behavior, Gemini can compile it into SenseScript, SENSE can validate it, and the physical system can begin behaving differently immediately.

That is the core idea behind SENSE:

Start with the idea, not the embedded code.

What's Next

The next direction I want to explore is automatic hardware discovery.

Instead of manually wiring and registering devices, I want users to be able to plug in a module and have SENSE automatically understand:

  • what the device is
  • what data it produces
  • what actions it supports
  • what electrical constraints it has

Longer term, I am interested in building a software-defined breadboard where physical components can be inserted into a programmable switching matrix.

A future workflow could look like:

insert components
        ↓
describe desired behavior
        ↓
SENSE generates a circuit netlist
        ↓
electrical safety validation
        ↓
board electronically configures wiring
        ↓
SenseScript defines behavior

The broader goal is to make physical prototyping start with:

"Here are the components I have. Here is what I want them to do."

instead of:

"Which GPIO pin does this go into?"

SENSE started as a hackathon experiment, but it made me interested in a much larger question:

What would robotics look like if experimenting with the physical world were as fast as experimenting with software?

Built With

  • Python
  • FastAPI
  • Gemini API
  • Raspberry Pi 5
  • ESP32
  • Luxonis OAK-1
  • DepthAI
  • YOLOv5n
  • OpenCV
  • JavaScript
  • HTML
  • CSS
  • Arduino / C++
  • PySerial
  • WebSockets
  • Custom AST validator
  • Custom SenseScript interpreter
  • GoDaddy Registry domain

Project Website

🌐 buildwithsense.design

Built With

Share this project:

Updates