HACKEAR is an interactive audiovisual installation that turns physical gesture into projected image, movement, and sound. The project uses an RPLIDAR C1 sensor to detect a participant’s presence near a projection surface, then translates those live spatial gestures into a sequence of responsive TouchDesigner scenes. The result is not a screen to watch from a distance: it is a performance environment in which the visitor’s body becomes an active part of the visual system.

Created at Constellation Labs, Espronceda institute of art and culture.

Stage 1: Defining the interaction

The first development stage was about identifying the central experience. Rather than building a conventional interface with buttons, keyboards, or camera-based tracking, we wanted interaction to happen through proximity and gesture. A person should be able to approach the wall, move a hand through space, and immediately see a visual response.

This decision shaped the whole project. The physical installation needed to be legible without instructions, reliable in a gallery context, and expressive enough to support an audiovisual performance. We also wanted the system to feel less like a technical demonstration and more like a living visual instrument. The projection wall became both a display and an interaction surface: a place where visitors could draw, disturb, activate, and transform the projected world.

Early references included generative drawing, hand-drawn animation, reactive graphics, and the atmosphere of digital systems revealed through movement. These ideas led to a structure in which each visual scene has its own character, while the same sensor input provides continuity across the complete experience.

Stage 2: Preparing visual and sonic source material

Before connecting live interaction, we gathered and organized the visual materials that would become the project’s vocabulary. A hand-drawn, black-line environment was prepared as a modular visual asset set. The scene includes familiar domestic elements: a door, window, plant, lamp, chair, wind, and birds. Instead of treating the illustration as one flat image, every major element was separated into independent transparent layers.

This separation was essential. It allowed each object to be animated, composited, and triggered independently inside TouchDesigner. The door can open, the window can move, the plant can react, the lamp can glow, the chair can shift, and birds or wind can appear as responsive details. The assets were kept in a consistent 1280 × 720 canvas so that all layers could be combined without positional corrections.

Each animated element was exported as a sequence of PNG frames. This made it possible to retain the hand-made character of the original drawings while enabling precise digital control. TouchDesigner could load these sequences as image sources, play them continuously, hold on their final frame, or trigger them once in response to interaction.

In parallel, the project’s audio material was developed and organized in Ableton Live. Sound was treated as another layer of the performance system: something that could support the changing intensity of the scenes and, in future iterations, be directly connected to sensor events through control messages.

Stage 3: Building the LiDAR acquisition system

The next challenge was turning the physical sensor into dependable live data. The RPLIDAR C1 produces a rotating scan of its environment, returning distance measurements around a full circle. Unlike a camera, it does not provide a conventional image. It provides points in space, which meant the project needed a custom pipeline to interpret those points as meaningful gestures.

To keep the installation stable, the sensor acquisition and the visual system were split into two separate TouchDesigner processes. One project is responsible only for reading the LiDAR through its serial connection. The second project receives complete scans locally and handles visual interpretation and projection. This separation prevents the visual project from competing with the acquisition process for direct access to the sensor and makes troubleshooting much easier during installation.

The acquisition process sends full scan data through TouchDesigner’s native local communication system. The visual process receives these scans as arrays of 360 samples. Maintaining the full scan, rather than only isolated events, gives the visual engine enough information to filter noise, detect clusters, and track movement over time.

Stage 4: Translating points into interaction

Raw LiDAR points are not yet interaction. A wall, a chair, a visitor’s body, and a hand can all appear in the scan. The central technical task was therefore to identify the intentional gesture and map it into the projected canvas.

The visual engine filters sensor data by quality and range, then groups nearby returns into candidate points. This is important because a hand may only create a small number of LiDAR returns rather than a dense shape. The system uses grouping thresholds and a minimum number of points so that isolated noise is less likely to become a false interaction.

Detected points are converted into screen coordinates, then smoothed over time. Smoothing makes the projected response feel continuous rather than jittery, while still preserving the live character of the gesture. The system can show up to four visible tracks, allowing multiple points of interaction without overloading the visual scenes.

A simulation mode was also included. This lets us test the visual system without requiring a person in front of the sensor and makes it easier to develop scenes away from the physical installation. We also added clear handling for sensor disconnection and reconnection so that a temporary hardware interruption does not leave an unwanted visual trace on screen.

Stage 5: Calibration for the physical space

Calibration became one of the most important stages of development. The LiDAR measures physical space, while the projector displays an image in a different coordinate system. For the interaction to feel convincing, a touch or gesture near a projected location needs to activate the corresponding visual position.

We built a sensor-calibration workflow directly into the show interface. The system projects a set of nine target crosses. The operator clears the sensing area, then touches each target position with a consistent part of the hand. From these captured measurements, the project calculates a spatial transformation between the sensor’s raw coordinates and the projection canvas.

The calibration workflow includes validation with three additional target points. The system only enables saving after these checks are within the selected tolerance. It also saves timestamped calibration profiles, allowing the installation to retain and revisit a successful mapping. This is particularly useful in an exhibition environment, where the projector or sensor may need to be moved between sessions.

The calibration process taught us that software alone cannot solve a poor physical setup. The LiDAR scan plane must be parallel to and slightly in front of the wall so that a hand touching the wall crosses the sensing plane. The system supports background filtering, but careful mounting and alignment remain the foundation of reliable interaction.

Stage 6: Designing the five-scene performance structure

Once sensing and calibration were working, we developed the project as a five-scene timeline rather than a single uninterrupted visual. This gave HACKEAR a performative arc. Each scene can have its own visual logic, image-processing approach, and response to the incoming sensor points.

The timeline controls the active scene, its duration, the overall show progress, and transitions between scenes. Only one scene is active at a time. This choice was both aesthetic and practical: it creates clear visual changes for the audience while reducing unnecessary processing from inactive scenes.

The first scene preserves the project’s colour and feedback-based visual language. Later scenes explore drawing, circles, hand interaction, and a native water-like surface. One scene allows continuous strokes, turning gesture into a form of projected drawing. Another uses local circle behavior to make touch points visible as distinct events. The water scene uses a custom GPU simulation so that interaction creates disturbances across a responsive surface rather than simply moving a cursor.

The scenes were exported as individual TouchDesigner components. This modular structure allows each one to be edited independently while keeping the shared sensor input, output resolution, and show controls consistent.

Stage 7: Interaction refinement and visual testing

The final development stage focused on refinement. We tested detection behavior, visual responses, scene controls, timelines, and calibration persistence. The project includes verification steps for the live sensor connection, simulation mode, sensor reconnection, point display, scene behavior, calibration targets, and timeline navigation.

Performance was also measured during a live visual test. The projection system rendered at approximately 59 frames per second while receiving 63 distinct LiDAR scans over eight seconds, with 360 samples per scan and no reported receiver or callback errors. These tests established that the visual system could maintain real-time responsiveness while processing live spatial data.

A show interface was added to bring essential controls into one place: scene timing, playback, calibration, visual status, and sensor telemetry. This interface turns a complex collection of components into a manageable installation tool for setup, rehearsal, and performance.

HACKEAR remains open to further expansion. Future versions can deepen the relationship between gesture and sound, add more scene-specific interaction rules, and explore more complex multi-participant behavior. Its current form is a working foundation: a hand-drawn audiovisual world activated by the movement of bodies in physical space.on

Built With

Share this project:

Updates