1) The idea of having a Voice Assistant like Alexa or Siri that could extend into the physical world for tasks around the house

2) Having an interactive robot for fun and immersive language education, she understands and can reply in Spanish

3) The Idea of having a personal robot that in the future as the ability to be a useful caregiver/ physical assistant

What it does

Rexana is a voice-activated personal assistant robot that currently does the following tasks:

Autonomous navigation around the house, using distance and location data (landmarks) from the 2D lidar, her cardinal/compass bearing, the wheel encoders that track each wheel's distance and object detection Rexana knows where she is around the house and using voice commands to navigate around the house. Why? Paranoid you left the oven on after leaving the house... Rexana can be accessed on my phone via a browser to give text or voice command to "go to the oven" I can then view the oven via the web cam. Forgot to water the plants "Rexana water the pot plant in the lounge room"..feeling lazy... "Rexana bring me the Pringles".

Rexana has custom "hands" that can be switched for purpose fit tasks, watering can, magnets, grippers.

More demo here:

As Rexana is navigating around the house she takes note of detected objects detection (powered by PyTorch) I divided her viewpoint into left, straight and right. So I can ask her about objects she can see, she stores data about each object (compass bearing, X,Y coordinates based on wheel encoders and distance-based on lidar) this allows her to recall objects or go to recently seen objects via voice or text command.

Using the above data I can also practice Spanish in a fun, immersive way with her by asking her what she can see "Que puedes ver aqui", "A la izquierda hay libros" a la derecha el television".

Programmatic training her hand movements is very tedious, using 3D human body pose estimation I can train her much faster and intuitively to do tasks, she can copy my actions waving, gestures or picking up objects.

She also has a retro-inspired dashboard for monitoring, training and manually controlling.

How I built it

Build Blog:

Rexana is a physical robot made from scratch using 3D printed parts, several plastic plant pots, 8 servo engines, camera, wheel encoders,2 dc engines a 2D LIDAR a magnetometer and a Raspberry PI onboard computer.

I used Pytorch Detectron2, 3D human pose estimation and experimented with Pytorch Geometric.

I used for training the voice commands (originally I was experimenting with RASA and Azure LUIS, has by far the lowest barrier to entry the team have done a great job at keeping it simple to yet powerful)

Data is captured and formatted via onboard Raspberry Pi Computer then sent to an AWS server for realtime inference over web sockets which then returns the pose/detected/inferred results.

Challenges I ran into

Her arm dimensions and joints are very different from a human so pose estimation is not very accurate (version 2 will be bigger and more closely resemble human joint positions and dimensions

Powering the 8 arm engines + 2 dc motors and the onboard Rasberry Pi computer was an unexpected challenge (getting the correct voltage /amperage and decent battery life)

Accomplishments that I'm proud of

Working proof of concept!

What I learned

I burnt our 3 servo engines trying to get the arms working well so learned a lot about servo motor torque and how to power them.

Autonomous, human-sized and genuinely useful robots are achievable, although some of the functionality is basic or rough I was able to complete a proof of concept and the lessons learned and existing groundwork will make the next version significantly better.

What's next for Rexana the Robot

Adding 3 x micro vacuum cleaners and mop extensions to so she can complete the "Vacuum Kitchen" / "Mop Kitchen" commands.

V2 bigger size, human dimension arms for better pose estimation, create docs, improve code and open-source.

Self annotation and improve automatic training by showing objects and giving names and locations.

More info and demos here:

Share this project: