Dance of Despair

For our physical computing midterm project, Sue Roh and I worked together to create a marionette that mirrors the user’s movements. Much like the concepts explored in the scene of “Craig’s Dance of Despair and Disillusionment” from Being John Malkovich (1999), users are to control the marionette’s joints with their own bodily movements—this convergence of the puppeteer and the puppet is represented by the virtual image of the user’s body on screen and through the physical actions of the marionette.

Technical Overview

Using the input captured from the computer’s camera, we run a machine learning library called PoseNet, which allows real-time pose estimation in the browser. Then, using P5.js, we overlay some shapes on top of the video capture input to visualize the data we are working with. In the spirit of Halloween, we show an image of a skull on the user’s face! 👻 Once the user is standing still in upright position with all limbs (left/right arms and left/right legs) visible to the camera, the screen will show that the system has been “calibrated.” Now, based on the incoming pose data for the user, we are able to calculate the angles for our motors to move in. The software portion of our project runs as a Node.js application that sends the motors’ angle information to Arduino Nano via serial communication.

Given the angle data for each of the six Servo motors, the micro-controller parses and writes those values to the respective motors. We have six motors for six joints: left elbow, right elbow, left wrist, right wrist, left ankle, and right ankle. Each of the motors have a rod extending out from its horn and a string connecting the end of the rod to the corresponding joint. As a result, the movements of the Servo motors can be translated to vertical movements of the limbs. The fabrication portion of our project is based on the marionette that Sue had created; it hangs in an enclosure with motors mounted on the top panel.

Process

The rods, motor mounts, and top panel were all produced from wood with the laser cutter. The rods were attached to the motor horns using hot glue and zip ties. The motors were mounted to the top panel by bolts and nuts.