The Microsoft Kinect for the Xbox 360, like a normal camera, gathers information about the color of everything it sees.
What makes it unlike a camera is that is also can calculate the distance between it and everything it sees.
Using Jean-Marc Pelletier's freenect extension to the programming language MaxMSP, I created this piece to explore the possibility of controlling sound with gesture. The video is an animation of my body in a virtual 3D space. I used an artificial intelligence algorithm called a self-organizing map to watch my movement and identify seven distinct poses. Each of these seven poses trigger a different note from the C major scale and a color with which to draw myself.
The self-organizing map identifies gestures without any instruction -- in the world of AI and machine learning, this is called an unsupervised learning task. Because no gestures are programmed in advance, I must move around and explore different positions in order to uncover different notes and colors.
For those interested in learning about self-organizing maps (SOMs), there is an article available here by Dr. Teuvo Kohonen, the inventor of the algorithm: scholarpedia.org/article/Kohonen_network
For those interested in the technical details of how I set up the SOM, I have an iPad sending OSC messages which allow me to adjust the learning rate and neighborhood radius of the SOM in realtime. At around 40 seconds into the video, I pull down the learning rate considerably and it is possible to observe the SOM switching very clearly between different gestures.