The "Tüb" (2010) was an instrument that I co-created with Björn Erlach and Michael Wilson for our "HCI Theory and Practice" class at Stanford in Fall 2010. The goal of the instrument was to sonify water waves in a tub. Thus, it combined the fun of bathtime with the experience of making music.
We accomplished the sonification process as follows: circular scans were taken of the images coming from the webcam, and the brightness values of the pixels were mapped onto a wave table, creating the shimmering sound that you hear. Since waves on the surface of the water created shadows on the tub, the sonic result was truly a sonification of the waves made in the water.
We also wanted to include in some way a collection of colorful plastic tubes that we found in the MAX-lab at CCRMA (where we built the instrument), which looked fun and playful and had nice acoustic properties. Ultimately, we incorporated them by triggering pre-sampled sounds of these tubes being struck, with the density and volume of the sample playback being determined by the turbulence in the tub at a given time. They also served a second purpose as tools with which to interact with the water.
The paper that we wrote, explaining the whole process in greater depth, can be found here: