Update (10/15). After that early 2013 test, the system was rewritten to make it more efficient and to incorporate live action videos. The revised system made its debut in September 2015 with "Rocket's Red Glare: Things Exploding on YouTube," an audiovisual improvisation with clarinetist/sound artist Curt Miller. The video of that first performance is at: vimeo.com/139920521
First rough demo of PIGS (Percussive Image Gestural System.)
PIGS research aims to develop instruments for visual performance that use percussion gestures - not sounds - to perform live visuals. When we perform a musical instrument, the challenge is to handle both performer-controllable things - like our own gestures - and more chaotic ones - like the vibration of a string or cymbal. Digital instruments don't inherently present these physically-based challenges. That avoids some degree of torture for the performer, but sometimes hurts the performance. PIGS approaches performative visuals as a combination of gesture and algorithmic process, analogous to the way a musical tone can be thought of in terms of fundamental (like the striking of a cymbal) and harmonic (the cymbal's "sh" sound as it fades away.) The video content is of course also an important component.
Although PIGS draws on elements of musical performance, it is not a sound/image mapping system. It's premised on parallels in performance, not on sonic/visual ones. While PIGS is designed to be performed image-only, I've left the accompanying sound in this demo for illustration purposes. (BTW, I'm performing here on a tabletop MIDI drum kit I've been using for development purposes, so performance is a bit awkward and limited as compared to a performance setup.)
I've just begun developing PIGS, and it's in a very rough stage. For example, there's lots of latency, and there's plenty of rough stuff in the "decay" effects I've created for this video. There's also lots for me to work out about what the triggered videos should be like, how they should be framed, timed, etc. (In this video, I've used "abstract"videos for most of the drums, and fireworks for the lowest tom-tom as an example of a representational video. And also, because fireworks are just lots of fun with drums.)