Historically, various strategies have been employed to systematize natural phenomena and translate extracted data into other forms of experience. Kandinsky used color and form in a highly theoretical way, associating tone with timbre (the sound’s character), hue with pitch, and saturation with the volume of sound. He claimed that “Music has a grammar, which, although modified from time to time, is of continual help and value as a kind of dictionary.” (Kandinsky 1914) Based on his own experiences as a synesthetician, in 1911 the Russian composer Alexander Scraibin (1872–1915) introduced a color classification system that he regarded as the “reunion of the arts separated over the course of time.” (Föllmer and Gerlach 2004) His ideas for a synthetic Gesamtkunstwerk (total work of art) was meant to immensely expand consciousness and lead to a state of ecstasy. Olivier Messiaen created a system of so called modes on which to base his music. Similar to the way in which certain functions were attached to the ecclesiastical keys within a liturgy, each of Messiaen’s modes are based on an individual scale and are assigned certain colors. His works existed as pure sound; the union of color and tone, of form and time constitute only as an idealogical basis.

What is learned from the quantity and variance of approaches to audio-visual translations is that media and technology serve to expand our understanding of ourselves. The future of digital media is built on experiments from the past and an understanding of human perception as the analog source for all development.

This software, created in Max/MSP/Jitter, extracts data from video feeds and generates sound based on the data extracted. There are a number of options for how this is done, based on the content of the video feed. The system of analysis is based on fundamental aspects of graphic design, including: luminance (tone), rate of change in frame (repetition), texture, color and contrast.

This video was analyzed by slicing the frame into 4 horizontal frames. The top slice contains the image of fence posts in the foreground and the luminance of this slice (because the posts are darker) is mapped to a drum hit. The other lower slices augment this basic rhythm with additional drum beats. Here is the basic Max/MSP/Jitter patch showing the slices:

One additional experiment with this software involves adding a line of text, one letter at a time. In the video below the ASCII code of each character of text is mapped to a quicktime instrument according to the tempo of the fundamental rhythm of the fence poles.