Dissertation
Action-Sound: Developing Methods and Tools to Study Music-related Body Movement
Abstract: Body movement is integral to both performance and perception of music, and this dissertation suggests that we also think about music as movement. Based on ideas of embodied music cognition, it is argued that ecological knowledge of action-sound couplings guide our experience of music, both in perception and performance. Then follows a taxonomy of music-related body movements, before various observation studies of perceiver’s music-movement correspondences are presented. Knowledge from the observation studies was used in the exploration of artificial action-sound relationships through the development of various prototype music controllers. The last part of the dissertation presents tools and methods that have been developed throughout the project, including the Musical Gestures Toolbox; techniques for creating motion history images and motiongrams of video material; and development of the Gesture Description Interchange Format (GDIF) for streaming and storing music-related movement data. These tools may be seen as an answer to many of the research questions posed in the dissertation, and have facilitated the analysis of music-related movement and creation of artificial action-sound relationships in the project.
Video recordings
Trial lecture, 30 January 2008:
Introduction of the the thesis during the disputation, 31 January 2008:
Audio and Video Examples
Please refer to the Ph.D. dissertation for descriptions of the files below.
Chapter 4
Video 4.1: Examples of ancillary movements seen in a performance of Beethoven's Tempest Sonata by pianist François-René Duchable (2003). The circular elbow movements can be seen as support movements for the circular hand movements. His head and torso movements can be seen as phrasing movements which help accentuate salient parts in the melody line.
Video 4.2: Examples of various types of music-related movement in the song We Work The Black Seam (Sting 1985). Sting shows several examples of performer-performer communicative movements, both in directing the others by turning and looking, as well as when walking over to Marsalis (tenor saxophone). All musicians show entrained (groove) movements.
Video 4.3: Examples of sound-facilitating and communicative movements in a solo by Rachelle Ferrel (The Manhattan Project, 1991), including air performance with the fingers, entrained movements in the right arm and the knees, and a phrasing movement when she bends forwards to create musical punctuation.
Video 4.4: Examples of Glenn Gould’s hand movements (Gould, 1974). These seem to be endogenous communicative movements like he is conducting himself.
Video 4.5: Examples of various types of communicative movements carried out by Nigel Kennedy (McFerrin and Kennedy, 2005), ranging from affect displays to theatrical movements.
Video 4.6: Examples of how Jimi Hendrix repeatedly performs a "swinging arm" movement with long sustained sounds (Hendrix, 1973).
Video 4.7: Examples of how music-related movements may be transferable between musicians (Clapton, 1999). Notice the similarities between the movements of Eric Clapton and Tim Carmon when soloing, here backwards bending movements with closed eyes occurring at salient points in the melody line.
Video 4.8: Examples of how music-related movements may be transferable between musicians (Clapton, 1999). Notice the similarities between the movements of Eric Clapton and Tim Carmon when soloing, here backwards bending movements with closed eyes occurring at salient points in the melody line.
Chapter 5
Video 5.1: A composite video of three subjects from the air piano observation study: novice (left), intermediate (middle), expert (right). The brightness, contrast and saturation of the video images have been increased to improve the quality of the motion images.
Video 5.2: The novice subject (5.2) was standing still and moving mainly her hands and arms, while the intermediate subject (5.3) moved back and forth with the whole body. This explains the large difference in the calculated QoM between these two subjects.
Video 5.3: The novice subject (5.2) was standing still and moving mainly her hands and arms, while the intermediate subject (5.3) moved back and forth with the whole body. This explains the large difference in the calculated QoM between these two subjects.
Video 5.4: An excerpt of the novice performing the upwards movement in the Scriabin excerpt (repeated three times). Although quite approximate, this sequence shows that the novice had a good understanding of the spatial and temporal characteristics of the sound-producing actions necessary to perform the sequence.
Video 5.5: Examples of the level of detail and expressivity found in the air performance by the expert subject. Here a preparatory lift is seen in the performance of the Chopin excerpt.
Video 5.6: Performing air piano on a table helped to structure the movements, and gave the performer tactile feedback. However, it restricted the overall movement compared to the other air performance recordings.
Video 5.7: Adding a spectrogram image of the audio below the video image helped navigate in the video files. However, it did not solve the problem of visualising the movement in the video.
Video 5.8: A composite video of three dance students performing free dance movements to music.
Video 5.9: The 9 subjects' tracings of the short sounds.
Video 5.10: The 9 subjects' tracings of the composite sounds.
Audio 5.1: Cymbal sound
Audio 5.2: Iterative sound
Audio 5.3: Three chirp sounds
Audio 5.4: Long decending sound
Chapter 6
Video 6.1: A quick overview of how a game controller can be used in MultiControl.
Video 6.2: Playing an acoustic music ball with a granular sound effect (the ringing sound is due to the poor recording equipment). From a performance at Spasibar, Oslo, August 2006.
Video 6.3: Playing an electroacoustic music ball with various types of delay sound effects. From the same performance as mentioned above.
Video 6.4: Playing an electroacoustic music ball with various types of delay sound effects. From the same performance as mentioned above.
Video 6.5: The Music Troll in the foyer at the Norwegian Academy of Music, January 2007. This short video was recorded at the end of the installation, and the fourth head was broken.
Chapter 7
Video 7.1: A recording of one of the dance students from our free dance observation study. This video has been used as the source material for many of the examples in Chapter 7 and 8.
Video 7.2: Excerpt from a performance of Whiteness I Remember at Blå, Oslo, December 2005. The dancer is controlling the grainy sounds through her movements.
Chapter 8
Video 8.1: Seven different types of motion history images all based on the original video presented in the top left corner.
Video 8.2: A quick tutorial of how to create motiongrams using MGT.
Video 8.3: A clarinettist performing the beginning of the Allegro appassionato from Clarinet Sonata Op.120, No.1 in F minor by Brahms (1894). Recorded for Workshop on Motion Capture for Music Performance at McGill University in October 2006.
Video 8.4: A percussionist performing an excerpt of J.S. Bach’s Concerto in A minor BWV 1041 on the xylophone (Bach, 1717). Recorded for the same workshop as mentioned above.
Video 8.5: Video of a SHR rat. The video is recorded from the upper right corner of the experimental cage. The two levers that the rats are supposed to push in the experiment can be seen on the left side, each having a light above them. The feeding mechanism is seen between the two levers.
Video 8.6: Video of a WKY ADD rat.
Video 8.7: Video of a WKY rat.