Audiovisual composition employing pitch-space visualisation based on the Lissajous mesh

Abstract

Melodic figures, when plotted on the visual space of a musical score, often come close to assuming rather distinctive visual configurations. Taking this phenomenon as an inspiration, this research aims to produce a set of audiovisual compositions under the guiding light of the metaphoric expression ‘the melody is a human face.’ The approach to realising this metaphor centres upon the employment of a pitch-space visualisation method underlined by the concept behind the Lissajous mesh. Audio tracks, along with their corresponding visualisation outputs obtained via a computational pitch-mapping engine, would become the source materials for audiovisual composition. These audiovisual compositions were developed in tandem with the iterative process of computational design and implementation of the visualisation approach. This paradigmatic visualisation strategy maps a musical phrase into a visual phrase, by imparting visual resolution using figurative forms, such as the human visage in particular. The musically derived visual phrase, as a unit of montage or cell, could in turn be combined and juxtaposed with other visual cells — be they live-action film segments or computer animation — to create a visual narrative. The first work, Au Sud du Sud, set the stage for creative exploration. The second composition, Concubine to My Farewell, was created as a visual montage of the cinematic scenes appropriated from the film Farewell My Concubine. Another composition, The Cutesification of Cries, went deeper into the exploration of the formal anthropomorphic potential of Lissajous visualisation, by making the graphic qualities of the visualisation gel more seamlessly with real-life video of human subjects, especially the close-up shots of the face. The series of creative experiments here revealed moderately convincing details indicating that more melodious pitch patterns do in general proffer a Lissajous visualisation mesh that is geometrically spaced in a way that better approximates the structure and proportion of the human face. It also does seem that some sort of primer in the form of an accompanying depiction of a visage superimposed over the Lissajous visualisation, could serve to persuade us to see the latent formal anthropomorphism of Lissajous figures. Without the foundation laid down by this visualisation method, the particular condition and mode of audiovisual poiesis — an experiential journey of doing, expressing itself and of becoming — seen here, would never have taken place. The same could be said about the idiosyncratic audiovisual thematic and temporal narratives that have coursed through each of these individual compositions. The audio tracks have dictated and structured the temporal pace, rhythmic juxtaposition and narrative flow of visual montage. There are also phases in the creative process where the narrative of the sampled video (such as that employed in Concubine to My Farewell), along with the graphic characteristics of the visualisation, would indirectly influence the direction of musical narrative.