Abstract

Does it make sense to sonify information that refers to an already audible phenomenon such as prosodic data? In order to be useful, a sonification of prosody should contribute to the comprehension of paralinguistic features that may not otherwise seize the attention of the listener.

Within this context, the paper illustrates a modular and flexible framework for the reduction and processing of prosodic data to be used for enhancing the perception of speaker’s intention, attitude and emotions. The model uses speech audio as input and provides MIDI and MusicXML data as output allowing samplers and notation software to auralize and display the information. The described architecture has been subjectively tested by the author over a multi-year period in composing for solo instruments, ensembles and orchestra. Two outcomes of the research are discussed: the advantages of an adaptive strategy for data reduction, and the auditory display of the deep pitch and temporal structures underlying prosodic processing.

Transcription steps in Background checks for pre-recorded voice, video and orchestra (2019)