The Musical Touch: Exploring Vibrotactile Augmentation of Music for Cochlear Implant Users

Music listening is a quintessential aspect of human life; it has been a constant companion of humanity throughout the ages, for as long as we can find evidence of people. Nowadays, the scientific community agrees that music has numerous benefits for the individual well-being, mental and even physical health, by encouraging social relationships, emotional healing, memory recollection, as well as it's intrinsic value --- music is good. A big obstacle in accessing music is hearing impairment that many individuals must live with. In the case of restoring severe hearing loss, the solution is usually found in a cochlear implant - a neuroprosthetic device that restores hearing by stimulating the inner ear directly. While these implants show fantastic results for speech related activities, indicated by the fact that most cochlear implant users live their daily life as normal hearing individuals, the music perception is severely degraded. As this segment of the population will only increase in the next 25 years, it is imperative to research new and innovative solutions that will work in cooperation with the inevitable advancement in hearing assistive devices.

One such opportunity can be found in using multiple sensory channels while ensuring the rules for multisensory integration are upheld. This idea is as old as the cochlear implant itself, as the creator of the first commercially available cochlear implant designed a tactile device called the "Tickle Talker", that was supposed to aid in the perception of sound by the hearing impaired user. This is the legacy this project continues, as advancements in the tactile technology afford designing significantly better devices, the research needs to explore and propose the best usage of those novel tools.

This dissertation is an interdisciplinary project conducted in collaboration with two institutions that have been immensely supportive of the work - The Royal Danish Academy of Music as well as the Copenhagen Center for Hearing and Balance from Rigshospitalet. Through an applied research project, we explored the possibilities of using vibrotactile stimulation designed around the needs of cochlear implant users in order to improve their music hearing performance and experience.

The primary contribution of this doctoral research is in the eight papers presented in part II, as well as the discussion presented in part I. The first paper in this series presents a scoping review of vibrotactile devices applicable to music, underscoring recurring themes and gaps within existing literature, and emphasizing the lack of standardization in this field. The subsequent two papers delve into the study of musical perceptual features through the use of vibrotactile displays, while also addressing the constraints inherent in single-actuator devices. The remaining five articles explore different facets of music listening experiences for cochlear implant users in social settings, particularly in concert environments. Among these, some studies are dedicated to the development of tactile displays specifically designed for concert use, which I refer to as 'concert furniture' while others focus on assessing and training the auditory performance of CI users in live music scenarios. These studies collectively aim to provide a deeper understanding of how cochlear implant users interact with and perceive music in concert scenarios and how vibrotactile technology can enhance these experiences.