|Home | About | Journals | Submit | Contact Us | Français|
Cochlear implantation is associated with poor music perception and enjoyment. Reducing music complexity has been shown to enhance music enjoyment in cochlear implant (CI) recipients. In this study, we assess the impact of harmonic series reduction on music enjoyment.
Prospective analysis of music enjoyment in normal-hearing (NH) individuals and CI recipients.
Single tertiary academic medical center.
NH adults (N=20) and CI users (N=8) rated the Happy Birthday song on three validated enjoyment modalities–musicality, pleasantness, and naturalness.
Subjective rating of music excerpts.
Participants listened to seven different instruments play the melody, each with five levels of harmonic reduction (Full|F3+F2+F1+F0|F2+F1+F0|F1+F0|F0). NH participants listened to the segments both with and without CI simulation. Linear mixed effect models (LME) and likelihood ratio tests were used to assess the impact of harmonic reduction on enjoyment.
NH listeners without simulation rated segments with the first four harmonics (F3+F2+F1+F0) most pleasant and natural (p<0.001|p=0.004). NH listeners with simulation rated the first harmonic alone (F0) most pleasant and natural (p<0.001|p=0.003). Their ratings demonstrated a positive linear relationship between harmonic reduction and both pleasantness (slope estimate=0.030|SE=0.004|p<0.001|LME) and naturalness (slope estimate=0.012|SE=0.003|p=0.003|LME). CI recipients also found the first harmonic alone (F0) to be most pleasant (p=0.003), with a positive linear relationship between harmonic reduction and pleasantness (slope estimate=0.029|SE=0.008|p<0.001|LME).
Harmonic series reduction increases music enjoyment in CI and NH individuals with or without CI simulation. Therefore, minimization of the harmonics may be a useful strategy for enhancing musical enjoyment among both NH and CI listeners.
Cochlear implantation (CI) restores sense of sound and improves quality of life for individuals with severe to profound sensorineural deafness.1 The majority of post-lingually deafened implantees using current devices now achieve close to 80% on sentence recognition tests in quiet listening conditions.2 Despite dramatic improvements in speech discernment, musical enjoyment among these individuals remains poor.3–5 In a study of 40 CI users, 50% of participants reported enjoyment in listening to music before implantation, while only 13% described continued enjoyment of music after implantation.6 This study was corroborated by others that also found that CI users spend less time listening to music after losing their hearing or receiving an implant.7
Reduced appreciation of music among CI users is thought to result from poor perception of basic music elements.8 The University of Washington Clinical Assessment of Music Perception test examined melody recognition, timbre recognition, and pitch perception in 42 postlingually deafened adult CI users and 10 normal-hearing (NH) individuals; they found CI users performed significantly worse than NH individuals in all three categories.3
Exploring the reasons and possible solutions for the decline in musical perception and enjoyment among CI users has been the subject of numerous research efforts. Analyzing different CI hardware and sound-processing strategies has been one approach to improving music appreciation. For example, one study of six CI users compared music perception through MED-EL implant device on one side and Cochlear Nucleus on the other. The two different CI systems were tested sequentially with the other turned off, but no significant differences in the enjoyment of music was found between the two devices.9
Despite hardware improvements, CI recipients still have reduced numbers of auditory neurons responsible for transmitting the electrical stimulation of CI, as well as a limited number of CI frequency channels. As a result, complex music may present a challenge for CI users and may ultimately be the most important limiting factor in music perception and enjoyment. In their study of 15 CI users and 24 hearing aid users eligible for CI implantation, Looi et al. found that music played by several instruments simultaneously was rated by both groups to be less pleasant than music played by a single instrument.10 Kohlberg et al. similarly showed that reengineering of a piece to include fewer musical elements (vocals, piano, guitar, fiddle, snare) enhanced enjoyment in CI listeners compared to normal-hearing listeners.11 Simplifying more technical aspects of music also seems to positively affect enjoyment in CI listeners. A study by Certo et al. demonstrated that minimization of reverberation time increased music enjoyment under CI conditions.12 Consequently, approaches that decrease music complexity (e.g. music reengineering) may prove fruitful in enhancing music enjoyment.
A particular characteristic of music that may impact musical enjoyment is the harmonic series. When a musical note is played, it produces modes of vibration at varying frequencies. These include a baseline vibration called the fundamental frequency, as well as integer multiple vibrations above this fundamental frequency called the harmonic series. Therefore, even though the fundamental frequency dominates listeners’ aural perception, they are, in reality, simultaneously hearing several different complimentary frequencies.13 Harmonics are a significant component of musical timbre and thus help distinguish one instrument from another. These naturally occurring overtone frequencies may make a considerable contribution to the aural complexity of music and increase the number of signals that the CI must process and transmit. The harmonics therefore represent a unique target for reduction of musical complexity via music engineering. With the end goal of enhancing the CI listening experience, we investigated the consequences of harmonic series reduction on music enjoyment in both normal-hearing and CI listeners.
This prospective study was approved by the Columbia University institutional review board. Twenty normal-hearing subjects and eight cochlear implantees were enrolled in the study. NH participants underwent a basic audiologic evaluation, including evaluation of pure tone thresholds, speech discrimination, and otoscopic evaluation. Inclusion criteria for NH listeners included an age of 18 years or older, fluency in English, no history of hearing loss, and pure-tone audiometric thresholds less than or equal to 25 dB hearing loss in both ears tested at 500, 1,000, 2,000, 4,000, and 8,000 Hz. Inclusion criteria for cochlear implantees included an age of 18 years or older, fluency in English, and postlingual deafness. A questionnaire inquiring about musical listening habits, general health, and previous musical training was completed by both groups.
Ivory II Grand Piano (Synthogy, Malibu, California) and Garritan Personal Orchestra 4 (MakeMusic, Boulder, Colorado) virtual instrument libraries were used to create the Happy Birthday song with seven different instruments. The piece was uniformly played at a tempo of 130 bpm in the key of C major, with G as the first note (the song does not start on the root note). This familiar song with its recognizable perfect 4th and 5th intervals in the first few measures was selected to diminish the influence of individual music genre bias on music enjoyment. It was also chosen for its rhythmic simplicity and its potential to be played by all seven instruments at the same pitch frequency (no innate transposition of notes up or down an octave). The seven instruments were chosen to represent a wide variety of musical instrument families as follows: bassoon (double reed), B-flat clarinet (single reed), grand piano (keyboard), marimba (pitched percussion), trumpet (brass), bowed violin (bowed string), and plucked violin (plucked string).
Using Logic Pro 9 (Apple, Cupertino, California) audio engineering software, the harmonic series of each note in the Happy Birthday song was individually reduced using low pass static filters. Five different harmonic levels were created using this method – an unadulterated level with the complete harmonic series (Full), a level with the first four harmonics (F3+F2+F1+F0), a level with the first three harmonics (F2+F1+F0), a level with the first two harmonics (F1+F0), and a level with the fundamental alone (F0). The resulting battery of 20-second MIDI files was then converted into a waveform audio file format for future playback.
The samples were passed through a CI simulation algorithm provided by Advanced Bionics Corporation using MATLAB version 8.4 (MathWorks, Natick, Massachusetts). Musical samples were presented at 65 dB SPL in an S121 Audiometric Suite (Eckel Industries, Cambridge, Massachusetts) using Sony MDR-CD60 Digital Reference Dynamic Stereo Headphones (Sony Corporation, New York, New York). NH subjects listened to both the original and CI processed samples, whereas cochlear implantees were presented original samples alone. All sound files were presented in randomized fashion within each instrument group. Each subject heard the 35 sound files one time without practice runs. Both groups rated the segments using a visual analog scale executed in MATLAB version 8.4 on an HP ENVY 14 Notebook PC Microsoft Windows 7 Home Premium (Hewlett-Packard, Palo Alto, California). Subjects were instructed to rate each sample on a scale from 0 to 1 in each of the following categories: “natural and unnatural,” “pleasant and unpleasant,” and “sounds like music and does not sound like music,” with higher numeric scores correlating to higher levels of naturalness, pleasantness, and musicality. The listening portion of the study lasted approximately 30 minutes.
Statistical analysis was performed in R Statistical Software (R Foundation for Statistical Computing, Vienna, Austria). Linear mixed effect models (LME) were used to obtain mean ratings for each enjoyment metric at each harmonic level and develop slope estimates for relationships that appeared linear. Likelihood ratio tests comparing these full models to intercept only models were employed to identify statistically significant differences in mean ratings between harmonic levels.
The CI simulation software is based on the spectral selectivity hypothesis, which states that an individual’s ability to identify the frequencies of spectral peaks, called formants, is crucial to accurate speech recognition. Any impairment in spectral resolution, whether from auditory system damage or the use of a CI, may consequently impair a listener’s ability to differentiate speech. Fu and Nogaki showed that this speech recognition impairment observed in CI subjects may be reproduced in normal-hearing listeners by smearing vocoder output bands.14,15
The CI simulation used in our study utilizes a multi-band vocoder to simulate both CI processing and the spread of excitation that may occur in the electrically stimulated cochlea.16 The sound signal is filtered into 15 logarithmically spaced channels covering the range from 350 to 5,500 Hz. The envelope signal is computed from each channel and used to modulate a noise band with a center frequency that corresponds to the center frequency of the channel and that simulates appropriate spread in the cochlea.16 This processing system is based on a simplified version of the Advanced Bionics Fidelity 120 coding strategy, which has been shown to reduce vowel and consonant recognition in normal-hearing individuals to that of CI listeners.17
The mean age of the 20 normal-hearing participants was 25 years (range 23–31 yr; SD=2.5 yr). Nine were female and 11 were male. All participants had pure tone audiometric thresholds less than or equal to 25 dB hearing loss in both ears at all tested frequencies and normal otoscopic exams. Thirteen subjects reported some level of formal musical training (range 2–26 yr; mean=9.57 yr; SD=7.17 yr), and two subjects had obtained a bachelor’s degree or beyond in music. The most commonly studied instrument was piano (5 subjects; mean=10.6 yr; SD=8.82 yr). No subject reported a history of ear surgery.
The mean age of the cochlear implantees was 62 years (range 35–74; SD=14.1 yr). Six were female and two were male. Three subjects reported some level of formal music training before implantation – two of which studied piano and another which studied guitar (range 42–30 yr; mean=11.6 yr; SD=15.8 yr). One subject had obtained a bachelor’s degree or beyond in music. The average time since implantation was 2.3 years (range 1–6 yr; SD=1.79 yr). Information on implant type, manufacturer, sound processor, processing strategy, and CI user performance are included in Table 1.
Music samples rated by NH listeners without CI simulation received higher mean ratings across all harmonic levels and for every enjoyment metric than those rated by NH listeners without CI simulation or CI listeners (p<0.001, LME). The music samples most enjoyed among NH listeners without CI simulation (natural, pleasant, musical; 0.866, 0.896, 0.940) received substantially higher mean ratings than the samples most enjoyed among NH listeners with CI simulation (natural, pleasant, musical; 0.133, 0.204, 0.179) or CI listeners (natural, pleasant, musical; 0.385, 0.416, 0.521).
Likelihood ratio tests demonstrated that NH listeners without CI simulation rated samples with the first four harmonic levels (F0+F1+F2+F3) to be most natural (0.866, p=0.004) and pleasant (0.896, p<0.001). Conversely, the first harmonic level (F0 only) was rated to be least natural (0.804, p=0.004) and pleasant (0.831, p<0.001). This correlates to 7.23% and 7.29% relative reductions in naturalness and pleasantness from the highest to the lowest rated harmonic level. Although the same harmonic levels (F0+F1+F2+F3 and F0 only) were also rated most and least musical respectively, these results were not statistically significant. NH mean ratings for each harmonic level obtained with linear mixed effect models and compared using likelihood ratio tests are shown in Figure 1.
Likelihood ratio tests showed that CI simulation samples with maximal reduction of harmonics (F0 only) were rated most natural (0.133, p=0.003) and pleasant (0.204, p<0.001). It also demonstrated that CI simulation samples without any reduction of harmonics (Full) were rated least natural (0.076, p=0.003) and pleasant (0.073, p<0.001). This correlates to 42.4% and 64.3% relative reductions in naturalness and pleasantness from the highest to the lowest rated harmonic level – a reduction that is substantially greater than the analogous reduction in the samples without simulation. Although the same harmonic levels (Full and F0 only) were also rated least and most musical respectively, results were not statistically significant.
Samples with CI simulation also showed a positive linear relationship between harmonic level reduction and both naturalness (slope estimate=0.012, standard error=0.003, p=0.003, LME model) and pleasantness (slope estimate=0.030, standard error=0.004, p<0.001, LME model). CI simulation mean ratings for each harmonic level obtained with linear mixed effect models and compared using likelihood ratio tests are shown in Figure 2.
CI listeners rated samples with maximal reduction of harmonics (F0 only) as most pleasant (0.436, p=0.003) and samples without harmonic reduction (Full) as least pleasant (0.307, p=0.003.) This correlates to a 29.4% relative reduction in pleasantness from the highest to the lowest rated harmonic level. Although samples without harmonic reduction (Full) were also rated least natural and musical, results were not statistically significant.
Samples rated by CI listeners showed a positive linear relationship between harmonic level reduction and pleasantness (slope estimate=0.029, standard error=0.008, p<0.001, LME model). CI listener mean ratings for each harmonic level obtained with linear mixed effect models and compared using likelihood ratio tests are shown in Figure 3.
Irrespective of the harmonics, mean enjoyment ratings assigned by NH listeners with CI simulation differed based on instrument (p<0.05). Marimba pieces obtained highest mean ratings with respect to naturalness (0.215), pleasantness (0.235), and musicality (0.316). Bowed violin was rated least natural (0.057) and musical (0.532), while trumpet was rated least pleasant (0.085). Mean enjoyment ratings assigned by actual CI listeners also differed based on instrument (p<0.05). Piano pieces received highest mean ratings with respect to naturalness (0.479), pleasantness (0.491), and musicality (0.600). Plucked violin was rated least musical (0.401) and natural (0.272), while trumpet was rated least pleasant (0.299).
To our knowledge, the impact of harmonics on musical enjoyment has not been previously studied with CI simulation or CI listeners. We found that samples under normal- hearing conditions were rated most favorably with the first four harmonics (F3+F2+F1+F0). Under CI simulation conditions, samples with minimal harmonics (F0 only) consistently received the highest mean ratings in naturalness and pleasantness, whereas those with the maximal number of harmonics (Full) consistently received the lowest mean ratings in the same two modalities. The same was true for actual CI listeners with regard to pleasantness and harmonic level. The relative reductions in pleasantness, naturalness, and musicality between the least and the most favored harmonic levels was also much greater for CI simulation or actual CI samples than that for original samples. Thus, manipulation of harmonics may be important for increasing enjoyment in both normal and CI listeners.
A greater number of harmonics likely causes a substantial increase in the number of auditory signals that the cochlear implant must process and transmit. The increased number of signals may compound the spectral smearing already produced by the implant, leading to even greater distortion of peak frequency. Music enjoyment under CI conditions may therefore be impaired via a mechanism similar to that noted in speech recognition. This impairment may explain the increased enjoyment with fewer harmonics under CI conditions.
In the absence of cochlear implant simulation, song versions including the first four harmonics were preferred over the full instrumental sound. This curious finding may reflect the role of harmonics with respect to consonance and dissonance. According to Helmholtz’s postulation, dissonance arises when two pitches are spaced close enough together in frequency that their harmonics interfere and create a perception of “roughness” or “beating.”18 Consonance presumably occurs in the absence of “beating,” when harmonics are spaced sufficiently far apart so as to not interact.19 Studies have suggested this phenomenon is related to cochlear mechanics and the critical band hypothesis, which infers that overall dissonance of a music segment depends on the total interaction of frequency components within single auditory filters.20 This interaction may be minimized with fewer harmonics and explain why full instrumental sound may not be the most preferred harmonic level in normal-hearing listeners.
Focusing specifically on musicality ratings, original samples with the first four harmonics were rated highest while CI simulated samples with the fundamental (F0 only) were rated most favorably. Unlike what was observed for naturalness and pleasantness, these results were not statistically significant. This may be in part due to the familiarity of the Happy Birthday song. A study looking at the effect of song familiarity on pleasure during music listening determined that increases in self-reported familiarity significantly enhanced emotional response to a given piece of music.21 Moreover, an experiment conducted by Vongpaisal et al. revealed that CI users were able to identify original versions of familiar pop songs by millennial artists like Backstreet Boys and Britney Spears with a mean accuracy level exceeding 80%.22 Happy Birthday may likewise be such a universally familiar song that regardless of any harmonic modifications made to the piece, it still sounds like music. In addition, it is important to acknowledge that harmonics may comprise only one piece of the puzzle with respect to musicality. Other elements of music such as pitch, rhythm, dynamics, and texture may make large contributions to the musical quality of a piece.
Like musicality, there are likely other factors besides harmonics that influence the larger experience of music enjoyment. In particular, reductions in reverberation time and instrumental complexity have been shown to enhance enjoyment under CI conditions.11,12 Studies have also shown that CI users may enhance their listening experience by controlling their listening environment (with quiet rooms and quality sound equipment) and using visual cues (like watching performers).23,24 Other considerations, including rhythmic complexity, musical tone duration, and instrument type may also have a role. Careful study of these components will be necessary to discern the ideal factors necessary for maximum enjoyment benefit.
Part of this study involved querying normal-hearing listeners’ enjoyment of music through a CI simulation validated for speech perception. This approach helped avoid the inherent heterogeneity of a CI population, especially with respect to variations in age, musical experience, and the etiology, duration, and severity of hearing loss. However, groups in the past have noted the limits of using CI simulation as a surrogate for the actual CI listening experience. In particular, Wright and Uchanski showed that music enjoyment was lower for normal-hearing listeners using CI simulation than for CI listeners.25 While this trend was observed in our data, our findings in actual CI users did in fact parallel those obtained with CI simulation with respect to the impact of harmonic series reduction on enjoyment. In addition, our findings concur with a previously published study by Kohlberg et al., which employed the same CI simulation software and demonstrated that enjoyment ratings collected under simulation conditions were good first approximations of those obtained from actual CI listeners.11
The outcomes of this study may in the future be implemented in reengineering music to exclude higher harmonics. Music may be modified by subtractive techniques such as high pass and low pass filters which select for passage of sound energy above or below given cutoffs respectively. Other common filters like band pass and band reject filters may modulate the passage of spectral music content inside or outside given ranges respectively. These devices serve as the basis of equalization – an already commonplace sound editing technique throughout the music industry.26
In addition to music reengineering, alternative CI coding strategies geared toward harmonic reduction are on the horizon. Sound filter banks that resolve the first harmonic of a complex stimulus via two adjacent filters have already been shown to decrease pitch detection thresholds in users with the LAURA cochlear implant.27 Furthermore, Francart et al. recently demonstrated that another fundamental frequency modulation (F0mod) processing strategy improves pitch perception, pitch ranking, and familiar melody identification without interfering with speech recognition.28 These frequency manipulation studies suggest that reduction of higher harmonics might also be incorporated into future CI coding strategies to further improve music perception and enjoyment.
Music enjoyment, with or without CI conditions is affected by the harmonic series. The first four harmonics are preferred in normal-hearing listeners, whereas minimal harmonics are preferred under CI conditions. This reduction in the harmonic series, coupled with further investigation of other determinants of music enjoyment, may therefore provide a useful strategy for reengineering and composing music that is more enjoyable for CI recipients.
The authors thank Leonid Litvak, Ph.D., and Abhjilt Kulkami, Ph.D., of Advanced Bionics Corporation for providing the cochlear implant simulation software.
Sources of Funding: Dr. Anil Lalwani serves on the Medical Advisory Board of Advanced Bionics Corporation. This project was supported by a grant from the National Institute on Aging (1T35AG044303-01).
Conflicts of Interest: The authors have no other funding, financial relationships, or conflicts of interest to disclose.