Integrating Chinese Phonemes and Phonetic Symbols using Multimodal Self-Organizing Networks

We present a model of integration of auditory and visual information as in the human cortex. More specifically, we demonstrate a possible way in which the phonetic symbols and associated Mandarin Chinese phonemes pronounced by different speakers are mapped onto the model of cortical areas. Our model has been strongly influenced by recent fMRI studies on integration of letters and speech sounds in the human brain. The model is based on multimodal self-organizing networks (MuSoNs) which were introduced in our previous works and proved to be a convenient tool to describe and study mapping and integration of sensory information as in the cortex. Our model demonstrates robustness of the bimodal percepts and shows that humans hear a noisy phoneme better when they see the corresponding uncorrupted letter. We also show how phonemes pronounced by different speakers are mapped onto overlapping cortical areas.