The Development of Intersensory Perception: Comparative Perspectives.

Most sensory experiences in our daily environment are multimodal in nature: Visible objects often come with a distinctive noise or smell; voice sounds are generated by someone with particular visual features. According to neurophysiology, stimuli of different sensory modalities evoke neural activity in vastly different parts of the brain. So how then are these scattered activity patterns integrated into a common percept of the object out there? The answer given in most textbooks is that they are integrated in some "higher areas of association cortex," where information from different modalities converges onto single neurons with multimodal properties. Unfortunately, neurobiological research in the last 25 years has led to a steady shrinkage of brain areas designated as multimodal association areas, whereas the number of (seemingly) unimodal visual, auditory, and somatosensory areas is on the rise. Some authors have even suggested doing away with the term "association cortex" altogether.

An alternative view of multisensory integration is that synchronous activity across separate, even remotely situated, unimodal areas of the brain signals the simultaneous presence of features in different modalities. This would be equivalent to the process of "feature binding" proposed to occur within the visual system (Singer, 1994). Associative learning could lead to very specific strengthening of synaptic connections between such synchronously activated areas, and objects are then encoded by the synaptic weight between these connections. The cell assemblies proposed by Hebb (1949) for the encoding of objects would thus extend over different modalities. Recall of the total object can be evoked by activation in just one of the modalities.

One way that may help to decide between these two alternative views is to take a developmental approach. According to the latter view above, such a concept was advocated by Helmholtz and Piaget, the sensory modalities should be largely separate at birth and associations among them would have to be learned. Initially unimodal representations would become progressively more multimodal during …