{ "items": [ "\n\n
Currently, one of the most controversial topics in the study of multisensory integration in humans (and in its implementation in the development of new technologies for human communication systems) concerns the question of whether or not attention is needed during (or can modulate) the integration of sensory signals that are presented in different sensory modalities. Here, we review the evidence on this question, focusing specifically on the integration of auditory and visual information during the perception of speech. Contrary to the mainstream view that has been prevalent for the last 30 years or so, recent studies have now started to reveal that attentional resources are, in fact, recruited during audiovisual multisensory integration, at least under certain conditions. Finally, considering all of the available evidence, we discuss the extent to which audiovisual speech perception should be considered to represent a 'special' case of audiovisual, and more generally, of multisensory integration. \u00a9 2009 Elsevier B.V. All rights reserved.
\n \n\n \n \nPrevious research has provided inconsistent results regarding the spatial modulation of auditory-somatosensory interactions. The present study reports three experiments designed to investigate the nature of these interactions in the space close to the head. Human participants made speeded detection responses to unimodal auditory, somatosensory, or simultaneous auditory-somatosensory stimuli. In Experiment 1, electrocutaneous stimuli were presented to either earlobe, while auditory stimuli were presented from the same versus opposite sides, and from one of two distances (20 vs. 70 cm) from the participant's head. The results demonstrated a spatial modulation of auditory-somatosensory interactions when auditory stimuli were presented from close to the head. In Experiment 2, electrocutaneous stimuli were delivered to the hands, which were placed either close to or far from the head, while the auditory stimuli were again presented at one of two distances. The results revealed that the spatial modulation observed in Experiment 1 was specific to the particular body part stimulated (head) rather than to the region of space (i.e. around the head) where the stimuli were presented. The results of Experiment 3 demonstrate that sounds that contain high-frequency components are particularly effective in eliciting this auditory-somatosensory spatial effect. Taken together, these findings help to resolve inconsistencies in the previous literature and suggest that auditory-somatosensory multisensory integration is modulated by the stimulated body surface and acoustic spectra of the stimuli presented.
\n \n\n \n \nSeeing one's own body (either directly or indirectly) can influence visuotactile crossmodal interactions. Previously, it has been shown that even viewing a simple line drawing of a hand can also modulate such crossmodal interactions, as if viewing the picture of a hand somehow primes the representation of one's own hand. However, factors other than the sight of a symbolic picture of a hand may have modulated the crossmodal interactions reported in previous research. In the present study, we examined the crossmodal modulatory effects of viewing five different visual images (photograph of a hand, line drawing of a hand, line drawing of a car, an U-shape, and an ellipse) on tactile performance. Participants made speeded discrimination responses regarding the location of brief vibrotactile targets presented to either the tip or base of their left index finger, while trying to ignore visual distractors presented to either the left or right of central fixation. We compared the visuotactile congruency effects elicited when the five different visual images were presented superimposed over the visual distractors. Participants' tactile discrimination performance was modulated to a significantly greater extent by viewing the photograph of a hand than when viewing the outline drawing of a hand. No such crossmodal congruency effects were reported in any of the other conditions. These results therefore suggest that visuotactile interactions are specifically modulated by the image of the hand rather than just by any simple orientation cues that may be provided by the image of a hand.
\n \n\n \n \nThe 'body schema' has traditionally been defined as a passively updated, proprioceptive representation of the body. However, recent work has suggested that body representations are more complex and flexible than previously thought. They may integrate current perceptual information from all sensory modalities, and can be extended to incorporate indirect representations of the body and functional portions of tools. In the present study, we investigate the source of a facilitatory effect of viewing the body on speeded visual discrimination reaction times. Participants responded to identical visual stimuli that varied only in their context: being presented on the participant's own body, on the experimenter's body, or in a neutral context. The stimuli were filmed and viewed in real-time on a projector screen. Careful controls for attention, biological saliency, and attribution confirmed that the facilitatory effect depends critically on participants attributing the context to a real body. An intermediate effect was observed when the stimuli were presented on another person's body, suggesting that the effect of viewing one's own body might represent a conjunction of an interpersonal body effect and an egocentric effect.
\n \n\n \n \nVatakis, A. and Spence, C. (in press) [Crossmodal binding: Evaluating the 'unity assumption' using audiovisual speech stimuli. Perception &Psychophysics] recently demonstrated that when two briefly presented speech signals (one auditory and the other visual) refer to the same audiovisual speech event, people find it harder to judge their temporal order than when they refer to different speech events. Vatakis and Spence argued that the 'unity assumption' facilitated crossmodal binding on the former (matching) trials by means of a process of temporal ventriloquism. In the present study, we investigated whether the 'unity assumption' would also affect the binding of non-speech stimuli (video clips of object action or musical notes). The auditory and visual stimuli were presented at a range of stimulus onset asynchronies (SOAs) using the method of constant stimuli. Participants made unspeeded temporal order judgments (TOJs) regarding which modality stream had been presented first. The auditory and visual musical and object action stimuli were either matched (e.g., the sight of a note being played on a piano together with the corresponding sound) or else mismatched (e.g., the sight of a note being played on a piano together with the sound of a guitar string being plucked). However, in contrast to the results of Vatakis and Spence's recent speech study, no significant difference in the accuracy of temporal discrimination performance for the matched versus mismatched video clips was observed. Reasons for this discrepancy are discussed.
\n \n\n \n \nWe examined whether or not increasing visual perceptual load or visual working memory (WM) load would affect the exogenous orienting of visuo-spatial attention, in order to assess whether or not exogenous orienting is genuinely automatic. In Experiment 1, we manipulated visual perceptual load by means of a central morphing shape that in some trials morphed into a particular target shape (a rectangle) that participants had to detect. In Experiment 2, the possibility that the presentation of any changing stimulus at fixation would eliminate exogenous orienting was ruled out, by presenting two alternating letters at fixation. In Experiment 3, we manipulated visual WM load by means of arrays consisting of three (low-load) or five (high-load) randomly located coloured squares. The participants had to remember these items in order to judge whether a cued square had been presented in the same or different colour at the end of each trial. In all the experiments, exogenous visuo-spatial attentional orienting was measured by means of an orthogonal spatial cuing task, in which the participants had to discriminate the elevation (up vs. down) of a visual target previously cued by a spatially nonpredictive visual cue. The results showed that increasing the perceptual load of the task eliminated the exogenous orienting of visuo-spatial attention. By contrast, increasing the WM load had no effect on spatial orienting. These results are discussed in terms of the light that they shed on claims regarding the automaticity of visuo-spatial exogenous orienting.
\n \n\n \n \nThe speeding-up of neural processing associated with attended events (i.e., the prior-entry effect) has long been proposed as a viable mechanism by which attention can prioritize our perception and action. In the brain, this has been thought to be regulated through a sensory gating mechanism, increasing the amplitudes of early evoked potentials while leaving their latencies unaffected. However, the majority of previous research has emphasized speeded responding and has failed to emphasize fine temporal discrimination, thereby potentially lacking the sensitivity to reveal putative modulations in the timing of neural processing. In the present study, we used a cross-modal temporal order judgment task while shifting attention between the visual and tactile modalities to investigate the mechanisms underlying selective attention electrophysiologically. Our results indicate that attention can indeed speed up neural processes during visual perception, thereby providing the first electrophysiological support for the existence of prior entry.
\n \n\n \n \nIn order to determine precisely the location of a tactile stimulus presented to the hand it is necessary to know not only which part of the body has been stimulated, but also where that part of the body lies in space. This involves the multisensory integration of visual, tactile, proprioceptive, and even auditory cues regarding limb position. In recent years, researchers have become increasingly interested in the question of how these various sensory cues are weighted and integrated in order to enable people to localize tactile stimuli, as well as to give rise to the 'felt' position of our limbs, and ultimately the multisensory representation of 3-D peripersonal space. We highlight recent research on this topic using the crossmodal congruency task, in which participants make speeded elevation discrimination responses to vibrotactile targets presented to the thumb or index finger, while simultaneously trying to ignore irrelevant visual distractors presented from either the same (i.e., congruent) or a different (i.e., incongruent) elevation. Crossmodal congruency effects (calculated as performance on incongruent-congruent trials) are greatest when visual and vibrotactile stimuli are presented from the same azimuthal location, thus providing an index of common position across different sensory modalities. The crossmodal congruency task has been used to investigate a number of questions related to the representation of space in both normal participants and brain-damaged patients. In this review, we detail the major findings from this research, and highlight areas of convergence with other cognitive neuroscience disciplines.
\n \n\n \n \nUsing functional magnetic resonance imaging (fMRI) in humans, we identified regions of cortex involved in the encoding of limb position. Tactile stimulation of the right hand, across the body midline, activated the right parietal cortex when the eyes were closed; activation shifted to a left parietofrontal network when the eyes were open. These data reveal important similarities between human and non-human primates in the network of brain areas involved in the multisensory representation of limb position.
\n \n\n \n \nWe studied the attentional blink (AB) and the repetition blindness (RB) effects using an audiovisual presentation procedure designed to overcome several potential methodological confounds in previous cross-modal research. In Experiment 1, two target digits were embedded amongst letter distractors in two concurrent streams (one visual and the other auditory) presented from the same spatial location. Targets appeared in either modality unpredictably at different temporal lags, and the participants' task was to recall the digits at the end of the trial. We evaluated both AB and RB for pairs of targets presented in either the same or different modalities. Under these conditions both AB and RB were observed in vision, AB but not RB was observed in audition, and there was no evidence o
\n \n\n \n \nRecent neurophysiological research in the monkey has revealed bimodal neuronal cells with both tactile receptive fields on the hand and visual receptive fields that follow the hands as they move, suggesting the existence of a bimodal map of visuotactile space. Using a cross-modal congruency task, we examined the representation of visuotactile space in normal people and in a split-brain patient (J. W.) as the right arm assumed different postures. The results showed that the congruency effects from distracting lights followed the hand around in space in normal people, but failed to do so in the split-brain patient when the hand crossed the midline. This suggests that cross-cortical connections are required to remap visual space to the current hand position when the hand crosses the midline.
\n \n\n \n \nTwo experiments examined any inhibition-of-return (IOR) effects from auditory cues and from preceding auditory targets upon reaction times (RTs) for detecting subsequent auditory targets. Auditory RT was delayed if the preceding auditory cue was on the same side as the target, but was unaffected by the location of the auditory target from the preceding trial, suggesting that response inhibition for the cue may have produced its effects. By contrast, visual detection RT was inhibited by the ipsilateral presentation of a visual target on the preceding trial. In a third experiment, targets could be unpredictably auditory or visual, and no peripheral cues intervened. Both auditory and visual detection RTs were now delayed following an ipsilateral versus contralateral target in either modality on the preceding trial, even when eye position was monitored to ensure central fixation throughout. These data suggest that auditory target-target IOR arises only when target modality is unpredictable. They also provide the first unequivocal evidence for cross-modal IOR, since, unlike other recent studies (e.g., Reuter-Lorenz, Jha, & Rosenquist, 1996; Tassinari & Berlucchi, 1995; Tassinari & Campara, 1996), the present cross-modal effects cannot be explained in terms of response inhibition for the cue. The results are discussed in relation to neurophysiological studies and audiovisual links in saccade programming.
\n \n\n \n \nSubjects judged the elevation (up vs. down, regardless of laterality) of peripheral auditory or visual targets, following uninformative cues on either side with an intermediate elevation. Judgements were better for targets on either modality when preceded by an uninformative auditory cue on the side of the target. Experiment 2 ruled out nonattentional accounts for these spatial cuing effects. Experiment 3 found that visual cues affected elevation judgments for visual but not auditory targets. Experiment 4 confirmed that the effect on visual targets was attentional. In Experiment 5, visual cues produced spatial cuing when targets were always auditory, but saccades toward the cue may have been responsible. No such visual-to-auditory cuing effects were found in Experiment 6 when saccades were prevented, though they were present when eye movements were not monitored. These results suggest a one-way cross-modal dependence in exogenous covert orienting whereby audition influences vision, but not vice versa. Possible reasons for this asymmetry are discussed in terms of the representation of space within the brain.
\n \n\n \n \nThe effects of tool-use on the brain's representation of the body and of the space surrounding the body ('peripersonal space') has recently been studied within a number of disciplines in cognitive neuroscience, and is also of great interest to philosophers and behavioural ecologists. To date, most experimental findings suggest that tool-use extends the boundary of peripersonal space-visual stimuli presented at the tips of tools interact more with simultaneous tactile stimuli presented at the hands than visual stimuli presented at the same distance, but not associated with the tools. We studied the proposed extension of peripersonal space by tool-use by measuring the effects of three different tool-use tasks on the integration of visual and tactile stimuli at three distances from participants' hands along two hand-held tools. When the tool-use task required using the shafts or the tips of the tools, visuotactile interactions were stronger at the tips of the tools than in the middle of the shaft. When the handles of the tools were used, however, visuotactile interactions were strongest near the hands and decreased with distance along the tools. These results suggest that tools do not simply 'extend' peripersonal space, but that just the tips of tools actively manipulated in extrapersonal space are incorporated into the brain's visuotactile representations of the body and of peripersonal space.
\n \n\n \n \nWe examined the visual capture of perceived hand position in forty-five 5- to 7-year-olds and in fifteen young adults, using a mirror illusion task. In this task, participants see their left hand on both the left and right (by virtue of a mirror placed at the midline facing the left arm, and obscuring the right). The accuracy of participants' reaching was measured when proprioceptive and visual cues to the location of the right arm were put into conflict (by placing the arms at different distances from the mirror), and also when only proprioceptive information was available (i.e., when the mirror was covered). Children in all age-groups (and adults) made reaching errors in the mirror condition in accordance with the visually-specified illusory starting position of their hand indicating a visual capture of perceived hand position. Data analysis indicated that visual capture increased substantially up until 6 years of age. These findings are interpreted with respect to the development of the visual guidance of action in early childhood.
\n \n\n \n \nVision tends to dominate over touch in the majority of experimental situations, particularly when visual information is presented on, or near to, the body. We combined two visual dominance paradigms in order to investigate crossmodal interactions between vision and touch for stimuli on versus off the body: 1) The Colavita visual dominance effect, which has recently been extended to vision and touch, and 2) The rubber hand illusion, which has often been used to probe visuotactile interactions. Specifically, we investigated whether moving a visual stimulus off the participant's body would affect visual dominance, and how this dominance would be mediated by the presence/absence of a rubber hand (given the rubber hand illusion provides a way of extending the representation of one's own body in space). Participants made speeded detection/discrimination responses to a random sequence of visual-only, tactile-only, and visuotactile targets. While participants responded near-perfectly on the unimodal target trials, their performance on the visuotactile target trials was deleteriously affected by the simultaneous presentation of a visual stimulus on (as opposed to away from) their body. In particular, when the visual stimulus was presented to their fingertip, participants failed to respond to far more of the tactile than visual stimuli on bimodal trials. The magnitude of this visual dominance effect decreased significantly when the visual stimulus was moved off the body. When a rubber hand was placed at the off-body location, a similar (albeit reduced) visual dominance effect was observed in both positions. These results therefore suggest that visuotactile interactions are strongest when visual stimuli are presented on a body (no matter whom that body, or body-part, belongs to).
\n \n\n \n \nA great deal is now known about the effects of spatial attention within individual sensory modalities, especially for vision and audition. However, there has been little previous study of possible cross-modal links in attention. Here, we review recent findings from our own experiments on this topic, which reveal extensive spatial links between the modalities. An irrelevant but salient event presented within touch, audition, or vision, can attract covert spatial attention in the other modalities (with the one exception that visual events do not attract auditory attention when saccades are prevented). By shifting receptors in one modality relative to another, the spatial coordinates of these cross-modal interactions can be examined. For instance, when a hand is placed in a new position, stimulation of it now draws visual attention to a correspondingly different location, although some aspects of attention do not spatially remap in this way. Cross-modal links are also evident in voluntary shifts of attention. When a person strongly expects a target in one modality (e.g. audition) to appear in a particular location, their judgements improve at that location not only for the expected modality but also for other modalities (e.g. vision), even if events in the latter modality are somewhat more likely elsewhere. Finally, some of our experiments suggest that information from different sensory modalities may be integrated preattentively, to produce the multimodal internal spatial representations in which attention can be directed. Such preattentive cross-modal integration can, in some cases, produce helpful illusions that increase the efficiency of selective attention in complex scenes.
\n \n\n \n \nAdopting an unusual posture can sometimes give rise to paradoxical experiences. For example, the subjective ordering of successive unseen tactile stimuli delivered to the two arms can be affected when people cross them. A growing body of evidence now highlights the role played by the parietal cortex in spatio-temporal information processing when sensory stimuli are delivered to the body or when actions are executed; however, little is known about the neural basis of such paradoxical feelings resulting from such unusual limb positions. Here, we demonstrate increased fMRI activation in the left posterior parietal cortex when human participants adopted a crossed hands posture with their eyes closed. Furthermore, by assessing tactile temporal order judgments (TOJs) in the same individuals, we observed a positive association between activity in this area and the degree of reversal in TOJs resulting from crossing arms. The strongest positive association was observed in the left intraparietal sulcus. This result implies that the left posterior parietal cortex may be critically involved in monitoring limb position and in spatio-temporal binding when serial events are delivered to the limbs.
\n \n\n \n \n