Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Enhancing allocation of visual attention with emotional cues presented in two sensory modalities

Enhancing allocation of visual attention with emotional cues presented in two sensory modalities Background: Responses to a visual target stimulus in an exogenous spatial cueing paradigm are usually faster if cue and target occur in the same rather than in different locations (i.e., valid vs. invalid), although perceptual conditions for cue and target processing are otherwise equivalent. This cueing validity effect can be increased by adding emotional (task-unrelated) content to the cue. In contrast, adding a secondary non-emotional sensory modality to the cue (bimodal), has not consistently yielded increased cueing effects in previous studies. Here, we examined the interplay of bimodally presented cue content (i.e., emotional vs. neutral), by using combined visual-auditory cues. Specifically, the current ERP-study investigated whether bimodal presentation of fear-related content amplifies deployment of spatial attention to the cued location. Results: A behavioral cueing validity effect occurred selectively in trials in which both aspects of the cue (i.e., face and voice) were related to fear. Likewise, the posterior contra-ipsilateral P1-activity in valid trials was significantly larger when both cues were fear-related than in all other cue conditions. Although the P3a component appeared uniformly increased in invalidly cued trials, regardless of cue content, a positive LPC deflection, starting about 450 ms after target onset, was, again, maximal for the validity contrast in trials associated with bimodal presentation of fear-related cues. Conclusions: Simultaneous presentation of fear-related stimulus information in the visual and auditory modality appears to increase sustained visual attention (impairing disengagement of attention from the cued location) and to affect relatively late stages of target processing. Keywords: Spatial attention, Multisensory, Emotion, Sensory modality, Fear, EEG, ERP Background The most relevant experimental protocol for investigat - In daily life, emotional stimuli, such as a fear-evoking ing stimulus-elicited allocation of attention is referred to dog, often feature salient properties in more than one as the spatial cueing paradigm. In this protocol, a target sensory modality (e.g., sound of barking and sight of stimulus is presented either at a location indicated by a aggressive posture). What are the consequences of  such preceding cue (i.e., valid condition) or at a different loca - bimodally corresponding signals for the deployment of tion (i.e., invalid condition) which can be considered attention to the stimulus object they belong to? equivalent concerning other factors of perceptual rel- evance. In studies of exogenous spatial cueing, the cue is presented at one of the potential target locations (usu- *Correspondence: ulrike.zimmer@medicalschool-hamburg.de ally one of them located on the left side and one located Faculty of Human Sciences, Department of Psychology, MSH Medical School on the right side of a central fixation point). Studies of Hamburg, Hamburg, Germany Full list of author information is available at the end of the article exogenous spatial cueing reliably yielded faster responses © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. The Creative Commons Public Domain Dedication waiver (http:// creat iveco mmons. org/ publi cdoma in/ zero/1. 0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 2 of 14 in valid than in invalid conditions [32, 54]. This cue - sensory input) fear-related or anger-related cues (i.e., ing validity effect has been ascribed to the deployment faces or voices). These studies demonstrated larger valid - of attention to the cued location. Such findings are not ity differences on the behavioral level with emotionally confined to situations, in which cue and target belong to compared to neutral cues [7, 23]. Extant findings also the same sensory modality but have also been observed suggest that the direction of an attentional shift by an in studies, in which an unisensory cue and an unisen- emotional cue depends on its specific emotional con - sory target consisted of stimuli presented in two differ - tent. For example, while fearful facial expressions appear ent modalities (e.g., an auditory cue with a visual target), to attract attention toward their location, disgusted demonstrating crossmodal cueing of spatial attention [5, expressions were associated with shifts of attention away 10, 67]. In contrast to crossmodal presentations, studies from their location, resulting in a reversed validity effect of bimodal spatial cueing have been conducted to inves- (i.e., faster responding in invalid than in valid trials [44, tigate the combined effects of attentional cues presented 72]). Noteably, a similar pattern was found when using simultanesouly in two sensory modalities (e.g., a visual- sound cues of fearful and disgusting content [74]. These auditory cue). results point to a modality independence of fear-related Although some bimodal spatial cueing studies (audio- attraction of attention and disgust avoidance, suggesting visual cues preceding visual targets) have been conducted that the modulation of attentional deployment is evoked (e.g., [47, 63]), these studies  did not investigate shifts of by the semantic content rather than by superficial per - spatial attention evoked by emotion. In an ERP study ceptual features of an emotional stimulus. [63], laterally presented bursts of white noise and LED Corresponding ERP data for emotional spatial cueing flashes served as cues that could be presented alone or demonstrated that the fear-related attraction of attention together (on the same side). These unisensory or bimodal can result in larger validity differences of P1-activation cues preceded triangular shaped LED-flashes presented for targets presented after fearful or angry compared to at the corresponding (valid) or opposite (invalid) side, neutral facial or voice cues [8, 55, 56]. Further, the target- which had to be categorized as pointing up or down. related P3-component, a positive ERP peaking around There were no gains for bimodal versus unimodal cueing 300–400  ms after target onset over occipital-parietal presentations on target-processing neither behaviorally electrodes, has been observed to be inversely correlated nor in terms of any target-related ERP-effect. However, with the behavioral validity effects associated with the ERP-results indicated a clearly enhanced amplitude of two types of emotional cue content [21, 56, 73–75]. This the cue-elicited P1 over contralateral visual areas related inverse P3-modulation corresponds to a redirection of to the processing of bi- versus unimodal cues. According spatial attention to the target, when a target followed a to the authors, the discrepancy of target-related behavio- cue on the previously unattended side (cf. [46]). In sum- ral and cue-related ERP results could be explained by the mary, whereas spatial cueing effects tend to be enhanced fact that target discrimination follwing the bimodal cue by emotional cue content, redundant bimodal cueing was not demanding enough to require specific shifts of yielded a more complex pattern indicating ERP-effects spatial attention. Another study involving bimodal spa- such as a modulation of the P1-component which were tial cueing used bilateral presentation of visual cues (i.e., not always accompanied by corresponding behavioral the face of a cat located on one side, and the face of a effects (possibly depending on task demands as assumed dog located on the other side of the screen) and added by [63]). an animal sound, presented via a centrally located loud- Spatial cueing with peripherally presented cues is sen- speaker [47]. Follwing such a bimodal cue, a visual tar- sitive to the timing of stimulus presentation. Specifically, get (Gabor-patch) was unpredictively presented to 50% the validity effect tends to decrease and often reverse (i.e., on the side indicated by the cueing-picture matching the faster responses in invalid compared to valid conditions) cueing sound. The task consisted of easy and difficult ori - when the interval separating cue and target is extended, entation changes in the visual Gabor-patch targets. The a phenomenon referred to as Inhibiton of Return (IOR, authors found that responses to targets were faster to [54], see [37], for an overview). IOR is of particular rel- the side where the animal sound matched with the ani- evance for studies investigating cueing effects of auditory mal picture, but this was confined to trials featuring high features which cannot be identified instantly. In con - demands of target discrimination. This result prompted trast to a remarkably fast emotional recognition of visual the authors to suggest that bimodal enhancement of spa- facial expressions (17–100  ms; [72]) or identification of tial cueing depends on the amount of perceptual load. elementary auditory features, emotional voice recogni- On the other hand, several spatial cueing studies inves- tion requires temporal integration of stimulus properties tigated emotional shifts of spatial attention by using changing dynamically across an extended period of time unisensory or crossmodal (i.e., cue and target of different [64]. In practice, the duration of an auditory fear-related Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 3 of 14 voice needs to be considerably longer than 600  ms to going ERP-slow wave which has been found particularly ensure a high level of correct identification if the task sensitive to two types of stimulation. On the one hand, requires distinguishing it from other emotional or neu- the presentation of emotional stimuli of negative ver- tral voices [22, 53]. sus positive valence (e.g. [30, 36, 51, 59]) revealed an Despite of using a long stimulus duration of 1000  ms enhancement in LPC-ativity. On the other hand, stimu- for voice cues in a previous study [75], we observed typi- lus-conflict between neutral non-emotional stimuli, such cal cueing validity effects when visual targets followed as incongruent versus congruent color-word conjunc- neutral voice cues after a short interstimulus-interval tions in a Stroop task, also revealed an increased LPC for (ISI, i.e., time between cue offset and target onset). In the conflicting condition (e.g., [13, 18]). Comparing these contrast, validity effects after neutral voices reversed fol - components among our four cueing conditions allowed lowing a long ISI, possibly indicating IOR. These findings us to estimate both the effects of emotional content pre - suggest that (at least for auditory cues) a long ISI rather sented in a single modality and redundant “emotional than a long cue-target onset asynchrony may be crucial cueing” in both modalities (Fig. 1). for IOR to occur. Taken together, adding identifiable emotional content to an auditory cue requires a suffi - Results ciently long voice duration to enable content identifica - Behavioral results tion, which, however, does not itself evoke IOR-effects as Our main research interest was to investigate whether long as combined with short ISI (cf. [75]). audio-visual cues with bimodally fearful coloring would In spatial cueing paradigms, P1-activity as well as enhance attentional shifts toward their location com- N170-activity, time-locked to visually presented face pared to audio-visual cues with only a single modality cues, has often been increased when comparing emo- of fear-related display or a fully bimodal neutral display. tional facial expressions to neutral ones (P1: [15], N170: Participants were instructed to ignore any bimodal face- e.g., [11, 69], see [31] for a review). Importantly, the P1 voice cue and to attend only to the arrow targets. Due to increase associated with emotional cue content seemed problems with the presentation of the auditory cues in to be independent of the type of stimulus used as cues as the first participant, this first data set was excluded from a similar P1-enhancement was also found when compar- the further analyses. ing frightening versus neutral pictures of objects other The analysis of reaction times of trials associated with than faces (i.e., IAPS—pictures, [9, 38]). The N170 com - a correct response failed to yield a significant main effect ponent is a component typically observed in connection of VALIDITY (F(1, 24) = 0.013; p = 0.909; ηp = 0.00063). with processing face (-like) stimuli which originates from The interaction of EMOTIONAL CUE COMBI - the fusiform face area (FFA) [26]. The increase of N170 NATION and VALIDITY was significant (F(2.525, activity evoked by emotional content occurred indepen- 72) = 3.363; p = 0.024; ηp = 0.138). The follow-up dently of specific task demands (i.e., passive viewing vs. detection/discrimination) ([25, 60, 66, 71], see [31] for a review). So far, there seems to be no evidence for an addi- tional enhancement of the N170 evoked by emotional faces when accompanied by vocal utterances match- ing the displayed emotion. Conversely, angry vs. neu- tral vocal expressions, presented during face encoding, reduced P1 and N170 to matching facial expressions [43]. In the present study, we used a spatial cueing para- digm with bimodal face-voice cues that varied concern- ing their degree of emotional content; i.e., fearful in face and voice (double emotional cue), fearful voice/neutral face or fearful face/neutral voice (single emotional cue), or both neutral (neutral cue). The aim was to investigate whether double emotional face-voice cues attract atten- Fig. 1 Task paradigm. A schematic example of a valid stimulus tion to their location more strongly than single emotional sequence is shown. Bimodal face-voice cues consisted of either both cues or purely neutral cues, respectively. If true, double fearful, one fearful (paired with neutral) or both neutral contents. 50% emotional cues should elicit an enhanced validity effect of the targets followed each cue type on the same side (valid) and in behavior (reaction times, hit rates) as well as more pro- 50% on the opposite side (invalid). Participants were required to push nounced target-related ERP-components like P1, P3, and one of two buttons indicating the pointing direction (up/down) of the target triangle LPC (late positive component). The LPC is a positively Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 4 of 14 Bonferroni-corrected posthoc tests indicated that the participants responded significantly faster in the valid than the invalid condition when both face and voice were fear-related (t(24) = 3.208; p_bonf = 0.034), but there was no significant validity effect following any other emo - tional cue combination. In addition, the main effect of EMOTIONAL CUE COMBINATION was significant (F(2.083, 72) = 4.268; p = 0.019; ηp = 0.169). Bonferroni- corrected post-hoc tests demonstrated first that reaction times after the voice-fear/face-neutral cues were faster than after voice-neutral/face-fear cues (t(24) = 3.182; p_bonf = 0.014). Secondly, reaction times to targets after the voice-fear/face-neutral cues were also faster com- pared to targets after the both neutral face-voice combi- Fig. 3 Early lateralized target activity: P1: Left panel: ERP-traces time-locked to the target over occipital O1/2-electrodes, the nations (t(24) = 2.988; p_bonf = 0.024) (Fig. 2). contra-versus-ipsi-lateral P1 amplitudes indicated a significant Considering hit-rates, neither the interaction of the increase for all valid compared all invalid targets (see upper versus factors nor any single main effects were significant. lower part of the figure). In addition, for valid conditions (upper channel), the contra-ipsilateral P1 activity of targets after the ERP results double fear combination was significantly increased compared to targets after combinations of both neutral or of neutral-voice The main aim of the study was to test whether emotion with fearful face combination ( VN/FF). The comparison of the valid shifts spatial attention more strongly when presented in double fear combination with the valid fear-voice/neutral-face two instead of one modality. To test, bimodal face-voice combination ( VF/FN) only showed an increase by trend, but was not cues consisted either both of fearful expression, or only significant. Right Panel: Topographies indicate the corresponding one of a fearful expression, or both with a neutral expres- contra-versus-ipsi-lateral P1-differences separately for in/valid targets and its associated cue-type. Contra-versus-ipsi-lateral amplitude sion (control condition) (Fig. 2). differences are summarized over hemispheres Contra‑ versus ipsilateral target activity at P1 We analyzed the P1 amplitude as contra-ipsilateral activ- ity differences over occipital electrode positions. First of of VALIDITY (F(1,24) = 27.844; p < 0.001; ηp = 0.537) all, the 4 × 2–ANOVA EMOTIONAL CUE COMBINA- with larger P1-activity for valid compared to invalid TION (both fear, voice fear/face neutral, voice neutral/ cue combinations. The main effect of EMOTIONAL face fear, both neutral) and VALIDITY (invalid, valid) was CUE COMBINATION was not significant (F(2.928, calculated (see Fig. 3). It revealed a significant main effect 70.277) = 2.088; p = 0.109; ηp = 0.080). Importantly, Fig. 2 Behavioral results. Left panel: Reaction times. The interaction of cue-type with validity was significant, mainly due a significant validity effect with valid faster than invalid combinations in the both fearful condition, but there was a trend for a reversed validity effect in the neutral condition. The validity difference of the both fear condition was significantly larger than the inverted one in the both neutral condition. Right panel: Hit-rates: Despite not being significant, the hit-rates mirrored a similar interaction, with improved performance after both fearful cues for valid targets, but for invalid after neutral targets. both fear face/voice both fearful, VF/FN voice fearful with neutral face, VN/FF voice neutral with fearful face, both neutral face/voice both neutral) Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 5 of 14 the interaction of both factors was significant (F(3, 72) = 2.888; p = 0.038; ηp = 0.169). Due to further inves- tigation of the significant interaction, two follow-up ANOVA’s considered valid and invalid P1-activity respec- tively. Only the ANOVA for the valid conditions indicated a significant effect of EMOTIONAL CUE COMBINA - TION (F = (3,72) = 3.347; p = 0.040; ηp = 0.176; see Fig. 4). A Bonferroni-corrected posthoc test revealed that the contra-ipsilateral activity of validly cued targets after the both fear combination was significantly increased compared to targets after the both neutral face-voice combination (t(24) = 3.012; p_bonf = 0.020). A further Bonferroni-corrected posthoc-test showed in addition a significant increase of ipsilateral activity of validly cued targets when comparing the both fear combination to the neutral-voice with fearful face combination (VN/FF; t(24) = 3.206; p_bonf = 0.019). The corresponding post - hoc comparison of targets after the both fear combina- tion versus targets after the fear-voice with neutral-face (VF/FN) combination was not significant, despite of showing a trend (t(24) = 1.425; p_bonf = 0.083). Centralized target activity at P3a and P3b We analyzed the modulation of P3a amplitude at the cen- tral-parietal ROI (Cz/Fp1/Fp2). Neither the interaction of EMOTIONAL CUE COMBINATION and VALIDITY (F(2.391,72) = 1.970; p = 0.126; ηp = 0.076) nor the main effect of EMOTIONAL CUE COMBINATION (F(1.833, 72) = 2.043; p = 0.145; ηp = 0.078) were significant. However, there was a significant main effect of VALID - ITY (F(1,24) = 13.375; p < 0.001; ηp = 0.358; Figs. 4, 5). Amplitude values at the P3b revealed a significant interaction of EMOTIONAL CUE COMBINATION by VALIDITY (F(2.27, 65.05) = 2.67; p = 0.049; ηp = 3.262). Fig. 4 ERP-traces of late centralized target activity time-locked to Follow up Bonferroni-corrected posthoc tests indicated the target over the central-parietal ROI. P3a: A significant main effect that validity effects were only significant in the neutral of validity was revealed, indicating that P3a-activity was increased for the invalid versus valid condition independent of cueing type. condition (t(24) = 3.088; p_bonf = 0.045), but in none of P3b: The interaction of validity by cueing type was significant. the other face-voice combinations (Figs.  4, 5). None of Posthoc-tests indicated that validity differences were only significant the other effects reached statistical significance. for the double neutral ( VN/FN) condition with a reverse effect of higher P3b amplitudes for the valid instead of invalid condition. LPC: Again, the interaction of validity by cueing type was significant. Centralized target activity at LPC Posthoc-tests indicated that validity differences were only significant Additionally, we analyzed the modulation of LPC ampli- for the double fear ( VF/FF) condition an increased LPC-amplitude for tude at the central- parietal ROI (Cz/Fp1/Fp2). The the invalid versus valid condition interaction of EMOTIONAL CUE COMBINATION and VALIDITY was not significant (F(3, 72) = 1.103; p = 0.354; ηp = 0.048). However, there was a signifi - combinations were not significant. Further, there was a cant main effect of EMOTIONAL CUE COMBINA - significant main effect of VALIDITY (F(1, 24) = 4.233; TION (F(2.155, 47.421) = 4.743; p = 0.012; ηp = 0.177) p = 0.049; ηp = 0.161). Importantly, Bonferroni-cor- with Bonferroni-corrected post-hoc tests indicating rected posthoc-tests indicated only a significant valid - that the both fear combination revealed significantly ity difference in the bimodal both fearful  condition higher LPC activation than the both neutral combina- (t(24) = 0.024; p_bonf = 0.024), whereas all other combi- tion (t(24) = 3.226; p_bonf = 0.023), whereas all other nations failed to reach significance. Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 6 of 14 Fig. 6 Cue activity (contra- versus ipsi): N170: upper panel: ERP-traces time-locked to the cue, the contra-versus ipsi-lateral N170 amplitudes were increased when face-voice combinations included a fearful face. Lower panel: Topographies over each hemisphere summarize the contra-versus-ipsi-lateral activation corresponding to N170 at 190-210 ms Fig. 5 ERP-topographies of the late centralized target activity showing validity differences of valid minus invalid conditions Discussion time-locked to the target over the central-parietal ROI. For P3a, a The current ERP-study investigated whether double emo - significant main effect of validity differences indicated an increased tional face-voice cues attract attention to their location P3a-activity for the invalid condition independent of cueing type (see more strongly than single emotional cues or purely neu- blue activity color). For P3b, validity effects were only significant for tral cues, respectively. To test, bimodal face-voice cues the double neutral condition ( VN/FN) and indicated a reverse validity difference with increased valid activity (see orange activity color). that varied concerning their degree of emotional content; For LPC, validity differences were only present in the double fear i.e., fearful in face and voice (both fearful), fearful voice/ condition ( VF/FF) increasing for the invalid versus valid condition neutral face or fearful face/neutral voice (single emo- tional cue), or both neutral (both neutral cue) preceded a visual target. Considering reaction times, the num- ber of cue components related to fear (both, one, none) Cue‑related N170 activation interacted with cue validity. This was mainly due to a Specifically asking for spatial effects due to the varying pronounced validity effect in the both fearful condition. emotional presence during the cue-target interval, we Target related ERP-effects started with a contra-versus- analyzed contra-versus-ipsilateral N170 activity differ - ipsi- lateralized P1 increase for valid versus invalid con- ences over occipital-parietal electrode positions (P7/ ditions which was largest for the both fearful condition. P8) separately for each emotional cue combination. The early P3a showed a main effect of increased activity ANOVA results showed a significant main effect of in invalidly cued compared to validly cued trials, inde- emotional cue combination (F(3, 72) = 2.807; p = 0.046; 2 pendently of presence and quantity of emotional content. ηp = 0.105 see Fig.  6). Bonferroni-corrected posthoc- In contrast, the P3b component was, again, differentially tests indicated that the N170 for the both-fear com- affected by the type of cue, displaying significantly larger bination was significantly more negative than for the amplitudes after valid than after invalid cues (i.e., revers- voice-fear with face-neutral combination (VF/FN; ing the P3a pattern) for all cueing conditions except for t(24) = 3.233; p_bonf = 0.012; see Fig.  6). Similarly, also the bimodal fearful condition. The former pattern of the face-fear with voice-neutral combination (VN/FF) results (i.e., P3a larger after valid than after invalid cue- revealed an increased N170 compared to the voice-fear ing, P3b larger after invalid than after valid cueing) thus with face-neutral combination (VF/FN; t(24) = 2.939; resembles the course of reaction times over the cue-tar- p_bonf = 0.045). Finally, the both-neutral combination get interval, that is, a regular validity effect after short was not significantly different when compared to any intervals which turns into a reversed validity effect other combination. Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 7 of 14 after longer periods. Viewed from this perspective, the (98.3%), this is a task of rather low perceptual demands. absence of the reversal effect in the P3b for the both fear - Instead, the results of the current study suggest that ful condition would be consistent with the assumption of bimodally presented fear-related content of spatial cues a reduced IOR mechanism. Finally, yet importantly, only may be sufficient to enhance the deployment of spatial in the both fearful condition, there was an increase of attention to the cued position compared to bimodal non- LPC for invalid compared to the valid condition, possi- emotional cues. bly indicating that evaluation of the emotional content is Cues of both neutral content indicated a slight non- enhanced by a mismatch of spatial positions. significant tendency for an opposite cueing effect of improved performance for invalid compared to valid Performance data targets. In the current study, the sound duration was set The present behavioral results indicated that emotional to 1000  ms to ensure the discrimination of the auditory content interacted with spatial validity. Regarding reac- content. Together with the current ISI of 190–240  ms, tion times, this interaction was mainly driven by a strong the SOA-time was therefore 1190–1240 ms. Considering validity effect in the both fearful condition, contrasting neutral sound cues of similar psychophysical structure with a descriptively slightly reversed validity effect of [75], we previously found typical spatial validity effects the neutral condition. Although the hit rate analysis did (i.e., improved performance to valid compared to invalid not yield any statistically significant interactions (which targets) with an interstimulus interval (ISI) of 50–150 ms, might not be surprising given the overall high level of but SOA of 1050–1150  ms due to the long sound dura- accuracy), the result displayed a trend of improved per- tion. Importantly however, neutral cueing effects formance after double fear cues for valid targets, and for inverted in an IOR-like manner (invalid improved com- invalid targets after neutral cues. Therefore, the interac - pared to valid) with an ISI of 650–750 ms, while an ISI of tion trend in the hit rates resembled the result pattern 350–450 ms indicated no cueing effects in neither direc - observed in reaction times. Enhancement of the cueing tion [75]. In the current study, the ISI was 190–240  ms, validity effect in the both fearful condition corresponds thus outside the time-range of typical validity effects for to previous findings using unimodal fearful cues. As pre - neutral cues as found in our previous study. It would thus viously shown, fearful facial expressions [72] as well as seem straightforward to assume that the interval admin- fear-related voice cues [74] enhanced performance differ - istered in the current study was too short to yield reli- ences in responding to targets presented at validly versus able IOR, while, on the other hand, sufficiently long for invalidly cued positions. Importantly, this cueing validity substantial undoing of the initial attention shift towards effect was found not only with short SOA for face cues the cued location, except for the bimodal fearful cueing (117 ms) but also with long SOA for voice cues (1200 ms), condition, in which disengagement of attention from the as long as the ISI was kept short, suggesting that the cued location was less feasible. occurrence of IOR depends on a long ISI. The results of the current study seem in line with this assumption as Validity differences for targets: ERP‑results there was no significant reversal of the validity effect— Lateralized contra‑vs‑ipsi P1‑effect Time-locked to tar- and a significant regular validity effect in the both fearful gets, contra-versus-ispi-lateralized P1-activity indicated condition—despite the SOA exceeded 1000 ms. an enhancement of valid versus invalid activity for all Further, effects of bimodal cues have so far only been conditions. Importantly however, targets after bimodal tested lacking emotional content [47, 63]. In the study fearful cues indicated a specifically large P1-validity dif - by Santangelo et  al. [63], bimodal cues evoked a statisti- ference due to a massive activity increase for validly cued cally non-significant behavioral enhancement of target targets. One the one hand, the general input of P1 for all processing compared to unisensory cues. In the study valid vs. invalid conditions fits well with a spatial bottom- of Mastrobernadino et  al. [47], enhanced validity effects up theory where early target processing is enhanced at were observed for bimodal cueing (i.e., correspondence positions preoccupied validly by spatial cues independent of the picture of a laterally presented animal face occur- of awareness or SOA duration [27, 54, 72]. For example, ring alongside another animal face on the opposite side in a spatial cueing paradigm, Giattino and colleagues [27] with a centrally presented sound) but these effects were used lateralized cues of short 17  ms duration hidden in confined to conditions of high task difficulty. While this a multiobject-presentation. Targets consisted of neutral may be the case for non-emotional cues, the results of quadratic shapes at valid or invalid positions of the cues. our study show that this is not the case for emotional Independently of whether participants consciously per- material. Our task consisted of an easy discrimination of ceived the cues or not, validly cued targets always evoked a lateralized arrow pointing up- or downwards. Accord- higher P1-activity over occipital areas than invalidly cued ing to the overall high accuracy rates of our participants targets. In addition, in a spatial cueing study with facial Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 8 of 14 anger cues and neutral targets, Liu et  al. [44] found that sus invalid position, the early spatial P3a-effects may also target-related P1-activity increased when validly versus point to automatic early bottom-up processes. invalidly cued. Similarly, Brosch and colleagues [8] found increased P1 for valid vs. invalid targets lateralized to side Central P3b‑late effect From 325 to 375  ms, the both of the fearful facial expression when double facial cues neutral condition and the fearful face with neutral voice with one fearful and one neutral expression were pre- condition (VN/FF) revealed validity effects with increased sented. In addition, P1-valdity differences were also found valid versus invalid activity. Firstly, considering the above- in crossmodal spatial cueing when an auditory anger voice mentioned P3a, these P3b validity-effects were inverted cued a visual neutral target [7]. All these P1-validity dif- in polarity as they now indicated more activation for valid ferences occurred early (around 100  ms) and showed a than invalid targets. This might indicate a disturbance by strong responsiveness of the spatial location of the target the valid instead of invalid spatial position and fits well in relation to the preceding position of the emotional cue with the current behavioral data indicating an IOR-ten- [8, 72]. These effects might therefore mirror a bottom-up dency for the fully neutral condition and VN/FF condition. process of sensory processing [8, 72]. Our current finding Secondly, IOR-effect with neutral stimuli can be found supports this hypothesis by extending previous data with over occipital-parietal electrodes at P3-time intervals [48, a specifically contra-vs-ipsilateral P1-validity difference 58, 75], indicating that attention needed to be boosted at and its enhancement by bimodally double fearful stimula- the validly instead of shifting away to the invalidly cued tion. and IOR-conform position. Thirdly, some recent research [39, 62] suggested that a late P3-activity mirrors conscious Central P3a‑effect ERP-data time-locked to targets, P3a- perception. Thus, the relatively long delay of target-onset activity increased for invalid versus valid double combi- after the neutral and mixed VN/FF-cues might support nations independent of condition. uTh s, independent an expectancy effect of target processing at the invalid of emotional presence or absence, spatial validity corre- position. In conclusion, the validity inversion of the late sponded inversely to the P3a-activity. With spatial cueing P3, respectively P3b, seems to fit well with the behavioral the P3a has been so far mainly investigated with predic- occurrence of a probable IOR-effect that is missing when tive designs (i.e., 75% valid vs. 25% invalid conditions), full emotional bimodal integration is required. where the low probability of invalid stimuli increases their novelty character of spatial position [16, 24, 28, 29]. How- LPC‑activity At around 500 ms time-locked to targets, ever, while the P3a validity differential activation might a positive slow wave (late positive component; LPC) be larger in predictive cueing, it was still present in the showed validity differences exclusively in  the bimodal current unpredictive cueing design (50% valid/invalid). both fearful combination again with an enhanced inva- The reason might be that even in unpredictive cuing an lid versus valid activity. This finding seems to combine invalid target position still needs to be spatially updated different views on the LPC (late positive component); a compared to the preceding cue position, thus leading to component that is sometimes thought to include the late increased P3a activation for invalid versus valid targets. positive potential (LPP; [12] for a review). Firstly, the Similarly, Lasaponara and colleagues [40] found increased LPC might be considered as a slow wave conflict poten - invalid P3a activity when using only a 50% ratio of valid tial [13, 18, 50]. For example, the LPC-activation appears stimulation compared to the sum of invalid and no-cue in non-emotional Stroop-tasks as well as non-emotional conditions. Considering the modulation by emotional Flanker-tasks and mirrors the conflict in semantic content in spatial cueing, there is often mentioned only meaning (Stroop) of an incongruent color-word-com- a general P3-activity without subdivisions of P3a and bination relative to congruent color-word-combination P3b [21, 44, 73, 75]. These emotional ERP-studies mainly respectively the perceptive conflict (Flanker) of incon - revealed P3 validity differences that increased in positive gruent versus congruent target-flanker combinations activity for invalid versus valid target stimuli when cued [1, 13, 18, 42, 70]. Secondly, the LPC/LPP is handled with fearful faces [21, 44]. Importantly, when targets were as an indicator of negative emotional processing [30, preceded by disgust sounds, such P3 effects inverted inde - 36, 51, 59]. In this sense, the LPC/LPP increased over pendently of the ISI-length, indicating an increased valid centro-parietal scalp regions for negative versus neutral versus invalid activity, which corresponded to early shifts emotional stimuli, when ERP-studies required a simple of attention away from the disgust evoking stimulus and emotional discrimination of a centrally presented pic- may therefore relate to an automatic early bottom-up pro- ture [30, 36, 51, 59]. In addition, a central cueing study cess of emotional context [73, 75]. As fearful like neutral [33] indicated validity effects in the same direction as stimulation supports processing of targets at the valid ver- the current study, e.g., increased LPP-activity after invalidly compared to validly cued negative emotional Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 9 of 14 pictures. In the aforementioned studies, the conflicting unspecifically enhance attention in general, at least at or negative enhanced LPC was time-locked to the onset this early time of processing. of the emotional stimulus itself. In contrast, the LPC difference in the present data was found time-locked Limitations to the presentation neutral target, importantly however Contrasting the spatial effects of fear and neutral content when following double negative cues and indicating spa- of bimodal cues, this study consisted of a 4 × 2 stimulus tial conflict. In addition, perception of stimulus conflict paradigm and thus eight conditions (e.g., even 16 condi- might yield aversive states ([17], for a review),  and fits tions when splitting into ipsi- and contralateral activity). therefore well to the just cued fearful bimodal percep- To ensure a relatively high signal-to-noise ratio of the tion. More specifically, the LPP/LPC complex is thought ERP-data (see [6]), we decided to confine experimental to be influenced by top-down processes (e.g., reinter - manipulations to a single ISI and a single type of emo- pretation of negative stimuli) and reflecting endogenous tional content. This resulted in two noteworthy caveats. spatial attention (cf. [12], for a review). Thus, our data First, adding a second even longer ISI might have yielded might integrate and extend present LPC-theories by stronger evidence for our interpretation of a delay of dis- showing that targets, invalidly cued by fearful bimodal engagement of attention from the cued location in the stimuli, are associated with enhanced spatial conflict. both fearful condition (i.e., if consistent IOR for all con- ditions could be observed with a sufficiently long SOA). Cue‑related N170‑activity Time-locked to cues, the Second, it should be stressed that our experiment was facial related N170 component was always enhanced confined to the investigation of one particular emotion when the facial cue expressed fear—independent of the (i.e., fear) and that the results may not transfer to visual- accompanying sound expression. Investigating bimodal auditory cues related to other emotions. As noted in the effects of neutral face-voice combinations, Latinus Introduction, studies of spatial cueing featuring disgust- and colleagues [41] found that the N170 did not differ related cues yielded a reversed validity effect (i.e., supe - between purely facial and combined facial-voice stimuli. rior performance in invalidly cued trials), consistent Considering emotional influence, purely visual spatial with a repelling or dispersing effect on spatial attention cueing studies found that N170 activity increased for (e.g., [44, 72, 74]). Assuming that the validity effect in the fearful versus neutral facial spatial cues independently of bimodal fearful condition of the current study reflects a whether their emotional content was task-relevant [11, reduction in attentional disengagement counteracting 69] or task-irrelevant ([15]; see also [31], for a review). IOR does not seem to allow predictions for situations in In contrast, some facial studies (e.g., [8, 55]) used a dou- which attentional engagement is absent in the first place. ble facial cue with one emotional and one neutral face. In this case, emotional enhancements at N170 might be rather hard to detect or at least weakened by nec- Conclusions essary subtraction analysis to extract fear-related acti- The present behavioral and ERP-data compared for the vation. In addition, asking for overt discrimination  of first  time spatial cueing effects elicited by bimodal face- facial emotion or gender, emotional discrimination also voice cues of varying emotional content (both fearful, revealed an enhanced N170 for fearful compared to face fearful with voice neutral, face neutral with fear- neutral expressions [3, 4, 61], but see [19] for no N170 ful voice, both neutral). Typical spatial cueing effects differences). So far, bimodal emotional differences of with faster reaction times to validly cued targets than to N170 have only been investigated with matching emo- invalidly cued targets selectively occurred in the both- tional face-voice combinations (e.g., bimodal fearful or fearful condition. This behavioral effect corresponded bimodal neutral), revealing that N170 was enhanced to P1-validity differences of target processing, that were for bimodally fearful compared to bimodally neutral largest in the both-fearful condition, as well as to a later face-voice combinations [68]. Only an increase in the LPC-validity difference, which was exclusively present N170 amplitude for angry versus neutral faces, how- in trials with bimodal fearful cues. In contrast, an early ever independently of voice, was found. Therefore, the general main effect of validity in the P3a component indi - present finding extends the results of previous studies, cated that spatial attention was initially directed towards by emphasizing the importance of facial expressions for the validly cued spatial position even in trials involving the N170, and its independence of the accompanying both neutral or single emotional types (albeit less pro- voice content. A possible explanation might relate to the nounced than in the both fearful condition). The pattern visual discrimination demands of the experimental set- of the P3b component suggested that spatial attention ting, where movement of the attentional focus is domi- was re-directed, but to a lesser extent in the bimodal nated by visual information and an added voice might both-fearful condition. In conclusion, our present data Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 10 of 14 therefore suggest that, at least when using long SOAs consisted of either two fearful expressions or two neu- with short ISI, only double fearful stimulation prevented tral expressions, whereas single emotion combinations or at least delayed the disengagement of spatial attention included one modality fearful and the other neutral. Dur- from an invalidly cued position. Single emotional stimu- ing the entire EEG-experiment, a fixation cross was pre - lation, by contrast, did not counteract such disengage- sented at the center of the screen to ensure exogenous ment and presumably the initiation of IOR, compared to and peripheral stimulation. Each trial started with the completely neutral stimulation, in a noticeable manner. presentation of a bimodal face-voice cue for 1000 ms. On Future bimodal cueing studies might corroborate this the offset of the face-voice cue followed a jittered cue-to- conclusion by extending the length of the ISI. Selective target-interval of 190-240 ms. Then a visual neutral target reduction of attentional disengagement from a cued loca- (a little white arrow pointing up or down) was presented tion after bimodal emotional cues should evidence itself for 100 ms. After an intertrial interval of 2–3 s, the next in selective absence or delay of IOR in this condition (see trial started (see Fig.  1). The task of the participants was [23], for a similar approach concerning the impact of to ignore the bimodal face-voice cues and to respond to emotional cue content in general). the pointing direction (up/down) of the target. All partic- ipants completed 10 runs. Each run consisted of 60 trials Methods and lasted about 3.5 min, leading to a total experimental Participants run time of about 35 min. Planning of sample size was based on previous, compa- rable spatial cueing studies that investigated the modu- Stimuli latory effects of emotion [73, 75]. As behavioral effects The emotional cueing voice was either a fearful voice were not evident in all studies [63], P1 differences (someone fearfully screaming) or a neutral voice (some- between valid and invalid trials for both fearful face-voice one biting into an apple). These voices were already used cues were considered the relevant endpoint. Indeed, a and rated in our previous studies (e.g. [73, 74]). In our post-hoc analysis with G*Power [20] showed that the previous study [74], the fear voice was rated as clearly achieved power in our study was 1-β = 0.99 given the negative (valence) with high arousal, whereas the neu- found difference between P1 peaks (two-tailed t-test, tral voice was rated as neutral (valence) with low arousal. α = 0.05, dz = 1.04) Provided the found effect size in this Each of the two voice stimuli had a duration of 1000 ms. study (dz = 1.04), analysis with G*Power indicated that a This duration guaranteed that the emotional content of sample size of 10 participant is sufficient to detect a sig - the voice was fully processed by the participants before nificant difference with 1-β = 0.80, α = 0.05. the presentation of the target (e.g., [52]) and was also Twenty-six participants (12 men, M = 23.4  years, chosen in our previous studies [73, 74]. The overall sound age SD = 4.5) took part in the ERP-experiment. All were level was normalized to 68  dB SPL for both emotional right-handed, had normal or corrected-to-normal vision, voices. To preserve the emotional character of the voices, normal hearing abilities and had no history of psychiat- the time–frequency structures of both voices were not ric or neurological disease. One of these participants was changed (cf. for a similar procedure with happy/sad emo- excluded from the final data analysis due to poor qual - tional voices: see [2, 34, 35]). For lateralized presentation, ity of the EEG signal (excessive movement and drifts). the originally stereo-recorded voices (someone fearfully All participants were students of the MSH and received screaming, someone biting into an apple) were converted study-participant-points. They gave written informed into mono-channel sounds by using “Au Adobe Audi- consent according to the ethical standards laid down in tion” (http:// www. adobe. com). During ERP/EEG, these the Declaration of Helsinki (BMJ 1991; 302; 1194). The mono-channel sounds were then delivered using the study was approved by the Ethics committee of the MSH. software Presentation (neurobehavioral systems; http:// www. neuro bs. com) to either the left or the right loud- Materials and procedure speaker. Both loudspeakers were positioned behind the We used a spatial cueing paradigm with bimodal face- right and left side of the screen to induce an associa- voice cues to investigate if and how congruent emo- tion between voice and target location. We did not use tional content in two modalities would lead to stronger headphones due to their interference with the EEG-cap engagement to the cue and therefore to enhanced target and -measurements. Similarly, the emotional faces con- processing when compared to single emotional pres- sisted either of a fearful or neutral expression. After the ence in only one modality or purely neutral combina- EEG-measurements, participants described the facial tions. Importantly, face-voice combinations were always expressions in their own words and applied ratings of bimodal cues, consisting of a facial expression with a emotional valence and arousal. First, they were asked to non-verbal voice expression. Congruent combinations name the facial expression (open answer). Indeed, the Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 11 of 14 fearful expression was described with fear-related words VALIDITY (valid, invalid). Greenhouse–Geisser correc- (e.g., “fearful”, “frightenend”, “horrified”, “panicking”) by tion was applied to all statistical comparisons for which 88% of the participants (22 out of 25) as well as the neu- the Mauchly test indicated violation of sphericity. In case tral expression with non-emotional related words (e.g., of significant interaction, follow-up posthoc Bonferroni “neutral”, “bored”, “well-balanced”) by 88% of the partici- corrected t-tests were used to indicate improvements pants. Secondly, participants rated valence and arousal versus deterioration between single conditions. Signifi - of the facial expressions on a scale ranging from 1 to 5 cance was inferred for corrected p-values < 0.05. (from lowest arousal/negative valence to highest arousal/ positive valence). For statistical analyses, the ratings for ERP‑data analysis valence and arousal, respectively, were averaged across The custom ERPSS software (Event-Related Potential participants for each facial expression. Paried t-tests indi- Software System, UCSD, San Diego, CA, USA, cf. [45] cated more negative valence (t(24) = 14.000; p < 0.001) was added-on to the open source EEGLAB software (an and higher arousal (t(24) = 6.385; p < 0.001) for the fearful open source environment for electrophysiological sig- than for the neutral expressions. To ensure fully bimodal nal processing, UCSD, San Diego, CA, USA, cf. [14] and stimulation during the EEG-experiement, onset and off - served as software for all ERP-analyses. ERP-analyses set of sounds and faces was simultaneously. started by transforming the raw-data of the 10 runs of each participant into ERPSS format and combining them EEG recording into one large data-file for further analysis. To avoid The BrainVision Recorder (Brain Products, Germany biasing to any electrode position, the reference for each http:// www. brain produ cts. com/) was used for record- of the 32 electrodes was set to the algebraic average of ing EEG/ERP-data in combination with a 32 channels all channels. For cues as well as targets, the continuous electrode cap of the electric shielded EEG-system “acti- EEG data were divided into 800  ms epochs, time-locked CAP snap”, (Brain Products, Germany). According to to the relating onsets including a pre-stimulus baseline the 10–20 EEG system, the 32-electrodes included seven of 200  ms. To discard epochs contaminated by large eye frontal electrodes (FP1, FP2, F7, F3, FZ, F4, F8), four movements, excessive muscle activity, drifts, or amplifier fronto-central electrodes (FC5, FC1, FC2, FC6), nine blocking, an artifact rejection was performed by reject- central-temporal electrodes (T7, C3, Cz, C4, T8, CP5, ing any voltage amplitudes under −  100  µV and over CP1, CP2, CP6), four lateral electrodes (FT9, FT10, TP9, 100  µV. Resulting artifact-free EEG epochs were aver- TP10) and eight parietal-occipital area electrodes (P7, aged together, separately for the various trial types (i.e., P3, PZ, P4, P8, O1, OZ, O2). All 32 electrodes were ref- for cues: right receptively left sided cues for both fear/ erenced to the algebraic average of all channels and were voice-fear with face-neutral (VF_FN)/voice-neutral with therefore unbiased to any electrode position. The ground face-fear (VN_FF)/both neutral; for targets: invalid, electrode was placed between electrode position Fp1 and receptively valid trial types corresponding to the four Fp2. Electrode impedances were kept below 10 kΩ for all emotional conditions time-locked to the visual target). electrodes. A dimly lit, sound-attenuated and electrically As a final step, all ERP-averages were digitally low-pass shielded chamber was used for EEG-recording. filtered (IIR-Butterworth) with a running-average filter of 30 Hz. Behavioral data The aim of our study was to investigate if and how Our main interest was to find evidence for enhancement bimodally presented double emotional stimulation (face of spatial cueing effects by emotional coloring in two ver - and voice both fearful) enhances reactions compared sus one or none sensory modalities of a bimodal audio- to bimodal audio-visual stimulations with fear solely in visual cue. Only trials with behavioral responses between one of two sensory modalities. Target-related analyses 200 and 1000 ms after the presentation of the arrow-tar- focused on the early spatial P1-component as well as get were considered for further behavioral analysis (98.3% the later P3-component as indicator of attentional shifts trials in total). For all cue conditions, reaction times (RTs) [73, 75]. Further, the LPC-component was analyzed as an to correctly judged arrow-targets as well as accucary rates emotional and conflict indicator [33, 65]. For early target were analyzed. Using the statistic program JASP version activity, we focused on contra-ipsilateral P1-differences at 0.16.1.0 (https:// jasp- stats. org/), for each performance O1/O2-electrodes similarly to McDonald and colleagues measure (RTs respectively accuracy rates), a repeated- (2013). Latency time window of 110–130  ms served as measures 4 × 2 ANOVA was performed with the within- the P1 for all subsequent analyses of emotional varia- subject factors EMOTIONAL CUE COMBINATION tions in contra-ipsi differences. Contra-ipsi analysis was (both fearful, voice-fearful with face neutral (VF_FN), performed by using the contra-ipsi function of erplab voice-neutral with face-fear (VN_FF, both neutral) and [45]. This function first averaged each condition’s contra Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 12 of 14 activation of right-sided stimulation at the left electrode fear/face neutral, voice neutral/face fear, both neutral) O1 with its contra activation of left-sided stimulation at and VALIDITY (invalid, valid) was calculated, this time the right O2. Similarly, the ipsi-averages for each condi- however including ERP-activation averaged over left and tion were calculated. In a second step, distraction of con- right target stimulation. For the N170 cue-locked activ- tra-versus ipsi-averaged stimulations was performed (cf. ity, we analyzed whether and how the N170-amplitude [45]). of contra versus ipsi-stimulations differed between the For the P3 analysis, spatial cueing literature as well as stimulations of varying fearful content (cue with: both our own studies had indicated that spatial cuing effects fear/voice-fear with face-neutral (VF_FN)/voice-neutral on visual targets are usually found over central-parieto- with face-fear (VN_FF)/both neutral). To test for dif- occipital electrode sites (for emotional cues: [44, 57, 73, ferences in contra-ipsi activity at the N170, a repeated 75], for neutral cues: [46, 49]). Thus, electrode positions measure ANOVA including the factor EMOTIONAL located symmetrically over the visual-parietal cortex CUE TYPE with the 4 repeated measures of the cue type areas (Cz/Fp1/Fp2) were combined to a parietal-occipi- (both fear/voice-fear with face-neutral (VF_FN)/voice- tal region of interest (ROI). To select the time windows neutral with face-fear (VN_FF)/both neutral) was per- for theP3, in each subject, the average of all target con- formed. For all analyses, significance was set to p < 0.05. ditions (i.e., averaged over validity and emotion appear- Further, in case of significant results in the interaction ance) indicated two peaks: a P3a-peak at around 260 ms of an ANOVA, subsequent posthoc-tests were added to and a later P3b peak at around 350 ms. For further analy- indicate the cause of significance. Greenhouse–Geisser ses, these peaks were chosen for P3a and P3b with plus/ corrections was applied to all statistical comparisons for minus 20  ms time windows. Also for the LPC-analyses, which the Mauchly Test indicated violation of sphericity. the ROI of the visual-parietal cortex areas (Cz/Fp1/Fp2; Acknowledgements cf. [33]) was used. The time-window for the LPC was We thank Mortatha Al Khafage for assistance in collecting the current set to 480–600  ms, thus covering temporal LPC of pre- EEG-data. vious literature [33, 65]. For the statistical analysis, we Author contributions used mean amplitude values computed over the length of UZ planned and designed the experimental task. UZ analyzed the data with these intervals. input by MW and MP. UZ and MW wrote the manuscript with intensive feed- back of MP. All authors read and approved the final manuscript. Cue-related analysis focused specifically on the ERP- component N170 as related to facial processing. The Funding N170 is generally located over visual-parietal areas Open Access funding enabled and organized by Projekt DEAL. including electrodes P7/8, with N170 around 170  ms Availability of data and materials (e.g., [11, 69]; see [31] for a review). To test for attentional The datasets used and analyzed during the current study are available from facial effects due to lateralized cue presentation, N170 the corresponding author on reasonable request. differences were investigated by using contra-ipsi-differ - ences. Contra-ipsi analysis was again performed by using Declarations the contra-ipsi function of erplab [45], however, this time Ethics approval and consent to participate comparing N170 ipsi/contra-activity of the left electrode This research was approved by the Ethics Committee of the Medical School P7 with the corresponding ipsi/contra-activity of the Hamburg MSH. Before starting the experimental task, participants received information about the purpose of the study, the task, and its duration and right electrode P8. gave their written informed consent. For all statistical analyses of the target-related P1- / P3- as well as cue-related N170 components, we used Consent for publication Not applicable. mean amplitude values computed over the length of the corresponding time-window intervals. All statisti- Competing interests cal ERP-analyses were performed by using JASP version The authors declare that they have no competing interests. 0.16.1.0 (https:// jasp- stats. org/). For target-activity, two Author details main analyses were performed separately. For the spatial 1 Faculty of Human Sciences, Department of Psychology, MSH Medical School P1-compenent, an repeated measurement 4 × 2 ANOVA Hamburg, Hamburg, Germany. ICAN Insitute of Cognitive and Aec ff tive Neuroscience, MSH Medical School Hamburg, Hamburg, Germany. F acult y included contra-versus-ipsi-activation for the factor of Psychology, Department of Biopsychology, Institute of Cognitive Neurosci- EMOTIONAL CUE TYPE (fear/voice-fear with face- ence, Ruhr-University Bochum, Bochum, Germany. neutral (VF_FN)/voice-neutral with face-fear (VN_FF)/ Received: 29 October 2021 Accepted: 27 August 2022 both neutral) and the factor VALIDITY (valid/invalid). To test for differences in both P3-peaks and LPC activa - tion, for each component a similar 4 × 2—ANOVA with the factors EMOTIONAL CUE TYPE (both fear, voice Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 13 of 14 References 22. Folyi T, Wentura D. Fast and unintentional evaluation of emotional 1. Appelbaum LG, Meyerhoff KL, Woldorff MG. Priming and backward sounds: evidence from brief segment ratings and the affective Simon influences in the human brain: processing interactions during the task. Cogn Emot. 2017;31(2):312–24. https:// doi. org/ 10. 1080/ 02699 931. stroop interference effect. Cereb Cortex. 2009;19(11):2508–21. https:// 2015. 11105 14. doi. org/ 10. 1093/ cercor/ bhp036. 23. Fox E, Russo R, Dutton K. Attentional bias for threat: evidence for delayed 2. Banse R, Scherer KR. Acoustic profiles in vocal emotion expression. J disengagement from emotional faces. Cogn Emot. 2002;16(3):355–79. Pers Soc Psychol. 1996;70:614–36. https:// doi. org/ 10. 1037/ 0022- 3514. https:// doi. org/ 10. 1080/ 02699 93014 30005 27. 70.3. 614. 24. Friedman D, Cycowicz YM, Gaeta H. The novelty P3: an event-related 3. Batty M, Taylor MJ. Early processing of the six basic facial emotional brain potential (ERP) sign of the brain’s evaluation of novelty. Neurosci expressions. Brain Res Cogn Brain Res. 2003;17(3):613–20. https:// doi. Biobehav Rev. 2001;25(4):355–73. https:// doi. org/ 10. 1016/ s0149- 7634(01) org/ 10. 1016/ s0926- 6410(03) 00174-5. 00019-7. 4. Blau VC, Maurer U, Tottenham N, McCandliss BD. The face-specific N170 25. Frühholz S, Jellinghaus A, Herrmann M. Time course of implicit process- component is modulated by emotional facial expression. Behav Brain ing and explicit processing of emotional faces and emotional words. Biol Funct. 2007;3:7. https:// doi. org/ 10. 1186/ 1744- 9081-3-7. Psychol. 2011;87(2):265–74. https:// doi. org/ 10. 1016/j. biops ycho. 2011. 03. 5. Blurton SP, Greenlee MW, Gondan M. Cross-modal cueing in audio- 008. visual spatial attention. Atten Percept Psychophys. 2015;77(7):2356–76. 26. Gao C, Conte S, Richards JE, Xie W, Hanayik T. The neural sources of N170: https:// doi. org/ 10. 3758/ s13414- 015- 0920-7. understanding timing of activation in face-selective areas. Psychophysiol- 6. Boudewyn MA, Luck SJ, Farrens JL, Kappenman ES. How many trials ogy. 2019;56(6): e13336. https:// doi. org/ 10. 1111/ psyp. 13336. does it take to get a significant ERP effect? It depends. Psychophysiol- 27. Giattino CM, Alam ZM, Woldorff MG. Neural processes underlying the ori- ogy. 2018;55: e13049. https:// doi. org/ 10. 1111/ psyp. 13049. enting of attention without awareness. Cortex. 2018;102:14–25. https:// 7. Brosch T, Grandjean D, Sander D, Scherer KR. Cross-modal emotional doi. org/ 10. 1016/j. cortex. 2017. 07. 010. attention: emotional voices modulate early stages of visual processing. 28. Gómez CM, Flores A, Digiacomo MR. Increased brain activation during J Cogn Neurosci. 2009;21(9):1670–9. https:// doi. org/ 10. 1162/ jocn. 2009. the processing of spatially invalidly cued targets. Open Neuroimag J. 21110. 2008;2:106–13. https:// doi. org/ 10. 2174/ 18744 40000 80201 0106. 8. Brosch T, Pourtois G, Sander D, Vuilleumier P. Additive effects of 29. Gómez CM, Flores A, Digiacomo MR, Vázquez-Marrufo M. Sequential P3 emotional, endogenous, and exogenous attention: behavioral and effects in a Posner’s spatial cueing paradigm: trial-by-trial learning of the electrophysiological evidence. Neuropsychologia. 2011;49(7):1779–87. predictive value of the cue. Acta Neurobiol Exp. 2009;69(2):155–67. https:// doi. org/ 10. 1016/j. neuro psych ologia. 2011. 02. 056. 30. Hajcak G, MacNamara A, Foti D, Ferri J, Keil A. The dynamic allocation of 9. Bublatzky F, Schupp HT. Pictures cueing threat: brain dynamics in attention to emotion: simultaneous and independent evidence from viewing explicitly instructed danger cues. Soc Cogn Aec ff t Neurosci. the late positive potential and steady state visual evoked potentials. Biol 2012;7(6):611–22. https:// doi. org/ 10. 1093/ scan/ nsr032. Psychol. 2013;92(3):447–55. https:// doi. org/ 10. 1016/j. biops ycho. 2011. 11. 10. Buchtel HA, Butter CM. Spatial attention shifts: implications for the role 012. of polysensory mechanisms. Neuropsychologia. 1988;26:499–509. 31. Hinojosa JA, Mercado F, Carretié L. N170 sensitivity to facial expression: a 11. Carlson JM, Reinke KS. Spatial attention-related modulation of the meta-analysis. Neurosci Biobehav Rev. 2015;55:498–509. https:// doi. org/ N170 by backward masked fearful faces. Brain Cogn. 2010;73(1):20–7. 10. 1016/j. neubi orev. 2015. 06. 002. https:// doi. org/ 10. 1016/j. bandc. 2010. 01. 007. 32. Jonides J. Voluntary versus automatic control over the mind’s eye’s move- 12. Carretié L. Exogenous (automatic) attention to emotional stimuli: a ment. In: Long JB, Baddeley AD, editors. Attention and performance IX. review. Cogn Aec ff t Behav Neurosci. 2014;14:1228–58. Hillsdale: Erlbaum; 1981. p. 187–203. 13. Coderre E, Conklin K, van Heuven WJ. Electrophysiological measures 33. Johnen AK, Harrison NR. The effects of valid and invalid expectations of conflict detection and resolution in the Stroop task. Brain Res. about stimulus valence on behavioural and electrophysiological 2011;1413:51–9. https:// doi. org/ 10. 1016/j. brain res. 2011. 07. 017. responses to emotional pictures. Int J Psychophysiol. 2019;144:47–55. 14. Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of https:// doi. org/ 10. 1016/j. ijpsy cho. 2019. 08. 002. single-trial EEG dynamics including independent component analysis. 34. Juslin PN, Laukka P. Impact of intended emotion intensity on cue utiliza- J Neurosci Methods. 2004;134(1):9–21. https:// doi. org/ 10. 1016/j. jneum tion and decoding accuracy in vocal expression of emotion. Emotion. eth. 2003. 10. 009. 2001;4:381–412. 15. Denefrio S, Simmons A, Jha A, Dennis-Tiwary TA. Emotional cue validity 35. Juslin PN, Laukka P. Communication of emotions in vocal expression effects: the role of neurocognitive responses to emotion. PLoS ONE. and music performance: different channels, same code? Psychol Bull. 2017;12(7): e0179714. https:// doi. org/ 10. 1371/ journ al. pone. 01797 14. 2003;12:770–814. 16. Dien J, Spencer KM, Donchin E. Localization of the event-related 36. Kappenman ES, MacNamara A, Proudfit GH. Electrocortical evidence for potential novelty response as defined by principal components rapid allocation of attention to threat in the dot-probe task. Soc Cogn analysis. Brain Res Cogn Brain Res. 2003;17(3):637–50. https:// doi. org/ Aec ff t Neurosci. 2015;10(4):577–83. https:// doi. org/ 10. 1093/ scan/ nsu098. 10. 1016/ s0926- 6410(03) 00188-5. 37. Klein RM. Inhibition of return. Trends Cogn Sci. 2000;4(4):138–47. https:// 17. Dignath D, Eder AB, Steinhauser M, Kiesel A. Conflict monitoring and doi. org/ 10. 1016/ s1364- 6613(00) 01452-2. the affective-signaling hypothesis—an integrative review. Psychon Bull 38. Kuniecki M, Pilarczyk J, Wichary S. The color red attracts attention in Rev. 2020;27(2):193–216. https:// doi. org/ 10. 3758/ s13423- 019- 01668-9. an emotional context. An ERP study. Front Hum Neurosci. 2015;9:212. 18. Donohue SE, Appelbaum LG, McKay CC, Woldorff MG. The neural https:// doi. org/ 10. 3389/ fnhum. 2015. 00212. dynamics of stimulus and response conflict processing as a func- 39. Lamy D, Salti M, Bar-Haim Y. Neural correlates of subjective awareness and tion of response complexity and task demands. Neuropsychologia. unconscious processing: an ERP study. J Cogn Neurosci. 2009;21(7):1435– 2016;84:14–28. https:// doi. org/ 10. 1016/j. neuro psych ologia. 2016. 01. 46. https:// doi. org/ 10. 1162/ jocn. 2009. 21064. 035. 40. Lasaponara S, D’Onofrio M, Pinto M, Dragone A, Menicagli D, Bueti D, De 19. Eimer M, Holmes A. An ERP study on the time course of emotional face Lucia M, Tomaiuolo F, Doricchi F. EEG correlates of preparatory orienting, processing. NeuroReport. 2002;13(4):427–31. https:// doi. org/ 10. 1097/ contextual updating, and inhibition of sensory processing in left spatial 00001 756- 20020 3250- 00013. neglect. J Neurosci. 2018;38(15):3792–808. https:// doi. org/ 10. 1523/ JNEUR 20. Faul F, Erdfelder E, LangAG, Buchner A. G*Power 3: a flexible statistical OSCI. 2817- 17. 2018. power analysis program for the social, behavioral, and biomedical sci- 41. Latinus M, VanRullen R, Taylor MJ. Top-down and bottom-up modula- ences. Behavior research methods. 2007;39(2):175–91. https:// doi. org/ 10. tion in processing bimodal face/voice stimuli. BMC Neurosci. 2010;11:36. 3758/ bf031 93146https:// doi. org/ 10. 1186/ 1471- 2202- 11- 36. 21. Fichtenholtz HM, Hopfinger JB, Graham R, Detwiler JM, LaBar KS. Happy 42. Liotti M, Woldorff MG, Perez R, Mayberg HS. An ERP study of the tempo - and fearful emotion in cues and targets modulate event-related potential ral course of the Stroop color-word interference effect. Neuropsycholo - indices of gaze-directed attentional orienting. Soc Cogn Aec ff t Neurosci. gia. 2000;38(5):701–11. https:// doi. org/ 10. 1016/ s0028- 3932(99) 00106-2 2007;2(4):323–33. https:// doi. org/ 10. 1093/ scan/ nsm026. Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 14 of 14 43. Lin H, Liang J. Contextual effects of angry vocal expressions on the 64. Schirmer A, Adolphs R. Emotion perception from face, voice, and touch: encoding and recognition of emotional faces: an event-related potential comparisons and convergence. Trends Cogn Sci. 2017;21(3):216–28. (ERP) study. Neuropsychologia. 2019;132: 107147. https:// doi. org/ 10. https:// doi. org/ 10. 1016/j. tics. 2017. 01. 001. 1016/j. neuro psych ologia. 2019. 107147. 65. Seib-Pfeifer LE, Gibbons H. Independent ERP predictors of affective prim- 44. Liu Y, Zhang D, Luo Y. How disgust facilitates avoidance: an ERP ing underline the importance of depth of prime and target processing study on attention modulation by threats. Soc Cogn Aec ff t Neurosci. and implicit affect misattribution. Brain Cogn. 2019;136: 103595. https:// 2015;10(4):598–604. https:// doi. org/ 10. 1093/ scan/ nsu094.doi. org/ 10. 1016/j. bandc. 2019. 103595. 45. Lopez-Calderon J, Luck SJ. ERPLAB: an open-source toolbox for the analy- 66. Smith ML. Rapid processing of emotional expressions without conscious sis of event-related potentials. Front Hum Neurosci. 2014;8:213. https:// awareness. Cerebral Cortex. 2012;22(8):1748–60. https:// doi. org/ 10. 1093/ doi. org/ 10. 3389/ fnhum. 2014. 00213.cercor/ bhr250. 46. Martín-Arévalo E, Chica AB, Lupiáñez J. No single electrophysiological 67. Spence C, Driver J. Audiovisual links in exogenous covert spatial orient- marker for facilitation and inhibition of return: a review. Behav Brain Res. ing. Percept Psychophys. 1997;59:1–22. 2016;300:1–10. https:// doi. org/ 10. 1016/j. bbr. 2015. 11. 030. 68. Stefanou ME, Dundon NM, Bestelmeyer P, Koldewyn K, Saville C, 47. Mastroberardino S, Santangelo V, Macaluso E. Crossmodal semantic Fleischhaker C, Feige B, Biscaldi M, Smyrnis N, Klein C. Electro-cortical congruence can affect visuo-spatial processing and activity of the fronto- correlates of multisensory integration using ecologically valid emotional parietal attention networks. Front Integr Neurosci. 2015;9:45. https:// doi. stimuli: differential effects for fear and disgust. Biol Psychol. 2019;142:132– org/ 10. 3389/ fnint. 2015. 00045. 9. https:// doi. org/ 10. 1016/j. biops ycho. 2019. 01. 011. 48. McDonald JJ, Ward LM, Kiehl KA. An event-related brain potential study of 69. Torrence RD, Troup LJ, Rojas DC, Carlson JM. Enhanced contralateral inhibition of return. Percept Psychophys. 1999;61(7):1411–23. https:// doi. theta oscillations and N170 amplitudes in occipitotemporal scalp regions org/ 10. 3758/ bf032 06190. underlie attentional bias to fearful faces. Int J Psychophysiol. 2021;165:84– 49. McDonald JJ, Störmer VS, Martinez A, Feng W, Hillyard SA. Salient sounds 91. https:// doi. org/ 10. 1016/j. ijpsy cho. 2021. 04. 002. activate human visual cortex automatically. J Neurosci. 2013;33(21):9194– 70. West R, Alain C. Eec ff ts of task context and fluctuations of attention on 201. https:// doi. org/ 10. 1523/ JNEUR OSCI. 5902- 12. 2013. neural activity supporting performance of the stroop task. Brain research. 50. McKay CC, van den Berg B, Woldorff MG. Neural cascade of conflict 2000;873(1):102–11. https:// doi. org/ 10. 1016/ s0006- 8993(00) 02530-0 processing: not just time-on-task. Neuropsychologia. 2017;96:184–91. 71. Williams LM, Palmer D, Liddell BJ, Song L, Gordon E. The “when” and https:// doi. org/ 10. 1016/j. neuro psych ologia. 2016. 12. 022. “where” of perceiving signals of threat versus non-threat. Neuroimage. 51. Myruski S, Bonanno GA, Cho H, Fan B, Dennis-Tiwary TA. The late positive 2006;31(1):458–67. https:// doi. org/ 10. 1016/j. neuro image. 2005. 12. 009. potential as a neurocognitive index of emotion regulatory flexibility. Biol 72. Zhang D, Liu Y, Wang L, Ai H, Luo Y. Mechanisms for attentional Psychol. 2019;148: 107768. https:// doi. org/ 10. 1016/j. biops ycho. 2019. modulation by threatening emotions of fear, anger, and disgust. Cogn 107768. Aec ff t Behav Neurosci. 2017;17(1):198–210. https:// doi. org/ 10. 3758/ 52. Paulmann S, Pell MD. Contextual influences of emotional speech prosody s13415- 016- 0473-9. on face processing: how much is enough? Cogn Aec ff t Behav Neurosci. 73. Zimmer U, Keppel MT, Poglitsch C, Ischebeck A. ERP evidence for spatial 2010;10:230–42. https:// doi. org/ 10. 3758/ CABN. 10.2. 230. attention being directed away from disgusting locations. Psychophysiol- 53. Pell MD, Kotz SA. On the time course of vocal emotion recognition. PLoS ogy. 2015;52(10):1317–27. https:// doi. org/ 10. 1111/ psyp. 12463. ONE. 2011;6(11): e27256. https:// doi. org/ 10. 1371/ journ al. pone. 00272 56. 74. Zimmer U, Höfler M, Koschutnig K, Ischebeck A. Neuronal interactions in 54. Posner MI, Cohen Y. Components of visual orienting. In: Bouma H, areas of spatial attention reflect avoidance of disgust, but orienting to Bonwhuis D, editors. Attention and performance X: control of language danger. Neuroimage. 2016;134:94–104. https:// doi. org/ 10. 1016/j. neuro processes. Hillsdale: Erlbaum; 1984. p. 551–6.image. 2016. 03. 050. 55. Pourtois G, Grandjean D, Sander D, Vuilleumier P. Electrophysiological 75. Zimmer U, Rosenzopf H, Poglitsch C, Ischebeck A. ERP-study on the time correlates of rapid spatial orienting towards fearful faces. Cerebral Cortex. course of disgust-motivated spatial avoidance. Biol Psychol. 2019;144:20– 2004;14(6):619–33. https:// doi. org/ 10. 1093/ cercor/ bhh023. 7. https:// doi. org/ 10. 1016/j. biops ycho. 2019. 02. 007. 56. Pourtois G, Thut G, Grave de Peralta R, Michel C, Vuilleumier P. Two electrophysiological stages of spatial orienting towards fearful faces: early Publisher’s Note temporo-parietal activation preceding gain control in extrastriate visual Springer Nature remains neutral with regard to jurisdictional claims in pub- cortex. Neuroimage. 2005;26(1):149–63. https:// doi. org/ 10. 1016/j. neuro lished maps and institutional affiliations. image. 2005. 01. 015. 57. Pourtois G, Vuilleumier P. Dynamics of emotional effects on spatial atten- tion in the human visual cortex. Prog Brain Res. 2006;156:67–91. https:// doi. org/ 10. 1016/ S0079- 6123(06) 56004-2. 58. Prime DJ, Jolicoeur P. On the relationship between occipital cortex activity and inhibition of return. Psychophysiology. 2009;46(6):1278–87. https:// doi. org/ 10. 1111/j. 1469- 8986. 2009. 00858.x. 59. Quiñones-Camacho L, Wu R, Davis E. Motivated attention to fear-related stimuli: evidence for the enhanced processing of fear in the late positive potential. Motiv Emot. 2018. https:// doi. org/ 10. 1007/ s11031- 018- 9670-x. 60. Righart R, de Gelder B. Context influences early perceptual analysis of faces—an electrophysiological study. Cerebral Cortex. 2006;16(9):1249– Re Read ady y to to submit y submit your our re researc search h ? Choose BMC and benefit fr ? Choose BMC and benefit from om: : 57. https:// doi. org/ 10. 1093/ cercor/ bhj066. 61. Rigoulot S, D’Hondt F, Defoort-Dhellemmes S, Despretz P, Honoré J, fast, convenient online submission Sequeira H. Fearful faces impact in peripheral vision: behavioral and thorough peer review by experienced researchers in your field neural evidence. Neuropsychologia. 2011;49(7):2013–21. https:// doi. org/ 10. 1016/j. neuro psych ologia. 2011. 03. 031. rapid publication on acceptance 62. Salti M, Bar-Haim Y, Lamy D. The P3 component of the ERP reflects con- support for research data, including large and complex data types scious perception, not confidence. Conscious Cogn. 2012;21(2):961–8. • gold Open Access which fosters wider collaboration and increased citations https:// doi. org/ 10. 1016/j. concog. 2012. 01. 012. 63. Santangelo V, Van der Lubbe RH, Olivetti Belardinelli M, Postma A. maximum visibility for your research: over 100M website views per year Multisensory integration affects ERP components elicited by exog- enous cues. Exp Brain Res. 2008;185(2):269–77. https:// doi. org/ 10. 1007/ At BMC, research is always in progress. s00221- 007- 1151-5. Learn more biomedcentral.com/submissions http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Behavioral and Brain Functions Springer Journals

Enhancing allocation of visual attention with emotional cues presented in two sensory modalities

Loading next page...
 
/lp/springer-journals/enhancing-allocation-of-visual-attention-with-emotional-cues-presented-9sNPXDqNw3
Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2022
eISSN
1744-9081
DOI
10.1186/s12993-022-00195-3
Publisher site
See Article on Publisher Site

Abstract

Background: Responses to a visual target stimulus in an exogenous spatial cueing paradigm are usually faster if cue and target occur in the same rather than in different locations (i.e., valid vs. invalid), although perceptual conditions for cue and target processing are otherwise equivalent. This cueing validity effect can be increased by adding emotional (task-unrelated) content to the cue. In contrast, adding a secondary non-emotional sensory modality to the cue (bimodal), has not consistently yielded increased cueing effects in previous studies. Here, we examined the interplay of bimodally presented cue content (i.e., emotional vs. neutral), by using combined visual-auditory cues. Specifically, the current ERP-study investigated whether bimodal presentation of fear-related content amplifies deployment of spatial attention to the cued location. Results: A behavioral cueing validity effect occurred selectively in trials in which both aspects of the cue (i.e., face and voice) were related to fear. Likewise, the posterior contra-ipsilateral P1-activity in valid trials was significantly larger when both cues were fear-related than in all other cue conditions. Although the P3a component appeared uniformly increased in invalidly cued trials, regardless of cue content, a positive LPC deflection, starting about 450 ms after target onset, was, again, maximal for the validity contrast in trials associated with bimodal presentation of fear-related cues. Conclusions: Simultaneous presentation of fear-related stimulus information in the visual and auditory modality appears to increase sustained visual attention (impairing disengagement of attention from the cued location) and to affect relatively late stages of target processing. Keywords: Spatial attention, Multisensory, Emotion, Sensory modality, Fear, EEG, ERP Background The most relevant experimental protocol for investigat - In daily life, emotional stimuli, such as a fear-evoking ing stimulus-elicited allocation of attention is referred to dog, often feature salient properties in more than one as the spatial cueing paradigm. In this protocol, a target sensory modality (e.g., sound of barking and sight of stimulus is presented either at a location indicated by a aggressive posture). What are the consequences of  such preceding cue (i.e., valid condition) or at a different loca - bimodally corresponding signals for the deployment of tion (i.e., invalid condition) which can be considered attention to the stimulus object they belong to? equivalent concerning other factors of perceptual rel- evance. In studies of exogenous spatial cueing, the cue is presented at one of the potential target locations (usu- *Correspondence: ulrike.zimmer@medicalschool-hamburg.de ally one of them located on the left side and one located Faculty of Human Sciences, Department of Psychology, MSH Medical School on the right side of a central fixation point). Studies of Hamburg, Hamburg, Germany Full list of author information is available at the end of the article exogenous spatial cueing reliably yielded faster responses © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. The Creative Commons Public Domain Dedication waiver (http:// creat iveco mmons. org/ publi cdoma in/ zero/1. 0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 2 of 14 in valid than in invalid conditions [32, 54]. This cue - sensory input) fear-related or anger-related cues (i.e., ing validity effect has been ascribed to the deployment faces or voices). These studies demonstrated larger valid - of attention to the cued location. Such findings are not ity differences on the behavioral level with emotionally confined to situations, in which cue and target belong to compared to neutral cues [7, 23]. Extant findings also the same sensory modality but have also been observed suggest that the direction of an attentional shift by an in studies, in which an unisensory cue and an unisen- emotional cue depends on its specific emotional con - sory target consisted of stimuli presented in two differ - tent. For example, while fearful facial expressions appear ent modalities (e.g., an auditory cue with a visual target), to attract attention toward their location, disgusted demonstrating crossmodal cueing of spatial attention [5, expressions were associated with shifts of attention away 10, 67]. In contrast to crossmodal presentations, studies from their location, resulting in a reversed validity effect of bimodal spatial cueing have been conducted to inves- (i.e., faster responding in invalid than in valid trials [44, tigate the combined effects of attentional cues presented 72]). Noteably, a similar pattern was found when using simultanesouly in two sensory modalities (e.g., a visual- sound cues of fearful and disgusting content [74]. These auditory cue). results point to a modality independence of fear-related Although some bimodal spatial cueing studies (audio- attraction of attention and disgust avoidance, suggesting visual cues preceding visual targets) have been conducted that the modulation of attentional deployment is evoked (e.g., [47, 63]), these studies  did not investigate shifts of by the semantic content rather than by superficial per - spatial attention evoked by emotion. In an ERP study ceptual features of an emotional stimulus. [63], laterally presented bursts of white noise and LED Corresponding ERP data for emotional spatial cueing flashes served as cues that could be presented alone or demonstrated that the fear-related attraction of attention together (on the same side). These unisensory or bimodal can result in larger validity differences of P1-activation cues preceded triangular shaped LED-flashes presented for targets presented after fearful or angry compared to at the corresponding (valid) or opposite (invalid) side, neutral facial or voice cues [8, 55, 56]. Further, the target- which had to be categorized as pointing up or down. related P3-component, a positive ERP peaking around There were no gains for bimodal versus unimodal cueing 300–400  ms after target onset over occipital-parietal presentations on target-processing neither behaviorally electrodes, has been observed to be inversely correlated nor in terms of any target-related ERP-effect. However, with the behavioral validity effects associated with the ERP-results indicated a clearly enhanced amplitude of two types of emotional cue content [21, 56, 73–75]. This the cue-elicited P1 over contralateral visual areas related inverse P3-modulation corresponds to a redirection of to the processing of bi- versus unimodal cues. According spatial attention to the target, when a target followed a to the authors, the discrepancy of target-related behavio- cue on the previously unattended side (cf. [46]). In sum- ral and cue-related ERP results could be explained by the mary, whereas spatial cueing effects tend to be enhanced fact that target discrimination follwing the bimodal cue by emotional cue content, redundant bimodal cueing was not demanding enough to require specific shifts of yielded a more complex pattern indicating ERP-effects spatial attention. Another study involving bimodal spa- such as a modulation of the P1-component which were tial cueing used bilateral presentation of visual cues (i.e., not always accompanied by corresponding behavioral the face of a cat located on one side, and the face of a effects (possibly depending on task demands as assumed dog located on the other side of the screen) and added by [63]). an animal sound, presented via a centrally located loud- Spatial cueing with peripherally presented cues is sen- speaker [47]. Follwing such a bimodal cue, a visual tar- sitive to the timing of stimulus presentation. Specifically, get (Gabor-patch) was unpredictively presented to 50% the validity effect tends to decrease and often reverse (i.e., on the side indicated by the cueing-picture matching the faster responses in invalid compared to valid conditions) cueing sound. The task consisted of easy and difficult ori - when the interval separating cue and target is extended, entation changes in the visual Gabor-patch targets. The a phenomenon referred to as Inhibiton of Return (IOR, authors found that responses to targets were faster to [54], see [37], for an overview). IOR is of particular rel- the side where the animal sound matched with the ani- evance for studies investigating cueing effects of auditory mal picture, but this was confined to trials featuring high features which cannot be identified instantly. In con - demands of target discrimination. This result prompted trast to a remarkably fast emotional recognition of visual the authors to suggest that bimodal enhancement of spa- facial expressions (17–100  ms; [72]) or identification of tial cueing depends on the amount of perceptual load. elementary auditory features, emotional voice recogni- On the other hand, several spatial cueing studies inves- tion requires temporal integration of stimulus properties tigated emotional shifts of spatial attention by using changing dynamically across an extended period of time unisensory or crossmodal (i.e., cue and target of different [64]. In practice, the duration of an auditory fear-related Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 3 of 14 voice needs to be considerably longer than 600  ms to going ERP-slow wave which has been found particularly ensure a high level of correct identification if the task sensitive to two types of stimulation. On the one hand, requires distinguishing it from other emotional or neu- the presentation of emotional stimuli of negative ver- tral voices [22, 53]. sus positive valence (e.g. [30, 36, 51, 59]) revealed an Despite of using a long stimulus duration of 1000  ms enhancement in LPC-ativity. On the other hand, stimu- for voice cues in a previous study [75], we observed typi- lus-conflict between neutral non-emotional stimuli, such cal cueing validity effects when visual targets followed as incongruent versus congruent color-word conjunc- neutral voice cues after a short interstimulus-interval tions in a Stroop task, also revealed an increased LPC for (ISI, i.e., time between cue offset and target onset). In the conflicting condition (e.g., [13, 18]). Comparing these contrast, validity effects after neutral voices reversed fol - components among our four cueing conditions allowed lowing a long ISI, possibly indicating IOR. These findings us to estimate both the effects of emotional content pre - suggest that (at least for auditory cues) a long ISI rather sented in a single modality and redundant “emotional than a long cue-target onset asynchrony may be crucial cueing” in both modalities (Fig. 1). for IOR to occur. Taken together, adding identifiable emotional content to an auditory cue requires a suffi - Results ciently long voice duration to enable content identifica - Behavioral results tion, which, however, does not itself evoke IOR-effects as Our main research interest was to investigate whether long as combined with short ISI (cf. [75]). audio-visual cues with bimodally fearful coloring would In spatial cueing paradigms, P1-activity as well as enhance attentional shifts toward their location com- N170-activity, time-locked to visually presented face pared to audio-visual cues with only a single modality cues, has often been increased when comparing emo- of fear-related display or a fully bimodal neutral display. tional facial expressions to neutral ones (P1: [15], N170: Participants were instructed to ignore any bimodal face- e.g., [11, 69], see [31] for a review). Importantly, the P1 voice cue and to attend only to the arrow targets. Due to increase associated with emotional cue content seemed problems with the presentation of the auditory cues in to be independent of the type of stimulus used as cues as the first participant, this first data set was excluded from a similar P1-enhancement was also found when compar- the further analyses. ing frightening versus neutral pictures of objects other The analysis of reaction times of trials associated with than faces (i.e., IAPS—pictures, [9, 38]). The N170 com - a correct response failed to yield a significant main effect ponent is a component typically observed in connection of VALIDITY (F(1, 24) = 0.013; p = 0.909; ηp = 0.00063). with processing face (-like) stimuli which originates from The interaction of EMOTIONAL CUE COMBI - the fusiform face area (FFA) [26]. The increase of N170 NATION and VALIDITY was significant (F(2.525, activity evoked by emotional content occurred indepen- 72) = 3.363; p = 0.024; ηp = 0.138). The follow-up dently of specific task demands (i.e., passive viewing vs. detection/discrimination) ([25, 60, 66, 71], see [31] for a review). So far, there seems to be no evidence for an addi- tional enhancement of the N170 evoked by emotional faces when accompanied by vocal utterances match- ing the displayed emotion. Conversely, angry vs. neu- tral vocal expressions, presented during face encoding, reduced P1 and N170 to matching facial expressions [43]. In the present study, we used a spatial cueing para- digm with bimodal face-voice cues that varied concern- ing their degree of emotional content; i.e., fearful in face and voice (double emotional cue), fearful voice/neutral face or fearful face/neutral voice (single emotional cue), or both neutral (neutral cue). The aim was to investigate whether double emotional face-voice cues attract atten- Fig. 1 Task paradigm. A schematic example of a valid stimulus tion to their location more strongly than single emotional sequence is shown. Bimodal face-voice cues consisted of either both cues or purely neutral cues, respectively. If true, double fearful, one fearful (paired with neutral) or both neutral contents. 50% emotional cues should elicit an enhanced validity effect of the targets followed each cue type on the same side (valid) and in behavior (reaction times, hit rates) as well as more pro- 50% on the opposite side (invalid). Participants were required to push nounced target-related ERP-components like P1, P3, and one of two buttons indicating the pointing direction (up/down) of the target triangle LPC (late positive component). The LPC is a positively Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 4 of 14 Bonferroni-corrected posthoc tests indicated that the participants responded significantly faster in the valid than the invalid condition when both face and voice were fear-related (t(24) = 3.208; p_bonf = 0.034), but there was no significant validity effect following any other emo - tional cue combination. In addition, the main effect of EMOTIONAL CUE COMBINATION was significant (F(2.083, 72) = 4.268; p = 0.019; ηp = 0.169). Bonferroni- corrected post-hoc tests demonstrated first that reaction times after the voice-fear/face-neutral cues were faster than after voice-neutral/face-fear cues (t(24) = 3.182; p_bonf = 0.014). Secondly, reaction times to targets after the voice-fear/face-neutral cues were also faster com- pared to targets after the both neutral face-voice combi- Fig. 3 Early lateralized target activity: P1: Left panel: ERP-traces time-locked to the target over occipital O1/2-electrodes, the nations (t(24) = 2.988; p_bonf = 0.024) (Fig. 2). contra-versus-ipsi-lateral P1 amplitudes indicated a significant Considering hit-rates, neither the interaction of the increase for all valid compared all invalid targets (see upper versus factors nor any single main effects were significant. lower part of the figure). In addition, for valid conditions (upper channel), the contra-ipsilateral P1 activity of targets after the ERP results double fear combination was significantly increased compared to targets after combinations of both neutral or of neutral-voice The main aim of the study was to test whether emotion with fearful face combination ( VN/FF). The comparison of the valid shifts spatial attention more strongly when presented in double fear combination with the valid fear-voice/neutral-face two instead of one modality. To test, bimodal face-voice combination ( VF/FN) only showed an increase by trend, but was not cues consisted either both of fearful expression, or only significant. Right Panel: Topographies indicate the corresponding one of a fearful expression, or both with a neutral expres- contra-versus-ipsi-lateral P1-differences separately for in/valid targets and its associated cue-type. Contra-versus-ipsi-lateral amplitude sion (control condition) (Fig. 2). differences are summarized over hemispheres Contra‑ versus ipsilateral target activity at P1 We analyzed the P1 amplitude as contra-ipsilateral activ- ity differences over occipital electrode positions. First of of VALIDITY (F(1,24) = 27.844; p < 0.001; ηp = 0.537) all, the 4 × 2–ANOVA EMOTIONAL CUE COMBINA- with larger P1-activity for valid compared to invalid TION (both fear, voice fear/face neutral, voice neutral/ cue combinations. The main effect of EMOTIONAL face fear, both neutral) and VALIDITY (invalid, valid) was CUE COMBINATION was not significant (F(2.928, calculated (see Fig. 3). It revealed a significant main effect 70.277) = 2.088; p = 0.109; ηp = 0.080). Importantly, Fig. 2 Behavioral results. Left panel: Reaction times. The interaction of cue-type with validity was significant, mainly due a significant validity effect with valid faster than invalid combinations in the both fearful condition, but there was a trend for a reversed validity effect in the neutral condition. The validity difference of the both fear condition was significantly larger than the inverted one in the both neutral condition. Right panel: Hit-rates: Despite not being significant, the hit-rates mirrored a similar interaction, with improved performance after both fearful cues for valid targets, but for invalid after neutral targets. both fear face/voice both fearful, VF/FN voice fearful with neutral face, VN/FF voice neutral with fearful face, both neutral face/voice both neutral) Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 5 of 14 the interaction of both factors was significant (F(3, 72) = 2.888; p = 0.038; ηp = 0.169). Due to further inves- tigation of the significant interaction, two follow-up ANOVA’s considered valid and invalid P1-activity respec- tively. Only the ANOVA for the valid conditions indicated a significant effect of EMOTIONAL CUE COMBINA - TION (F = (3,72) = 3.347; p = 0.040; ηp = 0.176; see Fig. 4). A Bonferroni-corrected posthoc test revealed that the contra-ipsilateral activity of validly cued targets after the both fear combination was significantly increased compared to targets after the both neutral face-voice combination (t(24) = 3.012; p_bonf = 0.020). A further Bonferroni-corrected posthoc-test showed in addition a significant increase of ipsilateral activity of validly cued targets when comparing the both fear combination to the neutral-voice with fearful face combination (VN/FF; t(24) = 3.206; p_bonf = 0.019). The corresponding post - hoc comparison of targets after the both fear combina- tion versus targets after the fear-voice with neutral-face (VF/FN) combination was not significant, despite of showing a trend (t(24) = 1.425; p_bonf = 0.083). Centralized target activity at P3a and P3b We analyzed the modulation of P3a amplitude at the cen- tral-parietal ROI (Cz/Fp1/Fp2). Neither the interaction of EMOTIONAL CUE COMBINATION and VALIDITY (F(2.391,72) = 1.970; p = 0.126; ηp = 0.076) nor the main effect of EMOTIONAL CUE COMBINATION (F(1.833, 72) = 2.043; p = 0.145; ηp = 0.078) were significant. However, there was a significant main effect of VALID - ITY (F(1,24) = 13.375; p < 0.001; ηp = 0.358; Figs. 4, 5). Amplitude values at the P3b revealed a significant interaction of EMOTIONAL CUE COMBINATION by VALIDITY (F(2.27, 65.05) = 2.67; p = 0.049; ηp = 3.262). Fig. 4 ERP-traces of late centralized target activity time-locked to Follow up Bonferroni-corrected posthoc tests indicated the target over the central-parietal ROI. P3a: A significant main effect that validity effects were only significant in the neutral of validity was revealed, indicating that P3a-activity was increased for the invalid versus valid condition independent of cueing type. condition (t(24) = 3.088; p_bonf = 0.045), but in none of P3b: The interaction of validity by cueing type was significant. the other face-voice combinations (Figs.  4, 5). None of Posthoc-tests indicated that validity differences were only significant the other effects reached statistical significance. for the double neutral ( VN/FN) condition with a reverse effect of higher P3b amplitudes for the valid instead of invalid condition. LPC: Again, the interaction of validity by cueing type was significant. Centralized target activity at LPC Posthoc-tests indicated that validity differences were only significant Additionally, we analyzed the modulation of LPC ampli- for the double fear ( VF/FF) condition an increased LPC-amplitude for tude at the central- parietal ROI (Cz/Fp1/Fp2). The the invalid versus valid condition interaction of EMOTIONAL CUE COMBINATION and VALIDITY was not significant (F(3, 72) = 1.103; p = 0.354; ηp = 0.048). However, there was a signifi - combinations were not significant. Further, there was a cant main effect of EMOTIONAL CUE COMBINA - significant main effect of VALIDITY (F(1, 24) = 4.233; TION (F(2.155, 47.421) = 4.743; p = 0.012; ηp = 0.177) p = 0.049; ηp = 0.161). Importantly, Bonferroni-cor- with Bonferroni-corrected post-hoc tests indicating rected posthoc-tests indicated only a significant valid - that the both fear combination revealed significantly ity difference in the bimodal both fearful  condition higher LPC activation than the both neutral combina- (t(24) = 0.024; p_bonf = 0.024), whereas all other combi- tion (t(24) = 3.226; p_bonf = 0.023), whereas all other nations failed to reach significance. Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 6 of 14 Fig. 6 Cue activity (contra- versus ipsi): N170: upper panel: ERP-traces time-locked to the cue, the contra-versus ipsi-lateral N170 amplitudes were increased when face-voice combinations included a fearful face. Lower panel: Topographies over each hemisphere summarize the contra-versus-ipsi-lateral activation corresponding to N170 at 190-210 ms Fig. 5 ERP-topographies of the late centralized target activity showing validity differences of valid minus invalid conditions Discussion time-locked to the target over the central-parietal ROI. For P3a, a The current ERP-study investigated whether double emo - significant main effect of validity differences indicated an increased tional face-voice cues attract attention to their location P3a-activity for the invalid condition independent of cueing type (see more strongly than single emotional cues or purely neu- blue activity color). For P3b, validity effects were only significant for tral cues, respectively. To test, bimodal face-voice cues the double neutral condition ( VN/FN) and indicated a reverse validity difference with increased valid activity (see orange activity color). that varied concerning their degree of emotional content; For LPC, validity differences were only present in the double fear i.e., fearful in face and voice (both fearful), fearful voice/ condition ( VF/FF) increasing for the invalid versus valid condition neutral face or fearful face/neutral voice (single emo- tional cue), or both neutral (both neutral cue) preceded a visual target. Considering reaction times, the num- ber of cue components related to fear (both, one, none) Cue‑related N170 activation interacted with cue validity. This was mainly due to a Specifically asking for spatial effects due to the varying pronounced validity effect in the both fearful condition. emotional presence during the cue-target interval, we Target related ERP-effects started with a contra-versus- analyzed contra-versus-ipsilateral N170 activity differ - ipsi- lateralized P1 increase for valid versus invalid con- ences over occipital-parietal electrode positions (P7/ ditions which was largest for the both fearful condition. P8) separately for each emotional cue combination. The early P3a showed a main effect of increased activity ANOVA results showed a significant main effect of in invalidly cued compared to validly cued trials, inde- emotional cue combination (F(3, 72) = 2.807; p = 0.046; 2 pendently of presence and quantity of emotional content. ηp = 0.105 see Fig.  6). Bonferroni-corrected posthoc- In contrast, the P3b component was, again, differentially tests indicated that the N170 for the both-fear com- affected by the type of cue, displaying significantly larger bination was significantly more negative than for the amplitudes after valid than after invalid cues (i.e., revers- voice-fear with face-neutral combination (VF/FN; ing the P3a pattern) for all cueing conditions except for t(24) = 3.233; p_bonf = 0.012; see Fig.  6). Similarly, also the bimodal fearful condition. The former pattern of the face-fear with voice-neutral combination (VN/FF) results (i.e., P3a larger after valid than after invalid cue- revealed an increased N170 compared to the voice-fear ing, P3b larger after invalid than after valid cueing) thus with face-neutral combination (VF/FN; t(24) = 2.939; resembles the course of reaction times over the cue-tar- p_bonf = 0.045). Finally, the both-neutral combination get interval, that is, a regular validity effect after short was not significantly different when compared to any intervals which turns into a reversed validity effect other combination. Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 7 of 14 after longer periods. Viewed from this perspective, the (98.3%), this is a task of rather low perceptual demands. absence of the reversal effect in the P3b for the both fear - Instead, the results of the current study suggest that ful condition would be consistent with the assumption of bimodally presented fear-related content of spatial cues a reduced IOR mechanism. Finally, yet importantly, only may be sufficient to enhance the deployment of spatial in the both fearful condition, there was an increase of attention to the cued position compared to bimodal non- LPC for invalid compared to the valid condition, possi- emotional cues. bly indicating that evaluation of the emotional content is Cues of both neutral content indicated a slight non- enhanced by a mismatch of spatial positions. significant tendency for an opposite cueing effect of improved performance for invalid compared to valid Performance data targets. In the current study, the sound duration was set The present behavioral results indicated that emotional to 1000  ms to ensure the discrimination of the auditory content interacted with spatial validity. Regarding reac- content. Together with the current ISI of 190–240  ms, tion times, this interaction was mainly driven by a strong the SOA-time was therefore 1190–1240 ms. Considering validity effect in the both fearful condition, contrasting neutral sound cues of similar psychophysical structure with a descriptively slightly reversed validity effect of [75], we previously found typical spatial validity effects the neutral condition. Although the hit rate analysis did (i.e., improved performance to valid compared to invalid not yield any statistically significant interactions (which targets) with an interstimulus interval (ISI) of 50–150 ms, might not be surprising given the overall high level of but SOA of 1050–1150  ms due to the long sound dura- accuracy), the result displayed a trend of improved per- tion. Importantly however, neutral cueing effects formance after double fear cues for valid targets, and for inverted in an IOR-like manner (invalid improved com- invalid targets after neutral cues. Therefore, the interac - pared to valid) with an ISI of 650–750 ms, while an ISI of tion trend in the hit rates resembled the result pattern 350–450 ms indicated no cueing effects in neither direc - observed in reaction times. Enhancement of the cueing tion [75]. In the current study, the ISI was 190–240  ms, validity effect in the both fearful condition corresponds thus outside the time-range of typical validity effects for to previous findings using unimodal fearful cues. As pre - neutral cues as found in our previous study. It would thus viously shown, fearful facial expressions [72] as well as seem straightforward to assume that the interval admin- fear-related voice cues [74] enhanced performance differ - istered in the current study was too short to yield reli- ences in responding to targets presented at validly versus able IOR, while, on the other hand, sufficiently long for invalidly cued positions. Importantly, this cueing validity substantial undoing of the initial attention shift towards effect was found not only with short SOA for face cues the cued location, except for the bimodal fearful cueing (117 ms) but also with long SOA for voice cues (1200 ms), condition, in which disengagement of attention from the as long as the ISI was kept short, suggesting that the cued location was less feasible. occurrence of IOR depends on a long ISI. The results of the current study seem in line with this assumption as Validity differences for targets: ERP‑results there was no significant reversal of the validity effect— Lateralized contra‑vs‑ipsi P1‑effect Time-locked to tar- and a significant regular validity effect in the both fearful gets, contra-versus-ispi-lateralized P1-activity indicated condition—despite the SOA exceeded 1000 ms. an enhancement of valid versus invalid activity for all Further, effects of bimodal cues have so far only been conditions. Importantly however, targets after bimodal tested lacking emotional content [47, 63]. In the study fearful cues indicated a specifically large P1-validity dif - by Santangelo et  al. [63], bimodal cues evoked a statisti- ference due to a massive activity increase for validly cued cally non-significant behavioral enhancement of target targets. One the one hand, the general input of P1 for all processing compared to unisensory cues. In the study valid vs. invalid conditions fits well with a spatial bottom- of Mastrobernadino et  al. [47], enhanced validity effects up theory where early target processing is enhanced at were observed for bimodal cueing (i.e., correspondence positions preoccupied validly by spatial cues independent of the picture of a laterally presented animal face occur- of awareness or SOA duration [27, 54, 72]. For example, ring alongside another animal face on the opposite side in a spatial cueing paradigm, Giattino and colleagues [27] with a centrally presented sound) but these effects were used lateralized cues of short 17  ms duration hidden in confined to conditions of high task difficulty. While this a multiobject-presentation. Targets consisted of neutral may be the case for non-emotional cues, the results of quadratic shapes at valid or invalid positions of the cues. our study show that this is not the case for emotional Independently of whether participants consciously per- material. Our task consisted of an easy discrimination of ceived the cues or not, validly cued targets always evoked a lateralized arrow pointing up- or downwards. Accord- higher P1-activity over occipital areas than invalidly cued ing to the overall high accuracy rates of our participants targets. In addition, in a spatial cueing study with facial Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 8 of 14 anger cues and neutral targets, Liu et  al. [44] found that sus invalid position, the early spatial P3a-effects may also target-related P1-activity increased when validly versus point to automatic early bottom-up processes. invalidly cued. Similarly, Brosch and colleagues [8] found increased P1 for valid vs. invalid targets lateralized to side Central P3b‑late effect From 325 to 375  ms, the both of the fearful facial expression when double facial cues neutral condition and the fearful face with neutral voice with one fearful and one neutral expression were pre- condition (VN/FF) revealed validity effects with increased sented. In addition, P1-valdity differences were also found valid versus invalid activity. Firstly, considering the above- in crossmodal spatial cueing when an auditory anger voice mentioned P3a, these P3b validity-effects were inverted cued a visual neutral target [7]. All these P1-validity dif- in polarity as they now indicated more activation for valid ferences occurred early (around 100  ms) and showed a than invalid targets. This might indicate a disturbance by strong responsiveness of the spatial location of the target the valid instead of invalid spatial position and fits well in relation to the preceding position of the emotional cue with the current behavioral data indicating an IOR-ten- [8, 72]. These effects might therefore mirror a bottom-up dency for the fully neutral condition and VN/FF condition. process of sensory processing [8, 72]. Our current finding Secondly, IOR-effect with neutral stimuli can be found supports this hypothesis by extending previous data with over occipital-parietal electrodes at P3-time intervals [48, a specifically contra-vs-ipsilateral P1-validity difference 58, 75], indicating that attention needed to be boosted at and its enhancement by bimodally double fearful stimula- the validly instead of shifting away to the invalidly cued tion. and IOR-conform position. Thirdly, some recent research [39, 62] suggested that a late P3-activity mirrors conscious Central P3a‑effect ERP-data time-locked to targets, P3a- perception. Thus, the relatively long delay of target-onset activity increased for invalid versus valid double combi- after the neutral and mixed VN/FF-cues might support nations independent of condition. uTh s, independent an expectancy effect of target processing at the invalid of emotional presence or absence, spatial validity corre- position. In conclusion, the validity inversion of the late sponded inversely to the P3a-activity. With spatial cueing P3, respectively P3b, seems to fit well with the behavioral the P3a has been so far mainly investigated with predic- occurrence of a probable IOR-effect that is missing when tive designs (i.e., 75% valid vs. 25% invalid conditions), full emotional bimodal integration is required. where the low probability of invalid stimuli increases their novelty character of spatial position [16, 24, 28, 29]. How- LPC‑activity At around 500 ms time-locked to targets, ever, while the P3a validity differential activation might a positive slow wave (late positive component; LPC) be larger in predictive cueing, it was still present in the showed validity differences exclusively in  the bimodal current unpredictive cueing design (50% valid/invalid). both fearful combination again with an enhanced inva- The reason might be that even in unpredictive cuing an lid versus valid activity. This finding seems to combine invalid target position still needs to be spatially updated different views on the LPC (late positive component); a compared to the preceding cue position, thus leading to component that is sometimes thought to include the late increased P3a activation for invalid versus valid targets. positive potential (LPP; [12] for a review). Firstly, the Similarly, Lasaponara and colleagues [40] found increased LPC might be considered as a slow wave conflict poten - invalid P3a activity when using only a 50% ratio of valid tial [13, 18, 50]. For example, the LPC-activation appears stimulation compared to the sum of invalid and no-cue in non-emotional Stroop-tasks as well as non-emotional conditions. Considering the modulation by emotional Flanker-tasks and mirrors the conflict in semantic content in spatial cueing, there is often mentioned only meaning (Stroop) of an incongruent color-word-com- a general P3-activity without subdivisions of P3a and bination relative to congruent color-word-combination P3b [21, 44, 73, 75]. These emotional ERP-studies mainly respectively the perceptive conflict (Flanker) of incon - revealed P3 validity differences that increased in positive gruent versus congruent target-flanker combinations activity for invalid versus valid target stimuli when cued [1, 13, 18, 42, 70]. Secondly, the LPC/LPP is handled with fearful faces [21, 44]. Importantly, when targets were as an indicator of negative emotional processing [30, preceded by disgust sounds, such P3 effects inverted inde - 36, 51, 59]. In this sense, the LPC/LPP increased over pendently of the ISI-length, indicating an increased valid centro-parietal scalp regions for negative versus neutral versus invalid activity, which corresponded to early shifts emotional stimuli, when ERP-studies required a simple of attention away from the disgust evoking stimulus and emotional discrimination of a centrally presented pic- may therefore relate to an automatic early bottom-up pro- ture [30, 36, 51, 59]. In addition, a central cueing study cess of emotional context [73, 75]. As fearful like neutral [33] indicated validity effects in the same direction as stimulation supports processing of targets at the valid ver- the current study, e.g., increased LPP-activity after invalidly compared to validly cued negative emotional Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 9 of 14 pictures. In the aforementioned studies, the conflicting unspecifically enhance attention in general, at least at or negative enhanced LPC was time-locked to the onset this early time of processing. of the emotional stimulus itself. In contrast, the LPC difference in the present data was found time-locked Limitations to the presentation neutral target, importantly however Contrasting the spatial effects of fear and neutral content when following double negative cues and indicating spa- of bimodal cues, this study consisted of a 4 × 2 stimulus tial conflict. In addition, perception of stimulus conflict paradigm and thus eight conditions (e.g., even 16 condi- might yield aversive states ([17], for a review),  and fits tions when splitting into ipsi- and contralateral activity). therefore well to the just cued fearful bimodal percep- To ensure a relatively high signal-to-noise ratio of the tion. More specifically, the LPP/LPC complex is thought ERP-data (see [6]), we decided to confine experimental to be influenced by top-down processes (e.g., reinter - manipulations to a single ISI and a single type of emo- pretation of negative stimuli) and reflecting endogenous tional content. This resulted in two noteworthy caveats. spatial attention (cf. [12], for a review). Thus, our data First, adding a second even longer ISI might have yielded might integrate and extend present LPC-theories by stronger evidence for our interpretation of a delay of dis- showing that targets, invalidly cued by fearful bimodal engagement of attention from the cued location in the stimuli, are associated with enhanced spatial conflict. both fearful condition (i.e., if consistent IOR for all con- ditions could be observed with a sufficiently long SOA). Cue‑related N170‑activity Time-locked to cues, the Second, it should be stressed that our experiment was facial related N170 component was always enhanced confined to the investigation of one particular emotion when the facial cue expressed fear—independent of the (i.e., fear) and that the results may not transfer to visual- accompanying sound expression. Investigating bimodal auditory cues related to other emotions. As noted in the effects of neutral face-voice combinations, Latinus Introduction, studies of spatial cueing featuring disgust- and colleagues [41] found that the N170 did not differ related cues yielded a reversed validity effect (i.e., supe - between purely facial and combined facial-voice stimuli. rior performance in invalidly cued trials), consistent Considering emotional influence, purely visual spatial with a repelling or dispersing effect on spatial attention cueing studies found that N170 activity increased for (e.g., [44, 72, 74]). Assuming that the validity effect in the fearful versus neutral facial spatial cues independently of bimodal fearful condition of the current study reflects a whether their emotional content was task-relevant [11, reduction in attentional disengagement counteracting 69] or task-irrelevant ([15]; see also [31], for a review). IOR does not seem to allow predictions for situations in In contrast, some facial studies (e.g., [8, 55]) used a dou- which attentional engagement is absent in the first place. ble facial cue with one emotional and one neutral face. In this case, emotional enhancements at N170 might be rather hard to detect or at least weakened by nec- Conclusions essary subtraction analysis to extract fear-related acti- The present behavioral and ERP-data compared for the vation. In addition, asking for overt discrimination  of first  time spatial cueing effects elicited by bimodal face- facial emotion or gender, emotional discrimination also voice cues of varying emotional content (both fearful, revealed an enhanced N170 for fearful compared to face fearful with voice neutral, face neutral with fear- neutral expressions [3, 4, 61], but see [19] for no N170 ful voice, both neutral). Typical spatial cueing effects differences). So far, bimodal emotional differences of with faster reaction times to validly cued targets than to N170 have only been investigated with matching emo- invalidly cued targets selectively occurred in the both- tional face-voice combinations (e.g., bimodal fearful or fearful condition. This behavioral effect corresponded bimodal neutral), revealing that N170 was enhanced to P1-validity differences of target processing, that were for bimodally fearful compared to bimodally neutral largest in the both-fearful condition, as well as to a later face-voice combinations [68]. Only an increase in the LPC-validity difference, which was exclusively present N170 amplitude for angry versus neutral faces, how- in trials with bimodal fearful cues. In contrast, an early ever independently of voice, was found. Therefore, the general main effect of validity in the P3a component indi - present finding extends the results of previous studies, cated that spatial attention was initially directed towards by emphasizing the importance of facial expressions for the validly cued spatial position even in trials involving the N170, and its independence of the accompanying both neutral or single emotional types (albeit less pro- voice content. A possible explanation might relate to the nounced than in the both fearful condition). The pattern visual discrimination demands of the experimental set- of the P3b component suggested that spatial attention ting, where movement of the attentional focus is domi- was re-directed, but to a lesser extent in the bimodal nated by visual information and an added voice might both-fearful condition. In conclusion, our present data Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 10 of 14 therefore suggest that, at least when using long SOAs consisted of either two fearful expressions or two neu- with short ISI, only double fearful stimulation prevented tral expressions, whereas single emotion combinations or at least delayed the disengagement of spatial attention included one modality fearful and the other neutral. Dur- from an invalidly cued position. Single emotional stimu- ing the entire EEG-experiment, a fixation cross was pre - lation, by contrast, did not counteract such disengage- sented at the center of the screen to ensure exogenous ment and presumably the initiation of IOR, compared to and peripheral stimulation. Each trial started with the completely neutral stimulation, in a noticeable manner. presentation of a bimodal face-voice cue for 1000 ms. On Future bimodal cueing studies might corroborate this the offset of the face-voice cue followed a jittered cue-to- conclusion by extending the length of the ISI. Selective target-interval of 190-240 ms. Then a visual neutral target reduction of attentional disengagement from a cued loca- (a little white arrow pointing up or down) was presented tion after bimodal emotional cues should evidence itself for 100 ms. After an intertrial interval of 2–3 s, the next in selective absence or delay of IOR in this condition (see trial started (see Fig.  1). The task of the participants was [23], for a similar approach concerning the impact of to ignore the bimodal face-voice cues and to respond to emotional cue content in general). the pointing direction (up/down) of the target. All partic- ipants completed 10 runs. Each run consisted of 60 trials Methods and lasted about 3.5 min, leading to a total experimental Participants run time of about 35 min. Planning of sample size was based on previous, compa- rable spatial cueing studies that investigated the modu- Stimuli latory effects of emotion [73, 75]. As behavioral effects The emotional cueing voice was either a fearful voice were not evident in all studies [63], P1 differences (someone fearfully screaming) or a neutral voice (some- between valid and invalid trials for both fearful face-voice one biting into an apple). These voices were already used cues were considered the relevant endpoint. Indeed, a and rated in our previous studies (e.g. [73, 74]). In our post-hoc analysis with G*Power [20] showed that the previous study [74], the fear voice was rated as clearly achieved power in our study was 1-β = 0.99 given the negative (valence) with high arousal, whereas the neu- found difference between P1 peaks (two-tailed t-test, tral voice was rated as neutral (valence) with low arousal. α = 0.05, dz = 1.04) Provided the found effect size in this Each of the two voice stimuli had a duration of 1000 ms. study (dz = 1.04), analysis with G*Power indicated that a This duration guaranteed that the emotional content of sample size of 10 participant is sufficient to detect a sig - the voice was fully processed by the participants before nificant difference with 1-β = 0.80, α = 0.05. the presentation of the target (e.g., [52]) and was also Twenty-six participants (12 men, M = 23.4  years, chosen in our previous studies [73, 74]. The overall sound age SD = 4.5) took part in the ERP-experiment. All were level was normalized to 68  dB SPL for both emotional right-handed, had normal or corrected-to-normal vision, voices. To preserve the emotional character of the voices, normal hearing abilities and had no history of psychiat- the time–frequency structures of both voices were not ric or neurological disease. One of these participants was changed (cf. for a similar procedure with happy/sad emo- excluded from the final data analysis due to poor qual - tional voices: see [2, 34, 35]). For lateralized presentation, ity of the EEG signal (excessive movement and drifts). the originally stereo-recorded voices (someone fearfully All participants were students of the MSH and received screaming, someone biting into an apple) were converted study-participant-points. They gave written informed into mono-channel sounds by using “Au Adobe Audi- consent according to the ethical standards laid down in tion” (http:// www. adobe. com). During ERP/EEG, these the Declaration of Helsinki (BMJ 1991; 302; 1194). The mono-channel sounds were then delivered using the study was approved by the Ethics committee of the MSH. software Presentation (neurobehavioral systems; http:// www. neuro bs. com) to either the left or the right loud- Materials and procedure speaker. Both loudspeakers were positioned behind the We used a spatial cueing paradigm with bimodal face- right and left side of the screen to induce an associa- voice cues to investigate if and how congruent emo- tion between voice and target location. We did not use tional content in two modalities would lead to stronger headphones due to their interference with the EEG-cap engagement to the cue and therefore to enhanced target and -measurements. Similarly, the emotional faces con- processing when compared to single emotional pres- sisted either of a fearful or neutral expression. After the ence in only one modality or purely neutral combina- EEG-measurements, participants described the facial tions. Importantly, face-voice combinations were always expressions in their own words and applied ratings of bimodal cues, consisting of a facial expression with a emotional valence and arousal. First, they were asked to non-verbal voice expression. Congruent combinations name the facial expression (open answer). Indeed, the Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 11 of 14 fearful expression was described with fear-related words VALIDITY (valid, invalid). Greenhouse–Geisser correc- (e.g., “fearful”, “frightenend”, “horrified”, “panicking”) by tion was applied to all statistical comparisons for which 88% of the participants (22 out of 25) as well as the neu- the Mauchly test indicated violation of sphericity. In case tral expression with non-emotional related words (e.g., of significant interaction, follow-up posthoc Bonferroni “neutral”, “bored”, “well-balanced”) by 88% of the partici- corrected t-tests were used to indicate improvements pants. Secondly, participants rated valence and arousal versus deterioration between single conditions. Signifi - of the facial expressions on a scale ranging from 1 to 5 cance was inferred for corrected p-values < 0.05. (from lowest arousal/negative valence to highest arousal/ positive valence). For statistical analyses, the ratings for ERP‑data analysis valence and arousal, respectively, were averaged across The custom ERPSS software (Event-Related Potential participants for each facial expression. Paried t-tests indi- Software System, UCSD, San Diego, CA, USA, cf. [45] cated more negative valence (t(24) = 14.000; p < 0.001) was added-on to the open source EEGLAB software (an and higher arousal (t(24) = 6.385; p < 0.001) for the fearful open source environment for electrophysiological sig- than for the neutral expressions. To ensure fully bimodal nal processing, UCSD, San Diego, CA, USA, cf. [14] and stimulation during the EEG-experiement, onset and off - served as software for all ERP-analyses. ERP-analyses set of sounds and faces was simultaneously. started by transforming the raw-data of the 10 runs of each participant into ERPSS format and combining them EEG recording into one large data-file for further analysis. To avoid The BrainVision Recorder (Brain Products, Germany biasing to any electrode position, the reference for each http:// www. brain produ cts. com/) was used for record- of the 32 electrodes was set to the algebraic average of ing EEG/ERP-data in combination with a 32 channels all channels. For cues as well as targets, the continuous electrode cap of the electric shielded EEG-system “acti- EEG data were divided into 800  ms epochs, time-locked CAP snap”, (Brain Products, Germany). According to to the relating onsets including a pre-stimulus baseline the 10–20 EEG system, the 32-electrodes included seven of 200  ms. To discard epochs contaminated by large eye frontal electrodes (FP1, FP2, F7, F3, FZ, F4, F8), four movements, excessive muscle activity, drifts, or amplifier fronto-central electrodes (FC5, FC1, FC2, FC6), nine blocking, an artifact rejection was performed by reject- central-temporal electrodes (T7, C3, Cz, C4, T8, CP5, ing any voltage amplitudes under −  100  µV and over CP1, CP2, CP6), four lateral electrodes (FT9, FT10, TP9, 100  µV. Resulting artifact-free EEG epochs were aver- TP10) and eight parietal-occipital area electrodes (P7, aged together, separately for the various trial types (i.e., P3, PZ, P4, P8, O1, OZ, O2). All 32 electrodes were ref- for cues: right receptively left sided cues for both fear/ erenced to the algebraic average of all channels and were voice-fear with face-neutral (VF_FN)/voice-neutral with therefore unbiased to any electrode position. The ground face-fear (VN_FF)/both neutral; for targets: invalid, electrode was placed between electrode position Fp1 and receptively valid trial types corresponding to the four Fp2. Electrode impedances were kept below 10 kΩ for all emotional conditions time-locked to the visual target). electrodes. A dimly lit, sound-attenuated and electrically As a final step, all ERP-averages were digitally low-pass shielded chamber was used for EEG-recording. filtered (IIR-Butterworth) with a running-average filter of 30 Hz. Behavioral data The aim of our study was to investigate if and how Our main interest was to find evidence for enhancement bimodally presented double emotional stimulation (face of spatial cueing effects by emotional coloring in two ver - and voice both fearful) enhances reactions compared sus one or none sensory modalities of a bimodal audio- to bimodal audio-visual stimulations with fear solely in visual cue. Only trials with behavioral responses between one of two sensory modalities. Target-related analyses 200 and 1000 ms after the presentation of the arrow-tar- focused on the early spatial P1-component as well as get were considered for further behavioral analysis (98.3% the later P3-component as indicator of attentional shifts trials in total). For all cue conditions, reaction times (RTs) [73, 75]. Further, the LPC-component was analyzed as an to correctly judged arrow-targets as well as accucary rates emotional and conflict indicator [33, 65]. For early target were analyzed. Using the statistic program JASP version activity, we focused on contra-ipsilateral P1-differences at 0.16.1.0 (https:// jasp- stats. org/), for each performance O1/O2-electrodes similarly to McDonald and colleagues measure (RTs respectively accuracy rates), a repeated- (2013). Latency time window of 110–130  ms served as measures 4 × 2 ANOVA was performed with the within- the P1 for all subsequent analyses of emotional varia- subject factors EMOTIONAL CUE COMBINATION tions in contra-ipsi differences. Contra-ipsi analysis was (both fearful, voice-fearful with face neutral (VF_FN), performed by using the contra-ipsi function of erplab voice-neutral with face-fear (VN_FF, both neutral) and [45]. This function first averaged each condition’s contra Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 12 of 14 activation of right-sided stimulation at the left electrode fear/face neutral, voice neutral/face fear, both neutral) O1 with its contra activation of left-sided stimulation at and VALIDITY (invalid, valid) was calculated, this time the right O2. Similarly, the ipsi-averages for each condi- however including ERP-activation averaged over left and tion were calculated. In a second step, distraction of con- right target stimulation. For the N170 cue-locked activ- tra-versus ipsi-averaged stimulations was performed (cf. ity, we analyzed whether and how the N170-amplitude [45]). of contra versus ipsi-stimulations differed between the For the P3 analysis, spatial cueing literature as well as stimulations of varying fearful content (cue with: both our own studies had indicated that spatial cuing effects fear/voice-fear with face-neutral (VF_FN)/voice-neutral on visual targets are usually found over central-parieto- with face-fear (VN_FF)/both neutral). To test for dif- occipital electrode sites (for emotional cues: [44, 57, 73, ferences in contra-ipsi activity at the N170, a repeated 75], for neutral cues: [46, 49]). Thus, electrode positions measure ANOVA including the factor EMOTIONAL located symmetrically over the visual-parietal cortex CUE TYPE with the 4 repeated measures of the cue type areas (Cz/Fp1/Fp2) were combined to a parietal-occipi- (both fear/voice-fear with face-neutral (VF_FN)/voice- tal region of interest (ROI). To select the time windows neutral with face-fear (VN_FF)/both neutral) was per- for theP3, in each subject, the average of all target con- formed. For all analyses, significance was set to p < 0.05. ditions (i.e., averaged over validity and emotion appear- Further, in case of significant results in the interaction ance) indicated two peaks: a P3a-peak at around 260 ms of an ANOVA, subsequent posthoc-tests were added to and a later P3b peak at around 350 ms. For further analy- indicate the cause of significance. Greenhouse–Geisser ses, these peaks were chosen for P3a and P3b with plus/ corrections was applied to all statistical comparisons for minus 20  ms time windows. Also for the LPC-analyses, which the Mauchly Test indicated violation of sphericity. the ROI of the visual-parietal cortex areas (Cz/Fp1/Fp2; Acknowledgements cf. [33]) was used. The time-window for the LPC was We thank Mortatha Al Khafage for assistance in collecting the current set to 480–600  ms, thus covering temporal LPC of pre- EEG-data. vious literature [33, 65]. For the statistical analysis, we Author contributions used mean amplitude values computed over the length of UZ planned and designed the experimental task. UZ analyzed the data with these intervals. input by MW and MP. UZ and MW wrote the manuscript with intensive feed- back of MP. All authors read and approved the final manuscript. Cue-related analysis focused specifically on the ERP- component N170 as related to facial processing. The Funding N170 is generally located over visual-parietal areas Open Access funding enabled and organized by Projekt DEAL. including electrodes P7/8, with N170 around 170  ms Availability of data and materials (e.g., [11, 69]; see [31] for a review). To test for attentional The datasets used and analyzed during the current study are available from facial effects due to lateralized cue presentation, N170 the corresponding author on reasonable request. differences were investigated by using contra-ipsi-differ - ences. Contra-ipsi analysis was again performed by using Declarations the contra-ipsi function of erplab [45], however, this time Ethics approval and consent to participate comparing N170 ipsi/contra-activity of the left electrode This research was approved by the Ethics Committee of the Medical School P7 with the corresponding ipsi/contra-activity of the Hamburg MSH. Before starting the experimental task, participants received information about the purpose of the study, the task, and its duration and right electrode P8. gave their written informed consent. For all statistical analyses of the target-related P1- / P3- as well as cue-related N170 components, we used Consent for publication Not applicable. mean amplitude values computed over the length of the corresponding time-window intervals. All statisti- Competing interests cal ERP-analyses were performed by using JASP version The authors declare that they have no competing interests. 0.16.1.0 (https:// jasp- stats. org/). For target-activity, two Author details main analyses were performed separately. For the spatial 1 Faculty of Human Sciences, Department of Psychology, MSH Medical School P1-compenent, an repeated measurement 4 × 2 ANOVA Hamburg, Hamburg, Germany. ICAN Insitute of Cognitive and Aec ff tive Neuroscience, MSH Medical School Hamburg, Hamburg, Germany. F acult y included contra-versus-ipsi-activation for the factor of Psychology, Department of Biopsychology, Institute of Cognitive Neurosci- EMOTIONAL CUE TYPE (fear/voice-fear with face- ence, Ruhr-University Bochum, Bochum, Germany. neutral (VF_FN)/voice-neutral with face-fear (VN_FF)/ Received: 29 October 2021 Accepted: 27 August 2022 both neutral) and the factor VALIDITY (valid/invalid). To test for differences in both P3-peaks and LPC activa - tion, for each component a similar 4 × 2—ANOVA with the factors EMOTIONAL CUE TYPE (both fear, voice Zimmer  et al. Behavioral and Brain Functions (2022) 18:10 Page 13 of 14 References 22. Folyi T, Wentura D. Fast and unintentional evaluation of emotional 1. Appelbaum LG, Meyerhoff KL, Woldorff MG. Priming and backward sounds: evidence from brief segment ratings and the affective Simon influences in the human brain: processing interactions during the task. Cogn Emot. 2017;31(2):312–24. https:// doi. org/ 10. 1080/ 02699 931. stroop interference effect. Cereb Cortex. 2009;19(11):2508–21. https:// 2015. 11105 14. doi. org/ 10. 1093/ cercor/ bhp036. 23. Fox E, Russo R, Dutton K. Attentional bias for threat: evidence for delayed 2. Banse R, Scherer KR. Acoustic profiles in vocal emotion expression. J disengagement from emotional faces. Cogn Emot. 2002;16(3):355–79. Pers Soc Psychol. 1996;70:614–36. https:// doi. org/ 10. 1037/ 0022- 3514. https:// doi. org/ 10. 1080/ 02699 93014 30005 27. 70.3. 614. 24. Friedman D, Cycowicz YM, Gaeta H. The novelty P3: an event-related 3. Batty M, Taylor MJ. Early processing of the six basic facial emotional brain potential (ERP) sign of the brain’s evaluation of novelty. Neurosci expressions. Brain Res Cogn Brain Res. 2003;17(3):613–20. https:// doi. Biobehav Rev. 2001;25(4):355–73. https:// doi. org/ 10. 1016/ s0149- 7634(01) org/ 10. 1016/ s0926- 6410(03) 00174-5. 00019-7. 4. Blau VC, Maurer U, Tottenham N, McCandliss BD. The face-specific N170 25. Frühholz S, Jellinghaus A, Herrmann M. Time course of implicit process- component is modulated by emotional facial expression. Behav Brain ing and explicit processing of emotional faces and emotional words. Biol Funct. 2007;3:7. https:// doi. org/ 10. 1186/ 1744- 9081-3-7. Psychol. 2011;87(2):265–74. https:// doi. org/ 10. 1016/j. biops ycho. 2011. 03. 5. Blurton SP, Greenlee MW, Gondan M. Cross-modal cueing in audio- 008. visual spatial attention. Atten Percept Psychophys. 2015;77(7):2356–76. 26. Gao C, Conte S, Richards JE, Xie W, Hanayik T. The neural sources of N170: https:// doi. org/ 10. 3758/ s13414- 015- 0920-7. understanding timing of activation in face-selective areas. Psychophysiol- 6. Boudewyn MA, Luck SJ, Farrens JL, Kappenman ES. How many trials ogy. 2019;56(6): e13336. https:// doi. org/ 10. 1111/ psyp. 13336. does it take to get a significant ERP effect? It depends. Psychophysiol- 27. Giattino CM, Alam ZM, Woldorff MG. Neural processes underlying the ori- ogy. 2018;55: e13049. https:// doi. org/ 10. 1111/ psyp. 13049. enting of attention without awareness. Cortex. 2018;102:14–25. https:// 7. Brosch T, Grandjean D, Sander D, Scherer KR. Cross-modal emotional doi. org/ 10. 1016/j. cortex. 2017. 07. 010. attention: emotional voices modulate early stages of visual processing. 28. Gómez CM, Flores A, Digiacomo MR. Increased brain activation during J Cogn Neurosci. 2009;21(9):1670–9. https:// doi. org/ 10. 1162/ jocn. 2009. the processing of spatially invalidly cued targets. Open Neuroimag J. 21110. 2008;2:106–13. https:// doi. org/ 10. 2174/ 18744 40000 80201 0106. 8. Brosch T, Pourtois G, Sander D, Vuilleumier P. Additive effects of 29. Gómez CM, Flores A, Digiacomo MR, Vázquez-Marrufo M. Sequential P3 emotional, endogenous, and exogenous attention: behavioral and effects in a Posner’s spatial cueing paradigm: trial-by-trial learning of the electrophysiological evidence. Neuropsychologia. 2011;49(7):1779–87. predictive value of the cue. Acta Neurobiol Exp. 2009;69(2):155–67. https:// doi. org/ 10. 1016/j. neuro psych ologia. 2011. 02. 056. 30. Hajcak G, MacNamara A, Foti D, Ferri J, Keil A. The dynamic allocation of 9. Bublatzky F, Schupp HT. Pictures cueing threat: brain dynamics in attention to emotion: simultaneous and independent evidence from viewing explicitly instructed danger cues. Soc Cogn Aec ff t Neurosci. the late positive potential and steady state visual evoked potentials. Biol 2012;7(6):611–22. https:// doi. org/ 10. 1093/ scan/ nsr032. Psychol. 2013;92(3):447–55. https:// doi. org/ 10. 1016/j. biops ycho. 2011. 11. 10. Buchtel HA, Butter CM. Spatial attention shifts: implications for the role 012. of polysensory mechanisms. Neuropsychologia. 1988;26:499–509. 31. Hinojosa JA, Mercado F, Carretié L. N170 sensitivity to facial expression: a 11. Carlson JM, Reinke KS. Spatial attention-related modulation of the meta-analysis. Neurosci Biobehav Rev. 2015;55:498–509. https:// doi. org/ N170 by backward masked fearful faces. Brain Cogn. 2010;73(1):20–7. 10. 1016/j. neubi orev. 2015. 06. 002. https:// doi. org/ 10. 1016/j. bandc. 2010. 01. 007. 32. Jonides J. Voluntary versus automatic control over the mind’s eye’s move- 12. Carretié L. Exogenous (automatic) attention to emotional stimuli: a ment. In: Long JB, Baddeley AD, editors. Attention and performance IX. review. Cogn Aec ff t Behav Neurosci. 2014;14:1228–58. Hillsdale: Erlbaum; 1981. p. 187–203. 13. Coderre E, Conklin K, van Heuven WJ. Electrophysiological measures 33. Johnen AK, Harrison NR. The effects of valid and invalid expectations of conflict detection and resolution in the Stroop task. Brain Res. about stimulus valence on behavioural and electrophysiological 2011;1413:51–9. https:// doi. org/ 10. 1016/j. brain res. 2011. 07. 017. responses to emotional pictures. Int J Psychophysiol. 2019;144:47–55. 14. Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of https:// doi. org/ 10. 1016/j. ijpsy cho. 2019. 08. 002. single-trial EEG dynamics including independent component analysis. 34. Juslin PN, Laukka P. Impact of intended emotion intensity on cue utiliza- J Neurosci Methods. 2004;134(1):9–21. https:// doi. org/ 10. 1016/j. jneum tion and decoding accuracy in vocal expression of emotion. Emotion. eth. 2003. 10. 009. 2001;4:381–412. 15. Denefrio S, Simmons A, Jha A, Dennis-Tiwary TA. Emotional cue validity 35. Juslin PN, Laukka P. Communication of emotions in vocal expression effects: the role of neurocognitive responses to emotion. PLoS ONE. and music performance: different channels, same code? Psychol Bull. 2017;12(7): e0179714. https:// doi. org/ 10. 1371/ journ al. pone. 01797 14. 2003;12:770–814. 16. Dien J, Spencer KM, Donchin E. Localization of the event-related 36. Kappenman ES, MacNamara A, Proudfit GH. Electrocortical evidence for potential novelty response as defined by principal components rapid allocation of attention to threat in the dot-probe task. Soc Cogn analysis. Brain Res Cogn Brain Res. 2003;17(3):637–50. https:// doi. org/ Aec ff t Neurosci. 2015;10(4):577–83. https:// doi. org/ 10. 1093/ scan/ nsu098. 10. 1016/ s0926- 6410(03) 00188-5. 37. Klein RM. Inhibition of return. Trends Cogn Sci. 2000;4(4):138–47. https:// 17. Dignath D, Eder AB, Steinhauser M, Kiesel A. Conflict monitoring and doi. org/ 10. 1016/ s1364- 6613(00) 01452-2. the affective-signaling hypothesis—an integrative review. Psychon Bull 38. Kuniecki M, Pilarczyk J, Wichary S. The color red attracts attention in Rev. 2020;27(2):193–216. https:// doi. org/ 10. 3758/ s13423- 019- 01668-9. an emotional context. An ERP study. Front Hum Neurosci. 2015;9:212. 18. Donohue SE, Appelbaum LG, McKay CC, Woldorff MG. The neural https:// doi. org/ 10. 3389/ fnhum. 2015. 00212. dynamics of stimulus and response conflict processing as a func- 39. Lamy D, Salti M, Bar-Haim Y. Neural correlates of subjective awareness and tion of response complexity and task demands. Neuropsychologia. unconscious processing: an ERP study. J Cogn Neurosci. 2009;21(7):1435– 2016;84:14–28. https:// doi. org/ 10. 1016/j. neuro psych ologia. 2016. 01. 46. https:// doi. org/ 10. 1162/ jocn. 2009. 21064. 035. 40. Lasaponara S, D’Onofrio M, Pinto M, Dragone A, Menicagli D, Bueti D, De 19. Eimer M, Holmes A. An ERP study on the time course of emotional face Lucia M, Tomaiuolo F, Doricchi F. EEG correlates of preparatory orienting, processing. NeuroReport. 2002;13(4):427–31. https:// doi. org/ 10. 1097/ contextual updating, and inhibition of sensory processing in left spatial 00001 756- 20020 3250- 00013. neglect. J Neurosci. 2018;38(15):3792–808. https:// doi. org/ 10. 1523/ JNEUR 20. Faul F, Erdfelder E, LangAG, Buchner A. G*Power 3: a flexible statistical OSCI. 2817- 17. 2018. power analysis program for the social, behavioral, and biomedical sci- 41. Latinus M, VanRullen R, Taylor MJ. Top-down and bottom-up modula- ences. Behavior research methods. 2007;39(2):175–91. https:// doi. org/ 10. tion in processing bimodal face/voice stimuli. BMC Neurosci. 2010;11:36. 3758/ bf031 93146https:// doi. org/ 10. 1186/ 1471- 2202- 11- 36. 21. Fichtenholtz HM, Hopfinger JB, Graham R, Detwiler JM, LaBar KS. Happy 42. Liotti M, Woldorff MG, Perez R, Mayberg HS. An ERP study of the tempo - and fearful emotion in cues and targets modulate event-related potential ral course of the Stroop color-word interference effect. Neuropsycholo - indices of gaze-directed attentional orienting. Soc Cogn Aec ff t Neurosci. gia. 2000;38(5):701–11. https:// doi. org/ 10. 1016/ s0028- 3932(99) 00106-2 2007;2(4):323–33. https:// doi. org/ 10. 1093/ scan/ nsm026. Zimmer et al. Behavioral and Brain Functions (2022) 18:10 Page 14 of 14 43. Lin H, Liang J. Contextual effects of angry vocal expressions on the 64. Schirmer A, Adolphs R. Emotion perception from face, voice, and touch: encoding and recognition of emotional faces: an event-related potential comparisons and convergence. Trends Cogn Sci. 2017;21(3):216–28. (ERP) study. Neuropsychologia. 2019;132: 107147. https:// doi. org/ 10. https:// doi. org/ 10. 1016/j. tics. 2017. 01. 001. 1016/j. neuro psych ologia. 2019. 107147. 65. Seib-Pfeifer LE, Gibbons H. Independent ERP predictors of affective prim- 44. Liu Y, Zhang D, Luo Y. How disgust facilitates avoidance: an ERP ing underline the importance of depth of prime and target processing study on attention modulation by threats. Soc Cogn Aec ff t Neurosci. and implicit affect misattribution. Brain Cogn. 2019;136: 103595. https:// 2015;10(4):598–604. https:// doi. org/ 10. 1093/ scan/ nsu094.doi. org/ 10. 1016/j. bandc. 2019. 103595. 45. Lopez-Calderon J, Luck SJ. ERPLAB: an open-source toolbox for the analy- 66. Smith ML. Rapid processing of emotional expressions without conscious sis of event-related potentials. Front Hum Neurosci. 2014;8:213. https:// awareness. Cerebral Cortex. 2012;22(8):1748–60. https:// doi. org/ 10. 1093/ doi. org/ 10. 3389/ fnhum. 2014. 00213.cercor/ bhr250. 46. Martín-Arévalo E, Chica AB, Lupiáñez J. No single electrophysiological 67. Spence C, Driver J. Audiovisual links in exogenous covert spatial orient- marker for facilitation and inhibition of return: a review. Behav Brain Res. ing. Percept Psychophys. 1997;59:1–22. 2016;300:1–10. https:// doi. org/ 10. 1016/j. bbr. 2015. 11. 030. 68. Stefanou ME, Dundon NM, Bestelmeyer P, Koldewyn K, Saville C, 47. Mastroberardino S, Santangelo V, Macaluso E. Crossmodal semantic Fleischhaker C, Feige B, Biscaldi M, Smyrnis N, Klein C. Electro-cortical congruence can affect visuo-spatial processing and activity of the fronto- correlates of multisensory integration using ecologically valid emotional parietal attention networks. Front Integr Neurosci. 2015;9:45. https:// doi. stimuli: differential effects for fear and disgust. Biol Psychol. 2019;142:132– org/ 10. 3389/ fnint. 2015. 00045. 9. https:// doi. org/ 10. 1016/j. biops ycho. 2019. 01. 011. 48. McDonald JJ, Ward LM, Kiehl KA. An event-related brain potential study of 69. Torrence RD, Troup LJ, Rojas DC, Carlson JM. Enhanced contralateral inhibition of return. Percept Psychophys. 1999;61(7):1411–23. https:// doi. theta oscillations and N170 amplitudes in occipitotemporal scalp regions org/ 10. 3758/ bf032 06190. underlie attentional bias to fearful faces. Int J Psychophysiol. 2021;165:84– 49. McDonald JJ, Störmer VS, Martinez A, Feng W, Hillyard SA. Salient sounds 91. https:// doi. org/ 10. 1016/j. ijpsy cho. 2021. 04. 002. activate human visual cortex automatically. J Neurosci. 2013;33(21):9194– 70. West R, Alain C. Eec ff ts of task context and fluctuations of attention on 201. https:// doi. org/ 10. 1523/ JNEUR OSCI. 5902- 12. 2013. neural activity supporting performance of the stroop task. Brain research. 50. McKay CC, van den Berg B, Woldorff MG. Neural cascade of conflict 2000;873(1):102–11. https:// doi. org/ 10. 1016/ s0006- 8993(00) 02530-0 processing: not just time-on-task. Neuropsychologia. 2017;96:184–91. 71. Williams LM, Palmer D, Liddell BJ, Song L, Gordon E. The “when” and https:// doi. org/ 10. 1016/j. neuro psych ologia. 2016. 12. 022. “where” of perceiving signals of threat versus non-threat. Neuroimage. 51. Myruski S, Bonanno GA, Cho H, Fan B, Dennis-Tiwary TA. The late positive 2006;31(1):458–67. https:// doi. org/ 10. 1016/j. neuro image. 2005. 12. 009. potential as a neurocognitive index of emotion regulatory flexibility. Biol 72. Zhang D, Liu Y, Wang L, Ai H, Luo Y. Mechanisms for attentional Psychol. 2019;148: 107768. https:// doi. org/ 10. 1016/j. biops ycho. 2019. modulation by threatening emotions of fear, anger, and disgust. Cogn 107768. Aec ff t Behav Neurosci. 2017;17(1):198–210. https:// doi. org/ 10. 3758/ 52. Paulmann S, Pell MD. Contextual influences of emotional speech prosody s13415- 016- 0473-9. on face processing: how much is enough? Cogn Aec ff t Behav Neurosci. 73. Zimmer U, Keppel MT, Poglitsch C, Ischebeck A. ERP evidence for spatial 2010;10:230–42. https:// doi. org/ 10. 3758/ CABN. 10.2. 230. attention being directed away from disgusting locations. Psychophysiol- 53. Pell MD, Kotz SA. On the time course of vocal emotion recognition. PLoS ogy. 2015;52(10):1317–27. https:// doi. org/ 10. 1111/ psyp. 12463. ONE. 2011;6(11): e27256. https:// doi. org/ 10. 1371/ journ al. pone. 00272 56. 74. Zimmer U, Höfler M, Koschutnig K, Ischebeck A. Neuronal interactions in 54. Posner MI, Cohen Y. Components of visual orienting. In: Bouma H, areas of spatial attention reflect avoidance of disgust, but orienting to Bonwhuis D, editors. Attention and performance X: control of language danger. Neuroimage. 2016;134:94–104. https:// doi. org/ 10. 1016/j. neuro processes. Hillsdale: Erlbaum; 1984. p. 551–6.image. 2016. 03. 050. 55. Pourtois G, Grandjean D, Sander D, Vuilleumier P. Electrophysiological 75. Zimmer U, Rosenzopf H, Poglitsch C, Ischebeck A. ERP-study on the time correlates of rapid spatial orienting towards fearful faces. Cerebral Cortex. course of disgust-motivated spatial avoidance. Biol Psychol. 2019;144:20– 2004;14(6):619–33. https:// doi. org/ 10. 1093/ cercor/ bhh023. 7. https:// doi. org/ 10. 1016/j. biops ycho. 2019. 02. 007. 56. Pourtois G, Thut G, Grave de Peralta R, Michel C, Vuilleumier P. Two electrophysiological stages of spatial orienting towards fearful faces: early Publisher’s Note temporo-parietal activation preceding gain control in extrastriate visual Springer Nature remains neutral with regard to jurisdictional claims in pub- cortex. Neuroimage. 2005;26(1):149–63. https:// doi. org/ 10. 1016/j. neuro lished maps and institutional affiliations. image. 2005. 01. 015. 57. Pourtois G, Vuilleumier P. Dynamics of emotional effects on spatial atten- tion in the human visual cortex. Prog Brain Res. 2006;156:67–91. https:// doi. org/ 10. 1016/ S0079- 6123(06) 56004-2. 58. Prime DJ, Jolicoeur P. On the relationship between occipital cortex activity and inhibition of return. Psychophysiology. 2009;46(6):1278–87. https:// doi. org/ 10. 1111/j. 1469- 8986. 2009. 00858.x. 59. Quiñones-Camacho L, Wu R, Davis E. Motivated attention to fear-related stimuli: evidence for the enhanced processing of fear in the late positive potential. Motiv Emot. 2018. https:// doi. org/ 10. 1007/ s11031- 018- 9670-x. 60. Righart R, de Gelder B. Context influences early perceptual analysis of faces—an electrophysiological study. Cerebral Cortex. 2006;16(9):1249– Re Read ady y to to submit y submit your our re researc search h ? Choose BMC and benefit fr ? Choose BMC and benefit from om: : 57. https:// doi. org/ 10. 1093/ cercor/ bhj066. 61. Rigoulot S, D’Hondt F, Defoort-Dhellemmes S, Despretz P, Honoré J, fast, convenient online submission Sequeira H. Fearful faces impact in peripheral vision: behavioral and thorough peer review by experienced researchers in your field neural evidence. Neuropsychologia. 2011;49(7):2013–21. https:// doi. org/ 10. 1016/j. neuro psych ologia. 2011. 03. 031. rapid publication on acceptance 62. Salti M, Bar-Haim Y, Lamy D. The P3 component of the ERP reflects con- support for research data, including large and complex data types scious perception, not confidence. Conscious Cogn. 2012;21(2):961–8. • gold Open Access which fosters wider collaboration and increased citations https:// doi. org/ 10. 1016/j. concog. 2012. 01. 012. 63. Santangelo V, Van der Lubbe RH, Olivetti Belardinelli M, Postma A. maximum visibility for your research: over 100M website views per year Multisensory integration affects ERP components elicited by exog- enous cues. Exp Brain Res. 2008;185(2):269–77. https:// doi. org/ 10. 1007/ At BMC, research is always in progress. s00221- 007- 1151-5. Learn more biomedcentral.com/submissions

Journal

Behavioral and Brain FunctionsSpringer Journals

Published: Sep 22, 2022

Keywords: Spatial attention; Multisensory; Emotion; Sensory modality; Fear; EEG; ERP

References