In recent years, the prediction of salient regions in RGB-D images has become a focus of research. Compared to its RGB counterpart, the saliency prediction of RGB-D images is more challenging. In this study, we propose a novel deep multimodal fusion autoencoder for the saliency prediction of RGB-D images. The core trainable autoencoder of the RGB-D saliency prediction model employs two raw modalities (RGB and depth/disparity information) as inputs and their corresponding eye-fixation attributes as labels. The autoencoder comprises four main networks: color channel network, disparity channel network, feature concatenated network, and feature learning network. The autoencoder can mine the complex relationship and make the utmost of the complementary characteristics between both color and disparity cues. Finally, the saliency map is predicted via a feature combination subnetwork, which combines the deep features extracted from a prior learning and convolutional feature learning subnetworks. We compare the proposed autoencoder with other saliency prediction models on two publicly available benchmark datasets. The results demonstrate that the proposed autoencoder outperforms these models by a significant margin.

Distracted driving has become a growing traffic safety concern. With advances in autonomous driving and connected vehicle technology, a mixture of various types of intelligent vehicles will become normal in the near future, while more factors that may cause driver cognitive distraction are emerging. However, there are rarely studies on distracted driving in mixed traffic environments. To fill this gap, we conducted a natural driving experiment with three representative events at a nonsignalized intersection in a mixed traffic environment and proposed a novel method of identifying cognitive distraction based on bidirectional long short-term memory (Bi-LSTM) with attention mechanism. Forty participants were recruited for each event, who completed three different cognitive distraction experiments induced by three different secondary tasks in contrast with a normal driving process when passing a nonsignalized intersection. Related driving performance and eye movement data were collected to train and test the Bi-LSTM with attention mechanism model. Compared with the support vector machine (SVM) model, its recognition accuracy rate is 94.33%, which is 3.83% higher than that of the SVM in the total event, which has reasonable applicability for distraction recognition in a mixed traffic environment. Potential applications of this model include distraction alarm and autonomous driving assistance systems, which could avoid road traffic accidents.

Aided communicators often have an opportunity to express themselves with speech-generating devices (SGDs) that produce symbol by symbol (SBS) and/or pre-stored (PS) utterances. Studies on the usage of PS utterances report that these utterances affect conversations positively, but it appears that aided communicators and professionals may have divergent views on their benefits. The aim of this study is to analyse how school-aged aided communicators, their mothers, peers, and speech and language therapists (SLTs) co-construct the social actions of PS utterances during their everyday interactions. The theoretical framework of this study is conversation analysis. This approach to analysing the data enhances our understanding of the linguistic and temporal resources of PS utterances and how they are used to reinforce various rich social actions that promote the progress of aided conversations to resemble natural spoken conversations. The results of this study will help SLTs and teachers in their planning content for SGDs as well as in teaching, and guiding aided communicators and their partners to utilize PS utterances in combination with SBS utterances during their conversations.

Most developmental research on Theory of Mind (ToM)—our ability to infer the beliefs, intentions, and desires of others—has focused on the preschool years. This is unsurprising as it was previously thought that ToM skills are developed between the ages of 2 and 7 years. Over the last couple of decades however, studies have provided evidence for significant structural and functional changes in the brain areas involved in ToM (the “social brain”) not only during childhood but also during adolescence. Importantly, some of these findings suggest that the use of ToM shows a prolonged development through middle childhood and adolescence. Although evidence from previous studies suggests a protracted development of ToM, the factors that constrain performance during middle childhood and adolescence are only just beginning to be explored. In this article, we report two visual-world eye-tracking studies that focus on the timecourse of predictive inferences. We establish that when the complexity of ToM inferences are at a level which is comparable with standard change-of-location false-belief tasks, then adolescents and adults generate predictions for other agents’ behaviour in the same timecourse. However, when inferences are socially more complex, requiring inferences about higher order mental states, adolescents generate predictive gaze bias at a marked delay relative to adults. Importantly, our results demonstrate that these developmental differences go beyond differences in executive functions (inhibitory control or working memory) and point to distinct expectations between groups and greater uncertainty when predicting actions based on conflicting desires.

The use of visual self-personalization by politicians to shape perceptions of their character and personality is a prominent form of modern political communication, especially on social media. Yet, little is known about the effect these portrayals have on the visual attention of voters and their impressions of the traits of the politicians depicted. This exploratory study examines the effects of visual self-personalization (professional vs. private focus) by Finnish political leaders of both genders on Instagram. Through a pilot laboratory study, viewers’ visual attention was measured using eye-tracking technology and, in combination with this, leader trait impressions were experimentally examined. The results indicate that photos depicting politicians in professional settings are more attention grabbing and effective in shaping trait impressions than photos showing politicians in private life. Among the recommendations for future research, it is noted that gender aspects should be further explored as this study, by focusing on a gender-egalitarian country, had a limited scope in that respect.

According to violation–compensation models of cognitive conflict, experiences that violate expected associations evoke a common, biologically based syndrome of aversive arousal, which in turn motivates compensation efforts to relieve this arousal. However, while substantial research shows that people indeed respond with increased arousal to expectancy violating events, evidence for the motivating role of arousal is rarely found. In two within-subjects studies ( N = 44 and N = 50), we demonstrate evidence for the motivating role of arousal in this violation–compensation process among university students. Using pupillometry and the hindsight bias phenomenon, we show that people respond with greater arousal when presented with expectancy violating information. In turn, we show that the pupillary response is positively related to the amount of hindsight bias being displayed. These findings provide further insights into the process underlying the hindsight bias and, crucially, support key predictions following from threat–compensation models.

A number of virtual reality head-mounted displays (HMDs) with integrated eye trackers have recently become commercially available. If their eye tracking latency is low and reliable enough for gaze-contingent rendering, this may open up many interesting opportunities for researchers. We measured eye tracking latencies for the Fove-0, the Varjo VR-1, and the High Tech Computer Corporation (HTC) Vive Pro Eye using simultaneous electrooculography measurements. We determined the time from the occurrence of an eye position change to its availability as a data sample from the eye tracker (delay) and the time from an eye position change to the earliest possible change of the display content (latency). For each test and each device, participants performed 60 saccades between two targets 20° of visual angle apart. The targets were continuously visible in the HMD, and the saccades were instructed by an auditory cue. Data collection and eye tracking calibration were done using the recommended scripts for each device in Unity3D. The Vive Pro Eye was recorded twice, once using the SteamVR SDK and once using the Tobii XR SDK. Our results show clear differences between the HMDs. Delays ranged from 15 ms to 52 ms, and the latencies ranged from 45 ms to 81 ms. The Fove-0 appears to be the fastest device and best suited for gaze-contingent rendering.

Eye tracking is the process of measuring either the point of gaze (where one is looking) or the motion of an eye relative to the head. This paper investigated use of eye gaze trackers in military aviation environment to automatically estimate pilot’s cognitive load from ocular parameters. We used a fixed base variable stability flight simulator with longitudinal tracking task and collected data from 14 military pilots. In a second study, we undertook three test flights with a BAES Hawk Trainer aircraft doing air to ground attack training missions and constant G level turn maneuvers up to +5G. Our study found that ocular parameters like rate of fixation is significantly different in different flying conditions and significantly correlate with altitude gradient during air to ground dive training task, normal load factor (G) of the aircraft during constant G level turn maneuvers and pilot’s control inceptor and tracking error in simulation tasks. Results from our studies can be used for real time estimation of pilots’ cognitive load, providing suitable warnings and alerts to the pilot in cockpit and training of military pilots on cognitive load management during operational missions.

This paper presents a new analysis approach for evaluating situation awareness in marine operation training. Taking advantage of eye tracking technology, the situation awareness reflected by visual attention can be visualized and analyzed. A scanpath similarity comparison method that allows group-wise comparisons is proposed. The term ‘Expert zone’ is introduced to evaluate the performance of novice operator based on expert operators’ eye movement. It is used to evaluate performance of novice operators in groups in certain segment of marine operation. A pilot study of crane lifting experiment was carried out. Two target stages of operation for the load descending until total immersion to the seabed were selected and analyzed for both novice and expert operators. The group-wise evaluation method is proven to be able to access the performance of the operator. Besides that, from data analysis of fixation-related source and scanpath, the similarities and dissimilarities of eye behavior between novice and expert is concluded with the scanpath mode in target segment.

Previous research has shown that exposure to Japanese gardens reduces physiological measures of stress, e.g. heart rate, in both healthy subjects and dementia patients. However, the correlation between subjects’ physiological responses and their visual behavior while viewing the garden has not yet been investigated. To address this, we developed a system to collect simultaneous measurements of eye gaze and three physiological indicators of autonomic nervous system activity: electrocardiogram, blood volume pulse, and galvanic skin response. We recorded healthy subjects’ physiological/behavioral responses when they viewed two environments (an empty courtyard and a Japanese garden) in two ways (directly or as a projected 2D photograph). Similar to past work, we found that differences in subject’s physiological responses to the two environments when viewed directly, but not as a photograph. We also found differences in their behavioral responses. We quantified subject’s behavioral responses using several gaze metrics commonly considered to be measures of engagement of focus: average fixation duration, saccade amplitude, spatial entropy and gaze transition entropy. We found decrease in gaze transition entropy, the only metric that accounts for both the spatial and temporal properties of gaze, to have a weak positive correlation with decrease in heart rate. This suggests a relationship between engagement/focus and relaxation. Finally, we found gender differences: females’ gaze patterns were more spatially distributed and had higher transition entropy than males.

Many studies deal with solution strategies in mental-rotation tests. The approaches range from global analysis, attention to object parts, holistic and piecemeal strategy to a combined strategy. Other studies do not speak of strategies, but of holistic or piecemeal processes or even of holistic or piecemeal rotation. The methodological approach used here is to identify mental-rotation strategies via gaze patterns derived from eye-tracking data when solving chronometric mental-rotation tasks with gender-stereotyped objects. The mental-rotation test consists of 3 male-stereotyped objects (locomotive, hammer, wrench) and 3 female-stereotyped objects (pram, hand mirror, brush) rotated at eight different angles. The sample consisted of 16 women and 10 men (age: M=21.58; SD=4.21). The results of a qualitative analysis with two individual objects (wrench and brush) showed four different gaze patterns. These gaze patterns appeared with different frequency in the two objects and correlated differently with performance and response time. The results indicate either an object-oriented or an egocentric mental-rotation strategy behind the gaze patterns. In general, a new methodological approach has been developed to identify mental-rotation strategies bottom-up which can also be used for other stimulus types.

Background: Individuals with Parkinson’s disease (PD) may need to spend more mental and physical effort (i.e., cognitive workload) to maintain postural control. Pupillary response reflects cognitive workload during postural control tasks in healthy controls but has not been investigated as a measure of postural demand in PD.Objectives: To compare pupillary response during increased postural demand using vision occlusion and dual tasking between individuals with PD and healthy controls.Methods: Thirty-three individuals with PD and thirty-five healthy controls were recruited. The four conditions lasted 60 s and involved single balance task with eyes open; single balance task with eyes occluded; dual task with eyes open; dual task with eyes occluded. The dual task comprised the Auditory Stroop test. Pupillary response was recorded using an eye tracker. The balance was assessed by using a force plate. Two-way Repeated Measures ANOVA and LSD post-hoc tests were employed to compare pupillary response and Center of Pressure (CoP) displacement across the four conditions and between individuals with PD and healthy controls.Results: Pupillary response was higher in individuals with PD compared to healthy controls (p = 0.009) and increased with more challenging postural conditions in both groups (p < 0.001). The post-hoc analysis demonstrated increased pupillary response in the single balance eyes occluded (p < 0.001), dual task eyes open (p = 0.01), and dual task eyes occluded (p < 0.001) conditions compared to single task eyes open condition.Conclusion: Overall, the PD group had increased pupillary response with increased postural demand compared to the healthy controls. In the future, pupillary response can be a potential tool to understand the neurophysiological underpinnings of falls risk in the PD population.

Comparative judgment (CJ) has been recently introduced in the educational field as a means of assessing competences. In this judgement process, assessors are presented with two pieces of student work and are asked to choose which one is better in relation to the competencies being assessed. However, since student work is heterogeneous and highly information loaded, it raises the question as to whether this type of assessment is too complex for assessors to use. Previous research on the topic has operationalized experienced complexity by employing self-report measures, which have been criticized for common problems associated with their use. In our study, we used eye tracking to study 23 high school teachers when they made 10 comparative judgments, and their pupil diameter was used as an indicator of the experienced complexity. This study builds on previous research that integrated Campbell’s theory on task complexity (1988) into CJ. Based on this framework, three hypotheses regarding the role of decision accuracy were formulated and empirically tested. Hypothesis one assumes that the distance between two pieces of student work on the rank-order (rank-order distance) is negatively related to experienced complexity, irrespective of decision accuracy. Hypothesis two assumes that decision accuracy moderates the relationship between rank-order distance and experienced complexity. Hypothesis three builds on hypothesis two by adding a negative relationship between experience and experienced complexity. In all three hypotheses, the average experienced complexity is assumed to vary between assessors, as is the strength of the expected relationships. An information-theoretic approach was used to test the holding of all three hypotheses. All hypotheses were translated into statistical models, and their relative and absolute fit were assessed. Results provided strong evidence for hypothesis three: both the moderating role of decision accuracy on the relationship between rank-order distance and experienced complexity, and the relationship between experience and experienced complexity, were confirmed.

A lot of research has been done on the detection of mental workload (MWL) using various bio-signals. Recently, deep learning has allowed for novel methods and results. A plethora of measurement modalities have proven to be valuable in this task, yet studies currently often only use a single modality to classify MWL. The goal of this research was to classify perceived mental workload (PMWL) using a deep neural network (DNN) that flexibly makes use of multiple modalities, in order to allow for feature sharing between modalities. To achieve this goal, an experiment was conducted in which MWL was simulated with the help of verbal logic puzzles. The puzzles came in five levels of difficulty and were presented in a random order. Participants had 1 h to solve as many puzzles as they could. Between puzzles, they gave a difficulty rating between 1 and 7, seven being the highest difficulty. Galvanic skin response, photoplethysmograms, functional near-infrared spectrograms and eye movements were collected simultaneously using LabStreamingLayer (LSL). Marker information from the puzzles was also streamed on LSL. We designed and evaluated a novel intermediate fusion multimodal DNN for the classification of PMWL using the aforementioned four modalities. Two main criteria that guided the design and implementation of our DNN are modularity and generalisability. We were able to classify PMWL within-level accurate (0.985 levels) on a seven-level workload scale using the aforementioned modalities. The model architecture allows for easy addition and removal of modalities without major structural implications because of the modular nature of the design. Furthermore, we showed that our neural network performed better when using multiple modalities, as opposed to a single modality. The dataset and code used in this paper are openly available.

The frontal alpha asymmetry (FAA) is a neurophysiological measure of motivation and preference. Despite the FAA is associated to commercial pleasantness, conflicting evidence emerged in the literature regarding its relationship with behavior. To study the association between FAA and consumers' decision, we manipulated a commercial script to elicit diverse consumers' attitudes and decisions and to evaluate whether the FAA score is associated to their final investment. A little informative script (S1) was used to polarize consumers' attitudes and investments toward unfavorable scores, while a more personalized message (S2) to elicit in customers a favorable attitude and higher investments. Twenty-one participants listened to the scripts, and their FAA, attitude, and monetary investment were measured. In S1, the FAA did not correlate with neither attitude nor the investment decision, while a robust negative correlation between these variables was found in S2. No other peripheral body and neural measures associated with attitude or final decision. Our data suggest that the FAA correlates with attitude and decision, when a commercial script is customized and provides an adequate information, likely leading the consumer to a more reasoned and planned decision-making process. When facilitating a favorable attitude toward an offer, the negative correlation of FAA and behavior may reflect the involvement of a control system, whose role is to monitor and govern possible conflicts between approach and avoidance motivations. This observation provides additional indication on the value of FAA as a marker of consumer behaviors, and how it could be affected by experimental and contextual bias.

The human pupil behavior has gained increased attention due to the discovery of the intrinsically photosensitive retinal ganglion cells and the afferent pupil control path’s role as a biomarker for cognitive processes. Diameter changes in the range of 10–2 mm are of interest, requiring reliable and characterized measurement equipment to accurately detect neurocognitive effects on the pupil. Mostly commercial solutions are used as measurement devices in pupillometry which is associated with high investments. Moreover, commercial systems rely on closed software, restricting conclusions about the used pupil-tracking algorithms. Here, we developed an open-source pupillometry platform consisting of hardware and software competitive with high-end commercial stereo eye-tracking systems. Our goal was to make a professional remote pupil measurement pipeline for laboratory conditions accessible for everyone. This work’s core outcome is an integrated cross-platform (macOS, Windows and Linux) pupillometry software called PupilEXT, featuring a user-friendly graphical interface covering the relevant requirements of professional pupil response research. We offer a selection of six state-of-the-art open-source pupil detection algorithms (Starburst, Swirski, ExCuSe, ElSe, PuRe and PuReST) to perform the pupil measurement. A developed 120-fps pupillometry demo system was able to achieve a calibration accuracy of 0.003 mm and an averaged temporal pupil measurement detection accuracy of 0.0059 mm in stereo mode. The PupilEXT software has extended features in pupil detection, measurement validation, image acquisition, data acquisition, offline pupil measurement, camera calibration, stereo vision, data visualization and system independence, all combined in a single open-source interface, available at https://github.com/openPupil/Open-PupilEXT.

In addition to photic reflex function, the temporal behavior of the pupil diameter reflects levels of arousal and attention and thus internal cognitive neural activity. Recent studies have reported that these behaviors are characterized by baseline activity, temporal complexity, and symmetricity (i.e., degree of symmetry) between the right and left pupil diameters. We hypothesized that experimental analysis to reveal relationships among these characteristics and model-based analysis focusing on the newly discovered contralateral projection from the locus coeruleus (LC) to the Edinger-Westphal nucleus (EWN) within the neural system for controlling pupil diameter could contribute to another dimension of understanding of complex pupil dynamics. In this study, we aimed to validate our hypothesis by analyzing the pupillary hippus in the healthy resting state in terms of sample entropy (SampEn), to capture complexity, and transfer entropy (TranEn), to capture symmetricity. We also constructed a neural model embedded with the new findings on neural pathways. The following results were observed: first, according to surrogate data analysis, the complexity and symmetricity of pupil diameter changes reflect a non-linear deterministic process. Second, both the complexity and the symmetricity are unimodal, peaking at intermediate pupil diameters. Third, according to simulation results, the neural network that controls pupil diameter has an inverted U-shaped profile of complexity and symmetricity vs. baseline LC activity; this tendency is enhanced by the contralateral synaptic projections from the LCs to the EWNs. Thus, we characterized the typical relationships between the baseline activity and the complexity and symmetricity of the pupillometric data in terms of SampEn and TranEn. Our evaluation method and findings may facilitate the development of estimation and diagnostic tools for exploring states of the healthy brain and psychiatric disorders based on measurements of pupil diameter.

This paper discusses the utilization of pilots' physiological indications such as electroencephalographic (EEG) signals, ocular parameters, and pilot performance-based quantitative metrics to estimate cognitive workload. The study aims to derive a non-invasive technique to estimate pilot's cognitive workload and study their correlation with standard physiological parameters. Initially, we conducted a set of user trials using well-established psychometric tests for evaluating the effectiveness of pupil and gaze-based ocular metrics for estimating cognitive workload at different levels of task difficulty and lighting conditions. Later, we conducted user trials with the NALSim flight simulator using a business class Learjet aircraft model. We analyzed participants' ocular parameters, power levels of different EEG frequency bands, and flight parameters for estimating variations in cognitive workload. Results indicate that introduction of secondary task increases pilot's cognitive workload significantly. The beta frequency band of EEG, nearest neighborhood index specifying distribution of gaze fixation, L1 Norm of power spectral density of pupil diameter, and the duty cycle metric indicated variations in cognitive workload.

Native language listeners engage in predictive processing in many processing situations and adapt their predictive processing to the statistics of the input. In contrast, second language listeners engage in predictive processing in fewer processing situations. The current study uses eye-tracking data from two experiments in bilinguals’ native language (L1) and second language (L2) to explore their predictive processing based on contrastive pitch accent cues, and their adaptation in the face of prediction errors. The results of the first experiment show inhibition effects for unpredicted referents in both the L1 and the L2 that can be modeled with a Bayesian adaptation model, suggesting that bilinguals adapt their prediction in the face of prediction errors in a way that is compatible with the model. In contrast, the results of the second experiment, after a training phase that increased the predictive validity of the cue, show inhibition effects for unpredicted referents only in the L1, but not in the L2. In addition, the Bayesian adaptation model significantly predicts only the L1, but not the L2 data. The results are discussed with respect to adaptation to the statistical properties of the input.

Diminished orienting to social stimuli, and particularly to faces, is a core feature of autism spectrum disorders (ASDs). Impaired face processing has been linked to atypical attention processes that trigger a cascade of pathological development contributing to impaired social communication. The aim of the present study is to explore the processing of emotional and neutral faces using an eye-tracking paradigm (the emotional faces task) with a group of 24 children with ASD aged 6 and under and a group of 22 age-matched typically developing (TD) children. We also measure habituation to faces in both groups based on the presentation of repeated facial expressions. Specifically, the task consists of 32 pairs of faces, a neutral face and an emotional face from the same identity, shown side by side on the screen. We observe differential exploration of emotional faces in preschoolers with ASD compared with TD. Participants with ASD make fewer fixations to emotional faces than their TD peers, and the duration of their first fixation on emotional faces is equivalent to their first fixation on neutral faces. These results suggest that emotional faces may be less interesting for children with ASD. We also observe a habituation process to neutral faces in both children with ASD and TD, who looked less at neutral faces during the last quarter of the task compared with the first quarter. By contrast, TD children show increased interest in emotional faces throughout the task, looking slightly more at emotional faces during the last quarter of the task than during the first quarter. Children with ASD demonstrate neither habituation nor increased interest in the changing emotional expressions over the course of the task, looking at the stimuli for equivalent time throughout the task. A lack of increased interest in emotional faces may suggest a lack of sensitivity to changes in expression in young children with ASD.