A Deep-Learning-Based Neural Decoding Framework for Emotional Brain-Computer Interfaces

Introduction

Decoding emotions from neural signals has long been popular in the fields of computational science, neuroscience, and psychology . Recent advances in brain-computer interface (BCI) further enable us to collect neural signals from a broader scale, enriching the foundation for establishing computational models .

Currently, BCIs can be classified as invasive (IBCIs) and non-invasive (Non-IBCIs).

However, due to technological challenges, many matured BCI applications aiming to decode human emotions are Non-IBCIs-based (e.g., electroencephalogram, EEG) .

Methodology

Traditionally, to enable the quantitative analysis and computational modeling of emotions, the neural representation of emotions is usually generated through cognitive and electrophysiological experiments. In this study, based on investigating the relations between labels, stimuli, emotions, and neural responses, we present a hybrid framework that combines CL and DL for spike decoding of emotions from neural responses obtained from IBCIs.

Study Design

Visual stimuli were presented on a monitor (VG248, ASUS) 57 cm in front of the subject.

Visual stimuli were faces of different species (human and monkey), different emotions (negative, positive, and neutral), and different spatial frequencies (broad, high, low, and a mix of high and low).

Results & Findings

Generally, IBCIs can collect signals from the deeper structure and generate cleaner signals than Non-IBCIs, thus can generate better accuracy when being used in neural decoding (e.g., motor imagination and language recovery) . For example, by assigning neural data as texts of “brain activity” pieces, some DL models achieve up to 95% accuracy on many classification and reconstruction tasks .

• Generally, IBCIs can collect signals from the deeper structure and generate cleaner signals than Non-IBCIs, thus can generate better accuracy when being used in neural decoding.
• For example, by assigning neural data as texts of “brain activity” pieces, some DL models achieve up to 95% accuracy on many classification and reconstruction tasks.
• To identify incorrect labels, methods such as confident learning (CL), which is a model-agnostic theory and algorithm for identifying labeling errors, have been proposed .
• Additionally, animal experiments normally do not allow unlimited data collection, making the data samples not large enough for DL model training.
• In the following, we demonstrate the decoding capability of this framework, its ability to perform end-to-end mapping of neural responses to emotions, and its robustness to.

Practical Applications

In particular, the emotion stimulus and the evoked emotion in subjects may not be consistently matched all the time. In addition, a piece of neural signals may encode multiple emotions.

However, it is not ideal to amplify data either as the labeling errors may spread or even amplify unexpectedly.

In other neural decoding scenarios, neural signal samples may have their characteristics.

The uncertainty of neural response samples

This section explains the concept of aleatoric uncertainty in neural response samples, focusing on the challenges of labeling errors in emotion-evoked neural data and the implications for supervised learning.

Emo-Net: A hybrid framework for emotion recognition

Emo-Net is introduced as a hybrid framework combining confidence learning and deep learning for emotion recognition. It assesses uncertainty in training data and proposes optimization strategies for improving model performance.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Figure 1 :: Figure 1: The illustration of the monkey experiment acquiring emotion-evoked neural signals using an invasive brain-machine interface. (A) The experimental paradigm. (B-C) The preprocessing of spiking signals and a single trial sample used for establishing datasets.
• Figure 2 :: Figure 2: An illustration of the generation of label errors in establishing emotionevoked neural datasets.
• Figure 3 :: Figure 3: An illustration of the Emo-Net framework. The training data is initially processed by a confident learning component. This component utilizes an external model θ2 to estimate true labels and provide a ranked uncertainty, which is derived from a combination of confidence thresholds and sparsity. After partitioning data, the weights of individual samples are re-adjusted and dirty data are pruned based on the level of uncertainty at both individual and overall levels. The cleaned data is then fed into the deep learning classifier θ1 to complete the remaining training process and achieve emotion classification.
• Algorithm- 1 :: Prune/reweight training data Input: training data X, training label Y, Emo-Net Net # Compute Quality Matrix Q of Samples Q ← Net.ComputeUncertainty(X, Y) # Compute reweight factor /prune mask matrix W using Q according to functions (1) and (2)W ← ComputeWeightMatrix(Q)# Select the minimum matrix w from W during every epoch for (x, y, w) in train epochs of (X, Y, Wp is the real probability density function, p(xij) is 0 or 1, and q(xij) is the predicted probability given by model θ.Finally, the DL model θ is employed to learn the neural representations of the induced emotions and complete the classification of neural responses to emotions, thereby achieving emotion recognition. The learning rate of the DL model is uniformly set to 0.001, and the batch size is set to 256.
• Figure 5: generated better performance in spike decoding after employing the Emo-Net. These results together demonstrated the efficacy of the CLP-embedded Emo-Net in spiking signal-based emotion decoding.