CRCC: Contrast-Based Robust Cross-Subject and Cross-Site Representation Learning for EEG

Introduction

Individual differences in human experiences cause major variability in neural signals. Zhiren Wang and Sen Song are the corresponding authors.

Cross-dataset passive brain-computer interface decoding studies suffer from stimulus material discrepancies in zero-shot settings.

Adopting simplified paradigms like resting-state recordings is important for investigating site effects.

Methodology

Neural data analysis is challenged by structural brain differences and diverse functional expression patterns. Performance gains in cross-paradigm generalization may arise from low-level perceptual responses rather than task-relevant neural representations.

Study Design

The method constrains design by reducing bias sources.

Final performance is reported using the subject as the minimum unit of analysis.

Results & Findings

Traditional algorithms show substantial performance degradation on cross-subject data. Recent methods extract cross-subject neural representations to alleviate individual variability, but site effects still hinder clinical translation.

• Traditional algorithms show substantial performance degradation on cross-subject data.
• Recent methods extract cross-subject neural representations to alleviate individual variability, but site effects still hinder clinical translation.
• Most existing studies use leave-one-site-out cross-validation or few-shot learning, while few evaluate cross-dataset zero-shot generalization.
• Inflated accuracy estimates are confounded by noise from heterogeneous acquisition conditions when healthy controls and patients are recruited from different environments.
• The observed total distributional shift is a composite effect of multi-level confounding factors.

Practical Applications

Models may capture spurious shortcuts for label discrimination rather than genuine neural signatures. Pushing intra-dataset performance to 0.9 or higher may be fundamentally unrealistic.

Related Work

Related Work 2.1. EEG dataset for site effects exploration: where the bias comes from? Deep learning algorithms and datasets for EEG decoding are no longer rare, with large-scale databases such as PRE-DICT (Cavanagh et al., 2019) and TDBRAIN(Van Dijk et al., 2022) becoming increasingly available. However, for a long time, algorithms have largely failed to achieve true clinical utility. A primary reason is that most models fail to demonstrate robust generalization in real-world scenarios. We remain uncertain whether this bias stems from the noise inherent in the labels themselves-previous research.

Rethinking Foundation Models for EEG

Rethinking Foundation Models for EEG The landscape of EEG decoding has been recently reshaped by the emergence of foundation models, such as LaBraM (Jiang et al., 2024) and EEGPT (Wang et al., 2024) , which leverage massive datasets and self-supervised learning to capture intricate neural representations. These models primarily follow a scaling-law trajectory, where escalating model complexity is prioritized to exhaustively learn highdimensional features. However, we argue that in real-world clinical applications, performance is often bottlenecked not by a lack of model capacity, but by the intrinsic domain shifts-specifically, the.

Pre-training with Masked Reconstruction

Pre-training with Masked Reconstruction The pre-training strategy employs MAE framework to learn robust representations from multi-channel neural signals. The model processes combined EEG data through a unified transformer architecture, where input features are organized as F ∈ R B×C×nsecond×dfeature for a batch with C channels. To enable the model to distinguish between modalities and encode spatial information, we incorporate three complementary embeddings. A type embedding differentiates EEG and other signal channels through a binary indicator T ∈ 0, 1 B×C . Spatial position embeddings encode the 3D coordinates L ∈.

Domain discriminators in Pre-training

Domain discriminators in Pre-training To address the domain shift challenge arising from diverse acquisition protocols, equipment and even paradigms across datasets, we employ domain adversarial training. The aggregate representation z ∈ R B×dembed obtained by pooling across non-padded channel tokens is processed through a gradient reversal layer GRLα before being fed to multiple domain discriminators D k k = 1 K , each targeting a specific aspect of domain variability such as data source, electrode density, or recording equipment. The adversarial loss is formulated as Ladv = k = 1.

Motivation

Motivation Fine-tuning Philosophy. The essence of our fine-tuning strategy lies in a fundamental shift from knowledge acquisition to feature alignment. Having broad neural representations during the pre-training phase, the objective is no longer to pursue the maximal complexity of features. Instead, we aim to guide the model toward distilling domain-invariant signatures by decreasing the errors in 1 .

Evaluation Strategy: Subject-level Metrics

Evaluation Strategy: Subject-level Metrics To ensure clinical relevance and a fair comparison across diverse architectures, we standardize the evaluation protocol. While all models are trained using their respective loss functions, the final performance is reported using the subject as the minimum unit of analysis. For a given subject k, let {x 1 , . . . , x m } be the set of EEG segments extracted from their recording. The subject-level prediction Ŷk is determined by aggregating the segmentlevel probabilities: Ŷk = argmax   1 m m j=1.

Experiment

Experiment The model weights yielding the highest average validation accuracy were selected and evaluated on the held-out test sites without further adaptation. Condition Separation. Recognizing that neural dynamics differ significantly between Eyes-Open (EO) and Eyes-Closed (EC) states, we treated these two conditions as distinct sub-pipelines throughout the fine-tuning and evaluation phases.

Impact of Site Diversity on Generalization

Impact of Site Diversity on Generalization To investigate how the breadth of training distributions influences model robustness, we conducted a scaling analysis by fine-tuning the model on an increasing number of sites, using Site 1 as the held-out target dataset. As illustrated in Figure 3 , the model’s performance exhibits a consistent upward trajectory as additional sites are integrated into the training phase. This observed scaling effect suggests that our architecture effectively leverages the biological variance present across different recording environments. Rather than over-parameterizing the model, the increase in site.

Ablation Studies

Ablation Studies To evaluate the individual contributions of our proposed components, we conducted systematic ablation experiments on Site 1 (Eyes-Open condition). We incrementally integrated: (i) multi-dataset generative pre-training, (ii) the Cross-Subject and Cross-Site (CSCS) contrastive loss, and (iii) the group-specific site-adversarial mechanism. To systematically evaluate the contribution of each proposed component, we conducted three sets of ablation experiments on Site 1 (EO), as summarized in Table 4 . Impact of CSCS Loss Design. We further examined the necessity of our Cross-Subject and Cross-Site contrastive loss and the site-adversarial mechanism (L).

Effect of Generative

Effect of Generative Contribution of Site-adversarial Loss Finally, we ablated the site-specific module. The results demonstrate that the inclusion of the discriminators is instrumental for stabilizing the latent space. Removing the adversarial loss leads to a performance drop across nearly all metrics, particularly reducing the B-Acc from 0.832 to 0.728. This highlights that the discriminators effectively functions as a robust feature filter, bridging the gap between real neural representation and clinical category knowledge. In summary, the ablation studies validate that the synergy of generative priors, site-invariant constraints, and the specialized.

Impact Statement

Impact Statement This paper presents a rigorous framework for robust EEG representation learning and can be transfered to other neuroscience field with strong domain shifts -a formidable barrier that currently prevents most EEG-based deep learning models from advancing into real-world clinical deployment. We recognize that the catastrophic performance drop of neural decoding models in unseen environments often stems from a reliance on site-specific artifacts rather than authentic neural signatures. By formulating domain bias through 3 factors, our work systematically addresses these confounding factors through integrated strategies: from meticulous data cohort.

D.2. Canonical Alignment for Cross-Subject Aggregation

D.2. Canonical Alignment for Cross-Subject Aggregation A significant challenge in multi-dataset EEG analysis is the heterogeneity of electrode montages (i.e., varying channel numbers and coordinates across subjects). To enable cross-subject comparison, we map subject-specific attributions to a standardized canonical space (e.g., the standard 10-20 system). Let P obs = {p 1 , . . . , p C } be the 3D coordinates of the observed channels for a specific subject, and P can = {p 1 , . . . , pK } be the coordinates of target canonical.

D.3. Multi-Level Cross-Domain Consistency Quantification

D.3. Multi-Level Cross-Domain Consistency Quantification To quantitatively demonstrate that our model learns domain-invariant biomarkers rather than overfitting to domain-specific noise, we compare the attribution patterns between the source domain (Training Set) and the target domain (Zero-Shot Test Set). Let M(S) and M(T ) denote the average aligned attribution matrices for the source and target domains, respectively. We propose a multi-level consistency evaluation using the Pearson Correlation Coefficient (PCC), denoted as r, at three levels of granularity: 1. Matrix-Level Consistency (r mat ): This metric measures the overall structural similarity of.

E. Contribution and Analysis of Site Adversarial Learning

E. Contribution and Analysis of Site Adversarial Learning We evaluate the relationship between accuracy and generalization performance for the HC and MDD discriminators of the site adversarial functions. We selected both site 1 and 5 for further evaluation. We applied the weights from every epoch of the 10-fold cross-validation to this test set and recorded the accuracies of the HC and MDD discriminators on the validation set. The experimental results provide preliminary evidence for our hypothesis: enhancing the model’s site-invariant capability (the ability to remove site-specific information) improves its generalization.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Figure 1 .: Figure 1. The CRCC Framework: our model comprises two primary stages: the pre-training phase (right) and the fine-tuning phase (left). During pre-training, the model takes 10-second EEG Differential Entropy (DE) features as input. These features are processed by an encoder, with gradients backpropagated through a joint objective involving a noise-reducing decoder and a domain discriminator, the latter of which facilitates robust representation learning across multiple open-source datasets. In the subsequent fine-tuning stage, the decoder and domain discriminator are detached. The core fine-tuning objective integrates a standard cross-entropy loss for classification with two specialized components: a contrastive learning module designed for cross-subject feature extraction and an adversarial loss aimed at mitigating inter-site variability.
• Figure 2: 5.1. Experimental SetupData processing. All EEG recordings were processed following a standardized pipeline, including downsampling, band-pass filtering, and Independent Component Analysis (ICA) for artifact removal. Details of the paradigm and preprocessing stages are provided in Appendix 6. The pretraining phase utilized a collection of entirely open-source datasets, the specifics of which are documented in Appendix B and C.Dataset Partitioning and Leakage Prevention. To ensure the integrity of our evaluation, all data partitioning during the fine-tuning stage was conducted on a per-subject basis. Strict protocols were implemented to prevent any potential subject leakage; for instance, multiple sessions from the same individual (e.g., longitudinal follow-ups) were strictly assigned to either the training or validation set as a single block.
• Figure 2 .: Figure 2. Scales values of the subjectsThe bars are color-coded, with blue representing HC and green representing MDD patients. For each group, the bars from left to right indicate: (i) the mean score for that category, (ii) the mean score of correctly classified subjects, and (iii) the mean score of misclassified subjects. The y-axis denotes the respective scale scores across four diagnostic dimensions: HAMD and BDI, and HAMA and SAS (Anxiety).
• Figure 3 .: Figure 3. Zero-shot generalization performance across varying training site scales. The trend demonstrates a clear scaling law: the balanced accuracy on the unseen test set increases monotonically as the number of training sites grows from 2 to 6. The model achieves peak performance (69.75%) with 6 sites, underscoring the benefits of data diversity in enhancing cross-site robustness.
• Figure 5: Pre-training. The first set of experiments underscores the critical role of our multi-dataset pre-training stage. Without pre-training (w/o Pre-training), the model struggles to achieve competitive results, with the balanced accuracy dropping significantly to 0.545. This suggests that the high-dimensional noise inherent in raw EEG signals poses a substantial challenge for supervised fine-tuning alone.

Conclusion

Matrix-Level Consistency measures the overall structural similarity of learned representations across the channel-feature space. Data from various sites in the model exhibit a higher degree of overall mixing aligning with clinical reality.

Therefore, the rigorous control design of Site 5, along with the model’s zero-shot generalization capability on this site, serves as the most critical metric for distinguishing model performance.