Average Rankings Mask Per-Subject Optimality: A Friedman-Nemenyi Benchmark of EEG Motor-Imagery BCI Decoders

Methods

Methods.

Datasets

Datasets We used three publicly available datasets through MOABB [34, 35] under a single harmonized paradigm. PhysionetMI [31, 38] comprises 109 participants recorded with 64 channels at 160 Hz; from the original protocol we retained the left-versus right-hand motor-imagery condition. Cho2017 [11] comprises 52 participants recorded with 64 channels at 512 Hz on a BioSemi ActiveTwo system, designed around left-and right-hand motor imagery. Zhou2016 [33] comprises only 4 participants recorded with 14 channels at 250 Hz across sessions separated by days to months; we retained the left\/right condition. Because four.

Feature extraction

Feature extraction Four methodologically distinct feature families were evaluated. (i) CSP [17, 39] derived lowdimensional spatial filters maximizing inter-class variance differences, computed separately within each band (no filter-bank CSP); we used two components, i.e. two features per trial. (ii) Covariance tangent-space projection (cov-tgsp) [19, 40] : per-epoch channel covariance matrices were projected into the tangent space at their Riemannian geometric mean, yielding Euclidean vectors that preserve covariance structure. (iii) Coherence-based connectivity projected into tangent space (con-tgsp) [25] , summarizing pairwise channel interactions in a Euclidean-compatible form. (iv) Nonlinear descriptors, HFD.

Scaling, classifiers, and the configuration count

Scaling, classifiers, and the configuration count Each feature representation was passed through one of four scalers (StandardScaler, RobustScaler, MinMaxScaler, or L2 normalization) and one of several classifiers: logistic regression with elastic-net regularization, linear discriminant analysis, linear and RBF support-vector machines, random forests, and a predefined family of 17 multilayer perceptrons differing in depth, width, activation, solver, and regularization. Classifier hyperparameters were fixed within each configuration; classifiers are thus treated as predefined pipeline components, not as the product of a separate tuning stage. Random states for stochastic optimizers were fixed (seed.

Evaluation and statistical analysis

Evaluation and statistical analysis Performance was assessed with the within-session protocol of MOABB [35] : each participant’s single-session data are split into training and test folds, and every pipeline is trained and tested on identical splits. This yields one accuracy per participant per pipeline per band. We use accuracy as the primary metric; because the LeftRightImagery paradigm is, by construction, a balanced two-class problem, accuracy and balanced accuracy nearly coincide here, and we report accuracy for direct comparability with MOABB and the wider literature. We emphasize that within-session evaluation measures.

Two families lead, but their order depends on the dataset

Two families lead, but their order depends on the dataset Across all datasets and bands, two feature families separated clearly from the rest. Covariance tangent-space projection (cov-tgsp) achieved the highest overall mean accuracy (0.687), followed by CSP (0.651); the connectivity and nonlinear families trailed (con-tgsp 0.599; Hjorth 0.567; SVD entropy 0.563; HFD 0.555). This reproduces the familiar finding that spatial and Riemannian methods are strong default choices for motor imagery. The ordering within the leading pair was, however, dataset-dependent (Table 2 ; Fig. 2 ). On Cho2017, cov-tgsp led decisively.

The leading pipelines are statistically tied on the most heterogeneous cohort

The leading pipelines are statistically tied on the most heterogeneous cohort Descriptive ranking is not evidence of an ordering. We therefore tested the six representative pipelines per dataset (Fig. 3 ). On both cohorts the Friedman omnibus test was significant (PhysionetMI \u03c7 2 = 58.3, p = 2.7\u00d710 -11 ; Cho2017 \u03c7 2 = 132.9, p = 5.9\u00d710 -27 ), confirming that not all pipelines perform equally. But the effect sizes tell sharply different stories. On Cho2017, concordance was moderate-to-strong (Kendall’s W = 0.51): cov-tgsp occupied a clean top rank.

At the individual level, the best pipeline varies across participants

At the individual level, the best pipeline varies across participants The tie at the top is a symptom of a deeper pattern: the identity of the best pipeline varies from participant to participant (Fig. 4 ). On PhysionetMI, the overall winner (cov-tgsp) was the best pipeline for only 38 of 109 participants (35%); CSP was best for 28 (26%); and nonlinear descriptors were best for 35 participants (32%), with Hjorth, HFD, and SVD entropy each leading for a non-trivial share. Grouped by family, spatial\/Riemannian methods won for 61% of participants.

The ranking is not an artefact of feature dimensionality

The ranking is not an artefact of feature dimensionality Because the families differ greatly in dimensionality (2 to 2,080 features) and share the same downstream classifiers, a natural concern is that the comparison merely rewards higher-dimensional representations. The data do not support this (Fig. 5 ). Performance is non-monotonic in dimensionality: CSP, with just 2 features, outperformed all three nonlinear families (64-192 features); and the two highest-dimensional methods, cov-tgsp and con-tgsp, share identical dimensionality (2,080) yet sit at opposite ends of the performance range (0.644 vs 0.542 on PhysionetMI). Across.

Data and code availability

Data and code availability The three datasets are publicly available through MOABB: PhysionetMI ( https:\/\/moabb.neurotec hx.com\/docs\/generated\/moabb.datasets.PhysionetMI.html ), Cho2017 (. . . Cho2017.html), and Zhou2016 (. . . Zhou2016.html). The complete per-subject score tables analysed here, and the scripts that reproduce all statistics and figures (Friedman\/Nemenyi\/Wilcoxon analyses, winner-per-subject and oracle computations, the dimensionality analysis, and the classifier and scaler comparisons), are provided as Supplementary Material and will be deposited in a public repository upon publication.

Author contributions

Author contributions All authors contributed to the conception and interpretation of the study and to writing the manuscript. The benchmark was executed within the MOABB framework; statistical re-analysis and figures were produced from the resulting per-subject score tables. 8 Competing interests O. Oullier and P. Barbaste are co-founders of Inclusive Brains, which maintains a research partnership with IBM on artificial intelligence, neurotechnologies, and quantum computing; this partnership did not influence the design, analysis, or interpretation of this study. X. Vasques is Vice-President and Chief Technology Officer of IBM Technology France.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Figure 1 :: Figure 1: Benchmark design. Three public motor-imagery datasets are harmonized through the MOABB LeftRightImagery paradigm, band-pass filtered (8-15 Hz and 8-30 Hz) and epoched (0.6-2.0 s post-cue). Four feature families (CSP; covariance and coherence tangent-space projections; nonlinear descriptors) are combined with four scalers and a set of classical classifiers, yielding 1,056 distinct configurations per band and dataset. Every configuration is fit and tested within a single recording session of a single participant; the resulting per-subject scores feed the statistical comparison. Zhou2016 (greyed) is illustrative only and excluded from inference.
• Figure 2 :: Figure 2: Per-subject decoding accuracy by pipeline and dataset. Boxplots show the distribution of per-participant within-session accuracy (band-averaged); points are individual participants; diamonds mark the family mean; the dashed line is chance (0.50). Pipelines are coloured by family. On PhysionetMI (left) the leading spatial pipelines overlap heavily with one another and with the nonlinear families; on Cho2017 (centre) the separation is clearer. Zhou2016 (right, n = 4) is illustrative only.
• Figure 3 :: Figure 3: Critical-difference diagrams (Nemenyi post-hoc, \u03b1 = 0.05). Pipelines are placed by mean rank (1 = best); horizontal bars connect groups that are not significantly different (critical difference CD shown top-left of each panel). On PhysionetMI (left) cov-tgsp and CSP fall in a single wide top clique, statistically tied, despite the significant omnibus test and weak overall concordance (W = 0.11). On Cho2017 (right) cov-tgsp stands apart as the sole top method (W = 0.51).
• Figure 4 :: Figure 4: Subject-level heterogeneity in the best-performing pipeline. Left and centre: the proportion of participants for whom each pipeline is the single best, by dataset; segment counts give the number of participants. The overall-best pipeline (cov-tgsp) is best for only 35% of PhysionetMI participants and 67% of Cho2017 participants. Right: the best single fixed pipeline versus the per-subject oracle; matching pipeline to participant would add about 7 accuracy points on PhysionetMI and about 1 on Cho2017.
• Figure 5 :: Figure5: Accuracy is not monotonic in feature dimensionality. Each point is one pipeline’s mean withinsession accuracy on a dataset, against the number of features per trial (log scale); colour denotes feature family and marker denotes dataset. Points sharing a dimensionality are given a small horizontal offset for legibility. CSP (2 features) beats the nonlinear families (64-192); cov-tgsp and con-tgsp share 2,080 features but differ widely in accuracy. The rank correlation between dimensionality and accuracy is weak and non-significant.

Limitations and Cautions

A useful limitation and caution is that this article summarizes the available paper text and extracted evidence; readers should consult the source paper before treating any interpretation as definitive.

The paper’s conclusions may depend on its source selection, definitions, assumptions, and the scope of its analysis, so follow-up reading is important.

Conclusion

Conclusion A large within-session benchmark, equipped with the statistics standard for multi-classifier comparison, shows that covariance tangent-space projection and CSP are the strongest motor-imagery decoders on average but are not universally best: their ordering depends on the dataset, they are statistically tied on the largest and most heterogeneous cohort, and the single best pipeline is optimal for only about a third of participants there, with nonlinear descriptors best for roughly another third. Feature dimensionality does not explain this pattern, and classifier and scaler choices are secondary to the feature representation.