DECAF: DYNAMIC ENVELOPE CONTEXT-AWARE FUSION FOR SPEECH-ENVELOPE RECONSTRUCTION FROM EEG

Introduction

Electroencephalography (EEG)-based auditory attention decoding (AAD) determines which speaker a listener is attending to in multi-speaker environments. Recent deep learning models based on CNNs, LSTMs, and Transformers have significantly improved reconstruction performance.

As illustrated in Fig. 1 (top), each window of EEG is processed in isolation, ignoring the powerful temporal dependencies inherent in a continuous signal like speech.

This limitation motivates the question of whether decoding can be enhanced by creating models that are aware of temporal context.

Methodology

A common intermediate task in AAD reconstructs the envelope of the attended audio stream from EEG signals to enable downstream correlation-based decoding pipelines. This inherent predictability suggests that the task of envelope reconstruction is fundamentally dynamic rather than static.

Study Design

This insight motivates a shift in approach from stateless regression to state-aware estimation.

Our contributions are threefold: (1) we introduce DECAF (Dynamic Envelope Context-Aware Fusion), a novel deep learning architecture that reframes auditory envelope decoding from a static regression problem into a dynamic state-estimation task by integrating direct neural evidence with a predictive temporal prior; (2) the proposed model achieves state-of-the-art performance on Task 2 of the IEEE.

Results & Findings

Accurate envelope reconstruction is critical for downstream AAD and neuro-steered hearing aids because the reconstructed envelope is correlated with audio streams to identify the attended speaker. These approaches are limited by treating envelope reconstruction as a direct mapping problem where each window of EEG is processed without an explicit, predictive model of the speech envelope’s temporal structure.

• Accurate envelope reconstruction is critical for downstream AAD and neuro-steered hearing aids because the reconstructed envelope is correlated with audio streams to identify the attended speaker.
• These approaches are limited by treating envelope reconstruction as a direct mapping problem where each window of EEG is processed without an explicit, predictive model of.
• The key to creating a context-aware model lies in leveraging the strong temporal structure of the speech signal itself, which is often available to a BCI.
• We introduce a state-space fusion model that combines EEG-derived estimates of the current envelope with an autoregressively generated temporal prior from past predictions, inspired by classical.
• A learned gating mechanism adaptively balances neural evidence and temporal context within a fully causal architecture that relies only on the history of the EEG signal.

Conclusion And Future Work

CONCLUSION AND FUTURE WORK This work was motivated by the hypothesis that making EEG to auditory envelope decoding models aware of temporal context can yield more accurate and coherent reconstructions. The results from our DECAF model affirm this, demonstrating how a recursive temporal prior fosters a synergistic fusion between past context and present neural evidence for robust envelope reconstruction. The proposed DECAF framework is methodological, modular, and data-agnostic, with a fully causal and recursive design that relies only on past EEG samples and previous model outputs, making it directly applicable.

Related Works

RELATED WORKS The foundational principle behind AAD is that low-frequency cortical responses entrain to the temporal envelope of the attended speech stream [10, 11] . This enables reconstructing the attended envelope from EEG and identifying the listener’s focus [1] . While early work focused on linear Temporal Response Function (TRF) models [12] , recent deep learning architectures such as CNNs [3] , LSTMs [4] , and Transformers [5, 13] have become the standard, while newer paradigms such as self-supervised learning are also being explored to learn robust deep representations [14.

Dataset and Baselines

Dataset and Baselines Dataset: For this study, we adhere strictly to the protocol and dataset from the ICASSP 2023 Auditory EEG Decoding Challenge (Task 2) [19] . The dataset features 64-channel EEG recordings from 85 subjects listening to narrated stories. We used the officially provided preprocessed data, downsampled to 64 Hz, and the prescribed training, validation, and test splits for all experiments to ensure reproducibility [20] . Evaluation is performed exclusively on the unseen-stimulus test set, which contains novel audio segments from the same subjects used during training, with no.

Proposed Model: DECAF

Proposed Model: DECAF The architecture of Dynamic Envelope Context-Aware Fusion model is designed to implement our dynamic, state-aware decoding paradigm. As illustrated in Fig. 2 , the model reframes envelope reconstruction as a state-space estimation problem, where a prediction from the past is updated by evidence from EEG to Envelope Module: This module provides a direct, neurally-driven estimate of the speech envelope. It uses the HappyQuokka model’s architecture as a powerful feature encoder [13] , which is trained from scratch as part of our endto-end system. Given a 3-second window.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Fig. 1 .: Fig. 1. Illustration of the shift from static to dynamic decoding. (Top) Static baselines perform stateless reconstruction using only an isolated ‘Present Window’ of EEG. (Bottom) Our dynamic model, DECAF, is state-aware, creating a temporal prior from past context and fusing it with present EEG information. The dashed loop indicates the model’s fully recursive operation, using its own past predictions.
• Fig. 2 .: Fig. 2. The system generates the current envelope prediction (A n ) by fusing a direct neural estimate from the EEG (E n ) with a temporal prediction derived from its own past output (A n-1 ).
• Fig. 3 .: Fig.3. The baseline models (left three panels) effectively capture low-frequency energy but fail to reconstruct higher-frequency details compared to the ground truth (black). The rightmost panel decomposes our proposed model, DECAF. The final Fusion output (blue) synergistically combines the low-frequency accuracy of the EEG branch (red) with the high-frequency information from the Envelope Forecaster (orange), allowing it to track the ground-truth spectrum with significantly higher fidelity.
• Fig. 4 .: Fig. 4. Reconstruction performance across varying EEG noise levels (SNR).chance (M = 0.016), confirming that the final model effectively integrates complementary information from both streams. The model also surpasses other baselines, including VLAAI and a linear mTRF model. Furthermore, we evaluated DECAF Oracle, an upper-bound version trained using a ground-truth past envelope as context, which achieved the highest performance (median \u2248 0.20), demonstrating the full potential of our fusion paradigm.Complementary Spectral Contributions: A Power Spectral Density (PSD) analysis (Figure3) reveals that while baseline models capture low-frequency neural entrainment (<10 Hz), they fail to reconstruct higher-frequency details. DECAF overcomes this through synergistic fusion: its EEG branch captures the low-frequency (1-8 Hz) neural signal, while the Envelope Forecaster's temporal prior preserves the higher-frequency components. By intelligently integrating these complementary streams, the final model restores high-frequency power while retaining low-frequency accuracy, resulting in a spectrally complete envelope that matches the ground truth with high fidelity and explains the model's state-of-the-art performance.Ablation: Effect of White Noise: To assess robustness under varying input quality, we tested models on EEG data.