UNetFormer: A UNet-like Transformer for Efficient Semantic Segmentation of Remote Sensing Urban Scene Imagery

Introduction

Driven by advances in sensor technology, fine-resolution remotely sensed urban scene images have been captured increasingly across the globe, with abundant spatial details and rich potential semantic contents. As for semantic segmentation, per-pixel classification is often ambiguous if only local information is modelled, while the semantic content of each pixel becomes more accurate with the help of global contextual information .

The global and local contextual information is illustrated in Fig. 1 .

The global contextual information is modelled by longrange window-wise dependencies (red).

Methodology

Urban scene images have been subjected extensively to semantic segmentation, the task of pixel-level segmentation and classification, leading to various urban-related applications, including land cover mapping , change detection , environmental protection , road and building extraction and many other practical applications . Recently, a growing wave of deep learning technology , in particular the convolutional neural network (CNN), has dominated the task of semantic segmentation .

Study Design

The trade-off between accuracy and efficiency as well as effective feature refinement allows the proposed method to exceed the state-of-the-art lightweight networks for efficient segmentation of remotely sensed urban scene images, demonstrated by four public datasets: the UAVid , ISPRS Vaihingen and Potsdam datasets, as well as the LoveDA .

Since then, CNN-based methods have dominated the semantic segmentation task in the remote sensing field .

Results & Findings

Compared with traditional machine learning methods for segmentation, such as the support vector machine (SVM) , random forest and conditional random field (CRF) , CNN-based methods are capable of capturing more fine-grained local context information, which underpins its huge capabilities in feature representation and pattern recognition . Despite the above advantages, the convolution operation with a fixed receptive view is designed to extract local patterns and lacks the ability to.

• Compared with traditional machine learning methods for segmentation, such as the support vector machine (SVM) , random forest and conditional random field (CRF) , CNN-based methods.
• Despite the above advantages, the convolution operation with a fixed receptive view is designed to extract local patterns and lacks the ability to model global contextual.
• Although the selfattention mechanism alleviates the above issue , they normally require significant computational time and memory to capture the global context, thus, reducing their efficiency.
• In this paper, we aim to achieve precise urban scene segmentation while ensuring the efficiency of the network simultaneously.
• In Section 2, we review the related work on CNN-based and Transformer-based urban scene segmentation and global context modelling.

Global Contextual Information Modelling

This section reviews various methods for integrating global contextual information into segmentation networks, particularly through attention mechanisms. It emphasizes the limitations of existing approaches that rely heavily on convolutional operations.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Fig. 1: Fig. 1 Illustration of the global and local contextual information. The local contextual information.
• Fig. 2: Fig. 2 Illustration of (a) the standard Transformer block and (b) the global-local Transformer block.
• Figure 3: related real-time applications. Thus, to fully harness the global context extraction ability of Transformers without resulting in high computational complexity, in this paper, we present a UNet-like Transformer with a CNN-based encoder and a Transformer-based decoder for efficient semantic segmentation of remotely sensed urban scene images. Specifically, for our UNetFormer, we select the lightweight backbone, i.e. ResNet18, as the encoder and develop an efficient globallocal attention mechanism to construct Transformer blocks in the decoder. The proposed efficient global-local attention mechanism adopts a dual-branch structure, i.e. a global branch and a local branch. Such a structure allows the attention block to capture both global and local contexts, thereby surpassing the single-branch efficient attention mechanisms in Transformers that only capture global contexts.
• Fig. 3 .: Fig. 3. An overview of the UNetFormer.
• Figure 5: global-local Transformer block consists of the global-local attention, multilayer perceptron, two batch normalization layers and two additional operations, as shown in Fig. 1 (b). Efficient Global-local attention: Although the global context is crucial for semantic segmentation of complex urban scenes, local information is still essential to preserve rich spatial details. In this regard, the proposed efficient global-local attention constructs two parallel branches to extract the global and local contexts, respectively, as shown in Fig. 4 (a). As a relatively shallow structure, the local branch employs two parallel convolutional layers with kernel sizes of 3 and 1 to extract the local context. Two batch normalization operations are then attached before the final sum operation. The global branch deploys the window-based multi-head self-attention to capture global context. As illustrated in Fig 4. (b), we first use a standard 1\uf0cd1 convolution to expand the channel dimension of the input 2D feature map \u2208 \u211d \ud835\udc35\ud835\udc35\u00d7\ud835\udc36\ud835\udc36\u00d7\ud835\udc3b\ud835\udc3b\u00d7\ud835\udc4a\ud835\udc4a to three times. Then, we apply the window partition operation to split the 1D sequence \u2208 and value (V) vectors. The channel dimension C is set to 64. The window size w and the number of heads h are both set to 8. The details of the window-based multi-head self-attention can refer to Swin Transformer (Liu et al., 2021) .

Conclusion

Section 6 is a summary and conclusion. Thus, the overall loss \u2112 can be formulated as:.