Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms.
Published in | American Journal of Remote Sensing (Volume 12, Issue 1) |
DOI | 10.11648/j.ajrs.20241201.11 |
Page(s) | 1-13 |
Creative Commons |
This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited. |
Copyright |
Copyright © The Author(s), 2024. Published by Science Publishing Group |
Pansharpening, Multiscale, Positive Feedback, Attention Mechanism
[1] | Vivone, G., Dalla Mura, M., Garzelli, A., Restaino, R., Scarpa, G., Ulfarsson, M.O., Alparone, L., Chanussot, J. A New Benchmark Based on Recent Advances in Multispectral Pansharpening: Revisiting Pansharpening With Classical and Emerging Pansharpening Methods. IEEE Geoscience and Remote Sensing Magazine. 2021, 9(1), 53-81. doi: 10.1109/MGRS.2020.3019315. |
[2] | Yilmaz, C. S., Yilmaz, V., Gungor, O. A theoretical and practical survey of image fusion methods for multispectral pansharpening. Information Fusion. 2022, 79, 1-43. doi: 10.1016/J.INFFUS.2021.10.001. |
[3] | Kwarteng, P., Chavez, A. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis. Photogrammetric Engineering and Remote Sensing. 1989, 55(3), 339-348. doi: 0099-1112/89/5503-339$02.25/0. |
[4] | Rahmani, S., Strait, M., Merkurjev, D., Moeller, M., Wittman, T. An Adaptive IHS Pan-Sharpening Method. IEEE Geoscience and Remote Sensing Letters. 2010, 7(4), 746-750. doi: 10.1109/LGRS.2010.2046715. |
[5] | Garzelli, A., Nencini, F., Capobianco, L. Optimal MMSE Pan Sharpening of Very High Resolution Multispectral Images. IEEE Transactions on Geoscience and Remote Sensing. 2008, 46(1), 228-236. doi: 10.1109/TGRS.2007.907604. |
[6] | Choi, J., Yu, K., Kim, Y. A new adaptive component- substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing. 2010, 49(1), 295-309. doi: 10.1109/TGRS.2010.2051674. |
[7] | Shensa, M. J. The discrete wavelet transform: wedding the a trous and Mallat algorithms. IEEE Transactions on Signal Processing. 1992, 40(10), 2464-2482. doi: 10.1109/78.157290. |
[8] | Otazu, X., Gonzalez-Audicana, M., Fors, O., Nunez, J. Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing. 2005, 43(10), 2376-2385. doi: 10.1109/TGRS.2005.856106. |
[9] | Palsson, F., Sveinsson, J. R., Ulfarsson, M. O., Benediktsson, J. A. Model-Based Fusion of Multi- and Hyperspectral Images Using PCA and Wavelets. IEEE Transactions on Geoscience and Remote Sensing. 2015, 53(5), 2652-2663. doi: 10.1109/TGRS.2014.2363477. |
[10] | Palsson, F., Sveinsson, J. R., Ulfarsson, M. O. A New Pansharpening Algorithm Based on Total Variation. IEEE Geoscience and Remote Sensing Letters. 2014, 11(1), 318-322. doi: 10.1109/LGRS.2013.2257669 |
[11] | Fu, X., Lin, Z., Huang, Y., Ding, X. A Variational Pan- Sharpening With Local Gradient Constraints. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019; pp. 10265-10274. doi: 10.1109/CVPR.2019.01051. |
[12] | Wang, X., Bai, S., Li, Z., Song, R., Tao, J. The PAN and MS Image Pansharpening Algorithm Based on Adaptive Neural Network and Sparse Representation in the NSST Domain. IEEE Access. 2019, 7, 52508-52521. doi: 10.1109/ACCESS.2019.2910656. |
[13] | Tian, X., Chen, Y., Yang, C., Gao, X., Ma, J. A Variational Pansharpening Method Based on Gradient Sparse Representation. IEEE Signal Processing Letters. 2020, 27, 1180-1184. doi: 10.1109/LSP.2020.3007325. |
[14] | Lang, W., Zhao, Z., Fang, S., Cao, Y., Wang, Y. Sparse representation-based detail-injection method for pan-sharpening. Journal of Applied Remote Sensing. 2020, 14(2), 026523. doi: 10.1117/1.JRS.14.026523. |
[15] | Gharbia, R., Hassanien, A. E., El-Baz, A. H., Elhoseny, M., Gunasekaran, M. Multi-spectral and panchromatic image fusion approach using stationary wavelet transform and swarm flower pollination optimization for remote sensing applications. Future Generation Computer Systems. 2018, 88, 501-511. doi: 10.1016/j.future.2018.06.022. |
[16] | Yilmaz, V., Yilmaz, C. S., Gungor, O. Genetic algorithm-based synthetic variable ratio image fusion. Geocarto International. 2019, 36(9), 1-18. doi: 10.1080/10106049.2019.1629649. |
[17] | Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G. Pansharpening by Convolutional Neural Networks. Remote Sensing. 2016, 8(7), 594. doi: 10.3390/rs8070594. |
[18] | Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., Paisley, J. PanNet: A Deep Network Architecture for Pan- Sharpening. In 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017; pp. 1753- 1761. doi: 10.1109/ICCV.2017.193. |
[19] | Xu, S., Zhang, J., Zhao, Z., Sun, K., Liu, J., Zhang, C. Deep Gradient Projection Networks for Pan-sharpening. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021; pp. 1366-1375. doi: 10.1109/CVPR46437.2021.00142. |
[20] | Xiang, Z., Xiao, L., Liu, P., Zhang, Y. A Multi-Scale Densely Deep Learning Method for Pansharpening. In IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan, 2019; pp. 2786-2789. doi: 10.1109/IGARSS.2019.8898095. |
[21] | Liu, Q., Zhou, H., Xu, Q., Liu, X., Wang, Y. PSGAN: A Generative Adversarial Network for Remote Sensing Image Pan-Sharpening. IEEE Transactions on Geoscience and Remote Sensing. 2021, 59(12), 10227- 10242. doi: 10.1109/TGRS.2020.3042974. |
[22] | Ma, J., Yu, W., Chen, C., Liang, P., Guo, X., Jiang, J. Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion. 2020, 62, 110-120. doi: 10.1016/j.inffus.2020.04.006. |
[23] | Wang, J., Shao, Z., Huang, X., Lu, T., Zhang, R., Ma, J. Pan-Sharpening Via High-Pass Modification Convolutional Neural Network. In 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA, 2021; pp. 1714-1718. doi: 10.1109/ICIP42928.2021.9506568. |
[24] | Hu, J., Shen, L., Sun, G. Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020, 42(8), 2011-2023. doi: 10.1109/TPAMI.2019.2913372. |
[25] | Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K. Spatial Transformer Networks. Advances in Neural Information Processing Systems. 2015, 28, 2015-2017. doi: 10.48550/arXiv.1506.02025. |
[26] | Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 2018; pp. 286-301. doi: 10.48550/arXiv.1807.02758. |
[27] | Wang, X., Girshick, R., Gupta, A., He, K. Non- local Neural Networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018; pp. 7794-7803. doi: 10.1109/CVPR.2018.00813. |
[28] | He, K., Zhang, X., Ren, S., Sun, J. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016; pp. 770-778. doi: 10.1109/CVPR.2016.90. |
[29] | Simonyan, K., Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences. 2014. doi: 10.48550/arXiv.1409.1556. |
[30] | Woo, S., Park, J., Lee, J. Y., Kweon, I. S. CBAM: Convolutional Block Attention Module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 2018; pp. 3-19. doi: 10.48550/arXiv.1807.06521. |
[31] | Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G.A., Restaino, R., Wald, L. A Critical Comparison Among Pansharpening Algorithms. IEEE Transactions on Geoscience and Remote Sensing. 2015, 53(5), 2565-2586. doi: 10.1109/TGRS.2014.2361734. |
[32] | Zhang, Y., Liu, C., Sun, M., Ou, Y. Pan-sharpening using an efficient bidirectional pyramid network. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8), 5549-5563. doi: 10.1109/TGRS.2019.2900419. |
[33] | Garzelli, A., Nencini, F. Hypercomplex Quality Assessment of Multi/Hyperspectral Images. IEEE Geoscience and Remote Sensing Letters. 2009, 6(4), 662-665. doi: 10.1109/LGRS.2009.2022650. |
[34] | Zhou, J., Civco, D.L., Silander, J. A wavelet transform method to merge Landsat TM and SPOT panchromatic data. International Journal of Remote Sensing. 1998, 19(4), 743-757. doi: 10.1080/014311698215973. |
[35] | Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F., Selva, M. Multispectral and panchromatic data fusion assessment without reference. Photogrammetric Engineering and Remote Sensing. 2008, 74(2), 193-200. doi: 10.14358/PERS.74.2.193. |
APA Style
Lei, D., Xiao, L., Li, Y., Huang, H., Chen, X., et al. (2024). A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening. American Journal of Remote Sensing, 12(1), 1-13. https://doi.org/10.11648/j.ajrs.20241201.11
ACS Style
Lei, D.; Xiao, L.; Li, Y.; Huang, H.; Chen, X., et al. A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening. Am. J. Remote Sens. 2024, 12(1), 1-13. doi: 10.11648/j.ajrs.20241201.11
@article{10.11648/j.ajrs.20241201.11, author = {Dajiang Lei and Lang Xiao and Yujia Li and Hefeng Huang and Xiaoyu Chen and Tingting Zhou and Shixing Ou and Liping Zhang}, title = {A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening}, journal = {American Journal of Remote Sensing}, volume = {12}, number = {1}, pages = {1-13}, doi = {10.11648/j.ajrs.20241201.11}, url = {https://doi.org/10.11648/j.ajrs.20241201.11}, eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajrs.20241201.11}, abstract = {Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms.}, year = {2024} }
TY - JOUR T1 - A Multiscale Delayed Channel Attention Network-Based Method for Pansharpening AU - Dajiang Lei AU - Lang Xiao AU - Yujia Li AU - Hefeng Huang AU - Xiaoyu Chen AU - Tingting Zhou AU - Shixing Ou AU - Liping Zhang Y1 - 2024/01/21 PY - 2024 N1 - https://doi.org/10.11648/j.ajrs.20241201.11 DO - 10.11648/j.ajrs.20241201.11 T2 - American Journal of Remote Sensing JF - American Journal of Remote Sensing JO - American Journal of Remote Sensing SP - 1 EP - 13 PB - Science Publishing Group SN - 2328-580X UR - https://doi.org/10.11648/j.ajrs.20241201.11 AB - Nowadays, remote sensing images are widely used in many fields. To obtain high-quality remote sensing images, remote sensing image fusion methods have attracted much attention. Although convolutional neural network-based pansharpening methods have good results, these methods focus on the forward propagation of the network, which cannot effectively seek the mapping relationship between images. Moreover, it is difficult to obtain global information due to the limitations of convolutional operations. In this paper, we propose a pansharpening method based on multiscale delayed channel attention networks. The method iteratively seeks to correlate high-resolution stage features with the original low-resolution multispectral image, providing a mechanism for error feedback to map the error of each stage. Meanwhile, it designs a multiscale feature fusion module to fuse feature information from different fields of view. The design of the delayed channel attention mechanism makes the network acquire the association relationship between low-frequency information and high-frequency information through adaptive learning, giving different weights to high-frequency information, and making it more flexible in dealing with different types of information. Finally, the feature aggregation module is used to generate fused images and adjust the corresponding feature information. The experimental results obtained from the Gaofen-2 and WorldView-2 experimental data show that the method achieves a significant improvement compared to the current fusion algorithms. VL - 12 IS - 1 ER -