AN APPROACH TO MULTIMODAL DATA FUSION IN TASKS OF REGRESSION AND CLASSIFICATION IN DRONE IMAGES

B.V. Pavlenko

About the authors

B.V. Pavlenko

Павленко Богдан Викторович – младший научный сотрудник, аспирант, ФГБНУ «Институт проблем искусственного интеллекта», 283048, г. Донецк, ул. Артема, д. 118 б. Область научных интересов: системы распознавания образов, нейронные сети, мультимодальные модели.

Download PDF

UDC: 004.932.2
DOI: 10.24412/2413-7383-2025-4-39-71-83
Language: Russian

Abstract: In this paper, a method of multimodal feature combination is proposed using images from an unmanned aerial vehicle (UAV) and encoded metadata. The applicability of the proposed approach to the tasks of UAV position regression and object classification in images with a minimal set of known metadata is considered. For regression, a mechanism for interpolating altitude values is introduced, which allows for a more accurate association between the visual representation of an image and its altitude, and provides greater stability to changes in altitude and camera rotation angles of the UAV. To regulate the contribution of textual modality, a method is used that ensures random ignoring of textual features and reduces overfitting on textual modality in addition to regularization by the feature alignment error and its coefficient of influence on the overall model error. The same approach is used for classification, with the comparison of several variants of textual templates. Experiments were conducted on the following datasets.

Keywords: multimodal neural networks, regression, classification, feature fusion, positioning, UAV.

References

1. Pikalyov, Ya. S. Key Object Detection and Cross-Geolocalization: Dataset Analysis and Methodological Prospects // Problems of Artificial Intelligence. - 2024. - Vol. 35. - No. 4. - Pp. 25-37.
2. Pikalyov, Ya. S. Ermoolenko, T.V. On Neural Feature Extraction Architectures for Object Recognition on Devices with Limited Computing Power // Problems of Artificial Intelligence. - 2023. - No. 3 (30). - Pp. 44-45
3. Sheng K. Zhan H. Xie Z. Chen H. Xu Y. Tang L. Luo B. GeoText-1652: A Benchmark for Language- Aware Cross-View Geo-Localization // European Conference on Computer Vision (ECCV). – 2024.
4. Xu Y. Zhang Z. He W. Wang J. Luo B. A Survey of Open-Vocabulary Object Detection for UAV Imagery // Drones. – 2025. – Vol. 9, No. 1. – P. 12. – DOI: 10.3390/drones9010012.
5. Sun X. et al. Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding //arXiv preprint arXiv:2505.12194. – 2025.
6. Ye J. et al. Where am I? Cross-View Geo-localization with Natural Language Descriptions //arXiv preprint arXiv:2412.17007. – 2024.
7. Pavlenko, B.V. Methodology for Creating a Set of Aerial Photographs for the Task of Cross- Geolocalization / B. V. Pavlenko, Ya. S. Pikalev // Problems of Artificial Intelligence. - 2024. - No. 4 (35). - Pp. 101-112. - DOI: 10.24412/2413-7383-2024-4-101-112.
8. Pikalyov, Ya. S. Detection of Key Objects and Cross-Geolocalization: Analysis of Datasets and Methodological Prospects // Problems of Artificial Intelligence. - 2024. - Vol. 35. - No. 4. - Pp. 25-37. П Павленко Б.В.
9. Zuev, V.M. Ivanova S. B. Estimation of the Device’s Own Location Based on Video Image Analysis // Problems of Artificial Intelligence. – 2024. – T. 33. – No. 2. – Pp. 21-28.
10. Ji Y. et al. Game4loc: A uav geo-localization benchmark from game data //Proceedings of the AAAI Conference on Artificial Intelligence. – 2025. – Т. 39. – №. 4. – С. 3913-3921.
11. Xu W. et al. Uav-visloc: A large-scale dataset for uav visual localization //arXiv preprint arXiv:2405.11936. – 2024.19:06
12. Yang Y. Newsam S. Bag-of-visual-words and spatial extensions for land-use classification //Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems. – 2010. – С. 270-279. Kim W. Son B. Kim I. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision // arXiv preprint, arXiv:2102.03334, 2021.
13. Huang S. et al. Language is not all you need: aligning perception with language models. arXiv //Preprint posted online February. – 2023. – Т. 27.
14. Tan H. Bansal M. LXMERT: Learning cross-modality encoder representations from transformers [Электронный ресурс] // arXiv preprint arXiv:1908.07490. – 2019. – URL: https://arxiv.org/abs/1908.07490 (дата обращения: 18.09.2025).
15. Radford, A. Kim, J. W. Hallacy, C. Ramesh, A. Goh, G. Agarwal, S. Sastry, G. Askell, A. Mishkin, P. Clark, J. Krueger, G. Sutskever, I. Learning Transferable Visual Models From Natural Language Supervision // arXiv preprint arXiv:2103.00020, 2021. URL: https://arxiv.org/abs/2103.00020
16. Qu G. et al. Stripnet: Towards topology consistent strip structure segmentation //Proceedings of the 26th ACM international conference on Multimedia. – 2018. – С. 283-291.
17. Faghri F. et al. MobileCLIP2: Improving Multi-Modal Reinforced Training //arXiv preprint arXiv:2508.20691. – 2025.
18. Cui B. Liu Z. Yang Q. UAV-YOLO12: A Multi-Scale Road Segmentation Model for UAV Remote Sensing Imagery //Drones. – 2025. – Т. 9. – №. 8. – С. 533.
19. Less Wright, Nestor Demeure et al. Ranger: synergistic combination of RAdam + Lookahead for the best of both [Электронный ресурс] // GitHub / arXiv preprint. – 2019. – URL: https://arxiv.org/abs/2004.01461 (дата обращения: 18.09.2025).
20. Liu L. Jiang H. He P. Chen W. Liu X. Gao J. Han J. On the Variance of the Adaptive Learning Rate and Beyond [Электронный ресурс] // arXiv preprint. – 2019. – № arXiv:1908.03265. – URL: https://arxiv.org/abs/1908.03265. (дата обращения: 18.06.2025).
21. Zhang M. R. Lucas J. Hinton G. Ba J. Lookahead Optimizer: k steps forward, 1 step back [Электронный

Issue: 4(39)'2025
Section: ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING
How to cite: B.V. Pavlenko. AN APPROACH TO MULTIMODAL DATA FUSION IN TASKS OF REGRESSION AND CLASSIFICATION IN DRONE IMAGES // Problems of Artificial Intelligence. - 2025. - № 4 (39). - P. 71-83. - https://paijournal.guiaidn.ru/en/2025/4(39)-7.html

Back to issue 4(39)'2025