ANALYZING THE EFFECTIVENESS OF DEEP LANGUAGE MODELS FOR THE TASK OF TONE DETECTION IN RUSSIAN-LANGUAGE TEXTS

About the magazine

News

Goals and sphere

Founder and publisher

Editorial Board

Licensing conditions

Confidentiality

Attitude towards plagiarism

Publication ethics

Archiving Policy

Subscription

For authors

Instructions for authors

The review process

Copyright

Agreement on the transfer of rights

Editorial fees

Archive

All issues

Search

Contacts

Contacts

Bondarenko Vitaly Ivanovich
Candidate of Technical Sciences, Associate Professor of the Department of Computer Technologies, Faculty of Physics and Technology
Federal State Budgetary Educational Institution of Higher Education "Donetsk State University"
Area of scientific interests: Artificial intelligence, intelligent data analysis, machine learning, mathematical modeling of hydro- and thermophysical processes, development of user interfaces for applied modeling programs.

Eliseev Vadim Olegovich
Research Intern, Laboratory of Intelligent Systems
Federal State Budgetary Scientific Institution "Institute of Applied Mathematics and Mechanics"
Area of scientific interests: Artificial intelligence, machine learning, neural networks, natural language processing, generative and large language models.

Ermolenko Tatyana Vladimirovna
Candidate of Technical Sciences, Associate Professor of the Department of Computer Technologies, Faculty of Physics and Technology
Federal State Budgetary Educational Institution of Higher Education "Donetsk State University"
Area of scientific interests: Digital signal processing, data analysis, discrete mathematics, algorithm theory, pattern recognition, natural language processing, computer vision, machine learning, neural networks.

Download pdf

UDC 004.912
Language: Russian
Annotation: The article decribes the process of solving the task of sentiment analysis across texts of varying lengths, such as customer reviews and news articles. A methodology involving fine-tuning machine learning models based on RuGPT-3 and RuBERT is proposed, achieved through the substitution of the last linear layer with a classification layer having outputs corresponding to the number of classes (neutral, positive, negative). Research indicates the advantages of utilizing RuGPT-3- based models, revealing a notable increase in predictive quality despite their lower operational speed. Additionally, a comparison of models trained on one text type to predict sentiments in another was conducted. The results show that models trained on news articles exhibit slightly superior classification of reviews. However, the resulting accuracy falls short for the multimodal application of trained models.
Keywords: language model, natural language processing, sentiment analysis, fine-tuning, GPT, BERT.

List of literature:
1. Radford A. Improving Language Understanding by Generative Pre-Training [Electronic resource]. URL: https://gwern.net/doc/www/s3-us-wes2.amazonaws.com/d73fdc5ffa8627bce44dcda2fc012da638ffb158.pdf.
2. Zmitrovich D. A Family of Pretrained Transformer Language Models for Russian / D. Zmitrovich, A. Abramov, A. Kalmykov, M. Tikhonova, E. Taktasheva, D. Astafurov, M. Baushenko, A. Snegirev, T. Shavrina, S. Markov, V. Mikhailov, A. Fenogenova. 2023.
3. Kuratov Y. Adaptation of deep bidirectional multilingual transformers for Russian language / Y. Kuratov, M. Arkhipov // Komp’juternaja Lingvistika i Intellektual’nye Tehnologii. 2019. Тт. 2019-May.
4. Yermolenko T.V. Development of algorithms and language models for a multi-language system of automatic summary of texts of different genres / T.V. Yermolenko, V.I. Bondarenko, Ya.S. Pikalyov // Vestnik of the Donetsk National Univesity. Series D: Technical sciences. 2023. – № 2. P. 22-43.
5. Humphrey A. Machine-learning classification of astronomical sources: estimating F1-score in the absence of ground truth / A. Humphrey, W. Kuberski, J. Bialek, N. Perrakis, W. Cools, N. Nuyttens, H. Elakhrass, P.A.C. Cunha // Monthly Notices of the Royal Astronomical Society: Letters. 2022. Т. 517. № 1.
6. Russian-language reviews | Kaggle [Electronic resource]. URL: https://www.kaggle.com/datasets/laytsw/reviews.
7. Sentiment Analysis in Russian | Kaggle [Electronic resource]. URL: https://www.kaggle.com/competitions/sentiment-analysis-in-russian/data.
8. Yermolenko T.V. Classification of errors in the text based on deep learning / T.V. Yermolenko // Problems of Artificial Intelligence. – 2019. – № 3(14). – P. 47-57.
9. Pikalyov, Ya. S. The development of a text corpora normalization system / Ya. S. Pikalyov // Problems of Artificial Intelligence. – 2022. – № 2(25). – P. 64-78.
10. Ryan Hoens T. Imbalanced datasets: From sampling to classifiers / T. Ryan Hoens, N. V. Chawla // Imbalanced Learning: Foundations, Algorithms, and Applications. – 2013.
11. Bondarenko V.I. Classification of scientific texts using deep machine learning methods / В.И. Бондаренко // Vestnik of the Donetsk National Univesity. Series D. Technical sciences. – 2021. – № 3. – С. 69-77.
12. Webster J.J. Tokenization as the initial phase in NLP / J.J. Webster, C. Kit. – 1992.
13. Pikalyov, Ya. S. Adaptation of ALBERT neural network model for language modeling problem/ Ya. S. Pikalyov, T. V. Yermolenko// Problems of Artificial Intelligence. – 2020. – №3(18). – С. 111-122.
14. Wang C. Neural machine translation with byte-level subwords / C. Wang, K. Cho, J. Gu // AAAI 2020 - 34th AAAI Conference on Artificial Intelligence. – 2020.
15. Radford Alec. Language Models are Unsupervised Multitask Learners | Enhanced Reader [Электронный ресурс]. URL: https://life-extension.github.io/2020/05/27/GPT技术初探/language-models.pdf
16. Pikalyov, Ya. S. The development of the automatic transformation of english accents in russian texts with the application of deep learning / Ya. S. Pikalyov, T.V. Yermolenko // Problems of Artificial Intelligence. – 2019. – № 2(13). – P. 74-86.
17. Mao A. Cross-Entropy Loss Functions: Theoretical Analysis and Applications / A. Mao, M. Mohri, Y. Zhong. – 2023.
18. Loshchilov I. Decoupled weight decay regularization / I. Loshchilov, F. Hutter // 7th International Conference on Learning Representations, ICLR 2019. – 2019.
19. Xie Z. On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective / Z. Xie, Z. Xu, J. Zhang, I. Sato, M. Sugiyama. – 2020.
20. Touvron H. et al. Llama 2: Open foundation and fine-tuned chat models //arXiv preprint arXiv:2307.09288. – 2023.
21. Takagi S. On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning / S. Takagi // Advances in Neural Information Processing Systems. 2022. Т. 35.
22. Eliseev Vadim. Sentiment Analysis Fine Tuned [Electronic resource]. URL: https://github.com/EliseevVadim/sentiment-analysis-fine-tuned.

Release: 1(32)'2024
Chapter: Informatics, Computer Engineering and Control
How to quote: Bondarenko V. I. ANALYZING THE EFFECTIVENESS OF DEEP LANGUAGE MODELS FOR THE TASK OF TONE DETECTION IN RUSSIAN-LANGUAGE TEXTS // V. I. Bondarenko, V. O. Eliseev, T. V. Yermolenko // Problems of artificial intelligence. - 2024. № 1 (32). - С. 51-62. - http://paijournal.guiaidn.ru/ru/2024/1(32)-4.html

Back to 1(32)'2024