AUTOMATIC TEXT SEGMENTATION INTO SEMANTICALLY HOMOGENEOUS FRAGMENTS (PARAGRAPHS)

A. V. Nicenko, V. Ju. Shelepov, S. A. Bolshakova

About the authors

A. V. Nicenko, V. Ju. Shelepov, S. A. Bolshakova

Ниценко Артём Владимирович – к.т.н. старший научный сотрудник отдела распознавания речевых образов, Федеральное государственное бюджетное научное учреждение «Институт проблем искусственного интеллекта», г. Донецк. Область научных интересов: искусственный интеллект, обработка естественного языка, компьютерная лингвистика. адрес: 283048, г. Донецк, ул. Артема, д. 118 б.

Шелепов Владислав Юрьевич – д.ф.-м.н. профессор, главный научный сотрудник отдела распознавания речевых образов, Федеральное государственное бюджет- ное научное учреждение «Институт проблем искусственного интеллекта», г. Донецк. Область научных интересов: искусственный интеллект, обработка естественного языка, компьютерная лингвистика. адрес: 283048, г. Донецк, ул. Артема, д. 118 б.

Большакова Светлана Анатольевна – младший научный сотрудник отдела распознавания речевых образов, Федеральное государственное бюджетное научное учреждение «Институт проблем искусственного интеллекта». Область научных интересов: искусственный интеллект, компьютерная лингвистика, интеллектуальный анализ информации. адрес: 283048, г. Донецк, ул. Артема, д. 118 б.

Download PDF

UDC: 004.89
DOI: 10.24412/2413-7383-2025-3-38-113-123
Language: Russian

Abstract: The work is devoted to the topical topic of segmentation of the Russian -language text into semantically homog eneous fragments. The paper provides an overview of a number of existing approaches to this task. An algorithm is proposed for automatically dividing text into paragraphs as thematically homogeneous fragments by using a ratio that takes into account the fr equency of occurrence of a word and the length of the text segment where it occurs. Using this software, experiments were conducted and the proposed algorithm was compared with other text segmentation methods. As a result, it was found that the proposed al gorithm demonstrates better performance than other compared approaches.

Keywords: semantic text segmentation, paragraph, word frequency, segment of occurrence.

References

1. Веб-сайт библиотеки NLTK. URL: https://www.nltk.org (дата обращения 18.09.2025). References
2. Chernobaev I.D. Surkova A.S. Overview of methods of thematic segmentation of text data // Information Systems and Technologies IST-2018: Proceedings of the XV International Scientific and Technical Conference dedicated to the 100th anniversary of the Nizhny Novgorod Radio Laboratory. Nizhnij Novgorod: NGTU im. R.E. Alekseeva, 2018. pp. 1079-1083.
3. Kirana R.P. Mukhrizal М. Jayanti F.G. Types of Lexical Cohesion and Grammatical Cohesion in Thesis Abstracts // Jadila: Journal of Development and Innovation in Language and Literature Education. 2020. vol. 1. no. 1. pp. 57-68. DOI: 10.52690/jadila.v1i1.14.
4. Muraj O.V. Cohesion and coherence in English speech in journalistic texts // Modern science: actual problems of theory and practice. Series: Humanities. 2021. no. 12-2. pp. 171-174. DOI: 10.37882/2223- 2982.2021.12-2.28.
5. Hearst M. A. TextTiling: Segmenting text into multiparagraph subtopic passages // Computational linguistics. 1997. vol. 23. no. 1. pp. 33-64.
6. Malioutov I. Barzilay R. Minimum cut model for spoken lecture segmentation // Proceedings of the ACL. 2006. pp. 25-32.
7. Galley M. McKeown K. Fosler-Lussier E. Jing H. Discourse segmentation of multi-party conversation // Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics. 2003. pp. 562-569.
8. Eisenstein J. Barzilay R. Bayesian unsupervised topic segmentation // Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2008. pp. 334-343.
9. Kharisudin I. Masri'an H. Topic Modeling on WhatsApp User Reviews Using Latent Dirichlet Allocation // Scandinavian Journal of Immunology. 2022. vol. 9. no. 1. pp. 51-62. DOI: 10.15294/sji.v9i1.34941.
10. Du L. Buntine W. Johnson M. Topic segmentation with a structured topic model // Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2013. pp. 190-200.
11. Avdeeva Z.K. Gavrilov M.S. Lemtjuzhnikova D.V. Sharafiev A.F. Methods for solving the problem of thematic segmentation of texts based on knowledge graphs // Proceedings of the Russian Academy of Sciences. Theory and management systems. 2024. № 4. С. 40-64. DOI: 10.31857/S0002338824040031.
12. Yu J. Shao H. Broadcast news story segmentation using sticky hierarchical Dirichlet process // Applied Intelligence. 2022. vol. 52. no. 11. pp. 12788-12800. DOI: 10.1007/s10489-021-03098-4.
13. Yu J. A DNN-HMM Approach to Story Segmentation //INTERSPEECH. 2016. pp. 1527-1531.
14. Vetráb M. Using Hybrid HMM/DNN Embedding Extractor Models in Computational Paralinguistic Tasks // Sensors. 2023. vol. 23, no. 11. pp. 5208. DOI: 10.3390/s23115208.
15. Baranov A. M. Judina T. A. An algorithm for segmenting scientific articles that combines the principles of teaching with and without a teacher // New information technologies in automated systems. 2019. No.
16. Available at: https://cyberleninka.ru/article/n/algoritm-segmentatsii-nauchnyh-statey-sochetayuschiy- printsipy-obucheniya-s-uchitelem-i-bez-uchitelya (accessed: 18.09.2025).
17. Javeed A. An LSTM model for extracting hierarchical relations between words for better topic modeling // Journal of Physics: Conference Series. 2021. vol. 1780, no. 1. pp. 12 -19. DOI: 10.1088/1742 - 6596/1780/1/012019. 3Н Ниценко А. В. Шелепов В. Ю. Большакова С.А.
18. Inzer S. Cheng K. Leung A. Shen X. Multi-scale Hybridized Topic Modeling: A Pipeline for Analyzing Unstructured Text Datasets via Topic Modeling // SIAM Undergraduate Research Online. 2023. vol. 16. DOI: 10.1137/22s1536832.
19. Lomakina L. S. Surkova A. S. Zhevnerchuk D. V. Chernobaev I. D. Text structures synthesis on the basis of their system-forming characteristics // IV International Research Conference "Information Technologies in Science, Management, Social Sphere and Medicine" (ITSMSSM 2017). Advances in Computer Science Research (ACSR). 2017. vol. 72. pp. 108-113.
20. Memon M. Q. Lu Yu. Chen P. An ensemble clustering approach for topic discovery using implicit text segmentation // Journal of Information Science. 2021. vol. 47, no. 4. pp. 431-457. DOI: 10.1177/0165551520911590.
21. Sonata I. Heryadi Ya. Tho C. Topic Segmentation using Transformer Model for Indonesian Text // Procedia Computer Science. 2023. vol. 227. pp. 159-167. DOI: 10.1016/j.procs.2023.10.513.
22. Iikura R. Okada M. Mori N. Improving BERT with Focal Loss for Paragraph Segmentation of Novels // Distributed Computing and Artificial Intelligence, 17th International Conference. DCAI 2020. Advances in Intelligent Systems and Computing. 2021. Vol. 1237. DOI: 10.1007/978-3-030-53036-5_3.
23. Solbiati A. Heffernan K. Damaskinos G. Poddar S. Modi S. Cali J. Unsupervised Topic Segmentation of Meetings with BERT Embeddings // arXiv preprint arXiv:2106.12978. 2021. DOI: 10.48550/arXiv.2106.12978.
24. Sokol V. Krykun V. Bilova M. Topic segmentation methods comparison on computer science texts // Bulletin of the National Technical University "KHPI". A series of System analysis, management and information technology. 2021. no. 2 (6). pp. 59-66. DOI: 10.20998/2079-0023.2021.02.10.
25. Hagen M. The complete paradigm. Morphology. Available at: https://ru.z-lib.fm/book/3305205/3a85b7/ (acessed: 15.09.2025).
26. Choi F. Y. Y. Advances in domain independent linear text segmentation. 2000. arxiv preprint arXiv: cs/ 0003083.
27. Pevzner L. Hearst M. A. A critique and improvement of an evaluation metric for text segmentation //

Issue: 3(38)'2025
Section: SYSTEM ANALYSIS, CONTROL AND INFORMATION PROCESSING, STATISTICS
How to cite: A. V. Nicenko, V. Ju. Shelepov, S. A. Bolshakova. AUTOMATIC TEXT SEGMENTATION INTO SEMANTICALLY HOMOGENEOUS FRAGMENTS (PARAGRAPHS) // Problems of Artificial Intelligence. - 2025. - № 3 (38). - P. 113-123. - https://paijournal.guiaidn.ru/en/2025/3(38)-10.html

Back to issue 3(38)'2025