РУС
ENG
SEMANTIC TEXT ANALYSIS USING ARTIFICIAL NEURAL NETWORKS BASED ON NEURAL-LIKE ELEMENTS WITH TEMPORAL SIGNAL SUMMATION
Aims and Scope
Founder and Publisher
Editorial board
Licensing terms
Privacy Statement
Plagiarism policy
Publication ethics
Archiving Policy
Subscription
Instructions for authors
Reviewing proccess
Copyright Notice
License agreement
Article processing charges
All issues
Search
Contacts
Kharlamov Alexander
Institute of Higher Nervous Activity and Neurophysiology, RAS, Moscow;
Moscow State Linguistic University, Moscow
HSE University, Moscow
Moscow Institute of Physics and Technology, Moscow Region, RF
Research interests: neuroinformatics, semantic representations, automatic text processing, integrated robots, physiology of sensory systems
Samaev Eugeniy
NPP Garant-Service-Universitet, Moscow
Kuznetsov Dmitriy
Positive Technologies, Moscow
Pantiukhin Dmitriy
MIREA - Russian Technological University, Moscow
Area of interest: neural network, neurocomputer, neuromorphic devices, memristor, information security, neural network management system, computer vision, natural language processing
UDC 528.013
DOI 10.34757/2413-7383.2023.30.3.001
Language: English
Annotation: Text as an image is analyzed in the human visual analyzer. In this case, the image is scanned along the points of the greatest informativity, which are the inflections of the contours of the equitextural areas, into which the image is roughly divided. In the case of text analysis, individual characters of the alphabet are analyzed in this wayNext, the text is analyzed as repetitive language elements of varying complexity. Dictionaries of level-forming elements of varying complexity are formed, the top of which is the level of acceptable com-patibility of the root stems of words (names) in sentences of the text, that is, the semantic level. The level of semantics represented by pairs of root stems is virtually a homo-geneous directed semantic network. Re-ranking the weights of the network vertices corresponding to the root stems of individual names, as occurs in the hippocampus, makes it possible to move from the frequency characteristics of the network to their semantic weights. Such networks can be used to analyze texts that represent them: one can compare them with each other, c lassify and use to identify the most significant parts of texts (generate abstracts of texts), etc.
Keywords: text analysis; language model; neural network; transformer model; semantic analysis of texts; artificial neural networks based on neurons with temporal summation of signals; language levels; semantic level; TextAnalyst technology for semantic text analysis; applications.я.
References:
1. Lena Voita. Language Modeling. // Available online: https://lena-voita.github.io/nlp_course/language_modeling (accessed on 20 June 2023).
2. Zhao, Wayne Xin, et al. "A survey of large language models." arXiv preprint arXiv:2303.18223 (2023).
3. Y. Zhu, R. Kiros, R. S. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” in 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015. IEEE Computer Society, 2015, pp. 19–27
4. “Project Gutenberg.” [Online]. Available: https://www.gutenberg.org/ 5. “Wikipedia.” [Online]. Available: https://en.wikipedia.org/wiki/Main Page
6. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, p. 9, 2019
7. A. Gokaslan, V. C. E. Pavlick, and S. Tellex, “Openwebtext corpus,” http://Skylion007.github.io/OpenWebTextCorpus, 2019.
8. J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, and J. Blackburn, “The pushshift reddit dataset,” in Proceedings of the Fourteenth International AAAI Conference on Web and Social Media, ICWSM 2020, Held Virtually, Original Venue: Atlanta, Georgia, USA, June 8-11, 2020. AAAI Press, 2020, pp. 830– 839.
9. “Common crawl.” [Online]. Available: https://commoncrawl.org/
10. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” in Proceedings of the 2021 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, 2021, pp.483–498.
11.R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi, “Defending against neural fake news,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alch´e Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 9051–9062.
12. The Russian National Corpus (ruscorpora.ru). 2003—2023.
13. “Bigquery dataset.” [Online]. Available: https://cloud.google.com/bigquery?hl=zh-cn
14. Gaulton, A. et al. The ChEMBL database in 2017. Nucleic Acids Res. 45, D945–D954 (2017)
15. Merk, D., Friedrich, L., Grisoni, F. & Schneider, G. De novo design of bioactive small molecules by artificial intelligence. Mol. Inf. 37, 1700153 (2018).
16.Chip Huyen. Evaluation Metrics for Language Modeling \\ The Gradient. 2019 [Online]. Available: https://thegradient.pub/understanding-evaluation-metrics-for-language-models/
17. Gu, Jiuxiang, et al. "Recent advances in convolutional neural networks." Pattern recognition 77 (2018): 354-377.
18. Dumoulin, Vincent, and Francesco Visin. "A guide to convolution arithmetic for deep learning." arXiv preprint arXiv:1603.07285 (2016).
19. Oord, Aaron van den, et al. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016).
20. Dauphin, Yann N., et al. "Language modeling with gated convolutional networks." International conference on machine learning. PMLR, 2017.
21. Staudemeyer, Ralf C., and Eric Rothstein Morris. "Understanding LSTM--a tutorial into long short-term memory recurrent neural networks." arXiv preprint arXiv:1909.09586 (2019).
22. Sepp; Hochreiter and J¨urgen Schmidhuber. Long Short-Term Memory. Neural computation, 9(8):1735– 1780, 1997.
23.Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In arXiv, pages 1–9, dec 2014.
24. Sharma, Sagar, Simone Sharma, and Anidhya Athaiya. "Activation functions in neural networks." Towards Data Sci 6.12 (2017): 310-316.
25.Jozefowicz, R., Zaremba, W., & Sutskever, I. (2015, June). An empirical exploration of recurrent network architectures. In International conference on machine learning (pp. 2342-2350). PMLR.
26. Alex Graves and Jurgen Schmidhuber. Framewise phoneme classification with bidirectional LSTM networks. In Proc. of the Int. Joint Conf. on Neural Networks, volume 18, pages 2047–2052, Oxford, UK, UK, jun 2005. Elsevier Science Ltd.
27.Rae, Jack, et al. "Fast parametric learning with activation memorization." International Conference on Machine Learning. PMLR, 2018.
28. Grave, E., Joulin, A., and Usunier, N. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426, 2016b.
29.Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling." arXiv preprint arXiv:1803.01271 (2018).
30. Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).
31. Lin, Tianyang, et al. "A survey of transformers." AI Open (2022).
32. Kalyan, Katikapalli Subramanyam, Ajit Rajasekharan, and Sivanesan Sangeetha. "Ammus: A survey of transformer-based pretrained models in natural language processing." arXiv preprint arXiv:2108.05542 (2021).
33. Baevski, Alexei, and Michael Auli. "Adaptive input representations for neural language modeling." arXiv preprint arXiv:1809.10853 (2018).
34. Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina (11 October 2018). "BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding". arXiv:1810.04805v2
35. Luo, Linkai, and Yue Wang. "Emotionx-hsu: Adopting pre-trained bert for emotion classification." arXiv preprint arXiv:1907.09669 (2019).
36.Radford, Alec, et al. "Language models are unsupervised multitask learners." OpenAI blog 1.8 (2019): 9.
37. Shoeybi, Mohammad, et al. "Megatron-lm: Training multi-billion parameter language models using model parallelism." arXiv preprint arXiv:1909.08053 (2019).
38.Ren, Xiaozhe; Zhou, Pingyi; Meng, Xinfan; Huang, Xinjing; Wang, Yadao; Wang, Weichao; Li, Pengfei; Zhang, Xiaoda; Podolskiy, Alexander; Arshinov, Grigory; Bout, Andrey; Piontkovskaya, Irina; Wei, Jiansheng; Jiang, Xin; Su, Teng; Liu, Qun; Yao, Jun (March 19, 2023). "PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing". arXiv:2303.10845
39.Dai, Andrew M; Du, Nan (December 9, 2021). "More Efficient In-Context Learning with GLaM". ai.googleblog.com. Retrieved 2023-03-09.
40. Dey, Nolan, et al. "Cerebras-GPT: Open compute-optimal language models trained on the Cerebras wafer-scale cluster." arXiv preprint arXiv:2304.03208 (2023).
41. Kharlamov A. A. Assotsiativnaya pamyat’ – sreda dlya formirovaniya prostranstva znanij. Ot biologii k prilozheniyam. [Associative memory as an environment for the formation of a space of knowledge. From biology to applications]. Dusseldorf, Germany: Palmarium Academic Publishing - 109 p. (in Russian) Available online: Ассоциативная память — среда для формирования пространства знаний: От биологии к приложениям (Russian Edition): Харламов, Александр: 9783639645491: Amazon.com: Books (accessed on 12 April 2017).
42. Neuroinformatics and Semantic Representations. Theory and Applications. Alexander Kharlamov & Maria Pilgun eds. 317 P. Cambridge Scholars Publishing. 2020. Available online: Neuroinformatics and Semantic Representations: Theory and Applications - Cambridge Scholars Publishing (accessed on 2020).
43.Rolls, E.T. Theoretical and Neurophysiological Analysis of the Functions of the Primate Hippocampus in Memory. In: Cold Spring Harbor Symposia on Quantitative Biology, Vol. LV, 1990, Cold Spring Harbor Laboratory Press. Pp. 995 – 1006. Available online: Dendritic organization in the neurons of the visual and motor cortices of the cat - PMC (nih.gov) (accessed on 1953).
44. Vinogradova O.S. Gippokamp i pamyat’. [Hippocampus and memory] Moscow: Nauka, 1975. - 336 p. (in Russian) Available online: Vinogradova, Okga Sergeevna – Gippokamp i pamyat’ [Текст] - Search RSL (accessed on 1975).
45. Hopfield, J.J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. 79, 1982. Pp. 2554 – 2558. Available online: Neural networks and physical systems with emergent collective computational abilities. - PMC (nih.gov) (accessed on 1982).
46.Buzikashvili N.E., Samoylov D.V., Krylova G.A. N-grammy v lingvistike. [N-grams in linguistics]// In Collection of papers: Methods and means of document management. Moscow: Editorial URRS. 2000. Pp. 91-130. (in Russian) Available online: Metody I sredstva raboty s dokumentami | | Editorial URSS | Knigi po reklame, marketingu, PR I dizayinu | Advertology.Ru (accessed on 2000).
47. Kharlamov A.A. Svidetelstvo o registratsii programmy “Programma dlya avtomaticheskogo smyslovogo analiza tekstov na osnove neyronnykh setey “TextAnalyst”” [Certificate of registration of the program "Program for automatic semantic text processing based on neural networks "TextAnalyst""]. Available online: ww.fips.ru/vse-servisy.php (accessed on 31 October 1997).
48. Kharlamov A.A. Sposob avtomatizirovannoj semanticheskoj indeksatsii teksta na estestvennom yazyke. [A method for automated semantic indexing of natural language text] Patent for invention No. 2518946, priority dated November 27, 2012. Registered April 11, 2014 (in Russian) Available online: A METHOD OF AUTOMATED SEMANTIC INDEXING OF TEXT IN NATURAL LANGUAGE. Patent No. RU 2518946 IPC G06F40/20 | Patent Exchange - Moscow Innovation Cluster (i.moscow) (accessed on 2014).
49.R-sistema. Vvedenie v ekonomicheskij shpionazh. Praktikum po ekonomicheskoj razvedke v sovremennom rossijskom predprinimatel’stve. [I-system. Introduction to economic espionage. Workshop on economic intelligence in modern Russian business] In 2 volumes. Moscow, Russia: “Hamtek Publisher”, 1997. (in Russian) Available online: Sergey Khich / R-system: introduction to economic espionage. Practicum on economic intelligence in modern Russian entrepreneurship. In 2 volumes. | Arbatkniga (arbatkniga.ru) (accessed on 1997).
50. Golenkov V.V., Gulyakina N.A. Printsipy postroeniya massovoj semanticheskoj tekhnologii komponentnogo proektirovaniya intellektualnykh sistem. [Principles of building a mass semantic technology of component design of intelligent systems] Proc. of the Conference “Open Semantic Technologies for Intelligent Systems” (OSTIS 2012). 2012. Pp. 23-24. (in Russian) Available online Golenkov_Printsipy.PDF (bsuir.by) (accessed on 2012).
51. Ivan Smirnov, Maksim Stankevich, Yulia Kuznetsova, Margarita Suvorova, Daniil Larionov, Elena Nikitina, Mikhail Savelov, and Oleg Grigoriev TITANIS: A Tool for Intelligent Text Analysis in Social Media. Springer Nature Switzerland AG 2021 S. M. Kovalev et al. (Eds.): RCAI 2021, LNAI 12948, pp. 232–247. 2021. Available online https://doi.org/10.1007/978-3-030-86855-0_16 (accessed on 2021)
Issues: 3(30)'2023
Section: Informatics, Computer Engineering and Control
Cite:
Kharlamov, A. SEMANTIC TEXT ANALYSIS USING ARTIFICIAL NEURAL NETWORKS BASED ON NEURAL-LIKE ELEMENTS WITH TEMPORAL SIGNAL SUMMATION // A. Kharlamov, E. Samaev, D. Kuznetsov и др. // Проблемы искусственного интеллекта. - 2023. № 3 (30). - http://search.rads-doi.org/project/13749/object/201177 doi: 10.34757/2413-7383.2023.30.3.001
