Contemporary Approaches in Evolving Language Models

Oralbekova D. Mamyrbayev O. Othman M. Kassymova D. Mukhsina K.
December 2023 Multidisciplinary Digital Publishing Institute (MDPI)

Applied Sciences (Switzerland)
2023 #13 Issue 23

This article provides a comprehensive survey of contemporary language modeling approaches within the realm of natural language processing (NLP) tasks. This paper conducts an analytical exploration of diverse methodologies employed in the creation of language models. This exploration encompasses the architecture, training processes, and optimization strategies inherent in these models. The detailed discussion covers various models ranging from traditional n-gram and hidden Markov models to state-of-the-art neural network approaches such as BERT, GPT, LLAMA, and Bard. This article delves into different modifications and enhancements applied to both standard and neural network architectures for constructing language models. Special attention is given to addressing challenges specific to agglutinative languages within the context of developing language models for various NLP tasks, particularly for Arabic and Turkish. The research highlights that contemporary transformer-based methods demonstrate results comparable to those achieved by traditional methods employing Hidden Markov Models. These transformer-based approaches boast simpler configurations and exhibit faster performance during both training and analysis. An integral component of the article is the examination of popular and actively evolving libraries and tools essential for constructing language models. Notable tools such as NLTK, TensorFlow, PyTorch, and Gensim are reviewed, with a comparative analysis considering their simplicity and accessibility for implementing diverse language models. The aim is to provide readers with insights into the landscape of contemporary language modeling methodologies and the tools available for their implementation.

BERT , GPT , language modeling , neural networks , NLP , transformer

Text of the article Перейти на текст статьи

Institute of Information and Computational Technologies, Almaty, 050060, Kazakhstan
Information and Communication Technologies Department, Institute Automation and Telecommunication, Academy of Logistics and Transport, Almaty, 050026, Kazakhstan
Department of Communication Technology and Networks, Universiti Putra Malaysia (UPM), Selangor D.E., Serdang, 43400, Malaysia

Institute of Information and Computational Technologies
Information and Communication Technologies Department
Department of Communication Technology and Networks

10 лет помогаем публиковать статьи Международный издатель

Книга Публикация научной статьи Волощук 2026 Book Publication of a scientific article 2026