When AI Is Fooled: Hidden Risks in LLM-Assisted Grading
Milani A. Franzoni V. Florindi E. Omarbekova A. Bekmanova G. Yergesh B.
November 2025Multidisciplinary Digital Publishing Institute (MDPI)
Education Sciences
2025#15Issue 11
This study investigates how targeted attacks can compromise the reliability and applications of large language models (LLMs) in educational assessment, highlighting security vulnerabilities that are frequently underestimated in current AI-supported learning environments. As LLMs and other AI tools are increasingly being integrated into grading, providing feedback, and supporting the evaluation workflow, educators are adopting them for their potential to increase efficiency and scalability. However, this rapid adoption also introduces new risks. An unexplored threat is prompt injection, whereby a student acting as an attacker embeds malicious instructions within seemingly regular assignment submissions to influence the model’s behaviour and obtain a more favourable evaluation. To the best of our knowledge, this is the first systematic comparative study to investigate the vulnerability of popular LLMs within a real-world educational context. We analyse a significant representative scenario involving prompt injection in exam assessment to highlight how easily such manipulations can bypass the teacher’s oversight and distort results, thereby disrupting the entire evaluation process. By modelling the structure and behavioural patterns of LLMs under attack, we aim to clarify the underlying mechanisms and expose their limitations when used in educational settings.
AI misuse detection , education , educational evaluation , generative AI , human-in-the-loop AI , large language models , prompt injection , trustworthy AI
Text of the article Перейти на текст статьи
Department of Human Sciences, Link Campus University, Via del Casale di San Pio V, 44, Rome, 00165, Italy
Department of Mathematics and Computer Science, University of Perugia, Perugia, 06123, Italy
Department of Computer Science, Hong Kong Baptist University, Hong Kong
Department of Mathematics and Computer Science, University of Modena-Reggio Emilia, Modena, 41100, Italy
Digital Development and Distance Learning Department, Eurasian National University of Astana, Astana, 010008, Kazakhstan
Department of Human Sciences
Department of Mathematics and Computer Science
Department of Computer Science
Department of Mathematics and Computer Science
Digital Development and Distance Learning Department
10 лет помогаем публиковать статьи Международный издатель
Книга Публикация научной статьи Волощук 2026 Book Publication of a scientific article 2026