A Survey on Evaluation of Large Language Models

Bibliographic Data

ID23322797
AuthorsYupeng Chang (0000-0001-7178-6088, Jilin University), Xu Wang (0000-0002-7634-6716, Jilin University), Jindong Wang (0000-0001-7040-9299, Microsoft Research Asia (China)), Yuan Wu (0000-0003-2167-2351, Jilin University), Linyi Yang (0000-0003-0667-7349, Westlake University), Kaijie Zhu (0009-0002-6220-1476, Chinese Academy of Sciences), Hao Chen (0000-0002-8873-8266, Carnegie Mellon University), Xiaoyuan Yi (0000-0003-2710-1613, Microsoft Research Asia (China)), Cunxiang Wang (0000-0002-3023-8082, Westlake University), Yidong Wang (0000-0001-6552-6466, Peking University), Wei Ye (0000-0001-7484-0609, Peking University), Yue Zhang (0000-0003-1198-882X, Westlake University), Yi Chang (0000-0003-3595-021X, Jilin University), Philip S Yu (0000-0002-3491-5968, University of Illinois Chicago), Qiang Yang (0000-0003-4210-9007, Hong Kong University of Science and Technology), Xing Xie (0000-0002-8608-8482, Microsoft Research Asia (China))
Year2024
Volume15
Issue3
Pages1-45
Publication date2024-06-30
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueACM Transactions on Intelligent Systems and Technology (JOURNAL)
Journal identifiersISSN: 2157-6904 • E-ISSN: 2157-6912
PublisherAssociation for Computing Machinery (ACM) (PUBLISHER)
DOI10.1145/3641289
OpenAlexW4391136507
LanguageEN
Citations received102
References cited51

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey

Popularity · Artificial Intelligence in Healthcare and Education · Computer Science · Natural Language Processing Techniques · Psychology · Social Psychology · Topic Modeling

Tools

Open DOI