Can generative AI reliably synthesise literature? exploring hallucination issues in ChatGPT

Bibliographic Data

ID20397175
AuthorsAmr Adel (0000-0002-0632-0940, Eastern Institute of Technology, corresponding author), Noor H S Alani (0000-0003-0449-7299, Eastern Institute of Technology, corresponding author), Noor Alani
Year2025
Volume40
Issue8
Pages6799-6812
Publication date2025-12-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueAI & Society (JOURNAL)
Journal identifiersISSN: 0951-5666 • E-ISSN: 1435-5655
PublisherSpringer Science and Business Media LLC (PUBLISHER)
DOI10.1007/s00146-025-02406-7
OpenAlexW4411506008
LanguageEN
Citations received9
References cited57

This study evaluates the capabilities and limitations of generative AI, specifically ChatGPT, in conducting systematic literature reviews. Using the PRISMA methodology, we analysed 124 recent studies, focusing in-depth on a subset of 40 selected through strict inclusion criteria. Findings show that ChatGPT can enhance efficiency, with reported workload reductions averaging around 60–65%, though accuracy varies widely by task and context. In structured domains such as clinical research, title and abstract screening sensitivity ranged from 80.6% to 96.2%, while precision dropped as low as 4.6% in more interpretive tasks. Hallucination rates reached 91%, underscoring the need for careful oversight. Comparative analysis shows that AI matches or exceeds human performance in simple screening but underperforms in nuanced synthesis. To support more reliable integration, we introduce the Systematic Research Processing Framework (SRPF) as a guiding model for hybrid AI–human collaboration in research review workflows

Data science · Generative grammar · Machine learning · MEDLINE · Systematic review · Workflow · Workload · Artificial Intelligence in Healthcare and Education · Computer Science · Engineering · Explainable Artificial Intelligence (XAI · Machine Learning in Healthcare · Artificial Intelligence

Tools

Open DOIOpen Access