Evaluating AI in Legal Operations
A Comparative Analysis of Accuracy, Completeness, and Hallucinations in ChatGPT-4, Copilot, DeepSeek, Lexis+ AI, and Llama 3
Bibliographic Data
| ID | 4221269 |
|---|---|
| Authors | Bakht Munir (University of Kansas), Muhammad Zubair Abbasi (0000-0003-1561-5457, University of London), W Blake Wilson (University of Kansas), W James Wilson (0000-0003-1341-2402), Alessandro Colombo (0009-0009-0244-7854), Allen Colombo (University of Kansas) |
| Year | 2025 |
| Volume | 53 |
| Issue | 2 |
| Pages | 103-114 |
| Publication date | 2025-01-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | International Journal of Legal Information (JOURNAL) |
| Journal identifiers | ISSN: 0731-1265 • E-ISSN: 2331-4117 |
| Publisher | Cambridge University Press (CUP) (PUBLISHER) |
| DOI | 10.1017/jli.2025.10052 |
| OpenAlex | W4411828919 |
| Language | EN |
| References cited | 3 |
The proliferation of Artificial Intelligence (AI) is significantly transforming conventional legal practice. The integration of AI into legal services is still in its infancy and faces challenges such as privacy concerns, bias, and the risk of fabricated responses. This research evaluates the performance of the following AI tools: (1) ChatGPT-4, (2) Copilot, (3) DeepSeek, (4) Lexis+ AI, and (5) Llama 3. Based on their comparison, the research demonstrates that Lexis+ AI outperforms the other AI solutions. All these tools still encounter hallucinations, despite claims that utilizing the Retrieval-Augmented Generation (RAG) model has resolved this issue. The RAG system is not the driving force behind the results; it is one component of the AI architecture that influences but does not solely account for the problems associated with the AI tools. This research explores RAG architecture and its inherent complexities, offering viable solutions for improving the performance of AI-powered solutions
Completeness (order theory · Lexis · Linguistics · Artificial Intelligence in Healthcare and Education · Artificial Intelligence in Law · Computer Science · Ethics and Social Impacts of AI · Mathematics · Philosophy · Psychology · Artificial Intelligence
| Highly cited | No |
|---|