•  
  •  
 

International Journal of Emerging and Disruptive Innovation in Education : VISIONARIUM

Research Highlights

  • The Problem: Artificial intelligence applications in the legal field generate hallucinated or fabricated case law and omit critical nuances necessary for a complete understanding of court decisions.

  • The Method: A double-blind experiment evaluated a sample size of 27 criminal justice and legal studies undergraduate students at the University of Central Missouri who were randomly assigned to read either an actual textbook excerpt of the U.S. Supreme Court case United States v. Miller (1976) or a Microsoft Copilot-generated summary before completing a 10-question multiple-choice comprehension test.

  • Quantitative Finding: In the artificial intelligence summary group, 6.67% of participants correctly answered Question 2 regarding the investigation's origin compared to 63.63% in the textbook group; 53.33% of the artificial intelligence group for Question 3 and 73.33% for Question 9 indicated they were unable to answer; the artificial intelligence group achieved a 73.33% correct response rate on Question 4 and a 53.33% correct response rate on Question 10.

  • Qualitative Finding: Artificial intelligence-generated legal summaries exhibit factual hallucinations, completely omit dissenting opinions, and exclude foundational case citations, leading to a degraded capacity to comprehend legal nuances and apply precedent.

Abstract

In recent years, the U.S. legal system has seen an increase of legal filings using fictitious court cases or legal propositions generated by artificial intelligence (AI) (Stokel-Walker, 2026). Rules of professional responsibility require that lawyers review filings in which AI was used to ensure their accuracy (Missouri Bar, Office of Legal Ethics Counsel, 2024, Opinion No. 2024‑11) Despite this mandate, filings with fictitious cases and incorrect statements of law are being filed. Besides posing a threat to the parties to an action, such filings may set an unwarranted precedent for future cases. They also tie up court resources searching for nonexistent law or unfounded propositions of law and erode public trust. Further, an AI generated summary of a legitimate court case presents a risk of being misleading as it may fail to capture a true understanding of the case. In this study, undergraduate criminal justice college students provided human review of AI generated legal material. The students were asked to complete a typical undergraduate criminal law and procedure class assignment. At random, students were either given an excerpt from a real legal case decision or an AI generated summary of the same. Students were then asked to read the excerpt or AI summary and answer the same questions based upon their reading. Not surprisingly, students using the AI summary were not able to correctly or fully answer the questions, as the AI summary did not capture the nuances or depth of the legal case that may be found by a full reading of the case excerpt.

Additional Files
Session 3.mp4 (176046 kB)
Download

Share

COinS
 
 

To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.