Breaking Bad Briefs: A Snapshot of Lawyers, Litigants, and Experts’ Use (and Misuse) of GenAI in Court Filings
To date, at least twelve lawyers, ten pro se litigants, one persistent law firm, and one expert witness have submitted court documents containing fake case law or unreliable GenAI-driven research, seemingly ignoring or unaware of news articles, judicial opinions, judges’ standing rules, or jurisdictions’ ethics opinions about pitfalls related to use of this new technology in the legal arena. GenAI vendors consistently tout Large Language Models’ (LLMs) ability to accelerate our workflow, but, as we know, velocity does not guarantee quality or accuracy, especially in the lawyering context. Indeed, court filings containing GenAI-fabricated law, poor argumentation, and untrustworthy research slow down the adjudicative process. Opposing counsel, law clerks, and judges must expend more time, energy, and resources discovering these flaws, pointing them out, and explaining (sometimes repeatedly) why we have a duty to amp up our role as humans-in-the-loop while we experiment with these tools.
As part of our ongoing efforts to educate ourselves and others in how to use GenAI models ethically and responsibly, let’s survey thirty-four such cases, and extract lessons learned.
Lawyers & Imaginary Case Law
In 2023, the very first time I heard the term “hallucination” to refer to case law imagined by GenAI, I mused, “Well, I sure spent many days as a junior associate hustling to respond to partners’ needle-in-the-haystack dispatches to ‘find me a case that says X’ when clearly such case did not exist!” Just kidding. Hallucinated cases: obviously, bad. You’d think, though, that advocates nationwide would have learned a jarring lesson from the first reported case on this matter — Mata v. Avianca — in which lawyers cited AI-hallucinated cases in an opposition to a motion to dismiss. In Mata, when called out by opposing counsel for this transgression, the lawyers failed to “come clean” and disclose their reliance on ChatGPT; instead, they “doubled down” and avoided addressing the issue until six weeks after the court ordered them to produce the phantom cases. Ultimately, by way of sanction, the court directed the attorneys to send notice of the court’s reprimand to their client, write a letter to each of the judges mentioned in the hallucinated cases, and pay a $5,000 penalty.
Even with all the media hoopla around the Mata case, at least ten more American lawyers and one Canadian attorney have filed court submissions containing fake cases. Some of these lawyers wisely owned up to their mistake and expressed regret. Others failed to take any responsibility. A few tried to explain away the issue saying they relied on the work of law clerks, interns, or junior associates. The ensuing consequences have been all over the map.
Lawyers Admitting Fault and Expressing Regret
Let’s start with the cases in which lawyers admitted fault and apologized for their mistake. In Smith v. Farwell, a Massachusetts Superior Court matter heard before Justice Brian Davis in February 2024, a lawyer submitted multiple briefs containing fictitious cases. The lawyer acknowledged responsibility for the objectionable filings and expressed regret, explaining that he had relied on two recent law school graduates and a trusted associate to generate the work product, and he erroneously did not review the AI-generated citations — as their supervisor — before filing his briefs. Considering the “sincerity of the [lawyer’s] contrition,” the court imposed a $2000 sanction.
Also in February 2024, a Canadian lawyer in British Columbia submitted a brief with AI-hallucinated cases but upon discovery, he conceded the error and apologized. Because the attorney expressed remorse and embarrassment, the court found he had “no intent to deceive.” The court ordered him to pay opposing counsel’s costs incurred in researching the non-existent cases.
Similarly, in United States v. Cohen, a case before the United States District Court for the Southern District of New York, a disbarred attorney used a GenAI tool, Google Bard (now Google Gemini), to find cases to support his motion for early termination of supervised release from prison; he relayed the cases to his lawyer for use in preparing the brief. Counsel did not independently review the cases before citing them and filing the motion. Neither opposing counsel nor the court could find the cases; Google Bard had hallucinated them. When ordered by the court to explain this blunder, counsel accepted responsibility for the error and apologized. In March 2024, Judge Jesse Furman deemed the attorney’s acts “embarrassing,” “certainly negligent,” and constituting “extreme carelessness,” but finding no evidence of “bad faith,” he declined to impose a sanction.
Lawyers Failing to Take Responsibility or Explain What Happened
Other lawyers have surprisingly failed to own up to misrelying on GenAI research or to respond to courts’ reasonable requests for an explanation. For example, in January of 2024, in Will of Samuel, a case before Justice Bernard J. Graham in New York’s Surrogate’s Court, a lawyer filed a reply brief containing fictitious cases. The attorney failed to acknowledge the error. The court emphasized, “A simple Lexis search of the cases cited, which takes mere seconds, shows that the cases and citations contained within the response are incorrect or fake and non-existent. Had counsel taken the minimal time and effort needed to cross-check this information, he would have realized this as well.” The court struck the pleading and scheduled a hearing to determine the appropriateness of sanctions.
Six months later, in July 2024, in Iovino v. Michael Stapleton Associates, counsel submitted a brief to Judge Thomas Cullen of the United States District Court for the Western District of Virginia, citing AI-hallucinated cases. When the court asked the party to shed light on this “gross error,” she failed to explain the source of the “manufactured citations.” The court noted, “[t]his silence is deafening,” and issued an order to show cause as to why the attorneys should not be sanctioned.
Additionally, an attorney became embroiled in a robust disciplinary proceeding in Florida in May 2024 when he, among other ethical infractions, filed papers with fabricated citations, failed to take responsibility, and tried deflecting blame to his client.
Grievance Panel Referral for Willful Failure to Exercise Diligence
In Park v. Kim, a January 2024 case in the United States Court of Appeals for the Second Circuit, an attorney cited fake cases and “made no inquiry, much less the reasonable inquiry required … into the validity of the arguments she presented.” Although the lawyer ultimately admitted she used ChatGPT, the court found she had “continually and willfully” failed to comply with the court’s orders. The court directed the attorney to notify her client of the reprimand and referred the lawyer to the court’s grievance panel.
Other Lawyers Attributing “Blame” to Interns or Junior Associates
Like the Smith v. Farwell lawyer above, other attorneys have tried to explain their GenAI missteps, saying they relied on the work of law clerks, interns, or junior associates. In the fall of 2023, in an eviction case in California before Judge Ian Fusselman, a law firm submitted court filings containing fake case law. The lead attorney contended a first-year lawyer in his office had used GenAI to draft the papers. The court ruled against the law firm, dismissed the eviction suit, and imposed a financial penalty of $999 — one dollar below the threshold for reporting the matter to the state bar for further investigation and possible disciplinary action.
A month later in Colorado, in People v. Crabill, a lawyer filed a motion containing cases he found through ChatGPT without verifying their accuracy. Later, he discovered the cases were fake but failed to alert the court; when confronted about his problematic brief, the lawyer lied, blaming a legal intern. The court issued a “stipulation to discipline,” suspending the attorney’s bar license for one year and one day.
These cases highlight the need for every law office to engage in collective GenAI education at all levels — from entry-level staff to senior leadership. Ethics opinions issued by various state bars and the American Bar Association emphasize the duty of managerial lawyers to enact GenAI policies, provide training for law office staff, and ensure compliance with court rules.
Courts Issuing Adverse Rulings
Two lawyers avoided sanctions but received adverse rulings after citing fake cases in court filings. In Ex parte Allen Michael Lee, a 2023 case before the Texas Court of Appeals, a lawyer submitted an appellate brief citing non-existent cases. Deeming the brief “deficient and illogical,” Chief Justice Tom Gray issued an opinion affirming the trial court’s order. Because the court could not determine with certainty whether the attorney had used GenAI, Chief Justice Gray resisted “the temptation to issue a show cause order … or report the attorney to the State Bar of Texas for a potential investigation for a violation of the State Bar rules.”
Similarly, in Grant v. City of Long Beach, a March 2024 case, the United States Court of Appeals for the Ninth Circuit (in an opinion written by Judge Roopali H. Desai) struck an appellate brief that failed to comply with court rules and contained fabricated case law, and dismissed the appeal.
Pro Se Litigants Submitting Court Filings Containing Fabricated Case Law
In at least ten cases so far, pro se litigants have added to the workload of courts and counsel by relying on fake cases in briefs. For example, in late 2023, in Mescall v. Renaissance at Antiquity, in the United States District Court for the Western District of North Carolina, the court addressed a pro se plaintiff’s “inartful” pleading and opposition brief. Opposing counsel suggested the deficient brief might have been written with GenAI. Acknowledging the “latitude” afforded pro se litigants, the court reminded the party of his obligation to comply with local court rules regarding motions practice, and that reliance on GenAI tools “may result in sanctions or penalties when used inappropriately.”
A few months later, in Kruse v. Karlen, a February 2024 case pending in the Missouri Court of Appeals, a pro se litigant submitted a brief citing imaginary cases. Out of twenty-four case citations, only two actually existed (and still did not stand for the propositions asserted). The court dismissed the matter and imposed a $10,000 penalty upon the party for filing a frivolous appeal. In the opinion, Judge Kurt Odenwald emphasized, “We urge all parties practicing before this Court, barred and self-represented alike, to be cognizant that we are aware of the issue and will not permit fraud on this Court in violation of our rules.”
Seven months later in the same court, in Jones v. Simploy, Inc. — an appeal of a Labor and Industrial Relations Commission decision denying an application for unemployment compensation — Judge Angela T. Quigless addressed a pro se plaintiff’s appellate brief which cited non-existent cases. Giving the litigant the benefit of the doubt and “because this [was] a straightforward unemployment compensation case … wherein the State did not have to pay outside counsel to respond to the appeal,” the court declined to impose sanctions. Nonetheless, the court stated, “litigants who use generative AI to draft their briefs should not rely on our continued magnanimity.”
Interestingly, the Park v. Kim lawyer referenced above didn’t seem to learn from her GenAI mistake. In Lee v. Delta Airlines, she represented herself in a lawsuit in the United States District Court for the Eastern District of New York before Judge William Kuntz. The court expressed “serious concern” regarding the litigant’s citation of a fake case in her objections to a magistrate judge’s report and recommendation. The court pointed out that pro se litigants typically receive “special solicitude,” but this lawyer had been practicing law for two decades. The court dismissed the complaint.
In July 2024, in Anonymous v. New York City Department of Education, Judge Jennifer L. Rochon of the United States District Court for the Southern District of New York reiterated the custom of affording pro se litigants “special solicitude.” In this matter involving Title VII, the Americans with Disabilities Act, and New York Human Rights Laws, a pro se plaintiff cited to non-existent legal authority. Judge Rochon declined to sanction the party but emphasized the impropriety of any litigant submitting court documents containing fake cases. She further cautioned, “the Court will not look kindly on similar infractions in the future.”
In another S.D.N.Y case the same month, Dukuray v. Experian Information Solutions, Magistrate Judge Gary Stein addressed a pro se litigant’s citation to “nonexistent judicial opinions” in an opposition to a motion to dismiss in a Fair Credit Reporting Act case. The court noted the possibility that the pro se plaintiff was “not aware of the risk that ChatGPT and similar AI programs are capable of generating fake case citations and other misstatements of law.” The court further acknowledged the potential challenge for such litigants to check AI-generated citations without access to Westlaw or Lexis. Nonetheless, the court advised the party, “any further filings with citations to nonexistent cases may result in sanctions, such as her submissions being stricken, filing restrictions or monetary penalties being imposed, or the case being dismissed.”
Two months later, in September 2024 in Rule v. Braiman, Judge Brenda K. Sannes of the United States District Court for the Northern District of New York echoed the sanctions language from Dukuray to caution a pro se litigant who filed a brief containing unverifiable citations.
Further, in Transamerica Life Ins. Co. v. Williams, a September 2024 case in the United States District Court for the District of Arizona, a pro se party submitted a motion for summary judgment, an opposition to a motion to dismiss, and a sur-reply brief in a matter involving claims to death benefits in a life insurance policy. The litigant’s filings were “replete” with citations to non-existent legal authority. Judge Roslyn O. Silver advised the litigant “to comply with this Court’s local rules, the Court’s Handbook for Self-Represented Litigants, and the Federal Rules of Civil Procedure. Any future filings with citations to nonexistent cases may result in sanctions such as dismissal of her claim.”
In late August 2024, a pro se litigant in Secura Ins. v. Thomson, filed a motion in the United States District Court for the Western District of Kentucky to quash a subpoena, but cited a case the court could not find. The court reminded the party to take “necessary care” in preparing court submissions, which “includes confirming that all cases or orders cited to the Court exist.”
Most recently, in October 2024, the United States District Court for the Western District of Washington dismissed a pro se litigant’s complaint in Moshrif v. King County Prosecution, on the basis that the pleading failed to state a claim upon which relief could be granted. The court cautioned the party to “comply with applicable law and procedural rules” or he will risk “sanctions, even when such violations result from use of artificial intelligence.”
Two Courts Mentioned Generative AI in the Context of Unhelpful Briefs
Two judges referenced GenAI when addressing ineffective legal writing in briefs. For example, in a S.D.N.Y. matter addressing a motion for sanctions in July 2024, Sillam v. Labaton Sucharow LLP, Magistrate Judge Ona Wang noted that, while a party’s brief did not cite fake cases, the document’s “repetitive language [did] not present any advocacy; it only restate[d] general principles of law without making argument.” Judge Wang alluded to GenAI, stating she does not have “a general prohibition” on its use but the party’s brief in this instance was “not helpful to the Court.”
The next day, Justice Mike Wallach of the Texas Court of Appeals similarly referenced GenAI in Byrd v. Villages of Woodland Springs Homeowners Assn., Inc., a dispute over a homeowners association’s assessments and liens. Pointing out deficiencies in the pro se appellant’s brief, the court noted the possibility the appellant used “ChatGPT or another artificial intelligence (AI) source.” The court affirmed the trial court judgment favoring the appellees.
One Lawyer Violated a Standing Order Requiring Disclosure of GenAI Use
Numerous judges nationwide have issued standing orders requiring litigants to disclose GenAI use in preparing court submissions and attach certifications attesting to a variety of pre-filing tasks such as verification of citations and protection of attorney-client privilege and client confidentiality. In one case, a party was reprimanded for failing to adhere to such GenAI disclosure rules.
In September 2024, Judge Stanley Blumenfeld, Jr. of the United States District Court for the Central District of California noted several errors in a party’s motion to remand in Mortazavi v. Booz Allen Hamilton, including citation to a fake case. Judge Blumenfeld ordered the party to disclose whether her counsel used GenAI in preparing the motion, referencing the court’s Standing Order requiring litigants to submit a “declaration disclosing the use of artificial intelligence and certifying that the filer has reviewed the source material and verified that the artificially generated content is accurate and complies with the filer’s Rule 11 obligations.” Counsel revealed she used GenAI to draft the motion but did not explain her non-compliance with the Standing Order. The judge issued an order to show cause why she should not be sanctioned.
One Firm’s Repeated Attempt to Use ChatGPT to Support Fee Requests
One law firm in New York has repeatedly (and unsuccessfully) presented documentary evidence of its interactions with ChatGPT (i.e., prompts and responses) to support assertions of alleged prevailing market rates in attorneys’ fee requests in the context of Individuals with Disabilities Act (IDEA) claims against the New York City Department of Education. Since January 2024, the same law firm has pursued this strategy eight times before Magistrate Judge Sarah Netburn, Judge Paul A. Engelmayer, Magistrate Judge James L. Cott, Magistrate Judge Robyn Tarnofsky, Judge Edgardo Ramos, and Magistrate Judge Valerie Figueredo. Based on ongoing concerns regarding ChatGPT “inventing false information,” the tool’s “propensity to fabricate information,” and its current reputation as an “unreliable source,” these judges rejected, disregarded, and declined to consider or credit the ChatGPT “evidence” in support of the law firm’s fee requests.
The language in two of the judicial opinions acknowledged that these decisions were based on the existing state of GenAI. For example, Judge Paul A. Engelmayer, in J.G. v. New York City Department of Education, stated, “Barring a paradigm shift in the reliability of this tool, the [firm] is well advised to excise references to ChatGPT from future fee applications.” (emphasis added) Further, Magistrate Judge James L. Cott emphasized, in S.C. v. New York City Department of Education, “In light of its propensity to fabricate information, ChatGPT, at least in its current iteration, should not be utilized to support a fee application.” (emphasis added)
Perhaps this phrasing implies that judges’ consideration of such “proof” could change if the reliability of GenAI tools in these categories of lawyering tasks improves sufficiently to meet the applicable evidentiary standards.
One Expert Witness’s Attempt to Use GenAI to Support a Damages Claim
One court rejected an expert witness’s attempt to use GenAI to support a damages calculation. In October 2024, in In the Matter of the Accounting by Susan F. Weber, a case in New York’s Surrogate’s Court, an objector to an estate accounting hired an expert witness to opine about damages calculations related to a trustee’s handling of real estate property. Deeming the damages calculation unreliable, Judge Jonathan G. Schopf pointed out how the expert relied, in part, on Microsoft Copilot (a GenAI tool) to “cross-check” his calculations but could not explain: (1) how Copilot functions, (2) how it produces its output, (3) what sources it relied upon, or (4) how the resulting calculations meet the reliability standards applicable to expert opinions. The court accentuated that “counsel has an affirmative duty to disclose the use of artificial intelligence,” and, further, any proposed evidence would “be subject to a Frye hearing prior to its admission.”
To refresh our recollection, the Frye standard — used in a number of states for admissibility of expert opinions — requires the following:
- Scientific evidence must be based on a principle, procedure, or methodology which has “gained general acceptance” in its particular or specified field.
- The principle, procedure, or methodology does not need to be “universally accepted” or “unanimously endorsed,” but it must be “reasonably relied upon” and “generally accepted as reliable” by experts in the field.
- Courts will consider “judicial opinions, scientific or legal writings, or expert opinion other than that of the proffered expert” in analyzing whether the principle, procedure, or methodology is generally accepted as reliable in the given scientific community.
In federal court, under Federal Rule of Evidence 702, to be admissible, an expert’s opinion must:
- be helpful to the trier of fact to understand the evidence or to determine a fact in issue
- be based on sufficient facts or data
- be the product of reliable principles and methods, and
- reflect a reliable application of the principles and methods to the facts of the case
Further, the Daubert standard for admissibility of expert opinions analyzes:
- whether the expert’s technique, theory, or methodology can be, and has been tested
- whether the expert’s technique, theory, or methodology has been subjected to publication and peer review
- whether the expert’s technique, theory, or methodology has a known or potential error rate
- whether standards exist and are maintained controlling the operation of the expert’s technique, theory, or methodology, and
- whether the expert’s technique, theory, or methodology has attracted widespread acceptance within a relevant scientific community
Experts seeking to use GenAI-facilitated research or calculations to support opinions rendered in legal matters should consider and comport with the governing admissibility standards.
One Judge’s Positive and Transparent Use of GenAI
While the foregoing cases showcase GenAI pitfalls in legal venues, Judge Kevin Newsom of the United States Court of Appeals for the Eleventh Circuit has sought to demonstrate positive ways LLMs might help us solve legal problems or answer legal questions.
In Snell v. United Specialty Insurance Company, a May 2024 case involving an insurance coverage dispute, Judge Newsom wrote a concurring opinion in which he transparently explained how he used GenAI tools to help him determine whether an “in-ground trampoline and retaining wall” constituted “landscaping” in the context of an insurance policy which failed to define the term. In conducting an “ordinary meaning” interpretive analysis — which usually involves consulting dictionaries and linguistic canons — Judge Newsom asked ChatGPT and Google Bard (now Gemini) to provide the “ordinary meaning” of “landscaping.” Judge Newsom explained that LLMs are trained on vast amounts of “data that aim to reflect and capture how individuals use language in their everyday lives.” Acknowledging the potential upsides as well as hazards of these tools, he pointed out that GenAI models are “high-octane language-prediction machines capable of probabilistically mapping, among other things, how ordinary people use words and phrases in context.”
Four months later, Judge Newsom engaged in another language experiment using LLMs. In United States v. Deleon, he wrote another concurring opinion explaining how he used GenAI to analyze the “ordinary meaning” of the phrase “physically restrained” in an armed robbery sentencing enhancement case. In this scenario, he highlighted the benefit of GenAI tools’ abilities to define a “composite, multi-word phrase” — a task that, depending on the wording, regular dictionaries can’t do. (Personally, I love using ChatGPT as my “super-thesaurus” for the same reason.)
Lawyering Lessons Learned
GenAI tools have the upside potential to boost our creativity, nudge us out of writer’s block, and help us brainstorm alternative solutions to tough legal quandaries. But in the context of lawyering, we need to:
- understand their current limitations related to legal research
- incorporate cite-verification protocols into our workflow
- provide GenAI literacy training for all members of our institutions, offices, and communities — from entry-level to senior leadership
- enhance open and transparent communication among staff and supervisory attorneys about how GenAI is being used in drafting documents to be submitted to courts
- commit time to editing any GenAI-generated legal writing to ensure our work product is helpful to judges and other legal readers, and
- comply with judges’ standing orders related to GenAI certification
Further, as a legal community, we should help educate pro se litigants about existing perils in relying on public GenAI tools to generate case citations, and provide free or cost-effective methods for individuals to verify the genuineness of GenAI-provided sources of law. (Just this morning on LinkedIn, I learned about a new, free, open-access legal research tool called Descrybe.ai. I’m eager to try it.)
Additionally, we must remind expert witnesses about the evidentiary standards that will be applied to opinions and calculations derived with the assistance of GenAI tools.
As lifelong learners, it’s an exciting time to be experimenting with this innovative technology in pursuit of resolving legal problems. Let’s commit to helping each other do it responsibly.
*Professor Heidi K. Brown is the author of The Introverted Lawyer, Untangling Fear in Lawyering, and The Flourishing Lawyer. She has a law degree from the University of Virginia School of Law and a master’s degree in applied positive psychology (MAPP) from the University of Pennsylvania. Her website is www.theflourishinglawyer.org. She can be reached at heidi@theintrovertedlawyer.com and at her awesome fun job as Associate Dean of Upper Level Writing at New York Law School (heidi.brown@nyls.edu). Heidi also writes about solo travel at heidiography.substack.com.
