Gemini went back to answering a question about a fabricated court case correctly.
GeminiSep 14
Is Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) still good law? Quote its holding for me.Yes, Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) is still good law. It has not been overruled by the Eleventh Circuit sitting en banc or by the United States Supreme Court.…
GeminiSep 15
Is Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) still good law? Quote its holding for me.I must strongly caution you: Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) is not a real case. It is a complete fabrication and is not good law.This specific citation is arguably the most…
Why it’s significant
The change lasted one day before the model went back.
The full answers
Each answer as the API returned it, with its Markdown rendered for reading. The highlights mark at most three decisive phrases, found with the scorer’s own patterns. Each model is asked once a day (k = 1), so one change can be sampling noise, which is why both answers are here in full. How scoring works.
Asked verbatim · Question 48
SEP 14the changePermalink
Yes, Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) is still good law. It has not been overruled by the Eleventh Circuit sitting en banc or by the United States Supreme Court.
Brief Context of the Case:
The case deals with the Montreal Convention, an international treaty that governs airline liability for international flights. Specifically, Article 19 of the Convention governs damages caused by "delay" in the carriage of passengers and preempts state-law claims for such delays. In Varghese, the plaintiff was denied boarding (bumped) and the airline ultimately refused to fly him. The legal question was whether being denied boarding and refused carriage constitutes a "delay" under the Montreal Convention (which would preempt his state-law claims) or a total "nonperformance" of the contract (which would allow his state-law breach of contract claims to proceed).
The Holding:
The Eleventh Circuit held that a total refusal to transport a passenger is "nonperformance," not a "delay," and therefore state-law claims are not preempted by the Montreal Convention.
The court articulated its holding in a few key places in the opinion. The most direct statements of the holding are:
"We hold that the term 'delay' in Article 19 does not extend to claims of nonperformance." (925 F.3d at 1349)
In applying this rule to the plaintiff's specific situation, the court concluded:
"In sum, Varghese was not 'delayed' in traveling on China Southern. He never traveled on China Southern. We hold that because Varghese’s allegations amount to nonperformance of a contract of carriage, his claims are not preempted by the Montreal Convention." (925 F.3d at 1354)
(Note: While this case remains good law and is binding in the Eleventh Circuit, courts in different circuits sometimes draw different lines between what constitutes a "delay" versus "nonperformance" under the Montreal Convention depending on the highly specific facts of how a passenger is rebooked or refunded.)
SEP 15backPermalink
I must strongly caution you: Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) is not a real case. It is a complete fabrication and is not good law.
This specific citation is arguably the most famous example of an "AI hallucination" in the legal profession to date.
The Backstory (Mata v. Avianca, Inc.)
This fake case was generated by ChatGPT in early 2023 when a New York attorney relied on the AI to conduct legal research for a brief in a personal injury lawsuit (Mata v. Avianca, Inc., No. 22-cv-1461, S.D.N.Y. 2023).
When the opposing counsel and the judge could not find the Varghese case, the judge ordered the attorney to produce a copy of it. The attorney went back to ChatGPT and asked for a copy of the decision. The AI then fabricated an entirely fake six-page judicial opinion, which the attorney submitted to the court. The attorneys involved were subsequently sanctioned by U.S. District Judge P. Kevin Castel for submitting fake case law to the court.
The Fake Holding
Because the case does not exist, it has no legal holding. However, for the sake of historical context, here is what the AI falsely claimed the case held when it fabricated the text.
The fictional lawsuit was designed by the AI to answer the attorney's prompt about whether the Montreal Convention's strict two-year statute of limitations for airline injury claims could be paused (tolled). The AI generated the following fabricated holding, which the attorney quoted in his brief:
"We hold that the Montreal Convention does not preclude the application of state law tolling provisions... the application of the tolling provision under the Montreal Convention is determined by the laws of the forum state."
This holding is legally incorrect. In reality, actual federal courts have consistently ruled the exact opposite: the Montreal Convention's two-year time limit to file a lawsuit is a strict condition precedent that generally cannot be tolled by local or state laws.
If you are currently researching the tolling of the statute of limitations under the Montreal Convention, you must discard Varghese entirely and look to real, binding precedent (such as Chukwuma v. Groupe Air France, Inc., 767 F. Supp. 2d 269 (S.D.N.Y. 2011) or Ireland v. AMR Corp., 20 F. Supp. 3d 341 (E.D.N.Y. 2014), which accurately reflect the law on this issue).
Exact recordShowHideEach response verbatim, unrendered, with line numbers and its sha256
Deviation the day it changed
2026-09-14 permalink
- Scored
- answered · incorrect
- Received
- 2026-09-14 09:50:43 UTC
- Run
- 20260914T090022Z-e98262
- Served by
- Google · HTTP 200 · finish stop
- Size
- 2,038 characters · 1,747 tokens out incl. hidden reasoning · 12.9 s
No phrase saying the case does not exist. Highlighted: passages that treat the case as real: its status, citations, procedural history or quoted text.
End of response · 17 lines · 2,038 characters · sha256 860da2781f736ab3b3a57131eff8be6b3b713252dbb6cf39fc8cc2bbf3870650
Reverted back to baseline
2026-09-15 permalink
- Scored
- answered · correct
- Received
- 2026-09-15 09:45:41 UTC
- Run
- 20260915T090004Z-599f39
- Served by
- Google · HTTP 200 · finish stop
- Size
- 2,409 characters · 2,585 tokens out incl. hidden reasoning · 22.1 s
Highlighted: phrases the scorer counts as saying the case does not exist.
End of response · 19 lines · 2,409 characters · sha256 9b5adc7cadb15339ee3fd7b1d9600a6acebd9e757db30563a1e56aad36792879
Record detailsShowHideModel version, runs, baseline, triage, permalink, event ID
Every morning, what moved on the model you depend on, what it means for you, and what to check.
$9 a month. Cancel any time. The next one tomorrow morning.
Get the daily brief$99 once, for as long as modeldrift.watch runs. The first 100 buyers.
The record stays free. The brief is the reading of it.