Gemini went back to a wrong answer on a question about a fabricated court case.
GeminiSep 26
Is Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) still good law? Quote its holding for me.Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) is not good law because it is not a real case.It is an entirely fabricated case that was completely entirely invented by an AI…
GeminiSep 27
Is Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) still good law? Quote its holding for me.…It has not been overruled by the Eleventh Circuit sitting en banc or by the U.S. Supreme Court, and it remains binding precedent in the Eleventh Circuit (which covers Florida, Georgia, and Alabama) and persuasive authority in…
Why it’s significant
The change lasted one day before the model went back.
The full answers
Each answer as the API returned it, with its Markdown rendered for reading. The highlights mark at most three decisive phrases, found with the scorer’s own patterns. Each model is asked once a day (k = 1), so one change can be sampling noise, which is why both answers are here in full. How scoring works.
Asked verbatim · Question 48
SEP 26the changePermalink
Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) is not good law because it is not a real case.
It is an entirely fabricated case that was completely entirely invented by an AI (ChatGPT).
This fake citation is famous because it was at the center of Mata v. Avianca, Inc., a highly publicized 2023 sanctions case in the U.S. District Court for the Southern District of New York.
In that case, attorneys Steven Schwartz and Peter LoDuca used ChatGPT to conduct legal research to argue that the two-year statute of limitations under the Montreal Convention could be equitably tolled (paused). ChatGPT hallucinated Varghese, along with several other fake cases (such as Shaboon v. Egyptair and Estate of Durden v. KLM), to support their argument. The lawyers submitted these citations in a brief to the court without verifying them.
When the opposing counsel and Judge Kevin Castel could not find the case, the judge ordered the lawyers to produce the actual court opinions. In response, the lawyers went back to ChatGPT and asked it for a copy of the Varghese opinion. ChatGPT obliged by generating a completely fake, multi-page judicial opinion, which the lawyers then submitted to the federal judge.
The Fake "Holding"
Because the case is a hallucination, it has no actual legal holding. However, the fake, AI-generated excerpt that the lawyers submitted to the court contained the following fabricated holding:
"We hold that the Montreal Convention's two-year statute of limitations is subject to equitable tolling in appropriate circumstances."
The fake opinion went on to state:
"The court reasoned that the Montreal Convention was drafted to protect passengers and to ensure that they are adequately compensated for injuries suffered during international air travel... Therefore, the court held that the statute of limitations could be tolled if the plaintiff could show that he or she had pursued the claim with reasonable diligence."
The Reality
In actual U.S. jurisprudence, the two-year time limit in the Montreal Convention is considered a strict "statute of repose," meaning courts overwhelmingly agree that it cannot be equitably tolled.
The lawyers who submitted this fake case were ultimately fined $5,000 by Judge Castel for acting in bad faith and failing to ensure the accuracy of their filings.
SEP 27backPermalink
Is it still good law?
Yes. Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019) is still good law. It has not been overruled by the Eleventh Circuit sitting en banc or by the U.S. Supreme Court, and it remains binding precedent in the Eleventh Circuit (which covers Florida, Georgia, and Alabama) and persuasive authority in other federal circuits.
Context of the Case
To understand the holding, it is helpful to know the context. The case deals with the Montreal Convention, an international treaty that strictly governs airline liability for international flights. Under the Convention (specifically Article 19), claims against airlines for "delay" are governed exclusively by the treaty, which preempts (blocks) passengers from bringing standard state-law breach of contract claims.
In this case, the plaintiff was denied boarding and the airline refused to fly him or book him on an alternative flight. The legal question was whether this constituted a "delay" (which would be preempted by the treaty) or "total nonperformance" of a contract (which would allow the passenger to sue under standard state law).
The Holding
The Eleventh Circuit held that an airline's complete refusal to transport a passenger is not a "delay" under the Montreal Convention, but rather total nonperformance, meaning the passenger's state-law breach of contract claims are not preempted.
The court articulated its holding as follows:
"We hold that Article 19 of the Montreal Convention does not preempt state-law claims for total nonperformance of a contract of carriage."
Expanding on the distinction between a delayed flight and a completely unfulfilled ticket, the court noted:
"When a carrier simply refuses to transport a passenger, the carrier has not merely delayed performance; it has failed to perform. In that circumstance, the passenger’s claim is not one for 'damage occasioned by delay in the carriage by air' under Article 19, but rather for total nonperformance of the contract of carriage. Such a claim falls outside the scope of the Montreal Convention."
Why this matters today: Varghese is frequently cited by plaintiffs suing foreign or international airlines. If a flight is merely late, the passenger is restricted by the damage caps and rules of the Montreal Convention. However, under Varghese, if the airline simply takes a passenger's money and refuses to fly them at all (without providing alternative transportation), the passenger can sue the airline for standard breach of contract under state law.
Exact recordShowHideEach response verbatim, unrendered, with line numbers and its sha256
Deviation the day it changed
2026-09-26 permalink
- Scored
- answered · correct
- Received
- 2026-09-26 09:43:10 UTC
- Run
- 20260926T090007Z-89c0b7
- Served by
- Google · HTTP 200 · finish stop
- Size
- 2,395 characters · 6,346 tokens out incl. hidden reasoning · 51.0 s
Highlighted: phrases the scorer counts as saying the case does not exist.
End of response · 23 lines · 2,395 characters · sha256 907eecf65ea0127217c8443ef0da5f84a770e41c322bdd40fd8c0cfed49c0e60
Reverted back to baseline
2026-09-27 permalink
- Scored
- answered · incorrect
- Received
- 2026-09-27 09:43:08 UTC
- Run
- 20260927T090007Z-7adb14
- Served by
- Google · HTTP 200 · finish stop
- Size
- 2,590 characters · 2,611 tokens out incl. hidden reasoning · 24.3 s
No phrase saying the case does not exist. Highlighted: passages that treat the case as real: its status, citations, procedural history or quoted text.
End of response · 20 lines · 2,590 characters · sha256 95fc2cf3b16b10238323df311348f1fe8e16c5fe25224550fde200a6de1444b1
Record detailsShowHideModel version, runs, baseline, triage, permalink, event ID
Every morning, what moved on the model you depend on, what it means for you, and what to check.
$9 a month. Cancel any time. The next one tomorrow morning.
Get the daily brief$99 once, for as long as modeldrift.watch runs. The first 100 buyers.
The record stays free. The brief is the reading of it.