SC orders zero tolerance for AI-hallucinated legal precedents

The Supreme Court set aside an NCLT insolvency verdict after finding it relied on AI-generated fake and hallucinated legal precedents. Stressing zero tolerance, it directed the Bar Council of India to frame guidelines and disciplinary measures against citing unverified AI-generated judicial material

New Delhi: Red flagging the use of “non-existent, fake, and hallucinated” judgements generated by artificial intelligence (AI), the Supreme Court on Thursday set aside a verdict of the National Company Law Tribunal (NCLT), saying that courts must adopt zero tolerance against citing or using AI-generated precedents.

The bench comprising Justices PS Narasimha and Alok Aradhe said, “The production of fake, non-existent, and hallucinated material and its utilisation as precedents in law, is like the release of methyl Isocyanate in the province of law and justice: invisible, insidious, and catastrophic by the time anyone notices. It not only contaminates but takes away the very lifeblood of judicial determination.”

---

It set aside an NCLT judgement on Essel Infraprojects insolvency after finding that the tribunal had relied on non-existent, fake and hallucinated judgement precedents generated through AI tools.

“It is necessary for Courts to adopt a zero-tolerance mode for producing, citing or using AI-generated precedents without verification. It is a misconduct on the part of an advocate to cite such judgments without verification,” the verdict said.

The bench said it was equally a “serious lapse” if a judge relies on such a fake or hallucinated AI-generated material as precedents in support of the determination.

“We have no hesitation in declaring that such a decision is no decision in the eyes of the law, irrespective of whether such material had a direct or indirect bearing on the decision-making. Such decisions are to be set aside even if an iota of fake or hallucinated material enters the decision-making process, as it would violate the sanctity of adjudication,” it said.

The bench said it is absolutely necessary to maintain integrity in decision-making. “And we reiterate and declare zero tolerance for the Bar as well as the Bench to cite, refer to, or rely on such material. It is also clarified that our judgment shall have no bearing on the rightful use of AI, but on the presentation or reliance on fake or hallucinated material as if it were a court precedent,” it said.

The bench said mere declaration of prohibitory action is not sufficient and there must be a consequential action following accountability.

“So far as the responsibility of the bar is concerned, we direct the Bar Council of India (BCI), being the apex statutory body, to constitute a committee and deliberate on this issue of members of the bar submitting such fake and hallucinated material before the Court as if they are precedents of law,” it said.

The verdict said the apex bar body must take up this issue with utmost seriousness, deliberate earnestly, and prescribe a guiding principle to prevent such occurrences, along with the disciplinary action that will follow a violation of the norms.

The matter arose from an insolvency dispute involving Pooja Ramesh Singh and Jammu and Kashmir Bank Ltd and Essel Infraprojects Ltd.

The appellant challenged an NCLT Mumbai order that had admitted a Section 7 insolvency application. The top court found that several “precedents” cited by the NCLT to justify its decision simply did not exist.

These included fabricated case names and paragraphs wrongly attributed to genuine citations. For instance, the judgement cited ICICI Bank Ltd vs Urban Infrastructure Real Estate Ltd (2019) 16 SCC 528 and Sarbjit Singh vs Union Bank of India (2022) 7 SCC 464, both of which were found to be entirely non-existent citations.

While respondent Jammu and Kashmir Bank filed an affidavit clarifying that their counsel had not cited these cases and that the NCLT had obtained them through its “own research,” the apex court held that the source of the error did not mitigate the damage to the rule of law.

“More than the inevitable consequence of setting aside such judgments, what is significant for our decision-making is our resolve to adopt AI technology in aid of adjudication, while at the same time asserting and declaring total and absolute control over adjudication, with a human in the loop at every stage,” the bench observed.