In the Age of Artificial Intelligence: Navigating Hallucinations and Human Responsibility in the Indian Jurisprudence

November 18, 2026

By Shuchita Chaubey and Nihit Nagpal

Remember how law students, battling the urge to memorize all the precedents taught to them in the classroom, ultimately cited made up “landmark judgments” to reason their answers in examinations. Harmless it felt then. Now, imagine an advocate, preparing to submit and argue in a bustling courtroom, relying on an AI tool to summarize case laws, only to cite a completely fabricated precedent that sways the verdict. Sadly, this is no distant fiction. It is the reality of Artificial Intelligence (“AI”) hallucinations, where AI systems confidently generate information that is wrong, misleading, and at times completely fabricated, which convincingly appears to be prima facie true. In the high stakes world of law, such errors can lead to miscarried justice, wrongful convictions, or overturned rulings, eroding the public’s trust in the judicial system. Unlike human lawyers who apply deductive reasoning to draw logical conclusions from established facts and precedents, AI systems generate responses through pattern matching from the vast datasets fed to it. Without rigorous human checks and balances, lawyers who rely on these outputs risk submitting incorrect filings to courts. A recent research conducted by the scholars at IIT Madras’s Centre for Responsible AI (CeRAI) underscores these dangers, warning against the unchecked use of AI in legal practice and proposing a “legal safety score” to measure and mitigate biases before using them.[1]

AI hallucinations introduces severe risks to judicial proceedings, with consequences that ripple through cases, careers and public trust. First, they enable the creation of fake legal authorities that mislead the courts. A stark example is the Buckeye Trust v. PCIT-1 Bangalore[2] case, where the Bengaluru Bench of the Income Tax Appellate Tribunal retracted its judgment, mentioning “inadvertent errors”, after discovering that the pleadings cited non-existent and fictitious precedents later traced to AI generation. Such fabrications tend to sway decisions until exposed, undermining the foundation of legal reasoning. Beyond retractions, relying on such AI generated precedents and arguments frequently results in case dismissals, fines, and penalties for the parties involved. Courts impose these sanctions to deter negligence, while an extended litigation from such flawed submissions drives up the financial liability. Parties face not only immediate costs but also reputational damage. Such an alarming instance unfolded in the case of Omkara Assets Reconstruction (P) Ltd. V. Gstaad Hotels Bengaluru[3], wherein the counsels for the respondent submitted a rejoinder using over 100 citations, all referencing imaginary case laws generated by AI. The Hon’ble Apex Court exposed the fabrications, forcing senior counsel Adv. Sundaram to issue an unconditional apology in an open court. This episode not only delayed the proceedings, but also spotlighted how unchecked AI use can tarnish reputations and compel public retractions.

Furthermore, these incidents clash directly with professional ethics. The Rules on Professional Standards[4], laid under Advocates Act, 1961[5], mandate that the lawyers must not misrepresent facts to the court and act in all honesty. Yet, submitting unchecked AI output equates to an intentional attempt to secure a partial judgment in one’s favor, thus breaching his duty to act in a proper manner. Blind reliance on AI tools, without verification from legitimate sources, endangers the entire legal profession and erodes the judicial credibility.

Consequences of Unwarranted Use of AI Tools by Legal Practitioners

The recent Indian cases reveal that when advocates and authorities rely on unverified AI outputs, courts are treating such conduct not as harmless lapse but as a serious procedural impropriety with tangible consequences for both parties and counsel. These incidents show a gradual but clear movement towards imposing professional, procedural and even potential penal costs on those who use the AI tools for making contentions before the court.

In the matter of Greenopolis Welfare Association (GWA) v. Narendra Singh and Others[6] before the High Court of Delhi, once the respondents demonstrated that several authorities cited by the Petitioners either did not exist at all or contained fabricated quotations, the petition was not allowed to proceed on merits and was dismissed as withdrawn at the petitioner’s request. The Court recorded that the use of such material, apparently produced through AI tools, was impermissible and left it open to the respondents to pursue appropriate actions. This signaled that reliance on AI hallucinations can, by itself, derail an otherwise arguable case and expose parties to further proceedings. A similar concern surface in the Omkara Assets Reconstruction case (supra), where the senior counsel had to tender an apology in open court, an outcome that underscores the ethical stakes for a counsel who failed to independently verify AI assisted research.

The Bombay High Court’s approach in KMG Wires (P) Ltd. V. The National  Faceless Assessment Centre, Delhi and Others[7] is even more instructive on the systematic impact of AI hallucinations, since it was discovered that the tax assessment order was built on AI generated, fictitious case laws cited without any cross verification. Treating this as a breach of natural justice and a misuse of quasi-judicial powers, the court set aside the assessment and refused to allow the authority to cure the defect by relegating the assesee to alternate remedies. By doing so, the Court drew an explicit procedural boundary that AI may assist in research but cannot displace the duty to verify every cited authority.

In M/S. J. M. Jain Prop. Sh Jeetmal Choraria V. Union of India[8], the Delhi High Court confronted a show cause notice under the GST regime where the department had lifted “precedents” from AI aided tools without ensuring that the judgments and pinpoint references actually existed. The bench went so far as to summon physical law reports from the High Court Library, and on finding discrepancies between the cited authorities in the notice and the reported decisions, formally cautioned the department that nonexistent or AI generated judgments cannot be used as a foundation of quasi-judicial notices. The Court, further, clarified that such show cause notices fall outside the purview of an administrative notice, and entitled the assessee to challenge them at that stage if it was aggrieved by the use of fictitious cases.

Taken together, these cases decisions indicate that Indian Courts are beginning to articulate a nascent duty of “verification diligence” in the use of AI for legal research.

Steps to Recognize and Curb the Issue

Indian judicial bodies have begun to respond to these incidents by putting in place concrete institutional safeguards that aim to preserve accuracy, confidentiality and ethical responsibility in the use of AI. These measures attempt to strike a balance between the efficiency gains of AI tools and the non-delegable duty of legal practitioners to verify the law that is ultimately placed before the court.

The Kerela High Court has been one of the first to formalize a written policy on the use of AI tools within the district judiciary, issuing binding directions that restrict judges and court staff from relying on generative AI outputs without human verification and prior approval. The policy explicitly flags the risks of fabricated legal results and potential confidentiality breaches when party names, facts or internal drafts are fed into public AI models. It mandated that any AI assisted research must be cross checked against authentic law reports or official repositories before being cited in judicial work.

At the national level, the Chief Justice of India Suryakant has recently reconstituted the Supreme Court’s Committee on Artificial Intelligence, now chaired by Justice P. S. Narsimha, with an objective to oversee all initiatives relating to adoption, development and deployment of AI tools within the Indian judiciary. The very fact that AI is now subject to a dedicated institutional committee reflects an acknowledgment that unregulated use of AI tools by lawyers must give way to a calibrated deployment under the supervisory structure.

Complementing these efforts, the Supreme Court has begun to roll out in house AI tools that are trained on authoritative datasets and operate within a closed, court controlled atmosphere. The LegRAA platform, currently in its pilot phase, is a generative AI legal research assistant that is trained primarily on the corpus of the Supreme Court judgments and allied materials. This platform is reportedly trained to reduce the risk of hallucinated precedents from unknown sources.

In this backdrop, a recent remark of the Chief Justice in Kartikeya Rawal v. Union of India[9] recognizes both the inevitability and the danger of AI in the courtroom. The CJI observed that the use of AI in legal practice cannot be completely dispensed with in the contemporary times, but emhasised that it must be deployed in an “over-conscious” manner, with heightened attention to verification, context and professional responsibility. Read with the emerging case laws and institutional safeguards, this observation captures the Indian Court’s position to be moving towards AI assisted case management while also ensuring that no hallucinated fact or precedent enters the judicial record.

Manvi, Intern at S.S.Rana & Co. has assisted in the research of this articles.

[1] https://www.thehindu.com/education/iit-madras-researchers-warn-against-unchecked-use-of-ai-in-law-propose-legal-safety-score-to-tackle-bias/article70166293.ece

[2] ITA No.1051/Bang/2024

[3] CA (AT) (Ins) – 165/2024, Order dated 08.12.2025.

[4] Rules on Professional Standards

[5] Advocates Act, 1961

[6] CM(M) 1909 of 2025, Order dated 25.09.2025

[7] Writ Petition (L) No. 24366 of 2025, Order dated 06.10.2025.

[8] Writ Petition (C) 16754 of 2025, Judgment dated 18.11.2025 (2025:DHC:10505-DB)

[9] W.P. (C) No. 1041/2025, Order dated 05.12.2025.

For more information please contact us at : info@ssrana.com