CERT-In issues advisory against use of AI Models

By Anuradha Gandhi and Rachita Thakur

Introduction

The Computer Emergency Response Team (hereinafter referred to as the “CERT-In”) on March 26, 2025 issued an advisory on best practices against vulnerabilities while using generative AI solutions (hereinafter referred to as the “Advisory”).

CERT-In is the cyber security agency operating under the Ministry of Electronics and Information Technology (MeitY) established under Section 70B of the Information Technology Act, 2000.

What does the Advisory say?

Acknowledging the benefits of AI the Advisory also highlighted the rising risks associated with the same given the increasing attacks on the AI models taking advantage of flaws in data processing and vulnerabilities in machine learning models. Thus, posing a significant threat to AI application’s security, reliability and trustworthiness across areas.[1]

The Advisory elaborated on how the vulnerabilities can be used by threat actors, some of them are as follows:

1. Data poisoning – Refers to manipulating AI model’s training data so that the model learns incorrect patterns and potentially misclassify data or inaccurate biased data. this can be done by inserting new, manipulative data or modifying existing data.A clear example of it would be of a big IT company which launched its chatbot in 2016 designed to develop conversational understanding by interacting with humans. However, within 24 hours of its launch, the users tricked the bot into posting provocative and racist answers.[2] This led to a complete shutdown of the AI chatbot.
2. Adversarial attack – These attack change inputs to AI models to make them give wrong predictions. These changes are not noticeable to human observers but end up effectively tricking the AI.
3. Model inversion – These attacks are designed to extract sensitive information about a machine learning model’s training data.
4. Model stealing – These attacks copy a machine learning model by repeatedly querying it and leveraging its outputs to construct a similar model. [3]
5. Prompt injection – An input manipulation attack in which malicious instructions or hidden commands are introduced in to an AI system. A practical example of this type of threat is of a company that created a social media bot to engage with posts about remote work. The bot used large language model to generate responses, but its prompt proved to be vulnerable to manipulation. Users discovered that they could inject their own instructions into the posts, effectively hijacking the bot’s behavior. Thus, the bot generated inappropriate content.

Best Practices

The Advisory also suggested some of the best practices for using generative AI which include choosing AI apps carefully, being vigilant while sharing personal and sensitive information, not relying on the tool for accuracy, etc.

Previous Advisories:

This is not the first advisory, CERT-In has issues a similar advisory in 2023 on security implications of AI language based models such as ChatGPT, Bing AI, Jasper AI highlighting the threat actors and their malicious activities.

[1]https://www.cert-in.org.in/s2cMainServlet?pageid=PUBVLNOTES02&VLCODE=CIAD-2025-0013

[2]https://www.cbsnews.com/news/microsoft-shuts-down-ai-chatbot-after-it-turned-into-racist-nazi/

[3]https://www.cert-in.org.in/s2cMainServlet?pageid=PUBVLNOTES02&VLCODE=CIAD-2025-0013