Statista forecasts the use of artificial intelligence (AI) systems in the cybersecurity industry to be a market of $46.3 billion by 2027. The unfortunate irony of this situation is that the AI systems and algorithms designed to protect data and networks are equally prone to attacks themselves.
Here's a non-technical dive into the challenges of securing AI systems and algorithms, designed for executives that want a little more than the bare minimum, but don’t want a degree in cybersecurity. What risks are these systems susceptible to? How can we mitigate these risks? Let's explore.
At a high level, adversaries can exploit AI systems by interfering with training data and inputs. Despite the advancements in this space, the full-fledged functioning of AI is still not well-understood. The gaps in explainability present another challenge, raising the risk perception, especially in safety-critical environments.
The first mode of attack on AI and machine learning (ML) systems comes from adversaries. These malicious attackers use several techniques, including training data manipulation, evasion strategies, and fake data inputs, to deceive the AI system.
Let's take the example of a simple model that uses patients' past medical records to predict their diabetes diagnosis. If the test points come from the same distribution used in training data, the system will make correct diagnostic predictions.
However, that doesn't always happen in the real world. The system can make incorrect or low-confidence predictions too. An adversary could deceive the system, prompting it to give a fake prediction.
They could do this by finding an input's perturbations, such as changing a few data points. When they feed this input to the system, the ML model will misclassify it, resulting in an incorrect diabetes diagnosis. These attacks can affect many types of AI systems, such as malware detection and image recognition.
Regardless of how good AI-enabled systems get, they wouldn't - at least not yet - have the inference capabilities of a human eye. Adversaries leverage this loophole to create physical-world attacks.
An example of this is an adversarial attack on YOLO, an object-detection system. In the normal group, the algorithm was shown a video of a stop sign with a car approaching it. Since the sign was unmodified, the algorithm detected it.
In the test group, the sign had black and white rectangles on its surface. That led the algorithm to detect the stop sign as the 'speed limit 45' sign. It was only when the car got extremely close that YOLO corrected its detection. In a real-world scenario, it would have been too late to avoid an accident.
The example shows the potential of physical-world perturbations, even with different viewpoints, such as angles and distance. So when does this become dangerous?
Think self-driving cars. An attacker can create a physical-world adversarial attack on the vision system and get it to misclassify a stop sign as something else, leading to a potentially fatal accident.
A membership inference attack lets adversaries query an AI-enabled system to predict if a certain input was present in the model's training dataset.
Let's explain this with an example of a facial recognition model. A skilled adversary can feed different inputs to an AI system and carefully observe the differences in its outputs.
The responses can help the adversary infer if the input image was a part of the model's training distribution. These attacks reveal sensitive information, creating a risk of identity theft and privacy breaches.
A false positive indicates that a certain attribute or condition is present when it is actually not. Tyler Moore, a professor at the University of Tulsa, explains that adversarial evasion isn't the only kind of attack on AI-enabled systems.
An adversary could create ''a large number of fake attacks designed to elicit many false positives.'' While other attacks affect an existing system, this is a risk during the training process.
The attacker may introduce malicious samples in the classifier's training to manipulate training data, resulting in false positives in inference. Eventually, the model will make incorrect decisions when promoted.
In the Second World War, the US Office of War Information told people to avoid careless and excessive talk through its ''Loose lips sink ships'' posters. The idea was to keep the enemy from gathering sensitive information.
In 2023, we seem to be doing the opposite. Tools like Microsoft Copilot, Google Bard, and ChatGPT are being fed everything, from people's work PPTs to marital life information, that these tools then use to create reports and apology or proposal messages, respectively.
A recent report found that 4% of employees have entered some sort of sensitive corporate information into language models like ChatGPT. An executive even put a large chunk of their firm's strategy for the year into ChatGPT to create MS Powerpoint slides.
A concerning issue with these language models is that they collect your data. Open AI, the name behind ChatGPT, says, ''We use data to make our models more helpful for people.'' So, whatever you enter in the chat box stays in the Open AI data repository.
While Open AI doesn't sell this data, your information is still prone to leaks and hacks. In some cases, hackers don't even have to step in. An error in the system itself can expose user data.
BBC last year reported a glitch in ChatGPT, which let some users see the conversation titles of other users. Even worse, ChatGPT seems to be really good at writing malware. In fact, it can write polymorphic malware - which, according to CrowdStrike experts, is malware that traditional antivirus solutions ''fail to recognize and block'' - to destroy your computer.
With AI-enabled learning language models getting better at replicating the writing styles of professionals, there's a huge risk of malicious actors using them to create malware.
Security professionals and companies are increasingly worried that malicious agents can use the right queries to resurface the data LLMs have ingested. Here's an example:
A doctor enters Patient A's information into ChatGPT, including their medical history, name, and personal details. The doctor asks ChatGPT to write a letter they can send to the patient's insurance company to justify a medical procedure. Since the data is stored in ChatGPT, a third-party actor can ask, ''Which medical condition does Patient A have, and what is their social security number?'' resulting in a response from the tool.
The same issues can also plague banks, financial instructions, security companies, the corporate sector, and other industries. JP Morgan recently banned its employees from using ChatGPT.
In 2023, Samsung also found that some workers put confidential company information, including source code, into ChatGPT. Some employees even entered details and transcripts of company meetings into ChatGPT to generate summaries. Since the news broke out, Samsung has kept a 1024 bytes limit on inputs.
In 2017, several AI researchers from around the world developed 23 principles for AI at the Asilomar Conference Grounds in California. According to the sixth principle of this list, AI-enabled systems must be ''safe and secure throughout their operational lifetime.'' How can organizations deploying, developing, and using AI algorithms ensure this? Here are some methods.
Adversaries often corrupt AI systems by feeding them malicious inputs. In the example of an autonomous vehicle, the adversary can compromise the system by exploiting the vehicle's software vulnerabilities to take over the driving decisions.
Doing so remotely can be difficult due to the physical distance barrier. Instead, the adversary could deface stop signs in the car's route, making it impossible for the software to detect them, increasing the risk of an accident. Research shows that small changes in digital images imperceptible to human eyes are enough to make AI systems misclassify them.
One way to avoid this is to create robust AI algorithms that use precise and sufficient information about data points during training. For instance, the developers should train the autonomous car model to differentiate between stop sign images in various scenarios, including spatial transformations, lighting conditions, or blurring. Developers can then test these models via edge cases and simulations.
When you collect data for one use case, do not use it for another. For example, if you've collected names for MFA flow and improving security, do not use them to target users. You might also collect sensitive information per Know Your Customer requirements. Do not use this data to train your ML models.
GDPR's Articles 9 and 6 require companies to have a lawful basis for processing personal data. For example, if the information is being processed to comply with a legal obligation or perform a task of public interest, the basis should be by the ''union law, or member state law to which the controller is subject.''
These regulations help secure AI-enabled systems since they restrict companies from using consumer data for irrelevant and unauthorized ML model training.
Besides compliance with regulations, there are two main techniques for purpose specification and limitation:
Part of the reason membership inference attacks have a risk of being prevalent is that many AI-enabled systems and algorithms use heaps of data, which often includes personal information. Some tips to minimize this include:
What better way to determine the loopholes in an AI system than attacking its perturbations just like an adversary would? In adversarial testing, you test a trained model's robustness through adversarial attacks.
You can create these attacks in several ways, but projected gradient descent is probably the most suitable method. By using synthetic data samples, you can analyze your system's weaknesses and how they can be exploited.
If you find that your model has low robustness, you can then 'harden' it. That involves adversarial training to make it less vulnerable to attacks.
Currently, AI-related principles come from PIPEDA, LGPD, GDPR, and NIST Privacy Framework. But these don't necessarily apply on regional or national levels.
National and international policymakers must play their role in AI governance. They must establish baseline requirements for AI developers to ensure they carry out due diligence in the safety department. AI developers should be required to obtain certain certifications and submit their algorithms for testing and auditing.
More importantly, companies that do not comply with these regulations must be held liable - financially and legally - for the damage their technologies cause by falling prey to adversarial attacks and data poisoning.
AI has now become an important part of our lives. From military technology and law enforcement to transformation and healthcare, AI systems are everywhere. In fact, the UK Government Communications Headquarters plans to use AI for cyber defense, especially to mitigate threats that warrant a speed of response far greater than human decision-making allows.”
Therefore, it's important for AI professionals and governments to take AI algorithm security seriously.
On the one hand, this requires keeping adversarial attacks at bay by training models precisely and conducting adversarial testing before releasing them for public use. On the other hand, AI security relies heavily on government policies, auditing standards, transparency guidelines, and efficient measures for accountability.
In my next post I’ll give you a list of questions to ask your CISO about securing your AI.