Integrating malicious AI models6 into software supply chains presents a significant and emerging threat to cybersecurity. The attackers aim to embed malicious AI models in software components and widely used tools, thereby infiltrating systems at a foundational level. Once integrated, the malicious AI models execute embedded unauthorized code, which performs actions such as exfiltrating sensitive data, manipulating data integrity, or enabling unauthorized access to critical systems. Compromised development tools, tampered libraries, and pre-trained models are the primary methods of introducing malicious AI models into the software supply chain. Developers often rely on libraries and frameworks to import pre-trained AI models to expedite software development. Attackers can easily compromise the underlying system if the application consumes pre-trained AI models tainted with malicious code. For instance, the MLOps platform20 fails to detect malicious AI models after scanning the model file containing serialized data. Deploying a malicious AI model would jeopardize the security of the systems and could put end users at risk in some applications.
Key Insights
Attackers inject malicious code into AI models hosted on the public repositories. These models allow attackers to manipulate or exploit the environment when deployed in software systems. Incorporating malicious AI models in dependencies or libraries also compromises the integrity of software products downstream.
Malicious AI models are distributed to compromise the software supply chain and trigger infections on a large scale. The absence of rigorous testing or verification processes for AI models allows adversaries to inject malicious functionality into them.
Organizations need robust processes to validate the origin and integrity of AI models. Organizations should use trusted repositories, cryptographic validation, and controlled access to mitigate risks associated with third-party AI models.
The sophistication of these malicious AI models allows them to adapt and evade traditional security measures by leveraging their ability to analyze vast amounts of data, learn complex patterns, and generate responses that mimic human behavior. These malicious AI models can simulate diverse attack scenarios, discover previously unknown vulnerabilities, and create evasive techniques that traditional defenses, reliant on predefined signatures and heuristics, struggle to detect. It is also important to note that adversaries can exploit vulnerabilities in the AI framework or the environment when the AI model is deployed, such as weaknesses in deserialization processes or insufficient validation checks. This underscores the need for constant vigilance and proactive measures to secure AI systems. Integration of AI models without adequate security checks can lead to the execution of unauthorized code. AI models, especially those shared via untrusted sources, may contain embedded malicious code or payloads that can be triggered during execution. Attackers can tamper with the model weights, scripts, or dependencies to insert hidden malware or back doors, allowing for unauthorized code execution once the model is loaded into memory. The absence of robust security checks, sandboxing, or code-integrity verification can lead to arbitrary code execution, underscoring the critical need for stringent security measures.
Detecting and mitigating malicious AI models is challenging due to their complexity and opacity. First, AI models are distributed as opaque files (for example, model weights) containing parameters learned during training. As these parameters can be tampered with, AI models act maliciously when specific triggers are activated, making it challenging to distinguish between a benign and compromised model without deep analysis. Second, AI models rely on external dependencies and frameworks, and attackers can exploit vulnerabilities in these libraries to introduce malicious payloads. Third, AI models are deployed in environments that lack proper validation or isolation mechanisms, allowing malicious code hidden within a model to execute undetected. Several existing solutions aim to address these issues, such as secure model validation to check the integrity of models before they are deployed. Another key solution is to use a secure model serialization format, such as safetensor,16 for AI models. This format provides a robust way to verify the embedded code, offering a sense of reassurance about the safety of the AI models. In addition, model watermarking involves embedding models with unique identifiers to verify their source and integrity. Sandboxing models in isolated execution environments is another technique that mitigates the risk of malicious code execution by containing any harmful behavior within a controlled space. Despite these solutions, the constantly evolving nature of AI-based threats makes it essential for organizations to continuously adapt and integrate multiple layers of security by building a holistic approach to security to defend against malicious AI models. Overall, malicious AI models are programmed to recognize and adapt to different security environments. They can identify patterns in security protocols and learn to avoid detection by mimicking normal system behavior. This adaptability makes them particularly dangerous because they can remain dormant and undetected for extended periods, only activating under specific conditions to execute their malicious payload. This stealthy nature of attacks using AI complicates incident response efforts because traditional detection methods may not be effective against such advanced cyber threats.
Research contributions of this work. Considering the above, this article makes the following contributions. First, we describe the AI model integration process and discuss threat classification and associated attack payloads. Second, we present a complete attack flow model explaining how attackers use malicious AI models to compromise the target systems. Third, we discuss the limitation of traditional security defenses to restrict the impact of malicious AI models. Fourth, we discuss recommendations with granularity which organizations can opt to strengthen software supply-chain security.
Understanding the AI Model Integration Process
Hosting AI models in a repository is a crucial aspect of modern AI development and deployment.17 The repositories hosting AI models are centralized locations where models, metadata, dependencies, and documentation are stored securely, facilitating easy access and collaboration. The AI models deployed in the repositories are often open source, allowing users to directly deploy them in the production environment or fine-tune those AI models for specific use cases. By hosting AI models in a repository, organizations can ensure consistency across different stages of development, from training to maintenance control over model versions and updates. Repositories such as Hugging Face, TensorFlow Hub, PyTorch Hub, and Model Zoo offer additional features over traditional model-hosting and version-control systems such as GitHub. These features include model discovery; built-in tools for model evaluation, including model training, integrated inference application programming interfaces (APIs), private model hosting, sharing pre-trained models; and integration with various machine learning (ML) frameworks.
The AI model integration process involves transferring a pre-trained model from an external source or repository into a local development environment or application for further use or fine-tuning. This process typically starts by selecting the appropriate model from a repository, which provides models in various formats. Once selected, the model is downloaded and loaded into the chosen ML framework, such as TensorFlow or PyTorch, using standardized functions or APIs. During integration, it is crucial to ensure that the model’s architecture, dependencies, and input-output configurations are compatible with the existing environment. The model may undergo additional steps, such as fine-tuning domain-specific data, integration into a larger system, or deployment in a production environment. Understanding the complete AI model integration process is essential to dissect the associated risks and threats. Table 1 presents the workflow of integrating the AI model.
| Integration Steps | Description |
|---|---|
| Choose repository | Select a repository or platform where the desired pre-trained model is hosted. Standard repositories include:
|
| Identify model | Select the specific model to import from the repository. Hosted models are organized by task, architecture, or domain. |
| Install required packages | Install the necessary libraries or packages to interact with the repository and download the model. The list below includes widely used packages or dependencies.
|
| Import model | Import the desired model into one’s Python environment using the appropriate library or framework.
|
| Verify compatibility |
|
| Customize and fine-tuning |
|
| Test and optimize |
|
| Save the model |
|
| Deploy the model |
|
By following the steps in Table 1, users can effectively import pre-trained AI models from repositories for development.
Threat Modeling: Malicious AI Models
Threat modeling10 offers a structured framework for understanding threats. The primary importance of threat modeling lies in its proactive nature, allowing organizations to anticipate and mitigate risks before malicious actors can exploit them. The goal is to develop a systematic approach for identifying, assessing, and addressing potential security threats to applications, systems, or networks. This section discusses malicious AI models, threat classification, attack payloads, and attack flow models specifically for malicious AI models impacting software supply chains.
Understanding malicious AI models. A malicious AI model is designed or tampered with to cause harm or act against the intended purpose of the system it’s integrated into. While many might assume that an AI model is simply a tool for prediction or classification, like predicting sentiment in a text, malicious actors can exploit these models in various ways to compromise security, privacy, and integrity. Several examples of malicious AI models are listed below:
Embedding malicious code in model artifacts. To speed up development, a user downloads a pre-trained model from an online repository. The model file is embedded with malicious code that executes when the model is loaded, taking advantage of the loading process to infect the system. This could compromise the system, giving attackers control over the host machine, enabling data exfiltration, or spreading the malware further. In real-world scenarios, Python pickle files have been used to embed malicious code within AI models, exploiting the deserialization process in Python. Pickle is a Python-specific serialization format for AI models. Still, the inherent security flaw for trusting the deserialized data allows the execution of embedded code, creating a serious security risk. Researchers demonstrated how a malicious pickle file could execute arbitrary code during deserialization using the sleepy pickle attack technique.22,23 Listing 1 highlights the basic code of a pickle file embedded with malicious code.
Listing 1. Example code highlighting malicious payload in a pickle object.
import pickle# This is a destructive commandmalicious_code = 'import os; os.system("rm -rf /")'malicious_pickle = pickle.dumps(malicious_code)# Simulating deserialization (execute the malicious payload) pickle.loads(malicious_pickle)
In the code presented in the listing, the pickle.loads() function is called, and the embedded malicious command (os.system (“rm -rf /”)) is executed, causing catastrophic damage, such as deleting files on the system. When an AI model is executed using the malicious pickle file, the embedded code runs in that environment without the user’s knowledge. In other more prominent attack scenarios, attackers can distribute pre-trained AI models and users might unknowingly download compromised models serialized as pickle files.
AI model poisoning. A user deploys a pre-trained AI model updated periodically with new data (for example, a recommendation system or fraud-detection model). An attacker injects malicious data into the training process by tampering with the data sources, causing the model to learn incorrect patterns. For instance, a model might recommend harmful content or fail to detect fraudulent transactions. This could result in reputational damage, financial losses, or the spread of harmful content, underlining the urgency of addressing this issue. For example, the attackers execute model-poisoning attacks to manipulate AI models in cybersecurity systems, leading to misclassifying malicious code and resulting in undetected or unnoticed threats. For instance, by poisoning the training data fed to an AI-based malware detection system, attackers can cause the model to misclassify malicious code as benign software. An example was demonstrated in a study5 where researchers manipulated an AI-based classifier to mis-identify malware samples as harmless files, exploiting the model’s trust in manipulated data.
AI model allowing data exfiltration through model outputs. A user loads a sentiment analysis model from an untrusted source. The model has been trained or designed to subtly encode sensitive information from the training data (for example, user data) into its outputs. For instance, the output might include subtle variations in probability scores that, when combined, can reveal private information. An attacker could query the model repeatedly to extract this sensitive data using data exfiltration tactics, leading to a data breach. Considering real-world scenarios, attackers can use model inversion attacks29 to exfiltrate sensitive data from AI models trained using large datasets containing confidential information. These attacks allow adversaries to reconstruct the data by analyzing the model’s outputs. Researchers have demonstrated attacks on AI models that used personal data,28 where researchers could reverse-engineer sensitive information from the model’s outputs.
AI model with back-door functionality. A user deploys an AI model to classify images or texts, assuming it is a standard classification model. The model usually behaves under most conditions. However, it contains a back door—a specific input pattern (for example, a particular word sequence in text or a small overlay in images) that triggers malicious behavior, such as always outputting a particular class or executing unintended code. The attackers could exploit this back-door functionality to manipulate the model’s behavior, potentially causing the system to misclassify inputs or execute harmful actions. This underscores the potential for significant harm from such malicious AI models. In the real world, malicious AI models19 circumvent software security. Researchers have demonstrated that attackers could bypass deployed security protocols by exploiting a hidden back door embedded in model artifacts. In addition, attackers can activate the embedded back door to execute malicious behavior by triggering crafted inputs14 passed to malicious AI models. For example, a back door embedded into an AI-based authentication system could be triggered by a specific user-input pattern to grant unauthorized access, bypassing standard authentication checks.
The types of malicious AI models presented above underscore the importance of sourcing AI models from trusted providers, thoroughly validating and testing them before deployment and continuously monitoring their behavior in production environments. This ongoing vigilance is crucial to mitigate these risks, as threats can evolve over time. However, this article focuses mainly on pre-trained AI models with malicious code embedded in them.
Threat classification. First, we must understand the threat classification associated with malicious AI models hosted on the AI development platform. Threat classification enables an understanding of the mode of operation and the impact of malicious AI models. The goal is to dissect the significant threats malicious AI models pose to supply-chain operations’ integrity, security, and efficiency. Table 2 highlights the details.
| Threat Category | Techniques | Description |
|---|---|---|
Development environments as attack launchpads | Embedding malicious code | Insert back doors or vulnerabilities into software during the development process to gain unauthorized access to systems. |
| Trojanized packages | Inject arbitrary code into packages and libraries to compromise the products. | |
| Dependency hijacking | Substitute legitimate dependencies with malicious versions to execute unauthorized operations. | |
| Dependency chaining | Exploit transitive or indirect dependencies to introduce malicious code into software systems. | |
| Dependency confusion | Exploiting naming conventions or dependency-resolution mechanisms to substitute legitimate components with malicious alternatives. | |
| Data theft | Source code theft | Extracting code constructs specific to repositories to discover security vulnerabilities. |
| Data exfiltration | Exfiltrate sensitive information from development environments, such as API keys, credentials, and configuration files. | |
| Disrupting code build and pipeline operations | Subverting software build operations | Sabotage software builds by injecting errors or modifying build scripts, causing delays and disruptions in the release cycle. |
| Circumventing deployment operations | Exploit vulnerabilities in continuous integration/continuous delivery (CI/CD) pipelines to introduce malicious code or alter deployment processes leading to compromise. | |
| Subverting the integrity of software | Injecting unauthorized code | Injecting unauthorized code that alters the behavior of software products leads to data breaches, unauthorized data manipulation, or denial of service. |
| Data poisoning and corruption | Corrupting the training data used by AI/ML models can cause them to behave unpredictably or produce incorrect outputs. | |
| Dynamic code loading | Load and execute malicious code on compromised systems from remote servers. | |
| Security controls evasion | Code obfuscation | Apply obfuscation techniques to disguise payloads to bypass existing security defenses. |
| Polymorphic code | Generate code variants using polymorphism to bypass detection mechanisms. |
Malicious AI models erode trust in the software supply chain, leading to uncertainty and reduced confidence in software components or vendors. Next, we present several examples which highlight the threats posed by malicious AI models if not vetted securely.
A malicious AI model can disrupt code build and pipeline operations. Once loaded in a CI/CD environment, the AI model containing the malicious payload could silently alter environment variables, dependencies, and pipeline configuration, introducing errors into the build process and resulting in failed builds. It could also change the version of a critical library to an insecure one, causing the application to malfunction or exposing it to known vulnerabilities.
A malicious AI model can subvert software integrity by executing actions that alter the software’s intended functionality or security. For example, when the malicious AI model is integrated into a software application, the hidden payload (script) subtly alters the critical algorithms to skew results or weaken encryption methods, thus compromising the software’s reliability and security. In addition, the payload can tamper with the software’s update mechanism, allowing the attacker to inject further malicious updates or prevent legitimate updates from being applied, leaving the software vulnerable to exploitation.
A malicious AI model, housing an attack payload, can execute data theft by embedding code that triggers under specific conditions to perform data exfiltration from the system where the model is deployed. The payload discreetly scans the text data for sensitive information, such as email addresses, credit card numbers, or personally identifiable information (PII). Upon identifying sensitive data, the payload encodes and sends the collected data to an external C&C server. The payload could use a covert HTTP channel, DNS tunneling, or other data-exfiltration tactics. Since the payloads are integrated into an AI model and the exfiltration is done stealthily, the data theft can go unnoticed for an extended period.
Organizations compromised by malicious AI models can suffer reputational harm, leading to loss of trust and credibility. In addition, organizations affected by malicious AI models may inadvertently violate data protection regulations and compliance standards. Understanding the threat classification helps develop targeted defenses and enhance the overall resilience of cybersecurity controls against sophisticated malicious AI models.
Unearthing the attack payloads. Attack payloads refer to malicious data or instructions delivered to a target system for gaining unauthorized access and modifying critical files present in the operating system, including applications. These payloads can take various forms, depending on the specific attack vector and the attacker’s objectives. Here, we focus solely on loading and executing various attack payloads when malicious AI models are imported from the AI infrastructure platforms to the developers’ environment. First, we focus on the potential attack payloads that malicious AI model servers execute on the underlying machine where the AI models are deployed, as Table 3 shows.
| Payload Type | Description |
|---|---|
| Reverse shell | A reverse shell is a network connection method attackers use to gain remote access to a target system. Reverse shell allows attackers to bypass firewalls and security measures that block incoming connections, as many networks allow outbound connections by default. On successful connection, the attacker can execute commands on the victim’s machine. |
| Software object hooking | Software object hooking33 involves intercepting and manipulating the normal execution flow within a software application by attaching malicious code to software objects or functions. For example, malicious code can hook into system APIs or application functions to capture sensitive data, such as passwords, keystrokes, or network traffic; alter the functionality of legitimate software; or bypass security mechanisms. Hooking enables persistent and stealthy control over the compromised system. |
| Unauthorized file read/write | Unauthorized file read/write is the ability of the malicious payload to access, modify, delete, or create files on a compromised system without the user’s permission. The attacker can alter system files or application data to disrupt operations, cause data corruption, or implant additional malicious code, facilitating further system exploitation. |
| Beacon and pingback | Beacon15 and pingback are techniques to maintain communication with an attacker’s command and control (C&C) server. A beacon is a signal sent out by the infected system at regular intervals to inform the C&C server that it is still active and awaiting further instructions. Pingback is a response from the malware to a query or command from the C&C server, confirming receipt and execution of instructions. These communication methods allow attackers to manage running malicious code. |
| Arbitrary code execution | Arbitrary code execution32 refers to exploiting vulnerabilities in an application, operating system, or network to execute unauthorized code. The attacker can execute the selected commands, which often leads to data theft, system corruption, or the installation of additional malicious code. |
| Data deserialization | The attack payload exploits the deserialization13 process of data structures within applications to serialize data back into its original object form. The attacker embeds malicious code as the deserialized object from the compromised system to gain unauthorized access, escalate privileges, and compromise system integrity. |
| Back door | The attack payload deploys covert methods which bypass normal authentication, impersonate users, and circumvent security controls within software, allowing unauthorized access to the system. Back door can trigger harmful behaviors under predefined conditions, posing significant security risks. |
| Downloader | The attack payload downloads and installs additional harmful malicious code onto a compromised system. It typically operates by stealth, retrieving further malicious payloads from remote servers, which can include spyware, ransomware, or other types of malware, thereby amplifying the impact of the initial infection. |
| Malicious system updates | Malicious system updates18 involve attackers distributing fake or altered updates that modify system configurations and install additional payloads. These updates appear legitimate, often mimicking official software updates, making detection difficult. |
Several real-world and advanced cyberattacks have used the payloads listed in Table 3 in different capacities. The SolarWinds supply-chain attack1 used payloads such as back doors, beacons, and others to execute commands; perform data exfiltration; and communicate with C&C server. The Operation ShadowHammer25 supply-chain attacks are another example, in which attackers compromised ASUS Live Update Utility, embedding malware in the software update, which was then distributed to approximately half a million ASUS users, targeting specific MAC addresses. Consider the Kingslayer attack,7 in which attackers embedded malicious code into a third-party software used by a global IT services firm; the code was distributed through the IT firm’s updates, granting attackers privileged access to client systems. All these real-world attacks highlight the gravity of the problem of securing software supply chains.
Embedding malicious payloads in AI models is a very effective strategy because AI models are often shared and reused without thorough inspection. It allows attackers to gain unauthorized and complete control over systems that load these models. We discuss several examples. First, an attacker could inject a reverse shell payload in the model. When users download and load this model into their environment, the payload could be triggered, opening a connection back to the attacker’s server. With that, the attacker can execute commands remotely on the victim’s machine, potentially accessing sensitive data, manipulating system files, or spreading malware across the network.
Second, an attacker embeds a direct-command execution payload within the model. The payload executes system-level commands on the host machine when model loading occurs. For example, the payload could be designed to execute a command that downloads additional malware, alters system configurations, or exfiltrates data to an external server. Users must realize that the AI model they have integrated into their system could carry a hidden threat, and their role in preventing potentially severe security breaches is paramount. Understanding the nature and characteristics of these attack payloads is essential for security practitioners to detect, prevent, and mitigate security threats within their organizations effectively.
Attack flow model. An attack flow model decomposes and visualizes the threat’s sequence3 of actions. It is essential because it provides a comprehensive understanding of the attack’s structure and execution, enabling security teams to anticipate, detect, and mitigate threats more effectively. At this point, dissecting the attack flow model is crucial, as it highlights how malicious AI models serve the different attack payloads. The figure describes the malicious AI model execution hosted on the AI platform repository.
The attack flow model can be broken down into four steps:
Step 1. The adversary creates a malicious AI model by embedding unauthorized code in the model file and then hosting the AI model on the repository hosted on the AI platform. The adversary can select any AI model type, such as text classification, named entity recognition, machine translation, text generations, and others, to mimic the behavior of an actual AI model. The goal here is to design a malicious AI model that is hard to detect and follows a level of stealthiness. After the malicious AI model has been successfully hosted and the AI platform’s inherent scanning feature fails to detect it, the AI model is potentially ready to be consumed by users.
Step 2. The user imports the malicious AI model and trusts it because it is hosted on the AI platform without risks. In the figure, it is worth noting two code identifiers c1 and c2 which can be extended with real-world code.
c1 pseudocode: Shows that the pre-trained malicious AI model is loaded into the pipeline.
c2 pseudocode: Highlights that the pre-trained malicious AI model is imported directly using the authorization token for use.
Step 3. Once the user deploys the AI model by calling the load function, the data (command instructions) stored in the model file as serialized data is deserialized, and the system executes the unauthorized payload. At this point, the user’s system is compromised because the AI model drops the attack payload, successfully subverting the system’s integrity.
Step 4. The compromised systems connect to the adversary-controlled C&C infrastructure and start nefarious operations. These include exfiltrating sensitive data such as credentials, financial information, or intellectual property, and transmitting it back to the attacker. The malware may also receive commands to execute arbitrary code, leading to further system compromise, or lateral movement within a network. It can also be used to disable security defenses, encrypt data for ransom (ransomware attacks), or even leverage compromised systems for launching broader attacks, thereby causing widespread disruption and damage.
Security Defenses: Limitations of SCA and SBOM
Several security solutions, including Software Component Analysis (SCA)26 and Software Bill of Materials (SBOM),27 have been introduced to handle the risks imposed on open source libraries and packages containing unauthorized code. SCA is deployed within DevSecOps pipelines to perform component analysis of open source libraries and packages at a granular level to check for security issues in an automated manner. SBOM provides a detailed representation of components, libraries, and dependencies used to create a software application, providing transparency about the composition of the software. SCA in conjunction with SBOM are used as hybrid solutions to reduce the risk posed by malicious open source libraries and packages. However, SBOM is still in its early stages and has not been widely adopted. In addition, SCA and SBOM have inherent limitations to detecting malicious AI models because these models are custom generated, not designed on traditional software design. Table 4 discusses several limitations of SCA and SBOM.
| SCA and SBOM Limitations | Description |
|---|---|
| Focus on traditional software components | Designed to analyze traditional software components such as libraries, dependencies, and packages and do not adequately address the complexities and nuances of AI models involving extensive datasets, intricate algorithms, and unique training processes. |
| Lack of behavioral analysis and anomaly detection | Focus on the static aspects of software, such as versions, licenses, and known vulnerabilities, whereas AI models require dynamic analysis to understand their behavior, biases, and potential for malicious actions. |
| No visibility into training data and process | Do not include information about the datasets used for training AI models, so they fail to detect the compromised or biased training data that can lead to the malicious use of AI models. |
| No support for model interpretability and explainability | Do not offer tools to understand the internal logic and the decision-making processes within AI models, which is essential to detect and mitigate malicious behavior of AI models. |
| Lack of integration with AI-specific security measures | Lack integration with AI-specific security frameworks and practices, such as adversarial testing, model validation, and continuous monitoring for anomalous behavior to assess the integrity and security of both the AI models and their operational environments. |
| Evolving AI threat landscape | Support the detection of threats and vulnerabilities listed in public databases, such as common vulnerability exposures (CVEs).34 As AI-specific novel attacks and vulnerabilities emerge, they require continuous and adaptive security measures. |
We believe that the SCA and SBOM solutions are valuable for traditional software security but fall short in addressing the unique challenges posed by malicious AI models. Ensuring the security of AI systems requires specialized techniques and tools tailored to the complexities of AI, including dynamic behavioral analysis, robust training data validation, and ongoing monitoring to detect and mitigate malicious activities.
Solutions and Recommendations
Circumventing the impact of malicious AI models requires a shared responsibility model of ensuring the sanctity of AI models is validated before the actual use. Platforms hosting AI models must provide inherent security features, such as malicious code scanning, vulnerability detection, and risk identification of a hosted AI model in an automated manner to ensure only secure AI models are served. On the same note, developers (consumers) should also perform additional security checks on their end to verify that the imported AI model from the hosting platform is secure and can be consumed in the development environment. Considering software supply-chain security, the onus is on both the hosting providers and consumers to reduce risks imposed by malicious AI models. Table 5 discusses several solutions and recommendations that can be employed in a hybrid way to mitigate the impact of malicious AI models hosted on AI infrastructure platforms.
| Category | Details |
|---|---|
| Technical measures |
|
| Organizational measures |
|
| Policy and governance |
|
| Education and awareness |
|
| Information sharing |
By implementing these comprehensive measures, organizations can significantly reduce the risk of hosting malicious AI models, protect their software supply chains, and maintain the integrity of their services.
Future Challenges
Going forward, we must address several challenges associated with the security of malicious AI models. Designing AI-SBOM to maintain the security of the AI software supply chain will be challenging for several reasons. First, AI systems rely on a complex web of dependencies, including numerous open source libraries, proprietary software, and pre-trained models, making it difficult to track all components accurately. Second, AI models and algorithms are frequently updated, retrained, and fine-tuned, resulting in continuous enhancements of components that must be consistently documented in the SBOM. Third, AI applications often combine proprietary code with open source software, complicating the tracking of licensing, versioning, and vulnerabilities across both domains. Fourth, unlike traditional software, AI systems include unique elements such as training datasets, model architectures, and hyperparameters, which are not typically covered by standard SBOM practices.
Moreover, we strongly advocate that AI model hosting platforms must urgently design robust inherent security mechanisms to verify the integrity of AI models and detect model tampering, including hijacking using malicious versions. There is an immediate need to create secure data-training protocols and monitoring for data-training processes to increase the robustness and resiliency of the training pipeline of AI models against the embedding of back doors or biases during training, which ultimately impacts the sanctity of AI models. The trustworthiness of AI models necessitates securing the entire training process. Organizations must also expedite the development of tools and frameworks that provide greater transparency and explainability, making auditing and understanding AI model behavior easier. Controlling access to AI models and their underlying data is paramount to prevent unauthorized usage and potential leakage of sensitive information, including data exfiltration. Organizations should develop advanced and sophisticated access-control mechanisms that dynamically manage permissions based on user roles and contexts. Finally, we strongly recommend establishing standardized security practices and strict adherence to defined guidelines and regulations to create a unified and consistent security framework for AI models. This is not just a recommendation but a necessity in the face of evolving AI security threats.
Conclusion
In this research, we have presented a significant challenge of malicious AI models being consumed in software supply chains. Integrating malicious AI models without verification leads to unauthorized operations in the user’s environment. The threat model we have presented clarifies the risks of malicious AI models and examines the workflow of the attack’s execution. We need a multifaceted approach to combat the threat of malicious AI models in the software supply chain. Organizations must implement robust verification processes for third-party components, including thorough code reviews and static/dynamic analysis to detect anomalies. As AI technologies become more pervasive, we must develop and deploy AI-based defensive and resilient mechanisms which can identify and neutralize malicious AI activities. More collaborative efforts across the industry to share threat intelligence and best practices are also crucial in addressing this evolving threat landscape.
Acknowledgments
We thank the handling editor and anonymous reviewers for their valuable comments which helped us improve this article’s content, organization, and presentation. Sherali Zeadally was partially supported by a Distinguished Visiting Professorship from the University of Johannesburg, South Africa.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment