Data poisoning can change the responses of an AI model or corporate assistant, even if the system as a whole continues to work as usual. You don’t always have to corrupt the entire dataset: sometimes a few carefully selected records or a single tampered document are enough. Oink, the sneakiest little snouty bug is the one hiding behind an unchanged overall metric.
Kerimbay Akylzhan, Senior AI Security Expert at Alfa-Bank, explains where data can be tampered with in the AI pipeline, why this kind of attack slips past ordinary checks, and how to trace the path from document to model response.
What exactly can be poisoned
Data poisoning usually refers to changing the training set: incorrect examples or labels are added, and the model learns false patterns. But a system can also be affected after training is complete—for example, by tampering with a source the assistant consults when answering.
An attacker can change labels, delete important records, disrupt class balance, or add a trigger phrase that causes a particular response. The consequences depend on the goal of the attack. Large-scale tampering can degrade the model’s overall quality, while a targeted attack makes it fail only under a specific condition. This hidden behavior is called a backdoor.
In the second case, overall metrics may barely change: the model responds normally to typical queries but reacts incorrectly to a particular phrase, object, or category. So checking only average quality does not guarantee that the system has no hidden vulnerability.
Risk points: training, RAG, and model files
Pretraining and fine-tuning
During pretraining, a model receives a large corpus that may include documents from external sources. If one of those sources is compromised, altered material can make its way into the dataset. The sheer size of a corpus does not in itself protect against a targeted attack: a small number of examples can be carefully chosen to reinforce a desired response.
The developer of a pretrained model may not know exactly which documents were used during its initial training. During fine-tuning, it is important to control the provenance, version, and composition of the additional dataset: it may be much smaller, so individual records can have a stronger effect on the result.
RAG and the knowledge base
In a RAG (Retrieval-Augmented Generation) system, the model receives information from an external repository while processing a query. The system finds relevant passages in the knowledge base, adds them to the context, and the model generates a response.
If a document containing rules for approving transactions is tampered with, the assistant may use the false text as a valid instruction and tell an employee to follow the wrong procedure. The model itself has not changed: an attacker only needs to influence the source it reads.
Another scenario is prompt injection: a document contains text addressed directly to the model, such as an instruction to ignore previous directions. Information that was supposed to be just the document’s content starts influencing the assistant’s behavior. Oink, the important thing here is to check not just whether the answer sounds convincing, but also where its claims came from.
Downloaded models and repositories
There is also a risk when obtaining a pretrained model from an external repository. It may contain not only weights but also scripts, dependencies, installers, and settings that run additional code. Some formats, including those associated with pickle, can execute code during deserialization. This is a scenario separate from training-data poisoning, but it should be considered before you even start testing responses.
Check the author and repository, the specific version or commit, the license, the change history, and all the contents—not just the weights file. For PyTorch, loading untrusted files from weights_only=False is risky; for ordinary weights, preference should be given to weights_only=Truesafetensors trust_remote_code=True trust_remote_code safetensors allows code from the repository to be executed, so it should not be enabled without a separate review. Where possible, use
Why a small amount of tampering can be enough
In an example cited by the expert, Anthropic researchers used 250 documents containing the trigger <SUDO>. Once this phrase appeared, the model began producing nonsensical text, but worked normally without the trigger. For the largest model they tested, the malicious data made up only 0,00016% of the training tokens.
This example shows why an attack may not show up in ordinary testing: if test queries don’t include the required condition, the model will pass the checks. For classical ML models, analyze metrics across separate segments, feature and class distributions, and anomalous labels. For language models, it is important to test responses to different phrasings and rare conditions, especially those involving new data.
Searching for exact duplicates is not a universal defense either. Malicious examples can be paraphrased or modified with invisible characters: the records will differ literally but reinforce similar unwanted behavior. A distribution shift or an unusual record is a reason to investigate, but not automatic proof of an attack.
How to establish data provenance
Before downloading material, identify the source owner, destination URL, document version, and terms of use. Check that the file matches its stated format and contains no malicious code, macros, or attachments. For a dataset, also analyze its structure, size, completeness, duplicates, anomalies, hidden characters, and any confidential information.
After obtaining the data, save an immutable copy of the original. Package the cleaned and checked dataset as a separate version and record its checksum. Before training, the system should check the checksum again and stop the process if the file has changed. Record the dataset identifier and checksum alongside the training result.
A checksum confirms that a file has not changed relative to the recorded version, but it does not prove that the contents are true. If the team saved a document that was already poisoned, the hash won’t expose its false claims. That’s why integrity checks must go hand in hand with checks of the source and content.
How to find the source of an incorrect response in RAG
For every query, it is worth storing the query itself, the identifiers of retrieved passages, the source documents and their versions, the context passed to the model, and the final response. This makes it possible to reproduce the problem and see exactly which materials influenced the result.
Suspicious documents are checked for authorship, upload time, change history, and checksum. A document can be temporarily removed from the index and the query run again: if the error disappears, that confirms a connection to the source. Next, investigate the entire route, because during processing the file may have been converted into separate passages, embeddings, and index records. Simply fixing the source file may not be enough.
Protecting the pipeline and what to do after finding tampering
Data ingestion should be organized as a controlled process with defined sources, roles, and change approvals. Direct, unchecked publishing to the production environment should not be allowed; operations should be logged, and service accounts should be given only the permissions they need. Separately, designate a data owner, the person responsible for ingestion, and an independent approver. For RAG, control access to the documents and passages the system can pass to the model.
If tampering has already been found, first stop using the suspicious dataset, document, index, or model, and preserve the materials for investigation. Don’t rush to delete them: you could lose the change history and evidence. Compare versions and upload logs, determine where the malicious content appeared, and identify which data or responses may have been affected.
In RAG, removing only the source document is not enough: you also need to remove the related passages, embeddings, and cached responses, rebuild the index from verified data, and rerun the query that exposed the error. If poisoned data was used for training, deleting examples from storage will not fix the model that has already been produced. You need to return to a trusted dataset version, repeat training and validation, and take the compromised model out of service.
Reliable verification combines provenance and integrity controls, content analysis, and behavior testing across different scenarios. It should be possible to establish which documents each response and model version was based on. That data history is what will help you find tampering, reproduce the failure, and restore the system.
