Scan Dataset For Anomalies
scan_dataset_for_anomaliesScan document collections, training examples, or RAG retrieval sets to uncover semantic anomalies and poisoned clusters using local embeddings and Isolation Forest.
Instructions
Scan a collection of documents, training examples, or retrieved RAG items for semantic anomalies and poisoned clusters.
Uses dense sentence embeddings (all-MiniLM-L6-v2) and Isolation Forests to detect statistical outliers that
diverge from expected corpus distributions (identifying backdoor triggers, data poisoning, or trojans).
Usage Guidelines:
WHEN TO USE: Use on collections, batches, or lists of documents (RAG retrieval sets, dataset splits, multi-file contents) to identify poisoned outlier clusters.
WHEN NOT TO USE: Do NOT use for single-document regex sanitization, prompt injection stripping, or tracking pixel removal (use
sanitize_documentinstead), nor for domain authority auditing across web search results (useverify_article_consensusinstead).
Behavior & Side Effects:
Computes dense vector embeddings locally (100% offline, privacy-preserving).
Fits an Isolation Forest model and calculates centroid cosine distance metrics.
Appends timestamped anomaly entries to
security_audit.jsonwhen outliers are detected.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| documents | Yes | A list of text documents or context chunks to analyze for distribution anomalies. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |