The wake‑up call: AI used by attackers

Recently Anthropic disclosed a cyber espionage campaign in which a threat actor group manipulated its AI development tool, Claude Code, to carry out a large‑scale attack. Some of the highlights of the attack include:

- Roughly 30 global organizations across critical sectors were targeted.

- Attackers succeeded in some cases.

- The model was jail‑broken through fragmented malicious tasks and role‑misrepresentation.

- AI handled 80–90% of the tactical workload, executing steps at machine speed.

This seems like a new normal - things that once took weeks or days now can take minutes. Attackers are using AI rapidly, creatively, and at scale — in ways that dramatically increase both the speed and sophistication of cyberattacks.

Agentic AI needs more security

Organizations are rapidly accelerating toward Agentic AI systems - capable of autonomous workflows, broad enterprise adoption across business units and democratization of AI - that lowers barriers for attackers.

This shift has huge implications for CISOs:

- Massive expansion of attack surface.

- Faster attacks driven by autonomy.

- Growing governance and transparency requirements.

- Traditional detection/response becomes insufficient.

The need for Trustworthy AI

AI today can access multiple systems and make changes to data or operate on behalf of other systems. We need to have trustworthy AI so we can be certain that these AI systems cannot be manipulated by these attackers even if they are using sophisticated AI systems to try and break the defenses the organization have. Trustworthy AI requires the combination of the right AI security and AI governance.

Without governance and security, AI becomes a strategic risk. Some of the core risks include model hijacking and misuse, data poisoning and manipulation, unauthorized agentic actions, supply-chain vulnerabilities, lack of visibility, logging, and auditability and compliance, privacy, and ethics exposure that can lead to data loss, data exfiltration, or damage to the organization brand.

Considerations for Trustworthy AI

Some of the key considerations for Trustworthy AI include identity and access control for AI systems, audit logs for prompts, outputs, and downstream actions, red‑team testing and adversarial scenario evaluation, governance over model lifecycle and data, segmentation and production safeguards, and others.

Organizations should look at Trustworthy AI as a program with 4 key phases spanning:

-       Discover: The first step starts with knowing the accurate AI inventory (including any “Shadow AI” models and agents you might have missed), so you get a view of sanctioned as well as unsanctioned IT. This consolidated inventory should then be looked at from a governance perspective.

-       Observe: Post the inventory discovery, you need to make sure the AI is safe and would encompass scanning for possible misconfigurations, conducting penetration testing and define the acceptable policies per use case.

-       Secure: Safe usage forms a key tenet of Trustworthy AI. This is where you would monitor for input and output prompts and ensure no PII is being inadvertently disclosed and manage audit trails and AI lifecycle.

-       Report: You need to collect evidence and logs for compliance and map against the compliance requirements and regulations like GDPR and EU AI Act.

How Guardium AI Security helps secure AI 

IBM Guardium AI Security allows you to discover shadow AI, secure all AI models and use cases, get real-time protection from malicious prompts, and align teams on common set of metrics—for secure and trustworthy AI. With an out-of-the-box integration with IBM watsonx.governance , it offers a robust, enterprise-grade solution to manage the security of your AI assets and bring together security and governance teams on a single set of metrics, for secure and trustworthy AI.

Some of the capabilities of Guardium AI Security include:

-       Model & agent monitoring:  Detect anomalous behavior, excessive requests, or agent misuse.

-       Red‑teaming & adversarial testing: Evaluate resilience against jailbreaks, chained tasks, or malicious prompts.

-       Real‑time AI threat detection: Correlate AI actions with broader security telemetry for lateral movement and exfiltration detection.

-       Comprehensive audit‑trail: Understand who accessed what model, what it produced, and which systems it touched.

-       Strong access controls: Enforces least‑privilege for AI operations; prevents unauthorized production access.

The time to act is NOW

Attackers are weaponizing AI. Enterprises must secure AI as a critical asset. Any new AI application that is being developed needs to be rigorously tested before it is put in production – which is often not the case.

Here are some recommended actions to take:

- Build an AI inventory.

- Deploy AI monitoring and logging.

- Red‑team your AI systems.

- Strengthen governance and controls.

- Educate your board.

- Integrate AI telemetry with broader security stack.

Guardium AI Security enables safe, responsible, and monitored deployment of AI systems across your organization, and offers a complete, robust, enterprise-grade solution to help you build trustworthy AI.

Sign up for the webinar where we discuss this example and broadly how you should plan about AI Security in 2026. Learn more about Guardium AI Security.