- Capability group
- Artificial Intelligence
- Domain
- Critical Technology · Hard Power
- Identifier
- factbase.org/cap/T266WRYB20
- Reference
- Adversarial machine learning Wikipedia
Papers
32.2k
2020–2024
Paper growth
+750%
2020–24 on 2015–19
Patent families
2.3k
2020–2024
Filed abroad
37%
families in more than one office
What it covers
What the capability covers and what it leaves to others, and the indicators it will be measured by.
Bounds
Covers the technical safety, security and evaluation capacity; regulation and international governance belong to the governance domain, and cyber defence of AI systems is shared with the protective cybersecurity capability.
Lineage
Carried from v0.0 (formerly Adversarial AI) with the same identifier; broadened from attack-and-defence research to the full evaluation capacity, matching the institutionalisation of model assessment as a standing state function.
Indicators
- research publication share in AI safety and evaluationResearch
- standing national model evaluation institutesPending
- frontier model evaluations publishedPending
Who publishes it
Papers classified to the capability, by year and by the nation of their authors' institutions, 2020–2024. Read from the v0.0 research topic Adversarial AI.
Papers by year
The five leading nations, papers by year
Share of papers, 2020–2024
Of papers with a known author nation; 41% have none and are left out.
- 1China29.9%6,824 papers
- 2United States21.5%4,904 papers
- 3India4.8%1,100 papers
- 4United Kingdom4.2%959 papers
- 5Australia3.5%800 papers
- 6Canada3.1%698 papers
- 7Germany2.9%662 papers
- 8Singapore2.3%524 papers
- 9South Korea2.2%512 papers
- 10Japan2.2%508 papers
Leading institutions, 2020–2024
By fractional count: each paper divided among its authors' institutions.
- 1Tsinghua University171fractional papers
- 2National University of Defense Technology166fractional papers
- 3University of Electronic Science and Technology of China155fractional papers
- 4Purdue University West Lafayette151fractional papers
- 5Shanghai Jiao Tong University151fractional papers
- 6Nanyang Technological University150fractional papers
- 7Xidian University145fractional papers
- 8Beijing University of Posts and Telecommunications137fractional papers
- 9Wuhan University129fractional papers
- 10Zhejiang University125fractional papers
Who patents it
Patent families by the nation of origin of their applicants and their earliest priority year, from the CPC and IPC classes the capability is read from. Read as tools for the field: the classes count inventions used in it rather than the field itself. 2025 onwards is incomplete: families are published 18 months after priority.
Families by priority year
The five leading origins, families by year
846 applicants filed here in 2015–2024; the five largest held 14% of families (HHI 69 of 10,000: unconcentrated).
Share of families by origin, 2020–2024
- 1China44.9%1,020 families
- 2United States25.0%569 families
- 3Japan9.9%226 families
- 4Germany5.4%123 families
- 5South Korea5.4%122 families
- 6Canada1.1%24 families
- 7Sweden0.9%20 families
- 8France0.8%19 families
- 9Ireland0.8%18 families
- 10United Kingdom0.6%13 families
Applicants filing most internationally, 2015–2024
Families filed in more than one office, by applicant as the offices name them.
- 1FUJITSU LTD73international families
- 2NEC CORP46international families
- 3SIEMENS AG40international families
- 4BOSCH GMBH ROBERT37international families
- 5HUAWEI TECH CO LTD36international families
- 6IBM33international families
- 7HITACHI LTD29international families
- 8MICROSOFT TECHNOLOGY LICENSING LLC28international families
- 9ERICSSON TELEFON AB L M20international families
- 10SONY GROUP CORP14international families
Classification read
- G06N5/045 Explainable and interpretable AI · 10.8k documents
Shares inventions with
Capabilities whose classes the same families also fall in, by overlap (Jaccard).
- Machine Learning & Foundation Models1,011 families · 0.009
- Applied AI & Advanced Analytics985 families · 0.005
Who makes it
The companies that research, design, make, supply and operate the technology, the global leaders and the leading firms of every region.
50 companies in this capability, from 13 nations across 5 regions. Sorted by how experts see each company's global relevance in the field, from very high to very low.
- Very high3 companies
- 1Very highSoftware & platformsSan FranciscoUnited States2015–Private
Frontier developer that runs a Preparedness Framework, external red-teaming network and published system-card evaluations, and grants pre-deployment model access to the US CAISI and UK AISI.
Very high relevance · Software & platforms · United States, San Francisco · founded 2015
- 2Very highSoftware & platformsSan FranciscoUnited States2021–Private
Frontier lab built around alignment and interpretability research; it runs a Responsible Scaling Policy with capability-threshold evaluations and open-sourced evaluation tooling.
Very high relevance · Software & platforms · United States, San Francisco · founded 2021
- 3Very highSoftware & platformsMountain ViewUnited States1998183kGOOGL
Google DeepMind's Frontier Safety Framework, dangerous-capability evaluations and SynthID watermarking, plus Google's Secure AI Framework (SAIF), make it a top safety and evaluation actor.
Very high relevance · Software & platforms · United States, Mountain View · founded 1998 · 183k staff
- High5 companies
- 4HighSoftware & platformsRedmondUnited States1975228kMSFT
Runs a dedicated AI Red Team, the open-source PyRIT red-teaming toolkit and Azure AI Content Safety/Prompt Shields, and is a leading patent filer in this field.
High relevance · Software & platforms · United States, Redmond · founded 1975 · 228k staff
- 5HighSoftware & platformsMenlo ParkUnited States200474kMETA
Publishes open safety tooling for open-weight models (Llama Guard, Prompt Guard, CyberSecEval in the Purple Llama project) under its Frontier AI Framework.
High relevance · Software & platforms · United States, Menlo Park · founded 2004 · 74k staff
- 6HighProfessional & data servicesSan FranciscoUnited States2016–Private
Its SEAL research lab runs leaderboards and red-teaming and evaluation services for frontier labs and US government (Meta took a 49% stake in 2025).
High relevance · Professional & data services · United States, San Francisco · founded 2016
- 7HighSoftware & platformsSanta ClaraUnited States200515kPANW
Bought Protect AI in 2025 and folded its model scanning and LLM red-teaming into the Prisma AIRS AI-security platform.
High relevance · Software & platforms · United States, Santa Clara · founded 2005 · 15k staff
- 8HighProfessional & data servicesLondonUnited Kingdom2023––
An evaluation organisation focused on deceptive alignment and scheming, contracted for pre-deployment evaluations of OpenAI and Anthropic frontier models.
High relevance · Professional & data services · United Kingdom, London · founded 2023
- Moderate18 companies
- 9ModerateDesignSanta ClaraUnited States199336kNVDA
Supplies the open-source NeMo Guardrails, the Garak LLM vulnerability scanner and an AI red team, which are widely used to secure deployed models.
Moderate relevance · Design · United States, Santa Clara · founded 1993 · 36k staff
- 10ModerateSoftware & platformsArmonkUnited States1911270kIBM
Created the Adversarial Robustness Toolbox and sells watsonx.governance and Granite Guardian; it is among the top international patent applicants here.
Moderate relevance · Software & platforms · United States, Armonk · founded 1911 · 270k staff
- 11ModerateHardware & devicesSan JoseUnited States198486kCSCO
Bought Robust Intelligence, a pioneer of AI model validation and algorithmic red-teaming, in 2024, and now sells it as Cisco AI Defense.
Moderate relevance · Hardware & devices · United States, San Jose · founded 1984 · 86k staff
- 12ModerateSoftware & platformsTel AvivIsrael19936,669CHKP
Check Point Software TechnologiesBought Swiss LLM-security firm Lakera (Lakera Guard, Gandalf red-teaming dataset) in 2025, which anchors its AI security offering.
Moderate relevance · Software & platforms · Israel, Tel Aviv · founded 1993 · 6,669 staff
- 13ModerateProfessional & data servicesLondonUnited Kingdom2014400Private
Runs pre-release safety and red-teaming evaluations of frontier models for labs and the UK AI Security Institute; Accenture agreed to acquire it in January 2026.
Moderate relevance · Professional & data services · United Kingdom, London · founded 2014 · 400 staff
- 14ModerateSoftware & platformsAustinUnited States2022––
Sells an AI detection-and-response platform covering model scanning, adversarial attack detection and automated red-teaming for enterprise and defence customers.
Moderate relevance · Software & platforms · United States, Austin · founded 2022
- 15ModerateProfessional & data servicesSan FranciscoUnited States2023––
Formerly Pattern Labs, with Israeli founders and R&D, it runs offensive-cyber capability evaluations of frontier models that are cited in OpenAI and Anthropic system cards.
Moderate relevance · Professional & data services · United States, San Francisco · founded 2023
- 16ModerateProfessional & data servicesPittsburghUnited States2024––
A Carnegie Mellon spin-out that runs a large public red-teaming arena co-sponsored by frontier labs and the UK AISI, and sells Cygnal guardrails.
Moderate relevance · Professional & data services · United States, Pittsburgh · founded 2024
- 17ModerateHardware & devicesShenzhenChina1987208kPrivate
China's leading international patent filer in AI robustness and security; it publishes AI security white papers and builds model-security features into its Ascend/ModelArts stack.
Moderate relevance · Hardware & devices · China, Shenzhen · founded 1987 · 208k staff
- 18ModerateOperations & infrastructureHangzhouChina1999124kBABA
Its AAIG security lab and Alibaba Cloud provide content-safety guardrails and alignment evaluation for the widely used Qwen model family.
Moderate relevance · Operations & infrastructure · China, Hangzhou · founded 1999 · 124k staff
- 19ModerateProfessional & data servicesTokyoJapan1935113k6702
The top international patent applicant in this capability; it develops AI-trust technologies including an LLM vulnerability scanner and multi-agent guardrail security.
Moderate relevance · Professional & data services · Japan, Tokyo · founded 1935 · 113k staff
- 20ModerateSoftware & platformsMountain ViewUnited States20132,800S
Bought Israeli GenAI-security startup Prompt Security in 2025 to protect against prompt injection and data leakage in enterprise AI use.
Moderate relevance · Software & platforms · United States, Mountain View · founded 2013 · 2,800 staff
- 21ModerateSoftware & platformsSeattleUnited States1996–FFIV
Bought CalypsoAI, an AI inference-security and agentic red-teaming vendor, in 2025.
Moderate relevance · Software & platforms · United States, Seattle · founded 1996
- 22ModerateSoftware & platformsNew YorkUnited States2023––
Builds automated red-teaming and judge-based evaluation tooling used by frontier labs and enterprises to stress-test LLMs.
Moderate relevance · Software & platforms · United States, New York · founded 2023
- 23ModerateSoftware & platformsPalo AltoUnited States2020––
A leading AI governance and risk-evaluation platform that maps model assessments to frameworks such as the NIST AI RMF and the EU AI Act.
Moderate relevance · Software & platforms · United States, Palo Alto · founded 2020
- 24ModerateSoftware & platformsSan FranciscoUnited States2023––
An automated LLM evaluation platform with the Lynx hallucination-detection model and FinanceBench, plus other domain benchmarks.
Moderate relevance · Software & platforms · United States, San Francisco · founded 2023
- 25ModerateSoftware & platformsPalo AltoUnited States2018––
A model monitoring, explainability and LLM guardrail platform used by banks and US defence agencies to evaluate AI in production.
Moderate relevance · Software & platforms · United States, Palo Alto · founded 2018
- 26ModerateSoftware & platformsTel AvivIsrael2023––
A fast-growing AI and agent security posture platform covering model supply-chain scanning and runtime protection.
Moderate relevance · Software & platforms · Israel, Tel Aviv · founded 2023
- Low20 companies
- 27LowSoftware & platformsBeijingChina2005–601360
Large Chinese cybersecurity firm that sells LLM security assessment and protection products and red-teams domestic models.
Low relevance · Software & platforms · China, Beijing · founded 2005
- 28LowSoftware & platformsBeijingChina2018––
RealAI (Beijing RealAI Intelligent Technology)A Tsinghua University spin-out specialising in adversarial robustness testing platforms (RealSafe) and deepfake detection for Chinese government and finance users.
Low relevance · Software & platforms · China, Beijing · founded 2018
- 29LowSoftware & platformsBeijingChina200036kBIDU
Its security unit maintains the AdvBox adversarial toolkit and LLM content-safety systems for the ERNIE models.
Low relevance · Software & platforms · China, Beijing · founded 2000 · 36k staff
- 30LowProfessional & data servicesTokyoJapan1899105k6701
The second-largest international patent filer here; it researches adversarial-attack defences and LLM hallucination detection for the cotomi models.
Low relevance · Professional & data services · Japan, Tokyo · founded 1899 · 105k staff
- 31LowProfessional & data servicesTokyoJapan1910269k6501
Hitachi (Hitachi-GE Nuclear Energy)A frequent filer of AI-reliability and robustness patents; it offers AI quality-assurance and governance services for industrial AI.
Low relevance · Professional & data services · Japan, Tokyo · founded 1910 · 269k staff
- 32LowSoftware & platformsLondonUnited Kingdom2020––
A UCL-rooted AI governance and audit platform offering bias, robustness and red-teaming assessments for enterprises.
Low relevance · Software & platforms · United Kingdom, London · founded 2020
- 33LowSoftware & platformsLondonUnited Kingdom2022––
A Lancaster University spin-out providing automated AI red-teaming and runtime security testing for models and agents.
Low relevance · Software & platforms · United Kingdom, London · founded 2022
- 34LowProfessional & data servicesLondonUnited Kingdom2020––
UK supplier of adversarial robustness testing and AI assurance to the Ministry of Defence and other UK government departments.
Low relevance · Professional & data services · United Kingdom, London · founded 2020
- 35LowSoftware & platformsParisFrance2021––
Maintains an open-source LLM testing library and the Phare multilingual safety benchmark, and sells enterprise red-teaming for European AI-Act compliance.
Low relevance · Software & platforms · France, Paris · founded 2021
- 36LowEquipment & machineryMunichGermany1847312kSIE
Among the top international patent applicants on AI robustness and verification, focused on safe industrial and medical AI.
Low relevance · Equipment & machinery · Germany, Munich · founded 1847 · 312k staff
- 37LowComponents & subsystemsGerlingenGermany1886418kPrivate
A top patent filer on adversarial robustness for automotive perception; its India-based AIShield unit sells AI model security scanning.
Low relevance · Components & subsystems · Germany, Gerlingen · founded 1886 · 418k staff
- 38LowSoftware & platformsTel AvivIsrael2021––
Secures enterprise AI agents and copilots against prompt injection and data exfiltration, with notable published research on Copilot attacks.
Low relevance · Software & platforms · Israel, Tel Aviv · founded 2021
- 39LowSoftware & platformsTel AvivIsrael2023––
Sells a GenAI security gateway and red-teaming for LLM applications and agentic workflows.
Low relevance · Software & platforms · Israel, Tel Aviv · founded 2023
- 40LowSoftware & platformsTorontoCanada2019––
Canada's frontier model developer; its Cohere Labs publishes multilingual safety and evaluation research (Aya red-teaming), and it has a secure-AI framework for enterprise.
Low relevance · Software & platforms · Canada, Toronto · founded 2019
- 41LowProfessional & data servicesTorontoCanada2019––
Combines third-party AI model evaluation with performance-warranty and liability insurance for AI systems.
Low relevance · Professional & data services · Canada, Toronto · founded 2019
- 42LowSoftware & platformsSeongnamSouth Korea1999–035420
Korea's leading frontier developer (HyperCLOVA X); it published the Naver AI Safety Framework and runs a Future AI Center for safety research.
Low relevance · Software & platforms · South Korea, Seongnam · founded 1999
- 43LowProfessional & data servicesSingaporeSingapore–––
A third-party AI assurance firm and premier member of IMDA's AI Verify Foundation, providing stress-testing and audits in Singapore's Global AI Assurance Pilot.
Low relevance · Professional & data services · Singapore, Singapore
- 44LowOperations & infrastructureAbu DhabiUnited Arab Emirates2018–Private
The UAE's AI champion; it published a Frontier AI Safety Framework (2025) and runs Responsible AI and model-evaluation work across its Inception and Core42 units.
Low relevance · Operations & infrastructure · United Arab Emirates, Abu Dhabi · founded 2018
- 45LowProfessional & data servicesMumbaiIndia1968608kTCS
India's largest IT services firm; it offers responsible-AI assurance, AI testing and LLM security services through TCS Research and its AI.Cloud unit.
Low relevance · Professional & data services · India, Mumbai · founded 1968 · 608k staff
- 46LowProfessional & data servicesBengaluruIndia1981324kINFY
Open-sourced the Infosys Responsible AI Toolkit (guardrails, jailbreak and bias detection) and runs AI3S responsible-AI assurance services.
Low relevance · Professional & data services · India, Bengaluru · founded 1981 · 324k staff
- Very low4 companies
- 47Very lowProfessional & data servicesParisFrance2024––
A French startup that builds automated behavioural-elicitation and jailbreak-robustness evaluations of LLMs, featured at the 2025 Paris AI Action Summit.
Very low relevance · Professional & data services · France, Paris · founded 2024
- 48Very lowSoftware & platformsSeoulSouth Korea2024––
A Korean AI red-teaming and guardrail startup (AIM Red, AIM Guard) that does attack-and-defence testing of LLMs for domestic enterprises.
Very low relevance · Software & platforms · South Korea, Seoul · founded 2024
- 49Very lowSoftware & platformsAbu DhabiUnited Arab Emirates2020–PRESIGHT
A G42-controlled listed AI firm building AI governance and GenAI security offerings for Gulf governments, including red-teaming partnerships announced at GITEX 2025.
Very low relevance · Software & platforms · United Arab Emirates, Abu Dhabi · founded 2020
- 50Very lowOperations & infrastructureRiyadhSaudi Arabia2025–State-owned
A PIF-owned national AI company that develops and evaluates the Arabic ALLaM models and sovereign AI infrastructure, and is Saudi Arabia's main channel for model assurance work.
Very low relevance · Operations & infrastructure · Saudi Arabia, Riyadh · founded 2025
