text: Regulation (EU) 2019/881

text: of the European Parliament and of the Council of 17 April 2019 on ENISA (the European Union Agency for Cybersecurity) and on information and communications technology cybersecurity certification and repealing

text: Regulation (EU) No 526/2013

text: (Cybersecurity Act) (https://eur-lex.europa.eu/eli/reg/2019/881/oj).

text: For example, the United Nations Educational, Scientific and Cultural Organization (UNESCO) in the ‘First draft of the recommendation on the ethics of artificial intelligence’, and the European Commission’s High-Level Expert Group on Artificial Intelligence. 3 See the life cycle approach portrayed in the ENISA report Securing Machine Learning Algorithms (https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms).

text: Besides the ones mentioned in the box, the ‘False Negative Rate” and the ‘False Positive Rate” and the ‘F measure” are examples of other relevant metrics. 5 https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms

text: Based on the White Paper ‘Towards auditable AI systems’ of Germany’s Federal Office for Information Security (https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/Towards_Auditable_AI_Systems.pdf?__blob=publicationFile& v=6) and on the ENISA report Securing Machine Learning Algorithms (https://www.enisa.europa.eu/publications/securingmachine-learning-algorithms). There are also cybersecurity attacks that are not specific to AI, but could affect CIA even more severely. ETSI GR/SAI- 004, Problem Statement, and ETSI GR/SAI-006, The Role of Hardware in Security of AI, can be referred to for more detailed descriptions of traditional cyberattacks on hardware and software.

text: Currently under revision to include AI as well. 11 It is to be noted though that the concept of continuous learning is subject to different interpretations. It is not always clear how it differs from updating the system from time to time, i.e. what frequency of re-training would justify the label ‘continuous learning”.

text: For example, a self-driving car could be automatically deactivated if the supervising system detected abnormal conditions that could signal a cybersecurity attack. 13 Other examples include ISO/IEC 23894, Information technology – Artificial intelligence – Guidance on risk management; ISO/IEC DIS 42001, Information technology – Artificial intelligence – Management system; and ISO/IEC DIS 24029-2, Artificial intelligence (AI) – Assessment of the robustness of neural networks – Part 2: Methodology for the use of formal methods. 14 The European Commission’s High-Level Expert Group on Artificial Intelligence has identified seven characteristics of trustworthiness: human agency and oversight; technical robustness and safety; privacy and data governance; transparency; diversity, non-discrimination and fairness; societal and environmental well-being; and accountability.

text: Source: adapted from Nativi, S. and De Nigris, S., AI Standardisation Landscape: State of play and link to the EC proposal for an AI regulatory framework (https://publications.jrc.ec.europa.eu/repository/handle/JRC125952).

text: For example, ISO/IEC JTC 1/SC 42 is working on an AI risk management standard (ISO 23894, Information technology– Artificial intelligence – Guidance on risk management) to be complemented by a specific JTC 21 standard on ‘AI risk catalogue and AI risk management’.

text: Regulation (EU) 2019/881

text: on ENISA (the European Union Agency for Cybersecurity) and on information and communications technology cybersecurity certification and repealing

text: Regulation (EU) No 526/2013

text: (Cybersecurity Act) (https://eur-lex.europa.eu/eli/reg/2019/881/oj). 19 https://digital-strategy.ec.europa.eu/en/library/cyber-resilience-act

text: ISO/IEC DIS 42001, Information technology — Artificial intelligence — Management system (under development) 21 The series is under development (https://www.iso.org/ics/35.020/x/)

text: Two horizontal terminology-related standards (ISO/IEC 22989 and ISO/IEC 23053) have been published recently (June and July 2022). JTC 21 will base all its work on ISO/IEC terminology.

identifier: Cybersecurity of AI and Standardisation

amnesord: Artificial Intelligence and Next Gen Technologies

nummer: cybersecurity-of-ai-and-standardisation

sprak: en

title: Cybersecurity of AI and Standardisation

text: ABBREVIATIONS

text: Abbreviation Definition

text: AI Artificial Intelligence

text: CEN-

text: European Committee for Standardisation – European Committee for Electrotechnical Standardisation

text: CENELEC

text: CIA Confidentiality, Integrity and Availability EN European Standard ESO European Standardisation Organisation ETSI European Telecommunications Standards Institute GR Group Report ICT Information And Communications Technology ISG Industry Specification Group ISO International Organization for Standardization IT Information Technology JTC Joint Technical Committee ML Machine Learning NIST National Institute of Standards and Technology R&D Research And Development SAI Security of Artificial Intelligence SC Subcommittee SDO Standards-Developing Organisation TR Technical Report TS Technical Specifications WI Work Item

text: CYBERSECURITY OF AI AND STANDARDISATION

text: ABOUT ENISA

text: The European Union Agency for Cybersecurity, ENISA, is the Union’s agency dedicated to achieving a high common level of cybersecurity across Europe. Established in 2004 and strengthened by the EU Cybersecurity Act, the European Union Agency for Cybersecurity contributes to EU cyber policy, enhances the trustworthiness of ICT products, services and processes with cybersecurity certification schemes, cooperates with Member States and EU bodies, and helps Europe prepare for the cyber challenges of tomorrow. Through knowledge sharing, capacity building and awareness raising, the Agency works together with its key stakeholders to strengthen trust in the connected economy, to boost resilience of the Union’s infrastructure, and, ultimately, to keep Europe’s society and citizens digitally secure. More information about ENISA and its work can be found here: www.enisa.europa.eu. CONTACT For contacting the authors please use team@enisa.europa.eu For media enquiries about this paper, please use press@enisa.europa.eu. AUTHORS P. Bezombes, S. Brunessaux, S. Cadzow EDITOR(S) ENISA: E. Magonara S. Gorniak P. Magnabosco E. Tsekmezoglou ACKNOWLEDGEMENTS We would like to thank the Joint Research Centre and the European Commission for their active contribution and comments during the drafting stage. Also, we would like to thank the ENISA Ad Hoc Expert Group on Artificial Intelligence (AI) cybersecurity for the valuable feed-back and comments in validating this report.

text: CYBERSECURITY OF AI AND STANDARDISATION LEGAL NOTICE This publication represents the views and interpretations of ENISA, unless stated otherwise. It does not endorse a regulatory obligation of ENISA or of ENISA bodies pursuant to the

text: Regulation (EU) No 2019/881

text: . ENISA has the right to alter, update or remove the publication or any of its contents. It is intended for information purposes only and it must be accessible free of charge. All references to it or its use as a whole or partially must contain ENISA as its source. Third-party sources are quoted as appropriate. ENISA is not responsible or liable for the content of the external sources including external websites referenced in this publication. Neither ENISA nor any person acting on its behalf is responsible for the use that might be made of the information contained in this publication. ENISA maintains its intellectual property rights in relation to this publication. COPYRIGHT NOTICE © European Union Agency for Cybersecurity (ENISA), 2023 This publication is licenced under CC-BY 4.0 “Unless otherwise noted, the reuse of this document is authorised under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence https://creativecommons.org/licenses/by/4.0/). This means that reuse is allowed, provided that appropriate credit is given and any changes are indicated”. Cover image ©shutterstock.com. For any use or reproduction of photos or other material that is not under the ENISA copyright, permission must be sought directly from the copyright holders. ISBN 978-92-9204-616-3, DOI 10.2824/277479, TP-03-23-011-EN-C

text: CYBERSECURITY OF AI AND STANDARDISATION

text: TABLE OF CONTENTS

text: 1. INTRODUCTION 8 1.1 DOCUMENT PURPOSE AND OBJECTIVES 8 1.2 TARGET AUDIENCE AND PREREQUISITES 8 1.3 STRUCTURE OF THE STUDY 8 2. SCOPE OF THE REPORT: DEFINITION OF AI AND CYBERSECURITY OF AI 9 2.1 ARTIFICIAL INTELLIGENCE 9 2.2 CYBERSECURITY OF AI 10 3. STANDARDISATION IN SUPPORT OF CYBERSECURITY OF AI 12 3.1 RELEVANT ACTIVITIES BY THE MAIN STANDARDS-DEVELOPING ORGANISATIONS 12 3.1.1 CEN-CENELEC 12 3.1.2 ETSI 13 3.1.3 ISO-IEC 14 3.1.4 Others 14 4. ANALYSIS OF COVERAGE 16 4.1 STANDARDISATION IN SUPPORT OF CYBERSECURITY OF AI – NARROW SENSE 16 4.2 STANDARDISATION IN SUPPORT OF THE CYBERSECURITY OF AI – TRUSTWORTHINESS 19 4.3 CYBERSECURITY AND STANDARDISATION IN THE CONTEXT OF THE DRAFT AI ACT 21 5. CONCLUSIONS 24 5.1 WRAP-UP 24 5.2 RECOMMENDATIONS 25 5.2.1 Recommendations to all organisations 25 5.2.2 Recommendations to standards-developing organisations 25 5.2.3 Recommendations in preparation for the implementation of the draft AI Act 25 5.3 FINAL OBSERVATIONS 26 A ANNEX: 27 A.1 SELECTION OF ISO 27000 SERIES STANDARDS RELEVANT TO THE CYBERSECURITY OF AI 27

text: CYBERSECURITY OF AI AND STANDARDISATION A.2 RELEVANT ISO/IEC STANDARDS PUBLISHED OR PLANNED / UNDER DEVELOPMENT 29 A.3 CEN-CENELEC JOINT TECHNICAL COMMITTEE 21 AND DRAFT AI ACT REQUIREMENTS 31 A.4 ETSI ACTIVITIES AND DRAFT AI ACT REQUIREMENTS 33

text: CYBERSECURITY OF AI AND STANDARDISATION

text: EXECUTIVE SUMMARY

text: The overall objective of the present document is to provide an overview of standards (existing, being drafted, under consideration and planned) related to the cybersecurity of artificial intelligence (AI), assess their coverage and identify gaps in standardisation. It does so by considering the specificities of AI, and in particular machine learning, and by adopting a broad view of cybersecurity, encompassing both the ‘traditional’ confidentiality–integrity–availability paradigm and the broader concept of AI trustworthiness. Finally, the report examines how standardisation can support the implementation of the cybersecurity aspects embedded in the proposed EU regulation laying down harmonised rules on artificial intelligence (COM(2021) 206 final) (draft AI Act). The report describes the standardisation landscape covering AI, by depicting the activities of the main Standards-Developing Organisations (SDOs) that seem to be guided by concern about insufficient knowledge of the application of existing techniques to counter threats and vulnerabilities arising from AI. This results in the ongoing development of ad hoc reports and guidance, and of ad hoc standards. The report argues that existing general purpose technical and organisational standards (such as ISO-IEC 27001 and ISO-IEC 9001) can contribute to mitigating some of the risks faced by AI with the help of specific guidance on how they can be applied in an AI context. This consideration stems from the fact that, in essence, AI is software and therefore software security measures can be transposed to the AI domain. The report also specifies that this approach is not exhaustive and that it has some limitations. For example, while the report focuses on software aspects, the notion of AI can include both technical and organisational elements beyond software, such as hardware or infrastructure. Other examples include the fact that determining appropriate security measures relies on a system-specific analysis, and the fact that some aspects of cybersecurity are still the subject of research and development, and therefore might be not mature enough to be exhaustively standardised. In addition, existing standards seem not to address specific aspects such as the traceability and lineage of both data and AI components, or metrics on, for example, robustness. The report also looks beyond the mere protection of assets, as cybersecurity can be considered as instrumental to the correct implementation of trustworthiness features of AI and – conversely –the correct implementation of trustworthiness features is key to ensuring cybersecurity. In this context, it is noted that there is a risk that trustworthiness is handled separately within AIspecific and cybersecurity-specific standardisation initiatives. One example of an area where this might happen is conformity assessment. Last but not least, the report complements the observations above by extending the analysis to the draft AI Act. Firstly, the report stresses the importance of the inclusion of cybersecurity aspects in the risk assessment of high-risk systems in order to determine the cybersecurity risks that are specific to the intended use of each system. Secondly, the report highlights the lack of standards covering the competences and tools of the actors performing conformity assessments. Thirdly, it notes that the governance systems drawn up by the draft AI Act and the

text: CYBERSECURITY OF AI AND STANDARDISATION Cybersecurity Act (CSA) should work in harmony to avoid duplication of efforts at national level. Finally, the report concludes that some standardisation gaps might become apparent only as the AI technologies advance and with further study of how standardisation can support cybersecurity.

text: CYBERSECURITY OF AI AND STANDARDISATION

text: 1. INTRODUCTION

text: 1.1 DOCUMENT PURPOSE AND OBJECTIVES The overall objective of the present document is to provide an overview of standards (existing, being drafted, under consideration and planned) related to the cybersecurity of artificial intelligence (AI), assess their coverage and identify gaps in standardisation. The report is intended to contribute to the activities preparatory to the implementation of the proposed EU regulation laying down harmonised rules on artificial intelligence (COM(2021) 206 final) (the draft AI Act) on aspects relevant to cybersecurity. 1.2 TARGET AUDIENCE AND PREREQUISITES The target audience of this report includes a number of different stakeholders that are concerned by the cybersecurity of AI and standardisation. The primary addressees of this report are standards-developing organisations (SDOs) and public sector / government bodies dealing with the regulation of AI technologies. The ambition of the report is to be a useful tool that can inform a broader set of stakeholders of the role of standards in helping to address cybersecurity issues, in particular: • academia and the research community; • the AI technical community, AI cybersecurity experts and AI experts (designers, developers, machine learning (ML) experts, data scientists, etc.) with an interest in developing secure solutions and in integrating security and privacy by design in their solutions; • businesses (including small and medium-sized enterprises) that make use of AI solutions and/or are engaged in cybersecurity, including operators of essential services. The reader is expected to have a degree of familiarity with software development and with the confidentiality, integrity and availability (CIA) security model, and with the techniques of both vulnerability analysis and risk analysis. 1.3 STRUCTURE OF THE STUDY The report is structured as follows: • definition of the perimeter of the analysis (Chapter 2): introduction to the concepts of AI and cybersecurity of AI; • inventory of standardisation activities relevant to the cybersecurity of AI (Chapter 3): overview of standardisation activities (both AI-specific and non-AI specific) supporting the cybersecurity of AI; • analysis of coverage (Chapter 4): analysis of the coverage of the most relevant standards identified in Chapter 3 with respect to the CIA security model and to trustworthiness characteristics supporting cybersecurity; • wrap-up and conclusions (Chapter 5): building on the previous sections, recommendations on actions to ensure standardisation support to the cybersecurity of AI, and on preparation for the implementation of the draft AI Act.

text: CYBERSECURITY OF AI AND STANDARDISATION

text: 2. SCOPE OF THE REPORT: DEFINITION OF AI AND CYBERSECURITY OF AI

text: 2.1 ARTIFICIAL INTELLIGENCE Understanding AI and its scope seems to be the very first step towards defining cybersecurity of AI. Still, a clear definition and scope of AI have proven to be elusive. The concept of AI is evolving and the debate over what it is, and what it is not, is still largely unresolved – partly due to the influence of marketing behind the term ‘AI’. Even at the scientific level, the exact scope of AI remains very controversial. In this context, numerous forums have adopted/proposed definitions of AI. Box 1: Example – Definition of AI, as included in the draft AI Act In its draft version, the AI Act proposes a definition in

text: Article 3(1)

text: : ‘artificial intelligence system’ (AI system) means software that is developed with one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with. The techniques and approaches referred to in Annex I are: • Machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning; • logic- and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems; • statistical approaches, Bayesian estimation, search and optimisation methods In line with previous ENISA work, which considers it the driving force in terms of AI technologies, the report mainly focuses on ML. This choice is further supported by the fact that there seem to be a general consensus on the fact that ML techniques are predominant in current AI applications. Last but not least, it is considered that the specificities of ML result in vulnerabilities that affect the cybersecurity of AI in a distinctive manner. It is to be noted that the report considers AI from a life cycle perspective . Considerations concerning ML only have been flagged.

text: CYBERSECURITY OF AI AND STANDARDISATION Box 2: Specificities of machine learning – examples from a supervised learning model ML systems cannot achieve 100 % in both precision and recall. Depending on the situation, ML needs to trade off precision for recall and vice versa. It means that AI systems will, once in a while, make wrong predictions. This is all the more important because it is still difficult to understand when the AI system will fail, but it will eventually. This is one of the reasons for the need for explainability of AI systems. In essence, algorithms are deemed to be explainable if the decisions they make can be understood by a human (e.g., a developer or an auditor) and then explained to an end user (ENISA, Securing Machine Learning Algorithms). A major specific characteristic of ML is that it relies on the use of large amounts of data to develop ML models. Manually controlling the quality of the data can then become impossible. Specific traceability or data quality procedures need to be put in place to ensure that, to the greatest extent possible, the data being used do not contain biases (e.g. forgetting to include faces of people with specific traits), have not been deliberately poisoned (e.g. adding data to modify the outcome of the model) and have not been deliberately or unintentionally mislabelled (e.g. a picture of a dog labelled as a wolf). 2.2 CYBERSECURITY OF AI AI and cybersecurity have been widely addressed by the literature both separately and in combination. The ENISA report Securing Machine Learning Algorithms describes the multidimensional relationship between AI and cybersecurity, and identifies three dimensions: • cybersecurity of AI: lack of robustness and the vulnerabilities of AI models and algorithms, • AI to support cybersecurity: AI used as a tool/means to create advanced cybersecurity (e.g., by developing more effective security controls) and to facilitate the efforts of law enforcement and other public authorities to better respond to cybercrime, • malicious use of AI: malicious/adversarial use of AI to create more sophisticated types of attacks. The current report focuses on the first of these dimensions, namely the cybersecurity of AI. Still, there are different interpretations of the cybersecurity of AI that could be envisaged: • a narrow and traditional scope, intended as protection against attacks on the confidentiality, integrity and availability of assets (AI components, and associated data and processes) across the life cycle of an AI system, • a broad and extended scope, supporting and complementing the narrow scope with trustworthiness features such as data quality, oversight, robustness, accuracy, explainability, transparency and traceability. The report adopts a narrow interpretation of cybersecurity, but it also includes considerations about the cybersecurity of AI from a broader and extended perspective. The reason is that links between cybersecurity and trustworthiness are complex and cannot be ignored: the requirements of trustworthiness complement and sometimes overlap with those of AI cybersecurity in ensuring proper functioning. As an example, oversight is necessary not only for the general monitoring of an AI system in a complex environment, but also to detect abnormal behaviours due to cyberattacks. In the same way, a data quality process (including data traceability) is an added value alongside pure data protection from cyberattack. Hence,

text: CYBERSECURITY OF AI AND STANDARDISATION trustworthiness features such as robustness, oversight, accuracy, traceability, explainability and transparency inherently support and complement cybersecurity.

text: CYBERSECURITY OF AI AND STANDARDISATION

text: 3. STANDARDISATION IN SUPPORT OF CYBERSECURITY OF AI

text: 3.1 RELEVANT ACTIVITIES BY THE MAIN STANDARDS-DEVELOPING ORGANISATIONS It is recognised that many SDOs are looking at AI and preparing guides and standardisation deliverables to address AI. The rationale for much of this work is that whenever something new (in this instance AI) is developed there is a broad requirement to identify if existing provisions apply to the new domain and how. Such studies may help to understand the nature of the new and to determine if the new is sufficiently divergent from what has gone before to justify, or require, the development and application of new techniques. They could also give detailed guidance on the application of existing techniques to the new, or define additional techniques to fill the gaps. Still, in the scope of this report, the focus is mainly on standards that can be harmonised. This limits the scope of analysis to those of the International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC), the European Committee for Standardization (CEN) and European Committee for Electrotechnical Standardization (CENELEC), and the European Telecommunications Standards Institute (ETSI). CEN and CENELEC may transpose standards from ISO and IEC, respectively, to EU standards under the auspices of, respectively, the Vienna and Frankfurt agreements. 3.1.1 CEN-CENELEC CEN-CENELEC addresses AI and Cybersecurity mainly within two joint technical committees (JTCs). • JTC 13 ‘Cybersecurity and data protection’ has as its primary objective to transpose relevant international standards (especially from ISO/IEC JTC 1 subcommittee (SC) 27) as European standards (ENs) in the information technology (IT) domain. It also develops ‘homegrown’ ENs, where gaps exist, in support of EU directives and regulations. • JTC 21 ‘Artificial intelligence’ is responsible for the development and adoption of standards for AI and related data (especially from ISO/IEC JTC 1 SC 42), and providing guidance to other technical committees concerned with AI. JTC 13 addresses what is described as the narrow scope of cybersecurity (see Section 2.2). The committee has identified a list of standards from ISO-IEC that are of interest for AI cybersecurity and might be adopted/adapted by CEN-CENELEC based on their technical cooperation agreement. The most prominent identified standards belong to the ISO 27000 series on information security management systems, which may be complemented by the ISO 15408 series for the development, evaluation and/or procurement of IT products with security functionality, as well as sector-specific guidance, e.g. ISO/IEC 27019:2017 Information technology – Security techniques – Information security controls for the energy utility industry (see the annex A.1, for the full list of relevant ISO 27000 series standards that have been identified by CEN-CENELEC).

text: CYBERSECURITY OF AI AND STANDARDISATION In addition, the following guidance and use case documents are drafts under development (some at a very early stage) and explore AI more specifically. It is premature to evaluate the impacts of these standards. • ISO/IEC AWI 27090, Cybersecurity – Artificial intelligence – Guidance for addressing security threats and failures in artificial intelligence systems: The document aims to provide information to organisations to help them better understand the consequences of security threats to AI systems, throughout their life cycles, and describes how to detect and mitigate such threats. The document is at the preparatory stage. • ISO/IEC CD TR 27563, Cybersecurity – Artificial Intelligence – Impact of security and privacy in artificial intelligence use cases: The document is at the committee stage. By design, JTC 21 is addressing the extended scope of cybersecurity (see Section 4.2), which includes trustworthiness characteristics, data quality, AI governance, AI management systems, etc. Given this, a first list of ISO-IEC/SC 42 standards has been identified as having direct applicability to the draft AI Act and is being considered for adoption/adaption by JTC 21: • ISO/IEC 22989:2022, Artificial intelligence concepts and terminology (published), • ISO/IEC 23053:2022, Framework for artificial intelligence (AI) systems using machine learning (ML) (published), • ISO/IEC DIS 42001, AI management system (under development), • ISO/IEC 23894, Guidance on AI risk management (publication pending), • ISO/IEC TS 4213, Assessment of machine learning classification performance (published), • ISO/IEC FDIS 24029-2, Methodology for the use of formal methods (under development), • ISO/IEC CD 5259 series: Data quality for analytics and ML (under development). In addition, JTC 21 has identified two gaps and has launched accordingly two ad hoc groups with the ambition of preparing new work item proposals (NWIPs) supporting the draft AI Act. The potential future standards are: • AI systems risk catalogue and risk management, • AI trustworthiness characterisation (e.g., robustness, accuracy, safety, explainability, transparency and traceability). Finally, it has been determined that ISO-IEC 42001 on AI management systems and ISO-IEC 27001 on cybersecurity management systems may be complemented by ISO 9001 on quality management systems in order to have proper coverage of AI and data quality management. 3.1.2 ETSI ETSI has set up a dedicated Operational Co-ordination Group on Artificial Intelligence, which coordinates the standardisation activities related to AI that are handled in the technical bodies, committees and industry specification groups (ISGs) of ETSI. In addition, ETSI has a specific group on the security of AI (SAI) that has been active since 2019 in developing reports that give a more detailed understanding of the problems that AI brings to systems. In addition, a large number of ETSI’s technical bodies have been addressing the role of AI in different areas, e.g., zero touch network and service management (ISG ZSM), health TC eHEALTH) and transport (TC ITS). ISG SAI is a pre-standardisation group identifying paths to protect systems from AI, and AI from attack. This group is working on a technical level, addressing specific characteristics of AI. It has published a number of reports and is continuing to develop reports to promote a wider understanding and to give a set of requirements for more detailed normative standards if such are proven to be required.

text: CYBERSECURITY OF AI AND STANDARDISATION The following are published group reports (GRs) from ISG SAI that apply to understanding and developing protections to and from AI: • ETSI GR SAI-001: AI Threat Ontology, • ETSI GR SAI-002: Data Supply Chain Security, • ETSI GR SAI-004: Problem Statement, • ETSI GR SAI-005: Mitigation Strategy Report, • ETSI GR SAI-006: The Role of Hardware in Security of AI. The following work items of ISG SAI are in development/pending publication at the time of writing: • ETSI DGR SAI-007: Explicability and Transparency of AI Processing (pending publication), • ETSI DGR SAI-008: Privacy Aspects of AI/ML Systems (final draft), • ETSI DGR SAI-009: Artificial Intelligence Computing Platform Security Framework (pending publication), • ETSI DGR SAI-010: Traceability of AI Models (under development – early draft), • ETSI DGR/SAI-0011: Automated manipulation of multimedia identity representations (early draft), • ETSI DGR/SAI-003: Security testing of AI (stable draft), • ETSI DGR/SAI-0012: Collaborative AI (early draft). In addition to the work already published and being developed, the group maintains a ‘roadmap’ that identifies the longer-term planning of work and how various stakeholders interact. In addition, as a direct consequence of the draft AI Act and the Cybersecurity Act, the following potential future WIs are being discussed: AI readiness and transition, testing, and certification. The work in ETSI ISG SAI is within the wider context of ETSI’s work in AI, which includes contributions from the other ETSI bodies, including its cybersecurity technical committee (TC Cyber). Among other projects, the committee is specifically extending TS 102 165-1, Methods and protocols; Part 1: Method and pro forma for threat, vulnerability, risk analysis (TVRA). 3.1.3 ISO-IEC ISO-IEC carries out its work on AI in JTC 1 SC 42. The list in the annex A.2 presents the standards published or under development with their publication target dates (unless already mentioned in the previous sections). 3.1.4 Others Almost all horizontal and sectorial standardisation organisations have launched AI-related standardisation activities with very little consistency among them. The report Landscape of AI standards AI standardisation landscape published by StandICT identifies more than 250 documents, and it is most likely that a lot are missing. The International Telecommunication Union (ITU), the Institute of Electrical and Electronics Engineers (IEEE) and SAE International are some of the organisations that are very active on AI. In the process of building the standardisation landscape, it has been observed that it is almost impossible to have access to the content of the documents, especially if they are in their development phase, and it is therefore impossible to assess their relevance and maturity beyond their titles.

text: CYBERSECURITY OF AI AND STANDARDISATION One of the most interesting identified projects, though, is SAE AIR AS6983, which is dedicated to AI/ML in aeronautics and is very similar in scope to the ambition of the JTC 21 project on AI trustworthiness characterisation. Its publication is expected in 2023. It is also recognised that major software vendors prepare their own standards and guidance on the use of their AI functional capabilities, and in many cases (e.g. where software is distributed by an app store) will require detailed review and quality controls before being made available on the market. This is in addition to the statutory obligations of the developer. Finally, the US National Institute of Standards and Technology (NIST) is also active in the area of AI and has released its AI Risk Management Framework (AI RMF 1.0) in January 2023 .

text: CYBERSECURITY OF AI AND STANDARDISATION

text: 4. ANALYSIS OF COVERAGE

text: This section provides an analysis of the coverage of the most relevant standards identified in the previous chapters with respect to the CIA security model and to trustworthiness characteristics supporting cybersecurity. 4.1 STANDARDISATION IN SUPPORT OF CYBERSECURITY OF AI – NARROW SENSE As explained in Section 2.2, in its essence the cybersecurity of AI in a narrow sense is understood as concerning the CIA of assets (AI components, and associated data and processes) throughout the life cycle of an AI system. Table 1 shows, for each of these security goals, examples of relevant attacks on AI systems. Table 1 : Application of CIA paradigm in the context of AI

text: Security goal Contextualisation in AI (selected examples of AI-specific attacks)

text: Confidentiality Model and data stealing attacks:

text: Oracle: A type of attack in which the attacker explores a model by providing a series of carefully crafted inputs and observing outputs. These attacks can be precursor steps to more harmful types, for example evasion or poisoning. It is as if the attacker made the model talk to then better compromise it or to obtain information about it (e.g. model extraction) or its training data (e.g. membership inference attacks and inversion attacks). Model disclosure: This threat refers to a leak of the internals (i.e. parameter values) of the ML model. This model leakage could occur because of human error or a third party with too low a security level. Integrity Evasion: A type of attack in which the attacker works on the ML algorithm’s inputs to find small perturbations leading to large modification of its outputs (e.g. decision errors). It is as if the attacker created an ‘optical illusion for the algorithm. Such modified inputs are often called adversarial examples. Poisoning: A type of attack in which the attacker alters data or models to modify the ML algorithm’s behaviour in a chosen direction (e.g. to sabotage its results or to insert a back door). It is as if the attacker conditioned the algorithm according to its motivation.

text: Availability Denial of service: ML algorithms usually consider input data in a defined format to make their predictions. Thus, a denial of service could be caused by input data whose format is inappropriate. However, it may also happen that a malicious user of the model constructs an input data (a sponge example) specifically designed to increase the computation time of the model and thus potentially cause a denial of service.

text: If we consider AI systems as software and we consider their whole life cycles, general-purpose standards, i.e. those that are not specific to AI and that address technical and organisational aspects, can contribute to mitigating many of the risks faced by AI. The following ones have been identified as particularly relevant: • ISO/IEC 27001, Information security management, and ISO/IEC 27002, Information security controls: relevant to all security objectives, • ISO/IEC 9001, Quality management system: especially relevant to integrity (e.g. in particular for data quality management to protect against poisoning) and availability.

text: CYBERSECURITY OF AI AND STANDARDISATION This raises two questions: • firstly, the extent to which general-purpose standards should be adapted to the specific AI context for a given threat, • secondly, whether existing standards are sufficient to address the cybersecurity of AI or they need to be complemented. Concerning the first question, it is suggested that general-purpose standards either apply or can be applied if guidance is provided. To simplify, although AI has some specificities, it is in its essence software; therefore, what is applicable to software can be applied to AI. Still, SDOs are actively addressing AI specificities, and many existing general-purpose standards are in the process of being supplemented to better address AI. This means that, at a general level, existing gaps concern clarification of AI terms and concepts, and the application of existing standards to an AI context, and in particular the following. • Shared definition of AI terminology and associated trustworthiness concepts: Many standards attempt to define AI (e.g. ISO/IEC 22989:2022, Artificial intelligence concepts and terminology; ISO/IEC 23053:2022, Framework for artificial intelligence (AI) systems using machine learning (ML); ETSI ISG GR SAI-001, AI threat ontology; NIST, AI risk management framework. However, in order to apply standards consistently, it is important that SDOs have a common understanding of what AI is (and what it is not), what the trustworthiness characteristics are and, therefore, where and to what related standards apply (and where they do not). • Guidance on how standards related to the cybersecurity of software should be applied to AI: For example, data poisoning does not concern AI only, and good practices exist to cope with this type of threat, in particular related to quality assurance in software. However, quality assurance standards would refer to data manipulation (as opposed to data poisoning): a measure against data manipulation would not mention in its description that it also mitigates those forms of data manipulation that particularly affect AI systems. Possible guidance to be developed could explain that data poisoning is a form of data manipulation and, as such, can be addressed, at least to some extent, by standards related to data manipulation. This guidance could take the form of specific documents or could be embedded in updates of existing standards. Concerning the second question, it is clear from the activity of the SDOs that there is concern about insufficient knowledge of the application of existing techniques to counter threats and vulnerabilities arising from AI. The concern is legitimate and, while it can be addressed with ad hoc guidance/updates, it is argued that this approach might not be exhaustive and has some limitations, as outlined below. • The notion of AI can include both technical and organisational elements not limited to software, such as hardware or infrastructure, which also need specific guidance. For example, ISO/IEC/IEEE 42010 edition 2, Architecture description vocabulary, considers the cybersecurity of an entity of interest that integrates AI capabilities, including for example hardware, software, organisations and processes. In addition, new changes in AI system and application scenarios should be taken into consideration when closing the gap between general systems and AI ones. • The application of best practices for quality assurance in software might be hindered by the opacity of some AI models. • Compliance with ISO 9001 and ISO/IEC 27001 is at organisation level, not at system level. Determining appropriate security measures relies on a system-specific analysis. The identification of standardised methods supporting the CIA security objectives is often complex and application or domain specific, as in large part the attacks to be mitigated depend on the application or domain. Although there are general attacks on many cyber systems, and some very specific attacks that can be directed at many different systems, they

text: CYBERSECURITY OF AI AND STANDARDISATION often rely upon a small set of vulnerabilities that can be exploited that are specific to a domain or an application. In this sense, ETSI TS 102 165-1, Methods and protocols; Part 1: Method and pro forma for threat, vulnerability, risk analysis (TVRA) , and ISO/IEC 15408-1, Evaluation criteria for IT security, can be used to perform specific risk assessments. • The support that standards can provide to secure AI is limited by the maturity of technological development, which should therefore be encouraged and monitored. In other words, in some areas existing standards cannot be adapted or new standards cannot be fully defined yet, as related technologies are still being developed and not yet quite mature enough to be standardised. In some cases, first standards can be drafted (e.g. ISO/IEC TR 24029-1:2021 on the robustness of deep neural networks) but will probably need to be regularly updated and adapted as research and development (R&D) progresses. For example, from the perspective of ML research, much of the work on adversarial examples, evasion attacks, measuring and certifying adversarial robustness, addressing specificities of data poisoning for ML models, etc. is still quite active R&D. Another challenge related to R&D on AI and standardisation is benchmarking: research results are often not comparable, resulting in a situation where it is not always clear what works under what conditions. Box 3: Example of technological gap: continuous learning Continuous learning is the ability of an AI component to evolve during its operational life through the use of in-operation data for retraining the AI component. This function is often perceived as the key ability of AI. Model poisoning is easy to do during continuous learning / in-operation learning. For example, during continuous learning, it is very challenging to check the quality of the data in real time. When it comes to high-risk AI components, the use of continuous learning would imply continuous validation of the data used for the training of the AI component (continuous data quality assessment), continuous monitoring of the AI component, continuous risk assessment, continuous validation and continuous certification if needed. While the issues with continuous learning have been described in ISO/IEC 22989, Information technology – Artificial intelligence – Artificial intelligence concepts and terminology, and the activities described above are conceptually feasible, their execution is still the object of R&D. • The traceability and lineage of both data and AI components are not fully addressed. The traceability of processes is addressed by several standards related to quality. In that regard, ISO 9001 is the cornerstone of quality management. However, the traceability of data and AI components throughout their life cycles remains an issue that cuts across most threats and remains largely unaddressed. Indeed, both data and AI components may have very complex life cycles, with data coming from many sources and being transformed and augmented, and, while AI components may reuse third parties’ components or even opensource components, all of those are obviously a source of increased risks. This aspect implies that technologies, techniques and procedures related to traceability need to be put in place to ensure the quality of AI systems, for instance that data being used do not contain biases (e.g. forgetting to include faces of people with specific traits), have not been deliberately poisoned (e.g. adding data to modify the outcome of the model) and have not been deliberately or unintentionally mislabelled (e.g. a picture of a dog labelled as a wolf). • The inherent features of ML are not fully reflected in existing standards. As introduced in Section 2.1, ML cannot, by design, be expected to be 100 % accurate. While this can also be true for (for example) ruled-based systems designed by humans, ML has a larger input space (making exhaustive testing difficult), black box properties and high sensitivity, meaning that small changes in inputs can lead to large changes in outputs. Therefore, it is even more

text: CYBERSECURITY OF AI AND STANDARDISATION important to understand, on the one hand, how the risk of failure can be mitigated and, on the other, if/when a failure is caused by a malicious actor. The most obvious aspects to be considered in existing/new standards can be summarised as follows. • AI/ML components may be associated with hardware or other software components in order to mitigate the risk of functional failure, therefore changing the cybersecurity risks associated with the resulting set-up . • Reliable metrics can help a potential user detect a failure. For example, with precision and recall metrics for AI systems relying on supervised classification, if users know the precision/recall thresholds of an AI system they should be able to detect anomalies when measuring values outside those thresholds, which may indicate a cybersecurity incident. While this would be a general check (more efficient for attacks on a massive scale than for specific attacks), the accurate definition of reliable metrics is a prerequisite to define more advanced measurements. • Testing procedures during the development process can lead to certain levels of accuracy/precision. It is to be noted that the subject of metrics for AI systems and of testing procedures is addressed by standardisation deliverables such as ISO/IEC DIS 5338-AI system life cycle processes (under development); ISO/IEC AWI TS 12791-Treatment of unwanted bias in classification and regression machine learning tasks (under development); ETSI TR 103 305-x, Critical security controls for effective cyber defence; and ETSI GR SAI-006, The role of hardware in security of AI . However, the coverage of the AI systems trustworthiness metrics that are needed is incomplete, which is one reason for the CEN-CENELEC initiative on the ‘AI trustworthiness characterisation’ project. 4.2 STANDARDISATION IN SUPPORT OF THE CYBERSECURITY OF AI – TRUSTWORTHINESS As explained in Section 2.2, cybersecurity can be understood as going beyond the mere protection of assets and be considered fundamental to the correct implementation of trustworthiness features of AI, and – conversely – the correct implementation of trustworthiness features is key to ensuring cybersecurity. Table 3 exemplifies this relation in the context of the draft AI Act. It shows the role of cybersecurity within a set of requirements outlined by the act that can be considered as referring to the trustworthiness of an AI ecosystem. In fact, some of them (e.g. quality management, risk management) contribute to building an AI ecosystem of trust indirectly, but have been included because they are considered equally important and they are requirements of the draft AI Act .

text: CYBERSECURITY OF AI AND STANDARDISATION Table 3 : Role of cybersecurity within a set of requirements outlined by the draft AI Act

text: Draft AI Act Description Relevance of cybersecurity Requirement

text: Data and data High-risk AI systems which make use of The requirements here address data quality, which is governance techniques involving the training of models with key to secure data feeds, processing and outputs. data shall be developed on the basis of training, Data quality can be reinforced by the use of tools that validation, and testing datasets that meet a set verify the source of data and the integrity of data (i.e. of quality criteria to prove that data have not been manipulated between source and sink), and by limiting access to data.

text: Record-keeping High-risk AI systems shall be designed and All of the major security management control developed with capabilities enabling the standards (e.g. ISO 27000 and ETSI TR 103 305) automatic recording of events (‘logs’) while the address the importance of event logging and having high-risk AI systems is operating. Those logging the staff to analyse the logs. These logs probably capabilities shall conform to recognised contain sensitive data, and appropriate standard standards or common specifications. cybersecurity measures, i.e. CIA, need to be deployed.

text: Transparency and High-risk AI systems shall be designed and As noted above, documentation in itself is not a provision of developed in such a way to ensure that their security requirement. However, as a security control, information to operation is sufficiently transparent to enable technical documentation is a key element in system users users to interpret the system’s output and use it transparency and in (high-level) explainability. appropriately. An appropriate type and degree of transparency shall be ensured, with a view to achieving compliance with the relevant obligations of the user and of the provider set out in Chapter 3 of [COM(2021) 206 final].

text: Human oversight High-risk AI systems shall be designed and This form of control is identified in ISO27001 and in developed in such a way, including with ETSI TS 103 305-1. ( ) Where human oversight is appropriate human–machine interface tools, that required, it should form an integral part of the design they can be effectively overseen by natural of the system, and performance and other constraints persons during the period in which the AI system should be added to the role of oversight. This may is in use. include the performance of mandatory actions and checks, and rules for escalation of an event assessment.

text: Risk management An assessment through internal checks for ISO/IEC 31000 is a framework for risk analysis and system ‘stand-alone’ high-risk AI systems would require the management of risk analysis systems. At a more a full, effective and properly documented ex ante detailed level, tools for vulnerability analysis (e.g. compliance with all requirements of the ETSI TS 102 165-1) may apply, as well as runtime regulation and compliance with robust quality analysis tools. Many development environments will and risk management systems and post-market perform both static and dynamic tests on software monitoring. that allow risks in the codebase to be identified. The suite of measures should operate in concert. A risk management system shall be established, implemented, documented and maintained in relation to high-risk AI systems.

text: Quality Providers of high-risk AI systems shall put a ISO 9001 is the overarching standard for the management quality management system in place that implementation of a quality management system in system ensures compliance with this Regulation. development environments, which should include security management aspects. The provider should establish a sound quality management system, ensure the accomplishment of the required conformity assessment procedure, draw up the relevant documentation and establish a robust postmarket monitoring system.

text: Conformity AI systems that create a high risk to the health This is necessary for the evaluation of all assessment and safety or fundamental rights of natural requirements, including cybersecurity. persons: in line with a risk-based approach, these high-risk AI systems are permitted on the European market subject to compliance with

text: CYBERSECURITY OF AI AND STANDARDISATION

text: certain mandatory requirements and an ex-ante conformity assessment. Robustness AI systems should be resilient against risks Cybersecurity is one of the key aspects – albeit not connected to the limitations of the system (e.g. the only one – of robustness. It concerns the errors, faults, inconsistencies, unexpected protection of the AI system against attacks as well as situations) as well as against malicious actions the capacity to recover from such attacks. that may compromise the security of the AI system and result in harmful or otherwise undesirable behaviour.

text: The general-purpose technical and organisational standards outlined in Section 3.1 cover these trustworthiness aspects to some extent. The SDOs are actively tackling the matter and are developing AI-specific standards in support of trustworthiness. In particular, ISO/IEC SC 42 is developing most of those aspects in multiple standards, and CEN-CENELEC JTC 21 is working towards adopting/adapting those standards (see annex A.3). This is normal and, to some extent, inevitable at first. Still, in a regulatory context, one could expect a unified comprehensive, coherent and synthetic approach to trustworthiness while avoiding the multiplication – and to some extent duplication – of efforts. Furthermore, it would be inefficient and even counterproductive to have multiple sets of standards for the same characteristics (robustness, explainability, etc.), some coming from the cybersecurity domain and some coming from the AI domain, with risks of discrepancy. The result is that a unified approach to trustworthiness characteristics is highly recommended. In particular, in order to bring coherency and comprehensiveness, it is necessary to clarify who is doing what, in order to avoid needless and confusing duplication, and a certain level of coordination and liaison is vital. Box 4: Example – Cybersecurity conformity assessment When it comes to AI systems, conformity assessment will be performed against all requirements outlined in the draft AI Act, trustworthiness, including its cybersecurity aspects, being among them. Existing standards on trustworthiness lack conformity assessment methods, sometimes including technical requirements and metrics. While there are a lot of activities in ISO/IEC SC 42 regarding trustworthiness characteristics, there are also a lot of gaps and very few developed requirements and metrics. Therefore, there is the risk that conformity assessment methods will be addressed by different standards depending on the characteristic being evaluated. Since some characteristics overlap each other, while others might be contradictory (e.g. there might be a trade-off between transparency and cybersecurity), a global and coherent approach is needed. Box 5: Example – Adversarial attacks For example, identified adversarial attack threats could be used in both the ML algorithm and the testing and validation process. In that specific case, the threats could have been identified by the AI system’s monitoring/oversight process and the testing process. It is likely that some technical requirements/adjustments coming from the cybersecurity threat assessment should find their place in the AI standards repository relating both to oversight and to testing. 4.3 CYBERSECURITY AND STANDARDISATION IN THE CONTEXT OF THE DRAFT AI ACT The draft AI Act refers explicitly to the cybersecurity of high-risk AI systems. High-risk AI systems are limited to AI systems intended to be used as safety components of products that are subject to third party ex ante conformity assessment, and stand-alone AI systems mainly with fundamental rights implications (e.g. for migration, asylum and border control management) and for the management and operation of critical infrastructure. More precisely, the draft AI Act builds upon a risk-based approach to identify whether an AI system is high risk on the basis of the system’s intended use and implications for health, safety and fundamental rights.

text: CYBERSECURITY OF AI AND STANDARDISATION It is important to note that this approach differs from the cybersecurity risk-based approach, which sees a cybersecurity risk as a function of its adverse impact and its likelihood of occurrence. Based on the draft AI Act, cybersecurity is a requirement that applies, and therefore is assessed, only once a system is identified as high risk. These high-risk systems are subject to a number of requirements, cybersecurity being one of them, as in

text: Article 15

text: , ‘Accuracy, robustness and cybersecurity’. The cybersecurity requirements outlined are legal and remain at a high level. Still, explicit reference is made to some technical aspects: High-risk AI systems shall be resilient as regards attempts by unauthorised third parties to alter their use or performance by exploiting the system vulnerabilities. […] The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent and control for attacks trying to manipulate the training dataset (‘data poisoning’), inputs designed to cause the model to make a mistake (‘adversarial examples’), or model flaws. The draft AI Act also lays down, in

text: Article 13

text: , ‘Transparency and provision of information to users’, that high-risk AI systems are to be accompanied by instructions for use, specifying, among other things, the ‘the level of accuracy, robustness and cybersecurity referred to in

text: Article 15

text: against which the high-risk AI system has been tested and validated and which can be expected, and any known and foreseeable circumstances that may have an impact on that expected level of accuracy, robustness and cybersecurity’. In addition, the draft AI Act refers to cybersecurity in its recitals. In particular, recital 51 mentions that, ‘To ensure a level of cybersecurity appropriate to the risks, suitable measures should therefore be taken by the providers of high-risk AI systems, also taking into account as appropriate the underlying ICT infrastructure’. Finally, the draft AI Act tackles cybersecurity through a number of other requirements, as exemplified in Table 2. The annexes (A.3 and A.4) contain an overview of activities of European standardisation organisations (ESOs) with respect to the requirements of the AI Act. Building on those, as well as on the previous sections, the following considerations have been outlined concerning the implementation of the draft AI Act from a cybersecurity perspective. • Given the applicability of AI in a wide range of domains, the identification of cybersecurity risks and the determination of appropriate security requirements should rely on a system-specific analysis and, where needed, on sectorial standards. Sectorial standards should build coherently and efficiently on horizontal ones. In turn, the assessment of compliance to security requirements can be based on AI-specific horizontal standards and on vertical/sector-specific standards as well. • It is important to develop the guidance necessary to back up existing technical and organisational standards that can support the cybersecurity of AI systems, while monitoring R&D advancements. Some aspects of cybersecurity can be addressed now by developing specific guidance, while others are still under R&D. For the purposes of the AI Act, the technological gaps described and ongoing R&D processes affect some aspects of the cybersecurity requirements outlined in

text: Article 15

text: (adversarial examples and data poisoning) and therefore might constitute standardisation gaps with respect to the draft AI Act, depending on how conformity assessment will be organised.

text: CYBERSECURITY OF AI AND STANDARDISATION • As explained in Section 3.1, SDOs are actively working on the standardisation of trustworthiness characteristics; however, it is unclear whether those standards will be adopted in time for the adoption of the draft AI Act. Therefore, it is recommended to monitor related developments closely. The draft AI Act also depicts a governance system upon which the conformity assessment of AI systems relies. Besides the specific recommendations on conformity assessment outlined above, the following are noted. • Ensure that the actors performing conformity assessment on AI systems have standardised tools and competences, including on cybersecurity. In certain cases, conformity assessment may be performed by notified bodies. AI trustworthiness will therefore rely partly on the competences of those bodies. If those bodies do not have the proper competences, they could make bad assessments and even bias the market. To date there are no standards that adequately cover cybersecurity and describing the competences of organisations for auditing, certification and testing of AI systems (and AI management systems) and their evaluators. This is crucial, as it is most likely that some AI algorithms will attack AI systems while other AI algorithms will protect them. The new AI threats (threats using AI) will probably be more and more efficient at exploiting existing vulnerabilities, while AI algorithms (cybersecurity using AI) could, for example, monitor the behaviour of an AI system to protect it. To sum up, there are standardisation gaps on competences for validation, testing, auditing, certification’ of AI systems and on ‘competences for auditing and certification of AI management systems (Although a project on this last point is being prepared by ISO/IEC SC 42, it is unclear to what extent it will be sufficient.) • Ensure regulatory coherence between the draft AI Act and legislation on cybersecurity. In particular, Article 42 of the draft AI Act sets out a presumption of conformity with cybersecurity requirements for high-risk AI systems that have been certified or for which a statement of conformity has been issued under a cybersecurity scheme pursuant to

text: Regulation (EU) 2019/881

text: (the Cybersecurity Act) . While no official request for a EU cybersecurity certification scheme for AI has been issued yet, it is important that, if developed, such a scheme would take due consideration of the draft AI Act – and vice versa. For example, the Cybersecurity Act sets out three levels of assurance (basic, substantial, high), which are commensurate with the level of the risk associated with the intended use of an ICT product/service/ process. These levels provide the rigour and depth of the evaluation of the ICT product/service/process and refer to technical specifications, standards and procedures, including those to mitigate or prevent incidents. It remains to be defined whether and how these assurance levels can apply in the context of the draft AI Act. • Another regulatory development that might affect the draft AI Act is the proposal COM(2022) 454 for a regulation on horizontal cybersecurity requirements for products with digital elements (the Cyber Resilience Act) . The proposal was presented in September 2022.

text: CYBERSECURITY OF AI AND STANDARDISATION

text: 5. CONCLUSIONS

text: This section sums up the report and recommends actions to ensure standardisation support to the cybersecurity of AI, and to the implementation of the draft AI Act. 5.1 WRAP-UP The study suggests that general-purpose standards for information security and quality management (in particular ISO/IEC 27001, ISO/IEC 27002 and ISO/IEC 9001) can partially mitigate the cybersecurity risks related to the confidentiality, integrity and availability of AI systems. This conclusion relies on the assumption that AI is in its essence software, and therefore what is applicable to software can be applied to AI, if adequate guidance is provided. This approach can suffice at a general level but needs to be complemented by a systemspecific analysis (e.g. relying on ISO/IEC 15408-1:2009), as the identification of standardised methods supporting the CIA security objectives is often domain specific. It is a matter of debate to what extent the assessment of compliance with the resulting security requirements can be based on AI-specific horizontal standards and to what extent it can be based on vertical/sectorspecific standards. Still, some standardisation gaps have been identified: • the traceability of processes is addressed by several standards, but the traceability of the data and AI components throughout their life cycles remains an issue that cuts across most threats and remains largely unaddressed in practice, despite being covered well in various standards or drafts (e.g. ISO/IEC DIS 42001 on AI management systems and the ISO/IEC CD 5259 series on data quality for analytics and ML ); • the inherent features of ML are not fully reflected in existing standards, especially in terms of metrics and testing procedures; • in some areas, existing standards cannot be adapted or new standards cannot be fully defined yet, as related technologies are still being developed and not yet quite mature enough to be standardised. Going beyond the mere CIA paradigm and considering the broader trustworthiness perspective, the main takeaway is that, since cybersecurity cuts across a number of trustworthiness requirements (e.g. data governance, transparency), it is important that standardisation activities around these requirements treat cybersecurity in a coherent manner. Concerning the implementation of the draft AI Act, besides the considerations above, the following gaps have been identified: • to date there are no standards that adequately cover cybersecurity and describe the competences of organisations for auditing, certification and testing of AI systems (and AI management systems) and their evaluators; • the abovementioned gap on areas that are the subject of R&D is relevant to the implementation of the draft AI Act, in particular with respect to data poisoning and adversarial examples.

text: CYBERSECURITY OF AI AND STANDARDISATION 5.2 RECOMMENDATIONS 5.2.1 Recommendations to all organisations The ESOs have made a commitment to standardisation in support of cybersecure AI, as is evidenced by ETSI’s ISG SAI and by CEN’s JTC 21. These actions are all positive and are to be encouraged and reinforced. While it is recognised that the ESOs have different operational models and different membership profiles, it is also recognised that the ESOs operate cooperatively in many fields, and this is, again, to be encouraged. Competitive effort to develop standards is to some extent inevitable and, while that is recognised, the ESOs are strongly discouraged from negative competition. One area where harmonisation is seen as essential is in the adoption of a common AI-related terminology and set of concepts not only across SDOs but also with other stakeholders. The present report does not suggest which SDO/ESO should initiate this activity but it is strongly suggested that, without a common set of cross-domain terminology and concepts, the first risk to cybersecurity would be not understanding each other . Recommendation 1: Use a standardised and harmonised AI terminology for cybersecurity, including trustworthiness characteristics and a taxonomy of different types of attacks specific to AI systems. 5.2.2 Recommendations to standards-developing organisations The following recommendations are to standardisation organisations. Recommendation 2: Develop specific/technical guidance on how existing standards related to the cybersecurity of software should be applied to AI. These should also include defences at different levels (before the AI system itself, e.g. infrastructure), for which the application of generic standards might be straightforward in many cases. At the same time, it is recommended to monitor and encourage areas where standardisation is limited by technological development, e.g. testing and validation for systems relying on continuous learning and mitigation of some AIspecific attacks. Recommendation 3: The inherent features of ML should be reflected in standards. The most obvious aspects to be considered relate to risk mitigation by associating hardware/software components with AI; reliable metrics; and testing procedures. The traceability and lineage of both data and AI components should also be reflected. Recommendation 4: Ensure that liaisons are established between cybersecurity technical committees and AI technical committees so that AI standards on trustworthiness characteristics (oversight, robustness, accuracy, explainability, transparency, etc.) and data quality include potential cybersecurity concerns. 5.2.3 Recommendations in preparation for the implementation of the draft AI Act The following recommendations are suggested to prepare for the implementation of the draft AI Act, and should be understood as complementary to the recommendations above. Recommendation 5: Given the applicability of AI in a wide range of domains, the identification of cybersecurity risks and the determination of appropriate security requirements should rely on

text: CYBERSECURITY OF AI AND STANDARDISATION a system-specific analysis and, where needed, sector-specific standards. Sectorial standards should build coherently and efficiently on horizontal ones. Recommendation 6: Encourage R&D in areas where standardisation is limited by technological development, on one hand by providing funding for the advancements in specific technologies (e.g. related to countermeasures against adversarial attacks) and on the other by raising awareness of the importance of integrating standardisation aspects in research activities. In addition, it is suggested to promote benchmarking by means of a systematic approach to guide R&D efforts, which are still characterised by a proliferation of specialised approaches that work under specialised conditions. Recommendation 7: Support the development of standards for the tools and competences of the actors performing conformity assessment. Recommendation 8: Ensure coherence between the draft AI Act and other legislative initiatives on cybersecurity, notably

text: Regulation (EU) 2019/881

text: (the Cybersecurity Act) and the proposal COM(2022) 454 for a regulation on horizontal cybersecurity requirements for products with digital elements (the Cyber Resilience Act). 5.3 FINAL OBSERVATIONS While the report gives an overview of the state of play of standardisation in support of AI, it is likely that additional standardisation gaps and needs may become apparent only as the AI technologies advance and with further study of how standardisation can support cybersecurity. Concerning the implementation of the AI Act, the importance of some gaps may vary depending on how the conformity assessment will be conceived. Last but not least, changes in the legislative landscape, with particular reference to the proposal for a Cyber Resilience Act, are expected to affect standardisation needs.

text: CYBERSECURITY OF AI AND STANDARDISATION

text: A ANNEX:

text: A.1 SELECTION OF ISO 27000 SERIES STANDARDS RELEVANT TO THE CYBERSECURITY OF AI

text: Name Document Reference

text: Information technology – Security techniques – Information security incident

text: ISO/IEC 27035-1:2016

text: management – Part 1: Principles of incident management

text: ISO/IEC 27035-2:2017

text: management – Part 2: Guidelines to plan and prepare for incident response

text: Information technology – Information security incident management – Part 3:

text: ISO/IEC 27035-3:2020

text: Guidelines for ICT incident response operations

text: Information technology – Security techniques – Guidelines for cybersecurity ISO/IEC 27032:2012

text: Information technology – Security techniques – Guidelines for information and

text: ISO/IEC 27031:2011

text: communication technology readiness for business continuity

text: Information technology – Security techniques – Mapping the revised editions of

text: ISO/IEC TR 27023:2015

text: ISO/IEC 27001 and ISO/IEC 27002

text: Information technology – Guidance on information security management system

text: ISO/IEC TS 27022:2021

text: processes

text: Information technology – Security techniques – Competence requirements for information security management systems professionals – Amendment 1:

text: ISO/IEC 27021:2017/AMD 1:2021

text: Addition of ISO/IEC 27001:2013 clauses or subclauses to competence requirements

text: Information technology – Security techniques – Competence requirements for

text: ISO/IEC 27021:2017

text: information security management systems professionals

text: Information technology – Security techniques – Code of practice for information

text: ISO/IEC 27017:2015

text: security controls based on ISO/IEC 27002 for cloud services

text: Information technology – Security techniques – Information security management

text: ISO/IEC TR 27016:2014

text: – Organizational economics

text: Information security, cybersecurity and privacy protection – Governance of

text: ISO/IEC 27014:2020

text: information security

text: Information security, cybersecurity and privacy protection – Guidance on the

text: ISO/IEC 27013:2021

text: integrated implementation of ISO/IEC 27001 and ISO/IEC 20000-1

text: Information technology – Security techniques – Code of practice for Information

text: security controls based on ISO/IEC 27002 for telecommunications organizations – ISO/IEC 27011:2016/Cor 1:2018

text: Technical Corrigendum 1

text: ISO/IEC 27010:2015

text: for inter-sector and inter-organizational communications

text: Information technology – Security techniques – Guidelines for the assessment of

text: ISO/IEC TS 27008:2019

text: information security controls

text: CYBERSECURITY OF AI AND STANDARDISATION

text: Information security, cybersecurity and privacy protection – Guidelines for

text: ISO/IEC 27007:2020

text: information security management systems auditing

text: Information technology – Security techniques – Requirements for bodies

text: providing audit and certification of information security management systems – ISO/IEC 27006:2015/AMD 1:2020

text: Amendment 1

text: ISO/IEC 27006:2015

text: providing audit and certification of information security management systems

text: Requirements for bodies providing audit and certification of information security

text: ISO/IEC TS 27006-2:2021

text: management systems – Part 2: Privacy information management systems

text: Information technology – Security techniques – Information security risk

text: ISO/IEC 27005:2018

text: management

text: ISO/IEC 27004:2016

text: – Monitoring, measurement, analysis and evaluation

text: ISO/IEC 27003:2017

text: systems – Guidance

text: Information security, cybersecurity and privacy protection – Information security

text: ISO/IEC 27002:2022

text: controls

text: ISO/IEC 27001:2013/Cor 2:2015

text: systems – Requirements – Technical Corrigendum 2

text: ISO/IEC 27001:2013/Cor 1:2014

text: systems – Requirements – Technical Corrigendum 1

text: ISO/IEC 27001:2013

text: systems – Requirements

text: CYBERSECURITY OF AI AND STANDARDISATION A.2 RELEVANT ISO/IEC STANDARDS PUBLISHED OR PLANNED / UNDER DEVELOPMENT

text: Name Expected Document Reference publication date (at the time of writing)

text: Artificial Intelligence (AI) — Assessment of the robustness of neural ISO/IEC TR 24029- Published networks — Part 1: Overview 1:2021

text: Assessment of machine learning classification performance ISO/IEC TS 4213 Published

text: Big data – Overview and vocabulary ISO/IEC 20546:2019 Published

text: Information technology — Artificial intelligence — Overview of ethical ISO/IEC Published and societal concerns TR 24368:2022

text: Information technology — Artificial intelligence — Overview of ISO/IEC Published trustworthiness in artificial intelligence TR 24028:2020

text: Information technology — Artificial intelligence — Process management

text: ISO/IEC 24668:2022 Published

text: framework for big data analytics

text: Information technology — Artificial intelligence (AI) — Bias in AI systems ISO/IEC Published and AI aided decision making TR 24027:2021

text: Information technology — Artificial intelligence (AI) — Overview of ISO/IEC Published computational approaches for AI systems TR 24372:2021

text: Information technology — Big data reference architecture — Part 1: ISO/IEC TR 20547- Published Framework and application process 1:2020

text: Information technology — Big data reference architecture — Part 2: Use ISO/IEC TR 20547- Published cases and derived requirements 2:2018

text: Information technology — Big data reference architecture — Part 3:

text: ISO/IEC 20547-3:2020 Published

text: Reference architecture

text: Information technology — Big data reference architecture — Part 4:

text: ISO/IEC 20547-4:2020 Published

text: Security and privacy

text: Information technology — Big data reference architecture — Part 5: ISO/IEC TR 20547- Published Standards roadmap 5:2018

text: Information technology — Governance of IT — Governance implications

text: ISO/IEC 38507:2022 Published

text: of the use of artificial intelligence by organizations

text: Safety of machinery – Relationship with ISO 12100 – Part 5: Implications

text: ISO/TR 22100-5:2021 Published

text: of embedded artificial intelligence machine learning

text: Published (to be ISO/IEC revised, new version

text: Artificial Intelligence (AI) Use cases

text: TR 24030:2021 expected in May 2023)

text: Artificial intelligence — Functional safety and AI systems ISO/IEC CD TR 5469 Apr-23

text: Software engineering — Systems and software Quality Requirements

text: ISO/IEC DIS 25059 May-23

text: and Evaluation (SQuaRE) — Quality model for AI systems

text: Artificial intelligence — Data quality for analytics and machine learning

text: ISO/IEC CD 5259-1 Jul-23

text: (ML) — Part 1: Overview, terminology, and examples

text: ISO/IEC CD 5259-3 Jul-23

text: (ML) — Part 3: Data quality management requirements and guidelines

text: CYBERSECURITY OF AI AND STANDARDISATION

text: ISO/IEC CD 5259-4 Jul-23

text: (ML) — Part 4: Data quality process framework

text: Information technology — Artificial intelligence — Controllability of

text: ISO/IEC AWI TS 8200 Jul-23

text: automated artificial intelligence systems

text: Information technology — Artificial intelligence — AI system life cycle

text: ISO/IEC DIS 5338 Aug-23

text: processes

text: Information technology — Artificial intelligence — Guidance for AI

text: ISO/IEC DIS 5339 Aug-23

text: applications

text: Information technology — Artificial intelligence — Overview of machine ISO/IEC AWI Nov-23 learning computing devices TR 17903

text: ISO/IEC CD 5259-2 Jan-24

text: (ML) — Part 2: Data quality measures

text: Information technology — Artificial intelligence — Objectives and

text: ISO/IEC AWI TS 6254 Feb-24

text: approaches for explainability of ML models and AI systems

text: Information technology — Artificial intelligence — Treatment of ISO/IEC AWI Feb-24 unwanted bias in classification and regression machine learning tasks TS 12791

text: Software and systems engineering — Software testing — Part 11: Testing ISO/IEC AWI Feb-24 of AI system TS 29119-11[1]

text: ISO/IEC AWI 5259-5 Feb-25

text: (ML) — Part 5: Data quality governance

text: Information technology — Artificial intelligence — Transparency

text: ISO/IEC AWI 12792 Feb-25

text: taxonomy of AI systems

text: Quality evaluation guidelines for AI systems ISO/IEC AWI TS 5471 Under consideration

text: Information technology — Artificial intelligence — Reference architecture

text: ISO/IEC DIS 5392 Under development

text: of knowledge engineering

text: CYBERSECURITY OF AI AND STANDARDISATION A.3 CEN-CENELEC JOINT TECHNICAL COMMITTEE 21 AND DRAFT AI ACT REQUIREMENTS

text: of m ion te m s te nce y nt s ovis s e TR, Action: adopt/adapt y rna s pr rs nt m Name TS, (ISO/IEC) – develop e s d s nt n ht e EN (ESOs) e use s gove ng a ig gem m a o s pi t rs ty e s na a dat e ncy uri a y e re on y c it nage k ti dat a a ove c nes e m nd n s a m a ty get m ord ura li k ta c nspa or c ber is a e um c obust y onform Tar R D R Tra inf H A R C Qua C

text: ISO/IEC 22989:2022

text: IS Adopt July 2022 x x x x x x x x x x

text: Artificial intelligence concepts and terminology

text: ISO/IEC 23053:2022

text: Framework for artificial intelligence (AI) systems using IS Adopt July 2022 x x x x x x x x x x

text: machine learning (ML)

text: ISO/IEC CD 5259-1

text: Data quality for analytics and machine learning (ML) – IS Adopt December 2023 x x x x x x x x x x

text: Part 1: Overview, terminology, and examples

text: ISO/IEC 9001:2015

text: IS 2015 x x

text: Quality management systems – Requirements

text: ISO/IEC 42001

text: IS Adopt December 2023 x x

text: Artificial intelligence – Management system

text: ISO/IEC 27001:2022

text: Information security management systems – IS 2022 x x

text: Requirements

text: ISO/IEC 23894

text: IS Adopt December 2023 x

text: Guidance on risk management

text: CYBERSECURITY OF AI AND STANDARDISATION

text: CEN-CENELEC

text: EN Develop Q1 2025 x x

text: Risk catalogue and risk management

text: ISO/IEC TR 24029-1

text: Assessment of the robustness of neural networks – TR Adopt December 2022 x

text: Part 1: Overview

text: ISO/IEC 24029-2

text: Assessment of the robustness of neural networks – IS Adopt December 2023 x

text: Part 2: Methodology for the use of formal methods

text: CEN-CENELEC

text: EN Develop Q1 2025 ? x x x x x x

text: AI trustworthiness characterisation

text: ISO/IEC CD 5259-2

text: Data quality for analytics and machine learning (ML) – IS Adopt Q2 2024 x

text: Part 2: Data quality measures

text: ISO/IEC CD 5259-3 Data quality for analytics and machine learning (ML) –

text: IS Adopt Q3 2023 x

text: Part 3: Data quality management requirements and guidelines

text: ISO/IEC CD 5259-4

text: Data quality for analytics and machine learning (ML) – IS Adopt Q4 2024 x

text: Part 4: Data quality process framework

text: CYBERSECURITY OF AI AND STANDARDISATION A.4 ETSI ACTIVITIES AND DRAFT AI ACT REQUIREMENTS

text: of m ion te m s te nce y nt s ovis s e TR, y rna pr rs nt m a s e s Name TS, Status ( ) d s nt n e EN e use ht gove a s m a ng o ig gem s t ty e pi rs s na a dat e ncy uri a y e on y c it nage k re ti dat a a ove c nes e m nd n s m a ty get m a ord ura li k ta c nspa or c ber is a e um c obust y onform Tar R D R Tra inf H A R C Qua C

text: DTR/MTS-103910 - MTS AI Testing Test Methodology and

text: TR Early draft July 2024 x x

text: Test Specification for AI-enabled Systems

text: TR WI Adopted TBC x x

text: MTS AI Testing AI-enabled Testing in Standardisation

text: TR 103 911

text: TR Under development (late) — x

text: MTS AI testing AI-enabled testing in standardisation

text: EN 303 645

text: Cyber security for consumer internet of things: Baseline EN Published — x

text: requirements

text: TR 103 304

text: Personally identifiable information (PII) protection in TR Published — x

text: mobile and cloud services

text: TR 103 305

text: TR Published — x x x x x x x

text: Critical security controls for effective cyber defence

text: TR 103 370

text: Practical introductory guide to technical standards for TR Published — x

text: privacy

text: CYBERSECURITY OF AI AND STANDARDISATION

text: TR 103 404 Network technologies (NTECH); Autonomic network

text: engineering for the self-managing future internet (AFI); TR Published — x x

text: Autonomicity and self-management in the backhaul and core network parts of the 3GPP architecture TR 103 473 Evolution of management towards autonomic future

text: TR Published — x x

text: internet (AFI); Autonomicity and self-management in the Broadband Forum (BBF) architectures TR 103 626 Autonomic network engineering for the self-managing future internet (AFI); An instantiation and implementation

text: TR Published — x x

text: of the generic autonomic network architecture (GANA) model onto heterogeneous wireless access technologies using cognitive algorithms

text: TR 103 627

text: Core network and interoperability testing (INT/WG AFI) TR Published — x x

text: autonomicity and self-management in IMS architecture

text: TR 103 629 Evolution of management towards autonomic future

text: TR Early draft — x x

text: internet (AFI); Confidence in autonomic functions; Guidelines for design and testability TR 103 747 Core network and interoperability testing (INT/WG AFI); Federated GANA knowledge planes (KPs) for multi-

text: TR Published — x x

text: domain autonomic management & control (AMC) of slices in the NGMN® 5G end-to-end architecture framework TR 103 748 Core network and interoperability testing (INT); Artificial

text: TR Published — x x x x

text: intelligence (AI) in test systems and testing of AI models; Use and benefits of AI technologies in testing

text: TR 103 749

text: INT artificial intelligence (AI) in test systems and testing TR Start of work May 2023 x x x x

text: AI models; Testing of AI with definition of quality metrics

text: TR 103 821 Autonomic network engineering for the self-managing

text: TR Start of work — x

text: future internet (AFI); Artificial intelligence (AI) in test systems and testing AI models TR 103 857 Autonomic management and control (AMC) intelligence for self-managed fixed & mobile integrated networks (AFI); Generic framework for E2E federated GANA

text: TR Stable draft March 2023 x x x x

text: knowledge planes for AI-powered closed-loop selfadaptive security management & control, across multiple 5G network slices, segments, services and administrative domains

text: CYBERSECURITY OF AI AND STANDARDISATION

text: TS 102 165-1

text: Methods and protocols; Part 1: Method and pro forma for TS Published — x x

text: threat, vulnerability, risk analysis (TVRA)

text: TS 103 195-2 Autonomic network engineering for the self-managing future internet (AFI); Generic autonomic network

text: TS Published — x x x x x x

text: architecture; Part 2: An architectural reference model for autonomic networking, cognitive networking and selfmanagement

text: TS 103 485

text: TS Published — x

text: Mechanisms for privacy assurance and verification

text: TS 103 701

text: Cyber security for consumer internet of things: TS Published — x

text: Conformance assessment of baseline requirements

text: -C -EN -011 3 -2 -03 TP ABOUT ENISA The European Union Agency for Cybersecurity, ENISA, is the Union’s agency dedicated to achieving a high common level of cybersecurity across Europe. Established in 2004 and strengthened by the EU Cybersecurity Act, the European Union Agency for Cybersecurity contributes to EU cyber policy, enhances the trustworthiness of ICT products, services and processes with cybersecurity certification schemes, cooperates with Member States and EU bodies, and helps Europe prepare for the cyber challenges of tomorrow. Through knowledge sharing, capacity building and awareness raising, the Agency works together with its key stakeholders to strengthen trust in the connected economy, to boost resilience of the Union’s infrastructure, and, ultimately, to keep Europe’s society and citizens digitally secure. More information about ENISA and its work can be found here: www.enisa.europa.eu. ISBN 978-92-9204-613-2 doi: 10.2824/308470