text: www.enisa.europa.eu European Union Agency For Network and Information Security

text: Towards a framework for policy development in cybersecurity v.1.0 | December 2018

text: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2017:228:FIN https://ec.europa.eu/digital-single-market/en/news/communication-artificial-intelligence-europe https://ec.europa.eu/digital-single-market/en/artificial-intelligence

text: ENISA programming document 2018-2020, https://www.enisa.europa.eu/publications/corporate-documents/enisaprogramming-document-2018-2020. Artificial Intelligence: A European Perspective, https://ec.europa.eu/jrc/en/publication/eur-scientific-and-technicalresearch-reports/artificial-intelligence-european-perspective https://www.enisa.europa.eu/topics/iot-and-smart-infrastructures

text: th During the 40 International Conference of Data Protection and Privacy Commissioners, a declaration on Ethics and Data Protection in Artificial Intelligence was signed . Among others, the declaration endorses a set of guiding principles in the development of artificial intelligence, and as an extension to autonomous agents, while establishing a working group on Ethics and Data Protection in Artificial Intelligence. Similar initiatives should be further encouraged and promoted, both at EU and International level, to ensure that future developments in the area take into account promotion and protection of human rights.

identifier: Towards a framework for policy development in cybersecurity - Security and privacy considerations in autonomous agents

nummer: considerations-in-autonomous-agents

publisher: Europeiska unionens cybersäkerhetsbyrå

sprak: en

title: Towards a framework for policy development in cybersecurity - Security and privacy considerations in autonomous agents

serie: rapporter

text: Security and privacy considerations in autonomous agents

type: rubrik

text: V.1.0 DECEMBER 2018 About ENISA

text: The European Union Agency for Network and Information Security (ENISA) is a centre of network and information security expertise for the EU, its member states, the private sector and EU citizens. ENISA works with these groups to develop advice and recommendations on good practice in information security. It assists member states in implementing relevant EU legislation and works to improve the resilience of Europe’s critical information infrastructure and networks. ENISA seeks to enhance existing expertise in member states by supporting the development of cross-border communities committed to improving network and information security throughout the EU. More information about ENISA and its work can be found at www.enisa.europa.eu.

type: stycke

text: Contributors

text: Kalloniatis Christos (University of the Aegean), Costas Lambrinoudakis (University of Piraeus), Konstantinos Kotis (University of Piraeus), Prokopios Drogkaris (ENISA)

text: Editors

text: Prokopios Drogkaris (ENISA), Athena Bourka (ENISA)

text: Contact

text: For queries in relation to this paper, please use isdp@enisa.europa.eu For media enquires about this paper, please use press@enisa.europa.eu.

text: Acknowledgements

text: We would like to thank Spyros Kokolakis (University of the Aegean), Stefanos Gritzalis (University of the Aegean) and Apostolos Malatras (ENISA) for their contributions during the preparation of this document.

text: Legal notice Notice must be taken that this publication represents the views and interpretations of ENISA, unless stated otherwise. This publication should not be construed to be a legal action of ENISA or the ENISA bodies unless adopted pursuant to the

text: Regulation (EU) No 526/2013

text: . This publication does not necessarily represent state-of the-art and ENISA may update it from time to time.

text: Third-party sources are quoted as appropriate. ENISA is not responsible for the content of the external sources including external websites referenced in this publication.

text: This publication is intended for information purposes only. It must be accessible free of charge. Neither ENISA nor any person acting on its behalf is responsible for the use that might be made of the information contained in this publication.

text: Copyright Notice © European Union Agency for Network and Information Security (ENISA), 2018 Reproduction is authorised provided the source is acknowledged.

text: ISBN 978-92-9204-282-0, DOI 10.2824/453043

text: Table of Contents

text: Executive Summary 5

text: 1. Introduction 6 Background 6 Initiatives at EU level 6 Scope – Objectives 7 Structure of the study 7

text: 2. Artificial Intelligence and Autonomous Agents 8 Autonomous agents as a business 8 Knowledge representation 9 The notion of autonomy 9 Learning 9 Neural Networks/Deep Learning 10 Self - management 10 Decision-making systems 11 Recognition Technology 11 Planning 12 Conflict resolution 12 Reasoning 12 Big Data 13

text: 3. Security and privacy considerations 14 Security considerations 14 Detection of rogue or unauthorized autonomous systems 14 Hijacking and misuse 14 Interference 14 Transparency and accountability 14 Adherence to security principles 14 Privacy considerations 15 Pervasiveness and the minimization principle 15 Data retention and protection 15 Data aggregation and repurposing 15 The opacity of processing 16

text: 4. Shaping a framework for policy development in cybersecurity 17

text: 5. Conclusions and Recommendations 18 Adoption of security and privacy by design principles 18 Development of baseline security requirements 18 Coordination of actions on highlighting and addressing ethical considerations 18

text: 6. Bibliography/References 20

text: Executive Summary

text: Over the last years, Artificial Intelligence (AI) has rapidly moved beyond the realms of research and academia to enter the commercial mainstream, with innovative autonomous agents utilizing AI and transforming how we access and leverage information. Autonomous agents are characterized by diversity, with applications varying from digital assistants residing in our smartphones to autonomous robots supporting the supply chain. Depending on the level of autonomy and the context of operations, security and privacy considerations may vary. One of the key aspects in autonomous systems is the data collected, mainly for supporting the demanding functionality in a qualitative and timely manner. However, due to the abundance of data processed, which may also include personal data, in addition to the relying on third party providers, introduces a number of security and privacy considerations. The current study highlights a number of relevant considerations, such as unauthorized autonomous systems, hijacking and misuse transparency and accountability, pervasiveness, retention and opacity of processing. During this analysis, a set of recommendations for shaping future EU policy initiative was deducted which includes:

text:  European Commission, relevant EU Bodies and relevant public and private sector stakeholders should further promote and support the adoption of security and privacy by design principles as a pre-requisite during the inception, design and implementation of autonomous agents and systems.  European Commission, relevant EU Bodies, public and private sector stakeholders should foster a collaborative approach on identification and exchange of best practices. Gradually such initiatives should put forward sets of baseline security requirements which can then be transposed to widely accepted technical specifications and standards.  European Commission, relevant EU Bodies, public and private sector stakeholders should further endorse and support existing initiatives on promotion and protection of human rights through establishing appropriate ethical parameters.

text: 1. Introduction Background

text: The rapid adoption of digital technologies in recent years has enabled many opportunities and allowed for real-time adjustments and decisions to be made, based on the amount of relevant available data and processing capabilities. Such an environment presents unlimited opportunities for innovation and interaction, but bring also along a number of challenges which should be addressed by future policy frameworks at EU level. Artificial intelligence (AI) and increasingly complex algorithms currently offer great potential and possibilities for a diverse range of application areas which influence our everyday lives more than ever before. In computer science, Artificial Intelligence (AI) is a way of making software think intelligently, in a similar manner to the way humans think (Garbhe, 2017). These tasks can be cognitively challenging and hence an observer might mistake the behaviour of the machine for the behaviour of an intelligent human. The field of AI however, attempts not only to understand an intelligent entity, and the way it perceives, understands, predicts, and manipulates a world far larger and more complicated than itself, but also attempts to build one (Russell, Norvig, & Davis, 2010). AI attempts to understand the way the human brain thinks and how humans learn, decide and work, while trying to solve a particular problem, and then make use of this understanding as a basis for developing intelligent software and systems.

text: Initiatives at EU level

text: The European Commission identified, in its mid-term review of the Digital Single Market strategy , the importance of building on Europe's scientific and industrial strengths, as well as on its innovative start-ups, to be in a leading position in the development of AI technologies, platforms, and applications. In mid-2018, the European Commission, through COM(2018) 237 , highlighted the need for the EU to become a leader in the AI revolution while acknowledging that “Like the steam engine or electricity in the past, AI is transforming our world, our society and our industry” . The Communication sets out a European initiative on AI, which aims to:

text:  Boost the EU's technological and industrial capacity and AI uptake across the economy, both by the private and public sectors, including investments in research and innovation and better access to data.  Prepare for socio-economic changes brought about by AI by encouraging the modernisation of education and training systems, nurturing talent, anticipating changes in the labour market, supporting labour market transitions and adaptation of social protection systems.  Ensure an appropriate ethical and legal framework, moving a step forward on ensuring legal clarity in AI-based applications. Lastly, the European Commission recently appointed the members to the High-Level Expert Group on Artificial Intelligence (AI HLEG) which seeks to support the implementation of the European strategy on Artificial Intelligence .

text: Scope – Objectives

text: Within the scope of the ENISA 2018 Annual Programming Document and towards supporting a framework for policy development in the area of cybersecurity, the Agency undertook the current study in an attempt to highlight the main security and privacy considerations that arise from the use of autonomous agents in today’s modern society. The objective of the study is on the one hand to provide an overview of representative application domains and on the other hand to contribute and complement relevant initiatives at EU level while also providing relevant insights, both for security and privacy, for future EU policy shaping initiatives.

text: As a first step towards sketching a point of reference for policy-makers tackling the issue of cybersecurity and establighing a relevant framework for policy development, emerging technologies and new application areas must be considered, embracing the new opportunities they bring along but in a way that they respond to European needs. As also acknowledged by the EC Joint Research Centre (JRC) report titled “Artificial Intelligence: A European Perspective” , “the EU Member States and the European Commission are developing coordinated national and European strategies, recognising that we can only succeed together”.

text: Structure of the study

text: Section 2 of the study outlines AI technology, which has been used in the context of autonomous agents, in several application domains through a representative, rather than an exhaustive, list. Section 3 discusses the main security and privacy considerations by aggregating and building up on already published ENISA studies in the area of IoT and Smart Infrastructures while in Section 4 a set of recommendations for relevant stakeholders and policy makers is discussed.

text: 2. Artificial Intelligence and Autonomous Agents

text: Modern Artificial Intelligence (AI) applications are pervasive and numerous to list. Frequently, when an application/technique of AI reaches mainstream use, it is no longer considered artificial intelligence. Some of the most widely used AI applications are included in domains such as:

text:  healthcare (e.g., knowledge-based diagnosis, medical personal assistants for elderly, decision-support systems for cancer treatment, robotic surgery),  medical robots for surgical purposes in cases that do not necessarily require human medical intervention,  automotive (e.g., driverless cars, distributed multi-agent coordination of autonomous vehicles),  finance/economics (e.g., fraud and financial crimes detection, AI based trading platforms),  gaming (e.g., dynamic purposeful behaviour in non-player characters (NPCs), pathfinding, deep learning prediction),  military (e.g., autonomous drones),  security (e.g., speech/image/object/face-recognition),  advertising (e.g., predicting the behaviour of customers),  art (e.g., e-auctions, automated story-telling creation, digital museum guide, automated music synthesis),  social life (e.g., digital personal assistants, smart homes).

text: An AI system is mainly composed of a software agent and the environment in which it operates (sense and act). Agents act in their environment, which may contain other agents too. The first two decades of the st 21 century are characterized by a number of examples of autonomous technology and AI. Most existing AI systems and robots are automatic but not autonomous i.e. they do not develop and maintain their internal structure and functioning through mechanisms like self-organization, evolution, adaptation, and learning. Driverless cars and drones, robots in deep sea and space exploration, weapon systems, bots in financial trade and deep learning in medical diagnosis, are among the most prominent, but certainly not the only examples of autonomous agents. AI, especially in the form of machine learning, and the increasing availability of Big Data, are important drivers of these developments. The coexistence and combination of these digital technologies is rapidly making them more powerful, and they are applied in an increasing number of new products and services, in public and private sectors, and can have both military and civilian use. The AI used in these systems can redefine work or improve work conditions for humans and reduce the need for human contribution, input and interference during operation. It can help to assist or replace humans with smart technology in time-consuming, expensive, difficult, dirty, dull or dangerous work, and even beyond.

text: The typology of such systems includes combined physical/logical ones such as a driverless car/boat/aircraft; and logical bots such as a software agent e.g. Microsoft Cortana, Amazon Alexa or Apple Siri.

text: Autonomous agents as a business

text: For the time being autonomous agents are addressed in the context of a specific business case e.g. delivery of packets by a drone or a public service e.g. surveillance of an area for fire hazard by a drone. Scoping the operation and the permitted actions by an agent is of paramount importance. Often contradicting requirements need to be balanced out to produce a meaningful outcome. The risk of breach of privacy is significant therefore when hovering in inhabited areas, specific limitations to collecting data should be put

text: in place; by the same token such limitations should be lifted in case of surveilling a critical infrastructure, and be imposed within reason for e.g. individuals who happen to walk in the area. Suitable provisions need to also be put in place to be able to collect data in case criminals must be identified by Law Enforcement Authorities (LEAs).

text: Knowledge representation

text: Intelligent agents are software entities that carry out a set of operations on behalf of a user or another program with some degree of independence or autonomy. Doing so, they employ some knowledge or representation of the world and the users’ goals or desires. Knowledge-based autonomous agents (Zhou, Liao, & Tang, 2012) are best understood as agents that have knowledge about their world and reason for their actions. They usually integrate a) a knowledge-base (KB) i.e. a set of representations of facts about the world, and b) a knowledge representation language i.e. a language whose sentences represent facts about the world. Specific operations for adding new sentences to the KB and querying what is known are required, as well as a reasoning mechanism for determining what follows from what has been specified to the knowledge base. We can build a knowledge-based agent by ‘telling’ it what it needs to know before it starts perceiving the world, and we can design learning mechanisms that output general knowledge about the environment given a series of perceptions. In other words, an autonomous agent is a knowledge-based agent with an integrated learning mechanism.

text: The notion of autonomy

text: Autonomy is rooted in the ability to set and abide by a self-imposed rules based system. In autonomous systems this notion translates to the ability to apply and interpret (to a certain extent) rules. This ability combines with a learning capacity that renders the outcome impossible to predict. Safeguards need to be put in place to curtail broad discretionary autonomy and render predictable the behaviour of such systems and introduce specific limits and boundaries. While meaningful human control might not be possible at all times, it is important to lay down the technical and procedural framework to render such controls possible.

text: Learning

text: An AI system needs large volumes of data to first analyze and then learn from them. In order to be able to learn, it first needs to be trained on how to do this. Generally, learning systems aim to a) recognize patterns (e.g., recognize a cat from a car), b) detect anomalies (e.g., detect fraud in expense reports), and c) predict (e.g., predict stocks). To accomplish these, the following types of learning may be used, as those are simply described in Layman’s intro to AI and Neural Networks (Preetham V V, 2016):

text:  Supervised Learning. In this type of learning, there is need to know what to expect from the system in the output. The systems, during their training, are given the output needs. For instance, the English alphabet can be used by the system as a training dataset to support the recognition of English handwriting. Given a set of documents, the system must be able to predict any English handwriting included in them. During the training of the system, a few handwritten alphabets are entered into the system as input, along with the desired matching output alphabet. Once the system gains accuracy in recognizing handwritten alphabets, it can be used, for instance, in autonomous driverless cars to read signs on the highway.  Unsupervised Learning. In this type of learning, the systems do not ‘know’ the output needs. On the other hand, they are allowed to expose the internal representation of the input. For instance, automatically categorizing images with cats in a different category/class of the recognized images of a car. However, the system cannot recognize and label the car category as a “car, vehicle” or the cat category as “cat, animal” since the semantics of the categories are not given as input in the system (unlike supervised learning, which can recognize and label).  Reinforcement Learning. In this type of learning, the systems are asked to increase the payoff of an activity by selecting the most optimal action. While there may not be a predefined expected output, the

text: systems can look at what payoffs are voted higher and optimize actions. For instance, recommending the music that we like based on our music preferences. In autonomous agents, learning trajectory patterns of autonomous vehicles, based on real-time analysis of large volumes of data, may result to recommending the most economic trajectory from origin A to destination B, given specific weather and air-traffic conditions. Latest reports demonstrate how an autonomous car earned how to drive itself in 20 minutes using reinforcement learning .

text: Neural Networks/Deep Learning

text: Artificial Neural Networks (ANN) is a significant technology for the control of autonomous agents. The major goal of the research on autonomous agents is to study intelligence as the result of a system environment interaction, rather than understanding intelligence on a computational level. Autonomous agents have to continuously learn while they are behaving in their environment, without distinguishing between a learning and a performance phase. Thus, a neural network for autonomous agents should feature incremental/continuous learning (Salomon, 1996).

text: In an autonomous agent scenario, the agent initially starts to explore its environment and, during its lifetime, it gains more and more experiences. In such a scenario it is clear that: i) an agent determines the pattern’s relevance at runtime, ii) it is not appropriate to select a set of training examples for off-line training, and iii) an agent needs a network (for example a neural one) that features incremental learning properties, since the agent acquires its experience over time.

text: For an autonomous agent, this means that it must update its network weights while it is operating. The network model for autonomous agents should be elastic since during its lifetime, the appropriate actions associated with a particular sensory information might change. This dynamic change can be caused by a change in the environment (perhaps due to the agent’s interaction with its environment) or by a dynamical change of the electrical characteristics of the sensors. Simply put, a Neural Network is a collection of nodes (or units) which are layered as input nodes, hidden nodes and output nodes. The input nodes can be, for instance, the pixels from a vehicle image, the hidden nodes shall learn the ‘knowledge weight’ of the vehicle-ness of a car, and then one of the output node is selected based on the type of the vehicle (car, bus, train, drone, etc.).

text: Self - management

text: In a self-managed autonomous system, the user, instead of controlling the system directly, defines general policies and rules that the guide system’s self-management process. Four types of self-management properties have been currently specified, referred to as self-star (also found as self-*, self-x, or auto-*) properties (Sanz, López, Bermejo, Chinchilla, & Conde, 2005)(Poslad, n.d.):

text:  Self-configuration: Automatic configuration of systems’ components  Self-healing: Automatic recognition and healing of systems’ faults  Self-optimization: Automatic optimization of functioning based on monitoring and control of resources with respect to the defined requirements  Self-protection: Proactive identification of and protection from arbitrary attacks.

text: We further outline an expanded set of self-star properties as proposed by Poslad (Poslad, n.d.) as well as by Nami and Bertel (Nami & Bertels, 2007):

text:  Self-regulation: Systems operate to maintain a particular level of some parameter without external control, e.g., Quality of service  Self-learning: Systems use Machine Learning techniques to learn how to process new data/information (unsupervised learning) without external control  Self-awareness (also called self-inspection and self-decision): Systems have knowledge of themselves, are aware of the internal components and external links in order to control and manage them  Self-governance: Systems manage themselves without external intervention. Self-governance (or selfmanagement) also refers to a set of self-star processes such as autonomous computing rather than a single self-star process  Self-description (also called self-explanation or self-representation): Systems explain themselves, they are capable of being understood (by humans) without further explanation.

text: Autonomous agents should be able to demonstrate one or more self-x properties in order to claim their ‘intelligence’.

text: Decision-making systems

text: A decision support system (DSS) is a computer-based application that collects, organizes and analyzes business data to facilitate quality business decision-making for management, operations and planning. A well-designed DSS should support the decision makers to analyse a variety of data (e.g., raw data, documents, personal knowledge from employees, etc.).

text: Research on DSSs mainly concerns the integration of agents in applications. Such work has been well documented, including studies that can lead to the successful integration of agents in DSS (Hess, Rees, Matheson, & Ragsdale, 1999)(Hess, Rees, & Rakes, 2008). AI techniques have been widely used in research for developing new tools for decision-making (e.g., clinical). In most of them, the computer supports users (e.g., clinicians, patients) who take the decisions. On the other hand, it is simple to turn decision support tools into autonomous agents, i.e., tools that make the decisions themselves, without external control.

text: For instance, the CREDO decision support platform (Fox & Khan, 2014), implementing data interpretation and decision support capabilities in a generic technology for clinical decision support and multidisciplinary patient care, builds on a cognitive model of decision making-based argumentation theory and formalized in agent specification and knowledge representation language (Fox & Khan, 2014). Such a system could be switched in an autonomous mode by properly facilitating: a) Intelligibility: a healthcare agent should be able to engage in natural and cooperative interaction with its users (clinicians, patients), b) Personalization: an agent should accommodate the users’ personal goals and preferences, as long as they don't conflict with specific principles, c) Justifiability: an understandable rationale must be available for all recommendations, and particularly for automated actions, at whatever level of detail the user may reasonably require, d) Controllability: the user must be able to modify the system’s assumptions and goals, and the system must be able to adapt appropriately and safely to such changes.

text: Recognition Technology

text: A recognition system integrates technology that is capable of identifying or verifying an entity or object (e.g., a person, a car, an animal, a packet, a sign, a lane, a light) from the analysis of (streaming) data that is collected by sensors (e.g., camera, microphone, laser, thermometer, pulse meter) and provide in the form of a digital file (such as an image, a video or sound file) or a stream. Recognition of traffic jams, traffic or car lights, notification or traffic signs, lanes, vehicles, pedestrians and ambient sounds are only a few examples of the recognition technology in the transportation domain for facilitating autonomy in driverless vehicles. Moreover, the integration and combination of such technology increases the performance of such complex systems in unique ways e.g., face recognition increases the safety of autonomous vehicles (Arar, 2016).

text: Planning

text: Planning in AI regards the realization of strategies or action sequences for execution of autonomous agents, thus, it is usually used in the phrase “automated planning and scheduling”. Technological approaches that provide solutions for AI planning include but are not limited to: dynamic programming, reinforcement learning and combinatorial optimization. Coordination endows rational agents with cooperation abilities to generate coherent activities and ensure a correct Multi-Agent System (MAS) behaviour. Such coordination requires an adequate plan representation (Fallah-Seghrouchni, 2005). Agents organize their activities and update their plans to cooperate and avoid conflicts. Multi-agent planning remains one of the main mechanisms for MAS coordination. It raises several interesting issues because of the autonomy feature of agents and also because of the environment changes. A model for multi-agent planning should be studied from several perspectives: dynamic (or reactive) planning, decentralized planning, task allocation and resource sharing, etc. Decentralized planning is addressing issues for autonomous agents (e.g., autonomous vehicles) cooperating in complex missions (e.g., in space) that often involve a set of tasks, each representing a component of the mission (Nam & Rhee, 2010). Task planning algorithms are used as part of the mission planner to assign agents to tasks.

text: Conflict resolution

text: In general, conflict resolution can be considered as the set of methods and processes involved in discovering and applying solutions for eliminating conflict between any kinds of agents. Committed, within a specific workgroup, agents attempt to resolve workgroup conflicts by actively communicating information about their conflicting motives or ideologies to the rest of the group (e.g., intentions; reasons for holding certain beliefs) and by engaging in collective negotiation.

text: As already stated, autonomous agents plan their paths through known and unknown environments to reach their goals. When multiple autonomous agents share the same area, conflict situations may occur that need to be solved. There are several approaches towards solving conflict resolution in autonomous agents (e.g., (Alshabi, Ramaswamy, Itmi, & Abdulrab, 2007; Düring & Pascheka, 2014; Jacak & Pröll, 2009)), with an emphasis in the decentralized decision making algorithms (Düring & Pascheka, 2014) based on two main ideas: Gathering individually available manoeuvres from other agents and choosing a cooperative manoeuvre combination. Such an approach combines the use of motion primitives to define a set of available manoeuvres with an intelligent decentralized decision making algorithm that is based on the operationalization of cooperative behaviour (ability of working together towards increasing the overall performance of the workgroup).

text: Applications of intelligent controllers solving conflict situations by coordinating autonomous agents include robotics, unmanned aerial and marine vehicles, air traffic management, automotive engineering. A conflict resolution mechanism is needed to let the vehicles determine the crossing order and optimal trajectories by themselves. A distributed conflict resolution mechanism that does not depend on a traffic manager is highly desired for small intersections and in the case that the manager is broken. A communicationenabled distributed conflict resolution mechanism is required in order for a group of connected autonomous vehicles to move safely and efficiently in intersections in the absence of a traffic manager (Liu, Lin, Shiraishi, & Tomizuka, 2018). Decision making (for the determination of passing order) as well as motion planning (for the computation of trajectories), are both required.

text: Reasoning

text: Reasoning is related to humans’ thinking, cognition, and wisdom. In AI, reasoning is divided into logical reasoning types such as deductive reasoning, inductive reasoning, abductive reasoning, and other less formal methods such as intuitive reasoning and verbal reasoning. Logical reasoning is about how agents understand information gathered by sensors within their environments, or how they conceptualize cause, effect, truth, or even how they create ideas regarding the notions of good and bad. Automated reasoning

text: is considered a subfield of AI with connections to theoretical computer science and philosophy. Formal logic played a major role in the field of automated reasoning, which itself led to the development of AI.

text: There are different kinds of AI reasoning in autonomous agents e.g., case-based reasoning (Vacek, Gindele, Zollner, & Dillmann, 2007), probabilistic reasoning (Lance Eliot, 2018). Vehicle guidance in complex scenarios (e.g. inner-city traffic) requires an in-depth understanding of the current situation of the vehicles and environment they interact (other vehicles, pedestrians, obstacles, etc.). To select the appropriate behaviour for an autonomous vehicle, a deep and real-time analysis of the current situation is needed. Such an analysis may consist of an estimation of the situation's evolution with respect to the selected behaviour. Case-based reasoning allows to utilize prior experiences for the assessment of situations. As in human driving, where there is a continuous adjustment of probabilities due to the obvious uncertainty of the environment, one would expect that self-driving cars would perform similarly. Today, most of the selfdriving cars that are developed do not integrate probabilities into their AI systems, since it is not as easy as it might seem at a first glance (Lance Eliot, 2018).

text: Big Data

text: Data is key to autonomous agents’ technology (Daniel Matthews, 2018). For instance, for an autonomous vehicle to plan and act, it is using machine learning algorithms which need high volume of data to be able to predict outcomes accurately (e.g., a pedestrian is getting ready to cross the street, and the vehicle has to be trained well enough to be able to react appropriately). Another example is real-world data from vehicles out on the road. Such vast amounts of data are not only data from sensors and from the available infrastructure, but also data from crowd-sources as well as personal data from drivers and passengers (Lynnette Reese, 2017). Crowd-sourced data include the reporting of new obstacles or construction. Connected self-driving car systems may upload their data to the cloud and share their data that will be then used to train the systems of other vehicles.

text: 3. Security and privacy considerations

text: Autonomous agents are characterized by diversity, with applications varying from digital assistants residing in our smartphones to autonomous robots supporting supply chain. Depending on the level of autonomy and the context of operations security and privacy considerations may vary. However, an overview of most common considerations identified is briefly presented below, based on a set of indispensable baseline security requirements as published by ENISA in 2016 .

text: Security considerations Detection of rogue or unauthorized autonomous systems

text: Malicious autonomous agents can masquerade as legitimate agents to avoid detection. Developers shall be able to provide means to ensure that the agent has indeed been authorised to perform its tasks, the confidentiality, integrity and availability of data processed is preserved, depending on the operation and the context, the agent cannot be tainted during its operation and its integrity is preserved and verifiable throughout its lifecycle.

text: Hijacking and misuse

text: Autonomous drones, vehicles, robots and other devices are controlled by software systems. The degree of vulnerability varies depending on the approach and quality of the software development process and the breadth and depth of testing. The developer shall be able to provide evidence that a managed security by design approach has been adopted, including documented secure software development, quality management and information security management processes.

text: Interference

text: Autonomous systems rely heavily on their sensory ability to perceive their environment, which incurs a security risk from sensors. Security researchers (Yan et al., 2016) have examined the security of sensors used to guide self-driving cars to find that they are vulnerable to contactless attacks. The provider shall design and pre-configure the delivered product such that functionalities are based on well-established security practices and are reduced to the strict minimum required for system operations.

text: Transparency and accountability

text: The behaviour of autonomous agents is not fully prescribed in their software code. Their behaviour is uncertain and results as a combination of their software, their training, and their perception of the environment. In most advanced autonomous agents, there is no distinction between the training and operation phases, as training continuous autonomously throughout the lifecycle of the agent. Thus, training is not fully controlled by the manufacturer. The manufacturer however shall be able to offer comprehensive and understandable documentation about the overall design of the agent, describing its architecture, functionalities and protocols, their realisation in hardware or software components, the interfaces and interactions of components with each other and with internal and external services, in order to be able to implement and deploy the agent in the most secure way possible.

text: Adherence to security principles

text: Similar to the basic security principles that are for information systems inception, development and deployment of an information system, a set of basic security principles has to be identified, defined and

text: adhered to through the lifecycle of an autonomous agent. Therefore, Components of the agent, services that it uses and services that it provides shall be secure:

text: • By design – the agent, or service, has been conceived, designed and implemented to ensure the key security properties are maintained: availability, confidentiality, integrity and accountability; • By default – the agent, or service, is supplied with the confirmed capability to support these security properties at initial deployment; • Throughout its lifecycle – security should be maintained from initial deployment through maintenance to decommissioning; • Each of the above principles should be verifiable.

text: Privacy considerations Pervasiveness and the minimization principle

text: The learning capabilities of autonomous agents are based on machine learning algorithms that by nature require large datasets as input. This means that they tend to collect as much data as possible, rather than a statistically relevant sample (Butterworth, 2018). Consider, for example, the case of autonomous vehicles. They are equipped with sensors that collect information from an area with radius of approximately 300 meters. They are equipped with cameras and microphones, as well as radars and laser sensors that create high-resolution 3D representations of objects in the environment. Autonomous agents do not seem to currently have the capacity to distinguish necessary from unnecessary data. The ability to act autonomously depends strongly on the ability to perceive the environment. As the environment is complex and dynamic, it is difficult to know a priori whether a piece of information will be useful or not.

text: Data retention and protection

text: Autonomous agents process real-time data that could be deleted after use without affecting their performance. However, these data are stored and retained for large periods of time for different reasons. One such reason is the investigation of accidents and other incidents. For example, in the case of autonomous vehicles it is very important to be able to investigate an accident, in order to identify the cause and provide a solution/patch. Training data sets are also retained for the same reason as it should be possible to investigate whether they were sufficient and appropriate. Data are also stored and used for research purposes, as they constitute a valuable input in the process of advancing the autonomous agents technologies.

text: Even when data are deleted, they leave traces that can be exploited. Researchers (Shokri et al., 2017) have shown that it is possible to determine whether a specific record was part of the training set of a machine learning algorithm, without having any access to the internal structure of the model (i.e., black box analysis). They have thus shown that machine learning models are vulnerable to inference attacks. Therefore, we should not consider the deletion of the training set to be an effective measure for privacy protection. The above data sets are accessible by engineers, developers, researchers, investigators and, possibly, many other stakeholders. As a result, the protection of data confidentiality is quite a challenging task.

text: Data aggregation and repurposing

text: In most of the applications of autonomous agent technologies data are aggregated by providers and used for several purposes. Autonomous vehicles and drones are connected to control centers owned by the companies that produce these vehicles and drones and data are transmitted for analysis to these centers. In a similar manner, software agents, such as intelligent assistants, share data with their manufacturers that analyse and process them to provide additional services to their users. Developers and providers collect data from numerous users and combine data from various sources. They can create detailed

text: profiles of people, they can make predictions about their behavior and guess their needs through data aggregation and profiling.

text: Data collected from one activity can be used and analysed for a wide range of purposes. This is often referred to as “repurposing”. Data that were primarily collected to be used for agent training could also be used for marketing purposes, data collected for the purpose of investigating accidents, could be used for user profiling, and so forth.

text: The opacity of processing

text: Traditional data processing systems comprise software that implements well-defined algorithms, such as decision trees. Data are stored separately in rigid structures, such as relational databases. Data subjects and data protection authorities can query databases and inspect processes and procedures to identify how a processing operation operates. Thus, the lawfulness, fairness and transparency of personal data processing can be observed.

text: On the contrary, machine learning processes operate as a “black box” to the user (Butterworth, 2018). Machine learning algorithms provide no explanation for their results. They cannot show whether a certain data instance has been necessary to achieve the purpose of processing or not and they cannot show the degree of significance of each piece of information. Thus, in the case of autonomous agents that base their autonomy on machine learning modules, it is currently challenging, to demonstrate the lawfulness, fairness and transparency of personal data processing. Hence, there is need to apportion responsibilities across different regulatory and enforcement agencies involved and determine competences across them.

text: 4. Shaping a framework for policy development in cybersecurity

text: As a first step towards sketching a point of reference for policy-makers tackling the issue of cybersecurity and establishing a relevant framework for policy development, this study provides an overview of representative application domains of autonomous agents and attempts to complement relevant initiatives at EU level by providing insights and considerations, both for security and privacy. The rapid adoption of digital technologies has enabled many opportunities for innovation and interaction but at the same time introduced a number of challenges that were not evident before. The cyber domain has evolved so swiftly that legal, economic, and societal mechanisms for maintaining cybersecurity have struggled to keep up.

text: Artificial intelligence (AI) and increasingly complex algorithms currently offer great potential and possibilities for a diverse range of application areas which influence our everyday lives more than ever before. Similarly, Industry 4.0 with the Internet of Things (IoT) at its core has already exerted an impact on society, transforming products, customer experience and the labour market. However, is not viable to expect that each technological advancement with a dedicated legislative instrument, mainly due to the fast pace of innovation. Therefore, a more broad approach must be pursued, able to balance the priorities of all involved stakeholders while putting forward participial solutions to emerging challenges.

text: In 2016, RAND published a study on “A Framework for Exploring Cybersecurity Policy Options” which aimed to “to develop an initial framework for cybersecurity that considers the roles of government, industry, advocacy organizations, and academic institutions and how these stakeholders’ concerns relate to each other. The main findings of the study pertained the lack of cybersecurity demand in the market, the limited incentives to promote cybersecurity best practices and finally the need for publicprivate partnerships able to identify, elaborate and eventually address emerging cybersecurity challenges.

text: Based on the analysis of numerous application domains of artificial intelligence in Section 2, it is evident that a framework for policy development in cybersecurity should adhere to two main principles, namely:

text:  Inclusiveness: reflecting and promoting interests and priorities from a wide range of stakeholders;  Openness: supporting the inclusion of future innovative digital technologies in diverse application domains.

text: Towards a more practical interpretation of these two principles, such a framework should:

text:  Promote the development and adoption of technical standards and specifications;  Encourage exchange of best practices and experience;  Facilitate the establishment of synergies and partnerships between private and public sector stakeholders.

text: 5. Conclusions and Recommendations

text: One of the key aspects in autonomous systems is the data collected, mainly for supporting the demanding functionality in a qualitative and timely manner. Given its distinctive characteristics, an envisioned policy framework will require forward thinking practices, both by private and public sector, and ensure that effective approaches are promoted while appropriate safeguards, requirements and obligations are also in place. Towards this direction, the main challenges identified are briefly presented below, in addition to respective recommendations, which are meant to complement ongoing work and initiatives at EU level.

text: Adoption of security and privacy by design principles

text: One of the key aspects in autonomous systems is the ability to process vast amounts of data and act upon them, mainly for supporting the demanding functionality in a qualitative and timely manner. However, the autonomous system must be conceived, designed and implemented in a way that ensures adherence key security properties availability, confidentiality, integrity and accountability. In addition, it must be supplied with the confirmed capability to support these security properties at installation throughout it’s lifecycle – from initial deployment through maintenance to decommissioning.

text: European Commission, relevant EU Bodies and relevant public and private sector stakeholders should further promote and support the adoption of security and privacy by design principles as a prerequisite during the inception, design and implementation of autonomous agents and systems.

text: Development of baseline security requirements

text: The rapid developments in the fields of artificial intelligence and autonomous agents reinforces the need for a coordinated approach and guidance during their inception, design and development. As such, developers could benefit from guidance on how to address identify and overcome identified security and privacy challenges. Such guidance could be provided through identification and exchange of best practises, development of baseline security requirements voluntary standards which will also provide guidance on conformity assessment.

text: Coordination of actions on highlighting and addressing ethical considerations

text: Autonomous agents and systems often need to make complex decisions, which apart from the aspect of accountability, introduce also ethical considerations with regard to human rights, human dignity and nondiscrimination. Addressing ethical issues and dilemmas in autonomous systems design requires an interdisciplinary and coordinated approach.

text: The cost of coping with machine autonomy is also an issue of concern as it far outpaces the ability of a single community to deal with the entire range of issues alone. The Internet has already challenged societies that have embraced it as effort and resources are dispensed off locally to counter threats originating from afar. In cases such as applications of machine autonomy it is likely that such issues will be

text: exacerbated further leading to a new set of policy challenges that might exhaust the capability of existing agencies to respond.

text: European Commission, relevant EU Bodies, public and private sector stakeholders should further promote and support existing initiatives on promotion and protection of human rights through establishing appropriate ethical parameters.

text: 6. Bibliography/References

text: Alshabi, W., Ramaswamy, S., Itmi, M., & Abdulrab, H. Coordination, cooperation and conflict resolution in multi-agent systems. Innovations and Advanced Techniques in Computer and Information Sciences and Engineering, 2007, pp. 495–500. Arar, S. Facial Recognition Increases Autonomous Vehicles Safety - News. 2016. Butterworth, M. The ICO and Artificial Intelligence. The Role of Fairness in the GDPR Framework. Computer Law & Security Review, 34, 2018, pp. 257-268. Daniel Matthews. Data Is Key to Autonomous Vehicle Technology, And Tesla Says It’s Winning, 2018. Düring, M., & Pascheka, P., ‘Cooperative decentralized decision making for conflict resolution among autonomous agents’. International Symposium on Innovations in Intelligent Systems and Applications INISTA 2014, IEEE, 2014, pp. 154–161. Fallah-Seghrouchni, A., Multi-agent Planning for Autonomous Agents’ Coordination. In Monitoring, Security, and Rescue Techniques in Multiagent Systems, Berlin/Heidelberg: Springer-Verlag, 2005, pp. 53– 68. Fox, J., & Khan, O., From decision support systems to autonomous agents: how can we ensure ethical practice? Doc.Gold.Ac.Uk, (April), 2014. Garbhe, S. What is Artificial Intelligence (AI) – Becoming Human: Artificial Intelligence Magazine, 2018. Hess, T. J., Rees, L. P., Matheson, L. A., & Ragsdale, C. T., Study of Autonomous Agents in Decision Support Systems, 1999. Hess, T. J., Rees, L. P., & Rakes, T. R., ‘Using Autonomous Software Agents in Decision Support Systems. In Handbook on Decision Support Systems 1’, Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 529– 555. Hirz, M., & Walzel, B., ‘Sensor and object recognition technologies for self-driving cars. Computer-Aided Design and Applications’, Vol. 15 No. 4, 2018, pp. 1–8. Jacak, W., & Pröll, K., ‘Conflict Resolution in Multiagent Systems Based on Wireless Sensor Networks’. Springer, Berlin, Heidelberg, 2009, pp. 753–760. Lance Eliot. Probabilistic Reasoning for AI Self-Driving Cars - AI Trends, 2018. Liu, C., Lin, C.-W., Shiraishi, S., & Tomizuka, M., ‘Distributed Conflict Resolution for Connected Autonomous Vehicles’. Transactions on Intelligent Vehicles, IEEE, Vol. 3 No.1, 2018, pp. 18–29. Lynnette Reese. Big data in autonomous driving | Solid State Technology, 2017. Nam, M., & Rhee, P. K., ‘Agent Based Object Recognition System for Autonomous Surveillance in Dynamic Environments’. 3rd International Conference on Human-Centric Computing, IEEE, 2010, pp. 1–5. Nami, M. R., & Bertels, K., ‘A Survey of Autonomic Computing Systems’. International Conference on Autonomic and Autonomous Systems ICAS’07 IEEE, 2007, pp. 26–26.

text: Poslad, S., Autonomous Systems and Artificial Life. In Ubiquitous Computing. Chichester, UK: John Wiley & Sons, Ltd., pp. 317–341.

text: Russell, S. J., Stuart J., Norvig, P., & Davis, E., Artificial intelligence: a modern approach. Prentice Hall, 2010.

text: Salomon, R. (1996). ‘Neural Networks in the Context of Autonomous Agents: Important Concepts Revisited’. Artificial Neural Networks in Engineering ANNIE’96, 1996, pp. 109–116.

text: Shokri, R., Stronati, M., Song, C. and Shmatikov, V., 2017, May. Membership inference attacks against machine learning models. In Security and Privacy (SP), 2017 IEEE Symposium on (pp. 3-18). IEEE.

text: Vacek, S., Gindele, T., Zollner, J. M., & Dillmann, R. ‘Using case-based reasoning for autonomous vehicle guidance’. International Conference on Intelligent Robots and Systems, IEEE/RSJ, IEEE, 2007, pp. 4271– 4276.

text: Yan, C., Xu, W., & Liu, J. (2016). Can you trust autonomous vehicles: Contactless attacks against sensors of self-driving vehicle. DEF CON, 24.

text: ENISA

text: European Union Agency for Network and Information Security 1 Vasilissis Sofias Marousi 151 24, Attiki, Greece

text: Heraklion Office

text: Science and Technology Park of Crete (ITE) Vassilika Vouton, 700 13, Heraklion, Greece Catalogue Number TP-06-18-399-EN-N

text: 1 Vasilissis Sofias Str, Maroussi 151 24, Attiki, Greece ISBN: 978-92-9204-282-0 DOI: 10.2824/453043 Tel: +30 28 14 40 9710 info@enisa.europa.eu www.enisa.europa.eu


text: http://www.computerworld.com/s/article/9227238/Consumerization_trend_creates_IT_worries_worker_benefits?taxonomy Id=220&pageNumber=1, accessed 4 September 2012. http://www.pwc.com/us/en/technology-innovation-center/consumerization-information-technology-transforming-ciorole.jhtml, accessed 4 September 2012.

text: www.cert.europa.eu, accessed continuously.

text: Work Package 1.1, that is: D2: Identification and analysis of specific areas to be assessed and D3: Opportunities and Risk Assessment.

text: http://en.wikipedia.org/wiki/Bring_your_own_device, http://www.webopedia.com/TERM/B/BYOD.html, accessed 4 September 2012.

text: http://en.wikipedia.org/wiki/Consumerization_of_IT, accessed 4 September 2012.

text: Guide to Risk and Opportunity Management, http://www.thurrock.gov.uk/i-know/pdf/perf_how_05_risk_2012.pdf, accessed 4 September 2012.

text: Based on available material from Turner and Townsend, URL: http://www.turnerandtownsend.com/risk.html, accessed 7 September 2012.

text: EMBRACING UNCERTAINTY IN DoD ACQUISITION, 1SG David E. Frick, USA, July 2010, in http://www.dau.mil, accessed 7 September 2012.

text: Interest of ENISA

text: Interest of ENISA Stakeholders has been expressed as follows:

text: - ENISA PSG: Proposed the consideration of mobility /mobile environments/mobile computing. Furthermore, addressing consumerization issues has also been mentioned. - ENISA Expert Groups: Given the trend of mobile computing (phones, gadgets), experts have proposed repeatedly to assess risks of mobile platforms.

text: Popularity The topic of COIT/BYOD is enjoying currently major attention within security, industry, government and consumer fora. Companies are currently rolling out solutions aiming at the implementation of COIT, while existing security controls are being modified in order to allow for the incorporation of COIT into professional IT-platforms.

text: Impact on Trust

text: Existing IT and security architectures were designed for a different Infrastructures computing paradigm. COIT introduces a number of changes impacting trust infrastructure. Such changes include: - Network perimeter changes - Approaches to Management of IT components - Revisions of existing user and service level/support agreements The impact of these changes on trust infrastructure is big and affects data security, perimeter security, identification and authentication functions. Novelty COIT changes will lead to novel approaches to: - Models for the protection of data accessed by devices - New application development, application and device management models - New types of service level, support and user agreements - New technology “mix” in the way to the end-user

text: Continuity/Coherence COIT advances and is the logical continuation of a variety of ENISA activities, in particular: - Smartphone Security - App Store Security - Cloud Computing

text: http://www.enisa.europa.eu/activities/identity-and-trust/risks-and-data-breaches/smartphones-information-security-risksopportunities-and-recommendations-for-users accessed continuously.

text: http://www.enisa.europa.eu/media/press-releases/app-store-security2013-the-five-lines-of-defence-new-report-by-eucyber-security-agency-enisa accessed continuously.

text: - Mobile Identity Management - Social Networking

text: http://www.enisa.europa.eu/activities/Resilience-and-CIIP/cloud-computing, accessed continuously.

text: CATEGORY RISK COMMENT Legal and Costs Data Regulatory

text: http://www.simplysecurity.com/2012/05/31/security-experts-shifting-focus-beyond-static-perimeter-defenses/, accessed 7 September 2012.

text: http://www.booz.com/media/uploads/FriendlyTakeoverVPFINAL.pdf, accessed 7 September 2012.

text: http://www.lancope.com/resource-center/market-briefs/stealthwatch-for-mobile-device-security/, accessed 7 September 2012.

text: Legend: X primarily counteraction of a risk; (X) secondary counteraction of risk Table 3: Risks counteracting the implementation of opportunities It is worth mentioning, that if these risks are correctly mitigated, they have a supportive effect towards implementation of an opportunity. Hence, if risk RLR3 “Lack of clear distinction between corporate and personal data on employee-owned devices will make e-discovery more difficult and may lead to litigation with employees” is mitigated in a satisfactory manner, this fact will encourage the implementation of opportunity O2 “Potential Human Resources benefits”.

identifier: Consumerization of IT: Top Risks and Opportunities

title: Consumerization of IT: Top Risks and Opportunities

text: Responding to the Evolving Threat Environment [Deliverable – 2012-09-28]

text: Consumerization of IT: Top Risks and Opportunities I Responding to the Evolving Threat Environment

text: Acknowledgements

text: This report has been produced by ENISA using input and comments from a group of experts from industry and academia and public organisations. The experts have been selected according to their engagement in the area of interest (e.g. security in emerging technologies, security in Bring Your Own Device (BYOD) and Consumerization of IT (COIT)). It should be noted that group members participate as individuals. Therefore, their contribution should not be taken as representing the views of any company or other organisation. The contributors are listed below in alphabetical order:  Jim Clarke, Waterford Institute of Technology, IR  Marcos Gomez Hidalgo, INTECO, ES  Antonio Lioy, Politecnico di Torino, IT  Milan Petkovic, Eindhoven University of Technology, Philips Research, NL  Claire Vishik, Intel Corporation, UK, US  Jeremy Ward, HP Enterprise Services, UK

text: Consumerization of IT: Top Risks and Opportunities II Responding to the Evolving Threat Environment

text: About ENISA

text: The European Network and Information Security Agency (ENISA) is a centre of network and information security expertise for the EU, its member states, the private sector and Europe’s citizens. ENISA works with these groups to develop advice and recommendations on good practice in information security. It assists EU member states in implementing relevant EU legislation and works to improve the resilience of Europe’s critical information infrastructure and networks. ENISA seeks to enhance existing expertise in EU member states by supporting the development of cross-border communities committed to improving network and information security throughout the EU. More information about ENISA and its work can be found at www.enisa.europa.eu.

text: Contact details

text: For contacting ENISA for general enquiries on this report, please use the following details:  E-mail: opsec@enisa.europa.eu  Internet: http://www.enisa.europa.eu

text: Legal notice

text: Notice must be taken that this publication represents the views and interpretations of the authors and editors, unless stated otherwise. This publication should not be construed to be a legal action of ENISA or the ENISA bodies unless adopted pursuant to the ENISA

text: Regulation (EC) No 460/2004

text: as lastly amended by

text: Regulation (EU) No 580/2011

text: . This publication does not necessarily represent state-of the-art and ENISA may update it from time to time. Third-party sources are quoted as appropriate. ENISA is not responsible for the content of the external sources including external websites referenced in this publication. This publication is intended for information purposes only. It must be accessible free of charge. Neither ENISA nor any person acting on its behalf is responsible for the use that might be made of the information contained in this publication. Reproduction is authorised provided the source is acknowledged. © European Network and Information Security Agency (ENISA), 2012

text: Consumerization of IT: Top Risks and Opportunities IV Responding to the Evolving Threat Environment

text: List of Tables

text: Consumerization of IT: Top Risks and Opportunities 1 Responding to the Evolving Threat Environment

text: 1 Executive Summary

text: This report is an ENISA deliverable in the area of “Identifying & Responding to the Evolving Threat Environment”. It delivers the results of a risk and opportunity assessment in the area of “Consumerization of IT” (COIT), that is, the recent trend where user-owned consumer oriented hard- and software spreads in business environments (see also definition in section Terminology below). COIT is considered as a term embracing the recent trend known as Bring- Your-Own-Device (BYOD). Further to the risk and opportunity assessment, this report presents the criteria and guidelines that were used for identification of COIT as an emerging area. The presented criteria and guidelines will be applied in the identification phase of future areas for the assessment of emerging risks and opportunities. The work has been conducted with the support of an Expert Group (see Acknowledgements). The risks and opportunities assessed are:

text: Risks related to costs:

text:  Increased risk of loss of value when employees bring the organisation’s brand into disrepute by uncontrolled use of consumerized services/devices  Increased variety and complexity of devices, systems and applications, all requiring management, will lead to increased costs  More use of mobile devices is likely to result in more lost devices and thus increased costs  Additional spending to ensure that security requirements do not act to either prevent appropriate consumerization or to encourage inappropriate use of consumer devices

text: Risks related to legal and regulatory issues:

text:  Corporate governance and compliance control over employee-owned devices will be weaker  Interoperation, usage models and change of security context between applications and systems will make enforcement of legal and regulatory compliance controls more difficult  Lack of clear distinction between corporate and personal data on employee-owned devices will make e-discovery more difficult and may lead to litigation with employees

text: Risks related to data confidentiality/integrity/availability:

text:  Potential loss of corporate data as a result of unauthorized sharing of information on employee’s devices and sharing of devices  Potential loss of corporate data as a result of access by unknown users and unmanaged devices to enterprise networks

text: Consumerization of IT: Top Risks and Opportunities 2 Responding to the Evolving Threat Environment

text:  Potential loss of corporate data as a result of difficulty of controlling security in application-rich mobile devices, especially if employee-owned  Increased risk of mobile devices being the target of attack for the acquisition of corporate data

text: Opportunities:

text:  Potential financial opportunities  Potential Human Resources benefits  Potential Data Management opportunities  Potential operational opportunities

text: Consumerization of IT: Top Risks and Opportunities 3 Responding to the Evolving Threat Environment

text: 2 Introduction

text: This report is the first ENISA deliverable in the area of “Identifying & Responding to the Evolving Threat Environment”. The objective of the report is to assess opportunities and risks associated with the consumerization of IT. In addition, where risks are identified, ENISA will recommend suitable mitigation strategies and provide guidelines on how such strategies can be implemented in operational environments. Consumerization and mobility have been identified both by ENISA stakeholders (i.e. ENISA Permanent Stakeholder Group and past ENISA expert groups) as areas to deliver the next evolution of security architectures and approaches. COIT provides a new emerging context to security approaches: through consumerization, security controls need to be installed outside the perimeter of an organisation in order to protect business assets on the move. Last but not least, the paradigm shift brought through the emerging consumerization trend of various IT components puts additional/further requirements on the security of the mobile and trust infrastructures. Collection, collation and aggregation of existing information are key to the performed assessments. A team of experts has performed information collection and aggregation of existing material in the subject area. For this purpose, ENISA has used information sources collected via the information selection facilities of the CERT-EU and through the involved experts. The delivered risk and opportunity assessment is the result of a collective exercise within the team of experts and ENISA. Based on the collected material, non-technical information regarding all the components of risks and opportunities has been generated. It is worth mentioning, that a systematic risk and opportunity assessment of the kind presented in this report has not been found in this form in any of the collected information sources. The content of this report contains two deliverables that are foreseen in the ENISA Work Programme 2012 for the area of COIT . While this report covers the risk and opportunity assessments, a forthcoming report in 2012 will cover risk mitigation and implementation strategies based on good practices (content of Work Stream 1, Work Package 1.2 of the Work Programme 2012). Around the end of year 2012, ENISA is going to re-validate the assessed risks and opportunities to reflect current developments in this rapidly evolving area.

text: Consumerization of IT: Top Risks and Opportunities 4 Responding to the Evolving Threat Environment

text: 3 Terminology

text: In the present section the terminology used within this assessment is being presented. BYOD – Bring Your Own Device: BYOD refers to the trend where employee-owned, privatelyused devices like smart phones, tablets, ultra-light laptops, etc., are used for job related tasks with permission/support of the employer . Consumerization of IT (COIT): Is a current trend in the area of IT where consumer-oriented, privately-used IT, like Social Networking, Cloud Storage, mail, smart phones, tablets, etc. is becoming part of professional IT. Given the positive user experience with these technologies, end users generate pressure to company IT to adopt similar functions/approaches. BYOD can be considered as the device-centric part of COIT. Risks: According to the widely accepted ISO 27005 definition risks emerge when: “Threats abuse vulnerabilities of assets to generate harm for the organization”. In more detailed terms, we consider risk as taking into account the following elements: Asset (Vulnerabilities, Controls), Threat (Threat Agent Profile, Likelihood) and Impact Opportunity: As regards opportunities, due to missing standardised definitions we use the definition according to which “An opportunity is an uncertainty that will enhance ability to achieve objectives” . Opportunities are related to the realisation of benefits for the organisation. An opportunity can include savings from increased organisational efficiency. In addition, an (business) opportunity is a gain for the organization as the result of a better exploitation of market /business conditions. In order to achieve opportunity management, elements that have to be considered are : driving improvements in an operational environment, balancing return and investment, obtaining change buy-in and manage reward. In addition, some resources argue that opportunity management might be the result of a risk management by focussing on positive consequences of a risk. Due to our focus on Information Security issues, the elements of both risks and opportunities should have an ICT context and be directly or indirectly related to Information Security.

text: Consumerization of IT: Top Risks and Opportunities 5 Responding to the Evolving Threat Environment

text: 4 Identification of specific areas to be assessed 4.1 Criteria for the identification of emerging areas for the assessment

text: A set of criteria is being used for the identification of emerging areas that will be subject to risk and opportunity assessment. These criteria are developed in such a way, so as to primarily cover the needs and requirements of our stakeholders (see first and second bullet points below). A further element taken into account is the relevance of an area with regard to emerging security requirements to existing trust infrastructure such as authentication, encryption, preventive and detective components, etc. (see third and fourth bullet points). Finally, the relevance of an area to ENISA’s work (current and past), is another criterion for selecting an area for the assessment (see last bullet point). In summary, the criteria used for the identification of areas are:  Interest of ENISA Stakeholders: ENISA steering committees, expert groups and other external stakeholders have expressed their interest in a particular area.  Popularity: Level of interest/ discussion within the security community is high (e. g. within Member States, Commission, market trends and developments, conferences, discussion fora, etc.).  Impact on Trust Infrastructures: Relevance of an area with regard to impact on existing trust infrastructures (e. g. posing additional requirements to infrastructures that are used to enforce/implement security controls).  Novelty: Novel security architectures and techniques are required to fulfil emerging security requirements.  Continuity/Coherence: An area is an important complement or logical advancement of other relevant ENISA work.

text: Besides the above mentioned criteria, a set of guidelines have been followed when conducting risk and opportunity assessments. In particular the performed assessments:  are based on collection and aggregation of existing quantitative data and/or take into account existing risk assessments,  provide non-technical information regarding all the components of the assessed risks and  take into account (business) opportunities, social, legal and trust aspects.

text: It is worth mentioning that the above mentioned guidelines have been approved by the ENISA Management Board and Permanent Stakeholder Group as part of the ENISA Work Programme 2012. The identification criteria have been derived from good practices of the past (e.g. followed during assessments in the areas Cloud Computing, Internet of Things and Future Internet, mobility issues, e-health, child online protection, supply chain security, browser security, etc.).

text: Consumerization of IT: Top Risks and Opportunities 6 Responding to the Evolving Threat Environment

text: 4.2 Selecting Consumerization of IT (COIT)

text: By applying the criteria mentioned in the previous section to various areas, we have identified the area of Consumerization of IT as being very relevant to security both from the user and the operator point of view. In this section, we indicate how the area of COIT matched the defined criteria by means of the following table:

text: Consumerization of IT: Top Risks and Opportunities 7 Responding to the Evolving Threat Environment

text: CRITERIA Consumerization of IT

text: Consumerization of IT: Top Risks and Opportunities 8 Responding to the Evolving Threat Environment

text: Table 1: Criteria for the selection of COIT Following the guidelines set, the material for the assessment has been collected from existing online contributions on this topic with the support of CERT-EU . Thus, more than 1500 relevant information sources have been taken into account and have been continuously traced for a period of 2 months (April-May 2012). In addition, individual searches from the members of the expert group have been performed. The list of collected sources includes some ca. 60 relevant publications. Interested individuals can obtain the list of information sources collected and processed by sending a request to ENISA. The selected material has greatly helped in the performance of the risk and opportunity assessment. However, a complete risk and opportunity assessment in the form presented in this report has not been found in any of the collected material.

text: Consumerization of IT: Top Risks and Opportunities 9 Responding to the Evolving Threat Environment

text: 5 Assessed Risks

text: The assessed risks have been categorised into three groups according to their relevance to impact areas, namely: costs, legal and regulatory issues and data confidentiality/integrity/availability/privacy. While not completely independent and exhaustive, these categories have been chosen as most suitable for a high-level overview. It is worth mentioning that additional aspects of risks are orthogonal to these categories, such as privacy risks, reputation risks, Intellectual Property risks, etc. Overlapping issues and dependencies across these categories are covered by means of overview table (see Table 2). In the sections below, 11 risks are identified within the three categories mentioned above. The risk numbering used carries the initial R standing for risk followed by the initial of its corresponding category (i.e. C for Costs, LR for Legal and Regulatory, D for Data), followed by a number. The assessed risks have NOT been presented in a specific order. This decision is motivated by the fact that different priorities of the assessed risks will exist for different type of organisations and businesses. Organisations, for example, dealing with confidential information (i.e. personal data, intellectual property etc.) will assign a higher priority to risks affecting data and compliance. Similarly, organisations that concentrate more on cost/performance issues of their workforce will try to mitigate risks related to costs and legal issues. Interested readers need to find out which risks will be more relevant to the type of their organisation and business models.

text: 5.1 Risks related to costs

text: RC1. Increased risk of loss of value when employees bring the organisation’s brand into disrepute by uncontrolled use of consumerized services/devices: This risk can arise through uncontrolled use of consumer IT services such as Cloud Computing, social media, drop boxes, browser data and software and applications installed or used in mobile devices. Through improper use of such services, users may neglect existing security policies and transfer company information outside the security domain, thus enabling access to non-authorized individuals. Alternatively, the risk can arise through sharing of devices (with family and friends, for example) or when a malicious individual gains physical access to the device through the use of a social engineering attack. It is worth mentioning, that such a risk will lead to loss of reputation and may cause significant costs to the organisation.

text: RC2. Increased variety and complexity of devices, systems and applications, all requiring management, will lead to increased costs: The result of this risk could be that, instead of achieving cost reduction through consumerization as initially assumed, businesses might need to invest more. Cost increases could be caused by additional IT management resources needed in order to accommodate the various systems (e.g. Mobile Device

text: Consumerization of IT: Top Risks and Opportunities 10 Responding to the Evolving Threat Environment

text: Management) required to manage all of the different mobile devices supported by the organization. In addition, costs will be incurred as a result of:

text: - Additional investments to achieve desired level of protection and compliance; - Support of employees using their own devices.

text: Moreover, given the dynamic nature of the technology deployed by end users, additional costs might be incurred due to the need for continuous adaptation and revision of policies (e.g. on opening network perimeter security). Last but not least, the risk of getting locked-in by a premature selection of technologies has been identified. This could result in a strategic failure that may cause significant additional costs to the organization.

text: RC3. More use of mobile devices is likely to result in more lost devices and thus increased costs: A large number of devices have been stolen, lost or have been left behind in the back of taxis around the world. Device loss will increase costs for an organization, as the device would have to be replaced and money would need to be spent on recovering data, assessing the risk associated with lost data and any consequent security and recovery measures. However, costs associated with loss of devices may be considered to be far less of a business risk than the loss of the information held on those devices; and this is considered in the data confidentiality/integrity/availability category below.

text: RC4. Additional spending to ensure that security requirements do not act to either prevent appropriate consumerization or to encourage inappropriate use of consumer devices: In order to ensure that the security requirements of consumerization are met, businesses will need to make their security architecture device-agnostic. This means abandoning rigid perimeter security and introducing end-to-end security that dynamically adapts to the characteristics of the user-owned device. It also means finding the balance between policies that are too liberal to deliver good security and policies that are too rigid to allow effective support. To achieve this implies significant enhancement of existing security policies while, at the same time increasing the understanding and ability of users through investment in security education and awareness training.

text: 5.2 Risks related to legal and regulatory issues

text: RLR1. Corporate governance and compliance control over employee-owned devices will be weaker: It is expected that difficulties will be faced in maintaining the traceability and manageability of user actions on consumer IT components that are not owned by the businesses. This includes the resolution of security incidents: businesses might have difficulties in managing an incident if no access to all parts of the consumer IT

text: Consumerization of IT: Top Risks and Opportunities 11 Responding to the Evolving Threat Environment

text: component is granted. In the same manner, absence of managed SLA agreements might with involved providers might deteriorate incident management.

text: Moreover, additional difficulties might arise in maintaining compliance with data protection regulation through loss of individual privacy, enterprise data and data integrity. As a result, businesses will struggle to enforce compliance on consumer devices and services that are not under their ownership. In a similar manner as RC1, this risk might cause a reputation loss to the organisation.

text: RLR2. Interoperation, usage models and change of security context between applications and systems will make enforcement of legal and regulatory compliance controls more difficult: Given the fact that consumer IT may be owned and operated entirely by end users, it will be difficult for businesses to enforce their entire suite of policies and working regimes, such as HR policies, legal scope and context and claims of ownership on intellectual property. Examples how such risks might materialize are: unofficial teleworking, working outside working hours, end user activities within different jurisdictions (e.g. use of cloud services and drop boxes), definition of sphere of influence regarding data and applications on the end-user devices.

text: RLR3. Lack of clear distinction between corporate and personal data on employee-owned devices will make e-discovery more difficult and may lead to litigation with employees: The risk of not being able to discriminate between user and company data stored on consumer devices is evident and may result in risks related to the intervention of businesses in the private life and property of employees. Examples of such data are user credentials, administration rights, access rights, etc. Risks from the inadvertent release of personal data might also arise, as deployed security controls could allow businesses to access user’s personal data stored on their device. This is clearly a privacy risk that could result in disgruntled employees who may be reluctant to use consumer devices in connection with their work; or who may even bring legal actions against their employer. In a similar manner as RC1 and RLR1, this risk might cause a reputation loss to the organisation.

text: 5.3 Risks affecting data (confidentiality, integrity, availability and privacy)

text: RD1. Potential loss of corporate data as a result of unauthorized sharing of information on employees’ devices and used services and sharing of devices: If there is weak implementation and enforcement of security policies in user-owned consumer devices (e.g. gadgets, home PC) and services (e.g. used cloud services, social networking) there

text: Consumerization of IT: Top Risks and Opportunities 12 Responding to the Evolving Threat Environment

text: is a significant risk of data loss. Available material shows that basic security controls such as automatic access locks are not implemented in many existing user-owned devices. Moreover, security protocols to protect data on the move are often not implemented or managed, leading to risks from the use of unsecured channels. A further source of risk is current relative immaturity and heterogeneity of consumer device software; where data loss can arise due to vulnerabilities, lack of robustness and stability of the devices, applications and services used. Such risks are rather related to the maturity of the technology of mobile devices. All these issues turn standardized implementation of security controls to a very difficult task.

text: RD2. Potential loss of corporate data as a result of access by unknown users and unmanaged devices to enterprise networks: Opening up the security perimeter to accommodate consumerization requirements might enable network intrusion. Data loss might also occur as a result of unauthorized third-party access to the network and data when that third-party is using the device of a legitimate user. Loss of corporate data may also cause loss of private data stored on the device and lead to yet another privacy risk.

text: RD3. Potential loss of corporate data as a result of difficulty of controlling security in application-rich mobile devices, especially if employee-owned: Risks are related to the security characteristics of the consumer IT components used. Businesses will be unable to check, influence or control the characteristics of these components (e.g. security precautions of App Stores, consumerized services, location tracking, etc.). Hence, existing security controls might be compromised, especially as a result of user behaviour patterns. This might lead to loss of corporate data. In a similar manner to the previous risk, loss of corporate data might also cause loss of private data stored on the relevant components (i.e. devices, cloud storage, etc.).

text: RD4. Increased risk of mobile devices being the target of attack for the acquisition of corporate data: Due to weak security controls in consumer devices and also in the functions available on these devices (such as location tracking, private mail, appstores, etc.) there is a risk that attack vectors, such as malware, phishing, identity theft, human engineering, spoofing and eavesdropping will become far more significant. Being targeted by this kind of attacks, users may also experience collateral loss of private information and be thus exposed to privacy risks.

text: 5.4 Overview of risks

text: The table below gives an overview of each of the assessed risks. The table shows both the primary category (Costs, Legal/Regulatory or Data) into which the risk falls – indicated by X; and the secondary category or categories – indicated by (X). The reasons for a risk falling into

text: Consumerization of IT: Top Risks and Opportunities 13 Responding to the Evolving Threat Environment

text: one or more secondary categories are indicated in the right hand column. The purpose of this table is to provide cross-functional information for those interested primarily in one kind of risk who may need to appreciate the relationship between that type of risk and others. It is expected, for example, that businesses coping with privacy issues, might also be interested in risks related to data loss. Other, more cost oriented businesses might also be interested in legal-related risks.

text: Consumerization of IT: Top Risks and Opportunities 14 Responding to the Evolving Threat Environment

text: Secondary categories due to effects RC1 X (X) (X) on compliance and data loss. Secondary categories due to effects RC2 X (X) (X) on compliance and data loss. Secondary categories due to effects RC3 X (X) (X) on compliance and data loss. Secondary categories due to effects RC4 X (X) on compliance and data loss. Secondary categories costs from possible fines and costs caused by loss RLR1 (X) X (X) of reputation due to compliance violations. Secondary category as costs from RLR2 (X) X possible fines due to compliance violations. Secondary categories as costs from possible fines and costs caused by loss RLR3 (X) X of reputation due to compliance violations. Secondary category legal/regulatory from possible privacy violations. RD1 (X) (X) X Secondary category as costs caused by loss of reputation. Secondary category legal/regulatory from possible privacy violations. RD2 (X) (X) X Secondary category costs caused by loss of reputation. Secondary category legal/regulatory RD3 (X) X from possible privacy violations. Secondary category as RD4 (X) X legal/regulatory from possible privacy violations. Legend: X is primary category; (X) is secondary category. Table 2: Primary and secondary classification/dependencies of identified risks

text: Consumerization of IT: Top Risks and Opportunities 15 Responding to the Evolving Threat Environment

text: 6 Opportunity assessment

text: The ongoing COIT trend has become significant mainly as a promise towards realization of opportunities. Combined with the issue of mobility COIT has come to put in question the usability of existing IT architectures as a result of positive user experience gained from consumerized IT components. A lot of discussion is going on regarding the opportunities of COIT that go beyond the pure use of a mobile hardware. The assessed opportunities show the various areas of the organization where benefits can be implemented by a proper introduction of COIT. The assessed opportunities are as follows: O1.Potential financial opportunities: COIT has the potential to save time and money by increasing productivity, reducing spending and increasing user/customer satisfaction. Workforce will be more productive due to permanent access to business data and transactions, and communication facilities. Cost cuts can be achieved by lower spending in hardware and building infrastructure, as employees will be more mobile and will use own devices and services. This will have a positive effect on customer satisfaction with the corresponding financial benefits. Organisations that see COIT as an opportunity to create a comprehensive strategy and clear governance model will be more likely to capitalise on the financial benefits of COIT.

text: O2.Potential Human Resources benefits: COIT has been initiated due to the trend of having employees bringing their own devices to business. COIT has thus gained an important role in increasing workforce creativity and ability to find/use tools for their business tasks and maintain lifelong learning. Modern organisations should try to use this opportunity in order to increase motivation of staff, support them in becoming literate in current technologies and use them also in business life. In this way businesses may attract talented individuals and achieve a better retention of employees by offering them job satisfaction and the freedom to unfold their creativity for the benefit of the business and their customers.

text: O3.Potential operational opportunities: COIT offers benefits with regard to optimization of operational aspects of businesses. Through the increased availability of staff, urgent matters can be better coordinated and resolved (e.g. emergency response). Similarly, by taking advantage of current technologies, the increased flexibility and mobility will have positive effect on working from remote locations, like home-working and working on the move. This will increase communication and collaboration attitude of staff, while enhance the staff capacity in working with virtual teams, increase peer influence and share knowledge by using modern channels (i.e. social networking, chatting, blogging). The ability to mobilize cross-disciplinary teams on the virtual space is essential for success. Finally, COIT offers a good reason to simplify and decentralize

text: Consumerization of IT: Top Risks and Opportunities 16 Responding to the Evolving Threat Environment

text: security policy and security governance, a trend that is irreversible and will lead in the middle term to the falling of traditional security models based on static perimeter 13 14 15 security .

text: O4.Potential Data Management opportunities: data management opportunities are emerging from the architecture behind a successful COIT deployment. In order to enhance data availability, the need to use Cloud Storage becomes evident. This will help employees to increase online interaction and online data access, while using tested applications (i.e. deployed via own app server). Frequent data interactions will increase data accuracy, while the degree of data sharing will be increased. Moreover, such storage architectures will allow for a better oversight and control of data flow within the organisation.

text: Consumerization of IT: Top Risks and Opportunities 17 Responding to the Evolving Threat Environment

text: 7 Dependencies between opportunities and risks

text: Having assessed COIT risks and opportunities this section elaborates on their dependencies, that is, which risks may counteract or (in case of non-materialisation) support the implementation of opportunities. These dependencies are demonstrated by means of the following table:

text: OPPORTUNITY

text: RISK O1 O2 O3 O4 COMMENT

text: RC1 X (X)

text: RC2 X

text: RC3 X

text: RC4 X

text: RLR1 (X) X (X)

text: RLR2 (X) X (X)

text: RLR3 X (X)

text: RD1 (X) X X

text: RD2 (X) (X) X X

text: RD3 (X) (X) X X

text: RD4 (X) (X) X X

text: Consumerization of IT: Top Risks and Opportunities 18 Responding to the Evolving Threat Environment

text: 8 Conclusions, further steps

text: The identified risks and opportunities provide a coherent view on similar aspects mentioned in numerous publications on this subject matter. The aim of this publication is to provide a more complete oversight of both opportunities and risks in this area by bringing together data from a collection of other, related studies. Based on this material, ENISA plans further interactions with stakeholders to be performed this year, such as participating in discussions in relevant national and international fora, working groups, interest groups, etc. The assessed risks and opportunities serve as input to the second part of the 2012 work, namely the report on recommendations for mitigating the risks considered and materialisation of opportunities. Around the end of the year 2012, the currently assessed risks and opportunities are going to be re-assessed in order to capture further developments at that particular time.

text: P.O. Box 1309, 71001 Heraklion, Greece www.enisa.europa.eu