text: https://www.enisa.europa.eu/news/cybersecurity-threats-fast-forward-2030 4 https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206 5 https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms

text: 7 The scenario description is aligned with ENISA’s previous work on AI , and provides the following information: • The purpose and the context • A high-level description of the scenario, highlighting the data encountered • The involved actors and their associated roles • A detailed description of the data • Machine learning algorithms • Other assets (besides data) associated with the described scenario • The overall process of the scenario • Security and privacy requirements

text: See https://www.rcr.ac.uk/sites/default/files/backlog_survey_feb_2016.pdf 10 Medical practice means a business entity in which physicians practice medicine together as partners, shareholders, owners, members or employees, or in which a single physician practices medicine. Furthermore, in our scenario, this includes at least public and private hospitals that offer radiology services.

text: Actor Role Description

text: Radiologists have the historical role of analysing the results of the radiographs. They have an important role to play as they can act as verifiers.

text: End Users and Data

text: Radiologists/medical The medical staff also has the role of providing the essential data for the

text: Owner (Data

text: practice training and testing of the algorithm, as it manages the X-rays database.

text: Radiologists also have their role in producing quality x-rays and images to ensure the right diagnosis can be made.

text: Large tech companies develop libraries and pre-trained models that are Large tech companies Model Provider published to be reused by the data science community. In this case, they provide a pre-trained model.

text: The historical patients are patients who were consulted in the medical

text: Historical Patients

text: practice before the use of AI for the diagnosis of osteoporosis. The X-ray (before the occurrence Data Provider images from their consultation have been anonymized and now form the

text: of the diagnosis)

text: database used to train the model.

text: New Patients (during New patients are those patients who have come to the radiologist's office for the occurrence of the Data Provider a consultation since the AI detection of the disease became available. Their diagnosis) data are also used in the future to better train the model.

text: The Cloud Provider is a third party that offers computational platforms, data Cloud Provider Cloud Provider 11 lakes and some data analysis capabilities or “Machine learning as service”.

text: Data Scientists are the spearhead of the algorithms. They oversee cleaning Data Scientists Data Scientists the data, preparing it, building the models, and ensuring the relevance of the results. They work with the medical staff to build a relevant model.

text: Developers and Data Developers and Developers and Data Engineers are responsible for routing data, Engineers Data Engineers transformation, and other technical operations as required.

text: Inspired from the definition of ENISA, AI Cybersecurity Challenges, December 2020

text: System and System and communication Network Administrators are responsible for the

text: communication network (flows etc.) and configuring the cloud servers that make the

text: Network Administrator application work or the data stored.

text: Data Data type Source / data provider Data Procurement

text: Data used for building the model

text: The data are already available. The images X-rays of patients with Database where historical need to be labelled (i.e., analysed by or without osteoporosis patients’ radiographies are Image experts) to determine if there are signs of stored. The data provider is (pseud ) osteoporosis in the image, and measure the onymised former patients. 12 13 performance of the model .

text: Data related to age, Database where historical sex, and body mass patient information is stored. For The data are already available. For each Xindex of historical Structured each patient, their radiograph is ray, this information is stored to facilitate patients of the medical data associated with this information. the diagnosis (age, sex, BMI) . Note that cabinet The data provider is former such information is pseudonymised . (pseudonymised) patients.

text: Data used once the model is in production

text: These data are used when a These data are obtained during the X-rays of patients who patient is consulted in the Image consultation when the patient is subject to come for consultation medical office. The data the X-Ray system provider is patients.

text: These data are used when a Data related to age, patient is consulted in the These data are obtained during the sex, and body mass Structured medical office. The data consultation when the patient is subject to index of who come for data provider is the patient, but it is the radiography system consultation the radiologist who seizes this data.

text: Soffer, Shelly, et al. Convolutional Neural Networks for Radiologic Images: A Radiologist’s Guide. s.l. : RSNA, 2019. https://pubmed.ncbi.nlm.nih.gov/30694159/ Lee, Jae-Seo, et al. Osteoporosis detection in panoramic radiographs using a deep convolutional neural network-based computer-assisted diagnosis system: a preliminary study. 2018. https://pubmed.ncbi.nlm.nih.gov/30004241/ Augmenting Osteoporosis Imaging with Machine Learning. Valentina Pedoia, Francesco Caliva, Galateia Kazakia, Andrew J. Burghardt, Sharmila Majumdar. 2021 https://pubmed.ncbi.nlm.nih.gov/34741729/ See https://datasaveslives.eu/pseudonymisation#:~:text=In%20many%20cases%20this%20may,always%20suitable%20for%20 healthcare%20research.

text: Learning paradigm Subtype Algorithm Type of data ingested Description

text: A convolutional neural network (CNN) is a class of artificial neural networks mainly focused on applications like Supervised Learning Classification CNN Image object detection, image classification, recommendation systems, and are also sometimes used for natural language processing .

text: Type of asset Asset Description

text: CNN, for classification In this scenario a pre-trained model is used. The weights of the neural of images between networks are calculated on a large public database for the model to be “osteoporosis” and “no able to classify objects between thousands of classes. These pre-trained osteoporosis” models are available open-source and downloadable via the internet.

text: The last layers of this pre-trained neural network are then re-trained. The

text: model takes as inputs: images of bones with osteoporosis, and images of healthy bones with the associated labels to specialise the model in detecting osteoporosis

text: The model is a CNN where personal data such as age, sex or BMI are added just before the affine layer to improve performances.

text: The data ingestion platforms that enable data engineers to store data and Data lake – in the cloud exploit them.

text: Model server – in the The model server is used to store the model and process the training and cloud retraining phases.

text: A diagnostic X-ray machine consisting of a tomography system and a Scanner computer that provides the results in the form of images.

text: X-ray computer-aided After each X-ray, the image and the results of the model are shown to an diagnostic system – onexpert thanks to this computer and its software.

text: Integrated Development Software application that provides comprehensive development facilities Environment to computer programmers.

text: Libraries (with Collection of precompiled code that can be used in a project to realise algorithms for well-defined operations. transformation)

text: Communication protocol and communication A system of rules allowing a group of entities to communicate with each Networks other to share or ask information.

text: Rajpurkar, Pranav, et al. CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning. 2017. https://ui.adsabs.harvard.edu/abs/2017arXiv171105225R/abstract

text: Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. Taha, Allan and Aziz, Abdel. 2015. https://bmcmedimaging.biomedcentral.com/articles/10.1186/s12880-015-0068-x 18 See https://towardsdatascience.com/should-i-look-at-precision-recall-or-specificity-sensitivity-3946158aace1

text: Steps Description Actors Assets

text: Historical data of images are Data Engineer/Developers annotated by radiologist and put in a Data Scientist Data lake table in a cloud data lake with information like gender. System and Programming Libraries Data Collection Communication Network All those Information (included Integrated development Administrators images) coming from other historical environment tables in the data lake and are Radiologist, Cloud anonymised. provider

text: Data Engineer/Developers Data lake Data, notably images, are cleaned Data Scientist Integrated development

text: Data Cleaning

text: to keep only the high-quality data. environment Radiologist Programming Libraries Cloud provider

text: The pre-processing phase consists Programming Libraries of standardising the size of all the Data Engineer/Developers panoramic radiographies. Indeed, Data lake, Model servers Data pre-processing Data Scientist the images are subsampled to a Integrated development uniform size of pixels using bilinear Cloud provider environment interpolation.

text: Data Scientist A pre-trained CNN is used for this Programming Libraries scenario. At the end of the CNN, an Large Company Model design and Data lake, Model servers affine layer is added to use the implementation Cloud Provider personal information of the patient Integrated development to evaluate his bone’s health. environment Large tech company

text: Programming Libraries Data Scientist The model is then trained on the Data lake, Model servers

text: Model training

text: model servers. Cloud provider Integrated development environment

text: One way to monitor is to supervise the new database with radiologist and see the diagnostic errors of the model.

text: Programming Libraries 10% of the most recent data are Data Scientist Data lake, Model servers

text: Model testing

text: used for testing. Cloud provider Integrated development environment

text: Programming Libraries The hyperparameters of the model Data Scientist Data lake, Model servers are optimised through different Optimization techniques such as Early stopping. Cloud provider Integrated development environment

text: Model is evaluated by some classification metrics like recall or Programming Libraries precision f-test. Radiologists prefer Data Scientist Data lake, Model servers to detect osteoporosis even if there Model evaluation is no osteoporosis – so emphasis Cloud provider Integrated development will be placed on reducing the environment number of false negatives.

text: Data Engineer/Developers Model server The model is put into production on Radiologist Computer aided diagnostic dedicated cloud servers and System and system incorporated in the computer aided communication network’s

text: Model deployment

text: diagnostic system. administrator Integrated development environment Cloud Provider

text: Data Scientist Data lake Radiologist make their diagnostic Radiologist Model server using the help of probability score return by the model. Data like image Patients Computer aided diagnostic are stored in a database in case of system System and Monitoring and re-training. Patients’ files are store communication network’s Scanner inference into a secure database in case of administrator trial. Integrated development Cloud Provider environment

text: Level Explanation

text: The model set up is an aid for the Radiologist. The system can be Availability Low unavailable for a few days if a radiologist is present to manually analyse the radiographs.

text: Data, whether used for training or made by the model, must be accurate Integrity Critical at all steps of the lifecycle. Alteration could lead to an erroneous prediction and thus errors in the detection and treatment of the disease.

text: Part of the processed data are medical data and are considered as Confidentiality Critical personal data. It is therefore confidential. It must be processed in such a way that it complies with the GDPR.

text: Actions linked to the process must at least be logged (traced and dated) to be able to follow up who did what on the algorithm. As personal data Traceability High are used by the process, actions linked must be imputable. I.e., we must be sure to trace all actions (even consultation actions) and to be able to attribute them without possibility of repudiation.

text: Health data are special categories of personal data as highlighted by Article 9 of the GDPR. Personal data collected (both in training and in production) in the purpose of this scenario must be processed lawfully. That is, the processing must be based on one of the legal bases provided by Article 6 of GDPR. In addition, individuals' personal data must be processed only in the way they reasonably expect, and any unexpected but justified processing must be clearly explained. Data processing must also comply with the transparency obligations of the right to be informed. Lawfulness: One of the main actors is the medical practice which is a private medical actor. The medical practice must justify that the processing of these data is necessary to safeguard its legitimate interest . As noted in the introduction, the processing of this data accelerates and improves the ability of the medical practice to detect osteoporosis. Indeed, radiologists make their diagnosis more quickly and no longer give patients very long appointment times. The data of the patients who come for consultation are stored in a different database than the one built for training the models. These data are kept for consultation in case of

text: legal disputes. The processing of this data is also of vital interest to patients, as the time needed to find an

text: appointment for a consultation is considerably reduced and the consultation is done much more quickly. The

text: patient knows at an early stage if he suffers from the disease and can take a treatment accordingly. In addition, user consent may be required not by the GDPR but by the public health code of the country in which the scenario applies . Fairness: even if the medical practice demonstrates that the data processing has a legal basis, it must only process personal data in the way that patients expect, i.e., only for the detection of osteoporosis, and not use it in a way that has an unjustified negative effect on them. Transparency: General information concerning the activities related to the scenario must be provided to the persons concerned (posting in the premises, mention in the reception booklet, etc.). In addition to this general information, individual information must be provided to the patient. This should be done by the radiologist for each patient during the first interviews with them. The practice must be transparent with patients and clearly inform them how and why their personal data are used throughout the machine learning lifecycle. Personal data collected (both in training and in production) need to be collected for specific, explicit, and Purpose legitimate purposes and is not further processed in a manner that is incompatible with those purposes. This limitation means that the medical data collected and treated should only be used to assist the radiologist in his osteoporosis diagnosis.

text: Health data are handled and are special categories of personal data whose processing is prohibited unless there are exceptions, but in the context of our scenario, we fall into one of these two exceptions:

text: article 9.2

text: .h of the GDPR: See https://gdpr-info.eu/art-9-gdpr/ See https://www.cnil.fr/fr/recherches-dans-le-domaine-de-la-sante-ce-qui-change-avec-les-nouvelles-methodologies-dereference

text: Personal data collected (both in training and in production) need to be adequate, relevant, and limited to Data what is necessary in relation to the purpose of detecting osteoporosis. The medical practice should justify minimisation and should indicate the procedures in place to ensure that only essential data such as X-rays are manipulated. Personal data collected (both in training and in production) must be accurate and up to date. The medical practice must ensure that the data manipulated remains accurate. Indeed, the radiographs manipulated by

text: the radiologists must be adequately annotated and verified by the other practitioners in the practice. No mistakes should be made and if there are any, they should quickly be detected and corrected. Personal data collected (both in training and in production) must be kept in a form which permits Storage authentication of data subjects for no longer than is necessary for the purposes for which the personal data limitation are processed. Specifically, data retention periods will be applied in accordance with the requirements of the local medical laws of countries and supervisory authorities’ recommendations where such scenario occurs. Personal data collected (both in training and in production) must be processed in a manner that ensures

text: appropriate security, including protection against unauthorised or unlawful processing etc. In addition, the

text: risks from data corruption and leakage are critical for patients. The practice must implement adequate

text: security measures to ensure data integrity and confidentiality (as there is a critical need for confidentiality and Confidentiality) integrity due to the nature of the data).

text: After personal data of the consulted patients is collected, they are stored in the medical practice database. This data is kept for consultation in case of legal disputes. Therefore, the medical practice Database creation must ensure that only authorised persons have access to the database and prevent data loss. The medical practice must also ensure that only the right data (real radiographs) are injected into the database. Compliance of the The medical practice must justify that the training model it has chosen to achieve its objectives is training model relevant and unbiased. The practice must ensure that before the system is put into production, it is not (i.e., before discriminatory and that radiologists can verify the effectiveness of the algorithm's decisions to reduce the potential risk of diagnostic errors.

text: Criteria Does it match the criteria? Justification

text: Evaluation or scoring

text: Automated decision making with legal or similar significant effect

text: Systematic monitoring

text: See https://www.cnil.fr/fr/intelligence-artificielle/la-cnil-publie-ressources-grand-public-professionnels See https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulationgdpr/accountability-and-governance/data-protection-impactassessments/#:~:text=DPIAs%20are%20a%20legal%20requirement,trust%20and%20engagement%20with%20indivi duals The condition for a DPIA to occur is that at least two conditions are satisfied.

text: Medical data is manipulated in addition Sensitive data or data of highly personal to the names, age, and sex of the X nature patients. This data is also stored in databases without being anonymised.

text: If the medical practice has thousands of Data processed on a large scale X patients, personal data is used on a large scale.

text: Matching or combining datasets

text: The data is from the patients. Patients can be considered vulnerable because Data concerning vulnerable data subjects X they can potentially have an illness and live in a state of stress, willing to do anything to get better.

text: Innovative use or applying new Prediction of osteoporosis with artificial X technological or organizational solutions intelligence is an innovative solution.

text: Preventing data subjects from exercising a right or using a service or contract

text: Level Explanation

text: The data, for the user, does not necessarily have to be available. An

text: Availability Low

text: unavailability of several days is acceptable.

text: The consequences of poor analysis caused by lack of data integrity Integrity Critical can be very serious. Alterations could lead to erroneous prediction, therefore errors in disease detection and treatment.

text: Part of the processed data are medical data and are considered as special categories of personal data. It is therefore confidential. It must be processed in such a way that it complies with the GDPR. This data must be kept and protected in a way that it does not leak or get stolen.

text: Confidentiality Critical

text: Only authorised persons should have access to this data. Since these manipulated data are personal data of highly sensitive nature, unauthorised persons could identify patients and make their illnesses public, which may have serious negative impact on the patients.

text: At a minimum, the actions related to the process must be recorded (traced and dated) to be able to follow the actors involved during the Traceability High stages of use of personal data. Particularly in the case of court cases, it is necessary to be able to attribute actions to an individual and to be able to impute them criminally.

text: Figure 14: Summary of threats and vulnerabilities

text: See the severity scales in the Annex

text: Vulnerabilities Threats Actors Assets Involved

text: Unlawful processing

text: Unfair processing

text: Lack of transparency Absence of an identified Diversion of purpose Medical practice Data

text: data controller

text: No respect of data minimisation

text: No respect of storage limitation

text: Compromise of diagnostic Contract with a low security system components Medical practice N/A

text: third party

text: Data disclosure

text: Disclosure of sensitive data

text: Data disclosure Data scientists Model

text: for ML algorithm training

text: Large tech companies Model

text: Existing biases in the ML

text: Diversion of purpose model or in the data Data scientists Data

text: Unlawful processing

text: Unfair processing Medical practice

text: Lack of transparency Data scientists Lack of auditability of Diversion of purpose Developers and data processing engineers No respect of data N/A minimisation System and communication network administrators No respect of storage limitation

text: Data scientists Data Lack of accuracy criteria No respect of accuracy Developers and Data Model Engineers

text: Lack of documentation Human error Medical Practice All assets

text: Unlawful processing Data scientists

text: Unfair processing Developers and Data Engineers Lack of transparency System and communication Diversion of purpose network administrators No respect of data minimisation

text: No respect of storage limitation

text: Diversion of purpose Medical practice Lack of pseudonymisation Data No respect of data Data Engineers minimisation

text: Denial of service due to Model inconsistent data Data lake Poisoning Model server Data disclosure Medical practice Scanner Unlawful processing Large tech companies Lack of consideration of X-ray computer-aided attacks to which diagnostic Unfair processing Data Scientists diagnostic system

text: systems could be exposed

text: Lack of transparency Developers and Data Integrated Development Diversion of purpose Engineers Environment

text: No respect of data Libraries minimisation Communication protocols and Label modification communication networks

text: Data scientists

text: Lack of consideration of Developers and Data real-life conditions in Diversion of purpose Engineers Model

text: training the model

text: System and communication network administrators

text: Lack of control for

text: Poisoning Data scientists Model

text: Lack of control of Data Diversion of purpose Medical practice processor (including Data external stakeholder) Data disclosure Cloud provider

text: Unfair processing Medical practice

text: Lack of control over model

text: Data performance No respect of accuracy Cloud provider

text: X-ray computer-aided

text: Lack of controls to ensure

text: Diversion of purpose Medical practice diagnostic system

text: that data is used only for

text: the purposes defined Data lake

text: Scanner Lack of controls to ensure Data scientists X-ray computer-aided that the data collected are No respect of data diagnostic system minimal for the purposes minimisation Developers and Data intended Engineers Data lake

text: Unlawful processing

text: Lack of controls to ensure

text: the adequacy of the Unfair processing Medical practice N/A

text: purpose and its current use

text: Lack of transparency

text: Lack of data deletion No respect of storage N/A Data lake mechanisms limitation

text: Lack of data for increasing

text: Poisoning N/A Data

text: robustness to poisoning

text: Lack of data retention No respect of storage Medical practice Data policy limitation

text: Unlawful processing

text: Lack of detail on the

text: purposes and justification Unfair processing Medical practice N/A

text: for their legitimacy

text: Lack of transparency

text: Data Scientists Data

text: Lack of detection of

text: Evasion Developers and Data Data lake

text: abnormal inputs

text: Engineers Model

text: Lack of justification and

text: traceability of decisions Diversion of purpose Medical practice Model

text: Lack of justification for the

text: No respect of data collection of individual Medical practice Data minimisation

text: personal data collected

text: Lack of measures to

text: No respect of data prevent further data Medical practice Data minimisation

text: Medical practice Lack of necessary data No respect of data Data scientists Data selection minimisation Data Engineers

text: Lack of practical means Unlawful processing Medical practice

text: and justification for the

text: Unfair processing Historical Patients All the assets

text: legal basis (legitimate

text: interest) Lack of transparency New Patients

text: Compromise diagnostic system components

text: Evasion Lack of security by design Medical practice All assets Human error

text: Data disclosure

text: Unlawful processing

text: Unfair processing

text: Lack of transparency

text: Diversion of purpose Lack of privacy by design Medical practice All assets No respect of data minimisation

text: No respect of accuracy

text: No respect of storage limitation

text: Unlawful processing

text: Lack of review of treatment

text: by a dedicated committee Unfair processing Medical practice N/A

text: to check fairness

text: Lack of transparency

text: Data lake

text: Model server

text: Scanner Lack of security process to Compromise of diagnostic maintain a good security X-ray computer-aided system components Medical practice level of the components of diagnostic system the diagnostic system Data disclosure Integrated Development Environment

text: Communication protocols and communication networks

text: Data lake

text: Model server

text: Lack of traceability of

text: No respect of accuracy actions and/or X-ray computer-aided modifications made to the No respect of storage N/A diagnostic system assets on which rely limitation personal data Integrated Development Environment Libraries

text: Lack of training based on

text: Evasion Data scientists Data

text: adversarial attacks

text: Lack of transparency on the purpose, the exact data

text: Diversion of purpose Medical practice N/A

text: that are collected, and how they are processed.

text: Lack of verification that the

text: data is adequate, relevant, No respect of data and not excessive for the Medical practice N/A minimisation

text: purpose of making a diagnostic

text: Medical practice

text: Data scientists Model easy to poison Poisoning Model Developers and data engineers

text: No detection of poisoned Data scientists samples in the training Poisoning Data dataset Data Engineers

text: Poor consideration of evasion attacks in the

text: Evasion Data scientists Model

text: model design implementation

text: Data lake

text: Model server

text: Medical Practice Scanner Poor access rights Compromise of diagnostic management process system components X-ray computer-aided Cloud provider diagnostic system Integrated Development Environment

text: Data disclosure Diversion of purpose Data scientist Poor data management Data No respect of data Developers and data minimisation engineers

text: No respect of storage limitation

text: Medical practice Too much information Compromise of diagnostic Large tech companies Model available on the model system components Data scientists

text: Medical practice

text: Data scientists Unprotected sensitive data Developers and Data Data disclosure Data on test environments Engineers

text: Medical practice Use of uncontrolled data Poisoning Data Data scientists

text: Use of unreliable sources

text: Label modification Medical practice Data

text: to label data

text: Medical practice

text: Use of unsafe data or

text: models (e.g., with transfer Poisoning Large tech companies Model

text: Historical Patients

text: Data lake

text: Model server

text: Scanner Use of vulnerable X-ray computer-aided Compromise of diagnostic components (Among the N/A diagnostic system system components

text: whole supply chain)

text: Integrated Development Environment

text: Large tech companies

text: Using of a widely known

text: Data Scientists model allowing the attacker Evasion Model to study it Developers and Data Engineers

text: Data lake Weak access protection Compromise of diagnostic mechanisms for machine system components Model server

text: learning model

text: Data disclosure Medical practice Integrated Development

text: components and for

text: personal data (encryption, Environment Poisoning

text: access control

text: Type Associated Vulnerabilities Threats it mitigates

text: Type Associated Vulnerabilities Threats it mitigates • Human error • Unlawful processing • Unfair processing Cybersecurity Lack of documentation on the diagnostic system • Lack of transparency & Privacy • Diversion of purpose • No respect of data minimization No respect of storage limitation

text: Type Associated Vulnerabilities Threats it mitigates • Lack of security process to maintain a good security level of the components of • Compromise of diagnostic system Cybersecurity the diagnostic system components & Privacy • Use of vulnerable components • Data disclosure • Disclosure of sensitive data for ML algorithm training

text: Type Associated Vulnerabilities Threats it mitigates Cybersecurity Lack of training based on adversarial attacks Evasion

text: Type Associated Vulnerabilities Threats it mitigates • No detection of poisoned samples in Cybersecurity the training dataset Poisoning • Lack of control for poisoning

text: Type Associated Vulnerabilities Threats it mitigates • Compromise of ML application components Cybersecurity Poor access rights management • Poisoning & Privacy • Data disclosure • Human error

text: Type Associated Vulnerabilities Threats it mitigates • Weak access protection mechanisms for • Compromise of diagnostic system machine learning model components components Cybersecurity Weak access protection mechanisms of the • Poisoning & Privacy personal data (encryption, access control • Data disclosure mechanism…) • Human error

text: Type Associated Vulnerabilities Threats it mitigates • Unfair processing Cybersecurity Lack of control over model performance • No respect of accuracy

text: https://www.enisa.europa.eu/news/enisa-news/tips-for-secure-user-authentication 27 See https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-63-3.pdf

text: Type Associated Vulnerabilities Threats it mitigates • Too much information available on Cybersecurity the model Compromise of diagnostic system & Privacy • Using a widely known model components allowing the attacker to study it

text: Type Associated Vulnerabilities Threats it mitigates • Unlawful processing Privacy Absence of an identified Data Controller • Unfair processing • Lack of transparency

text: Type Associated Vulnerabilities Threats it mitigates • Lack of pseudonymisation Privacy • Unprotected sensitive data on test No respect of data minimisation environments

text: Type Associated Vulnerabilities Threats it mitigates • Lack of auditability of processing • Unlawful processing • Lack of traceability of actions and/or • Unfair processing modifications made to the assets on • Lack of transparency Privacy • Diversion of purpose which rely personal data • No respect of data minimisation • Lack of explanation and traceability of • No respect of storage limitation decisions taken

text: Type Associated Vulnerabilities Threats it mitigates • Lack of control of Data processor • Diversion of purpose Privacy (including external stakeholder) • Data disclosure • Contract with a low security third party

text: Type Associated Vulnerabilities Threats it mitigates • Lack of controls to ensure that data is used only for the purposes defined • Lack of controls to ensure the adequacy Privacy Diversion of purpose of the purpose and its current use • Lack of detail on the purposes and justification for their legitimacy

text: Type Associated Vulnerabilities Threats it mitigates • Lack of data deletion mechanisms Privacy No respect of storage limitation • Lack of data retention policy

text: Type Associated Vulnerabilities Threats it mitigates • Lack of justification for the collection of individual personal data collected • Unlawful processing Privacy • Lack of transparency on the purpose, the • Lack of transparency exact personal and medical data that are • No respect of data minimisation extracted, and how they are processed.

text: Type Associated Vulnerabilities Threats it mitigates • Lack of legal basis related to legitimate interest Privacy Unlawful processing • Lack of practical means and justification for the legal basis

text: Type Associated Vulnerabilities Threats it mitigates • Lack of necessary data selection • Lack of verification that the data is adequate, relevant, and not excessive for Privacy No respect of data minimisation the purpose of delivering a diagnostic • Lack of measures to prevent further data collection

text: Type Associated Vulnerabilities Threats it mitigates • Unlawful processing • Unfair processing Privacy Lack of Privacy by Design • Lack of transparency • Diversion of purpose • No respect of data minimisation

text: Type Associated Vulnerabilities Threats it mitigates Lack of review of treatment by a dedicated Privacy Unfair processing committee to check fairness

text: Type Associated Vulnerabilities Threats it mitigates Cybersecurity Lack of consideration of real-life condition in Failure or malfunction of ML application & Privacy training the model

text: Type Associated Vulnerabilities Threats it mitigates • Denial of service due to inconsistent data • Poisoning • Data disclosure • Unlawful processing Cybersecurity Lack of consideration of attacks to which • Unfair processing & privacy diagnostic systems could be exposed • Lack of transparency • Diversion of purpose • No respect of data minimisation • Label modification The Medical Practice must ensure that such threats are comprehended by the teams. For instance, the company must ensure that all its teams (including Data Scientists) are trained in the specificities of machine learning and the new associated cybersecurity and privacy risks.

text: Type Associated Vulnerabilities Threats it mitigates Cybersecurity Use of unreliable sources to label data Label modification

text: Type Associated Vulnerabilities Threats it mitigates Privacy Existing biases in the ML model or in the data Unfair processing

text: Control name and Privacy and security Associated Vulnerabilities Threat mitigated type requirements addressed

text: Based on « Agence Nationale de la Securité des Systèmes d’Information » (ANSSI). See: https://www.ssi.gouv.fr/uploads/2019/11/anssi-guide-ebios_risk_manager-en-v1.0.pdf 30 Metrics based on the National Commission on Informatics and Liberty (CNIL) -an independent French Administrative regulatory body. See: https://www.cnil.fr/sites/default/files/atoms/files/cnil-pia-3-enknowledgebases.pdf

text: Such as the CNIL. It proposes an analysis grid to enable organizations to assess the maturity of their artificial intelligence systems regarding the GDPR available on https://www.cnil.fr/fr/intelligence-artificielle/guide. 32 see the various CNIL method sheets on this subject, such as this one: https://www.cnil.fr/fr/intelligenceartificielle/guide/developper-et-entrainer-un-algorithme

identifier: Cybersecurity and privacy in AI - Medical imaging diagnosis

publisher: Europeiska unionens cybersäkerhetsbyrå

sprak: en

title: Cybersecurity and privacy in AI - Medical imaging diagnosis

serie: rapporter

text: ABOUT ENISA

type: rubrik

text: The European Union Agency for Cybersecurity, ENISA, is the Union’s agency dedicated to achieving a high common level of cybersecurity across Europe. Established in 2004 and strengthened by the EU Cybersecurity Act, the European Union Agency for Cybersecurity contributes to EU cyber policy, enhances the trustworthiness of ICT products, services and processes with cybersecurity certification schemes, cooperates with Member States and EU bodies, and helps Europe prepare for the cyber challenges of tomorrow. Through knowledge sharing, capacity building and awareness raising, the Agency works together with its key stakeholders to strengthen trust in the connected economy, to boost resilience of the Union’s infrastructure, and, ultimately, to keep Europe’s society and citizens digitally secure. More information about ENISA and its work can be found here: www.enisa.europa.eu.

type: stycke

text: CONTACT

text: For contacting the authors, please use info@enisa.europa.eu For media enquiries about this paper, please use press@enisa.europa.eu.

text: EDITORS

text: Monika Adamczyk, Apostolos Malatras, Ioannis Agrafiotis, ENISA

text: LEGAL NOTICE

text: This publication represents the views and interpretations of ENISA, unless stated otherwise. It does not endorse a regulatory obligation of ENISA or of ENISA bodies pursuant to the

text: Regulation (EU) No 2019/881

text: ENISA has the right to alter, update or remove the publication or any of its contents. It is intended for information purposes only and it must be accessible free of charge. All references to it or its use as a whole or partially must contain ENISA as its source.

text: Third-party sources are quoted as appropriate. ENISA is not responsible or liable for the content of the external sources including external websites referenced in this publication.

text: Neither ENISA nor any person acting on its behalf is responsible for the use that might be made of the information contained in this publication.

text: ENISA maintains its intellectual property rights in relation to this publication.

text: COPYRIGHT NOTICE

text: © European Union Agency for Cybersecurity (ENISA), 2023 Reproduction is authorised provided the source is acknowledged.

text: This publication is licenced under CC-BY 4.0. Unless otherwise noted, the reuse of this document is authorised under the Creative Commons Attribution 4.0 International (CC-BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/). This means that reuse is allowed, provided that appropriate credit is given and any changes are indicated.

text: ISBN: 978-92-9204-641-5 – DOI: 10.2824/25285 – Catalogue Nr: TP-03-23-362-EN-N

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS

text: TABLE OF CONTENTS

text: 1. INTRODUCTION 6

text: 1.1 STUDY OBJECTIVES 6

text: 1.2 METHODOLOGY 6

text: 1.2.1 Description of the scenario 7 1.2.2 Identification of cybersecurity and privacy threats and vulnerabilities 7 1.2.3 Identification of cybersecurity and privacy controls 7

text: 1.3 TARGET AUDIENCE 7

text: 1.4 USING THIS DOCUMENT 8

text: 2. SCENARIO DESCRIPTION 9

text: 2.1 PURPOSE AND CONTEXT 10

text: 2.2 HIGH-LEVEL DESCRIPTION 10

text: 2.3 ACTORS AND ROLES 12

text: 2.4 PROCESSED DATA 13

text: 2.5 MACHINE LEARNING ALGORITHMS 13

text: 2.6 ASSETS 14

text: 2.7 OVERALL PROCESS 14

text: 2.8 PRIVACY AND CYBERSECURITY REQUIREMENTS 18

text: 3. SECURITY AND PRIVACY THREATS AND VULNERABILITIES 22

text: 3.1 THREAT CONTEXTUALISATION 22

text: 3.1.1 Compromise of diagnostic system components 23 3.1.2 Evasion 24 3.1.3 Human error 24 3.1.4 Data disclosure 24 3.1.5 Poisoning (by label modification) 24 3.1.6 Unlawful Processing 25 3.1.7 Unfair processing 25 3.1.8 Lack of transparency 25 3.1.9 Diversion of purpose 25 3.1.10 No respect of data minimisation 25 3.1.11 No respect of accuracy 25 3.1.12 No respect of storage limitation 26 3.1.13 No respect of compliance of the training model 26

text: 3.1.14 Synthesis of possible impacts and associated threats 26

text: 3.2 VULNERABILITIES ASSOCIATED TO THREATS AND AFFECTED ASSETS 27

text: 4. CYBERSECURITY AND PRIVACY CONTROLS 32

text: 4.1 IMPLEMENT A SECURITY BY DESIGN PROCESS 33

text: 4.2 DOCUMENT THE DIAGNOSTIC SYSTEM 33

text: 4.3 CHECK THE VULNERABILITIES OF THE COMPONENTS USED AND IMPLEMENT PROCESSES

text: TO MAINTAIN SECURITY LEVELS OF ML COMPONENTS OVER TIME 34

text: 4.4 ADD SOME ADVERSARIAL EXAMPLES TO THE DATASET 34

text: 4.5 CHOOSE AND DEFINE A MORE RESILIENT MODEL DESIGN 35

text: 4.6 INTEGRATE POISONING CONTROL IN THE TRAINING DATASET 35

text: 4.7 ENLARGE THE TRAINING DATASET 35

text: 4.8 SECURE THE TRANSIT OF THE COLLECTED DATA 35

text: 4.9 CONTROL ALL DATA USED BY THE ML MODEL 36

text: 4.10 IMPLEMENT ACCESS RIGHT MANAGEMENT PROCESS 36

text: 4.11 ENSURE ALL SYSTEMS AND DEVICES COMPLY WITH AUTHENTICATION, AND ACCESS

text: CONTROL POLICIES 37

text: 4.12 MONITOR THE PERFORMANCE OF THE MODEL 37

text: 4.13 REDUCE THE AVAILABLE INFORMATION ABOUT THE MODEL 38

text: 4.14 IDENTIFY A DATA CONTROLLER FOR THE MEDICAL DATA PROCESSING 38

text: 4.15 PSEUDONYMISE DATA COMING FROM THE HISTORICAL PATIENT 39

text: 4.16 GENERATE LOGS AND PERFORM INTERNAL AUDIT 39

text: 4.17 IDENTIFY ALL THE DATA PROCESSORS FOR THE MEDICAL DATA PROCESSING AND PERFORM THE

text: CONTROL ACTIONS NECESSARY TO GIVE REASONABLE ASSURANCE THAT THEY ARE COMPLIANT 40

text: 4.18 PERFORM A PRIVACY IMPACT ASSESSMENT 40

text: 4.19 DEFINE AND IMPLEMENT A DATA RETENTION POLICY 41

text: 4.20 STUDY ON DATA FIELDS NECESSITY AND JUSTIFICATION IN THE PRIVACY POLICY 41

text: 4.21 FORMALIZE A LIA (LEGITIMATE INTEREST ASSESSMENT) 41

text: 4.22 MINIMISE DATA AT EACH STEP OF THE PROCESSING; COLLECT ONLY WHAT IS NEEDED

text: WHEN NEEDED 42

text: 4.23 IMPLEMENT A PRIVACY BY DESIGN PROCESS 42

text: 4.24 CALL ON ETHICAL COMMITTEE AND EXTERNAL AUDITS 43

text: 4.25 DEFINE ACCURACY CRITERIA 43

text: 4.26 ENSURE THAT THE MODEL IS SUFFICIENTLY RESILIENT TO THE ENVIRONMENT

text: IN WHICH IT WILL OPERATE 43

text: 4.27 RAISE AWARENESS OF SECURITY AND PRIVACY ISSUES AMONG ALL STAKEHOLDERS 44

text: 4.28 USE RELIABLE SOURCES TO LABEL DATA 44

text: 4.29 ENSURE THAT MODELS ARE UNBIASED 44

text: 4.30 SUMMARY 45

text: 5. CONCLUSION 51 A ANNEX: SECURITY AND PRIVACY SCALES AND REQUIREMENTS 52

text: A.1 CYBERSECURITY AND PRIVACY SEVERITY SCALES 52

text: A.2 CYBERSECURITY SCALE OF IMPACT 53

text: A.3 PRIVACY SCALE OF IMPACT 53

text: A.4 PRIVACY REQUIREMENTS CRITERIA 54

text: EXECUTIVE SUMMARY

text: Given the great influence of artificial intelligence (AI) in people's daily lives due to the key role it plays in digital transformation through its automated decision-making capabilities, ENISA aims to raise awareness of cybersecurity and privacy threats related to various scenarios using artificial intelligence. To this end, ENISA, with the support of the Ad-Hoc Working Group on Artificial Intelligence Cybersecurity, has published two reports in the last two years: 1 2 Cybersecurity Challenges of Artificial Intelligence and Securing Machine Learning Algorithms .

text: ENISA continues its momentum with a new report on cybersecurity and privacy in medical imaging diagnosis, which is supported by AI. An in-depth study of the scenario has been conducted by identifying first the assets, the actors and their roles, relevant processes, the AI algorithms used, as well as the requirements in terms of cybersecurity and privacy needed for it. Building upon previous ENISA work such as the “Securing Machine Learning Algorithms” report cited above, in addition to legislation such as GDPR and literature searches, this report has identified cybersecurity and privacy threats and vulnerabilities that can be exploited in the examined scenario. While focus is on ML-related threats and vulnerabilities, broader AI considerations were also taken into account. Lastly, corresponding cybersecurity and privacy controls that consider the context of the scenario and the impact of the associated threats/vulnerabilities were defined. The specificities within the implementation of these controls are described, including possible trade-offs between cybersecurity, privacy, and performance. Each control is classified as a cybersecurity control, a privacy control, or a mixture of both, depending on the threats it mitigates and their associated impacts (cybersecurity impacts, privacy impacts, or both).

text: This report allows better assessment of the reality that artificial intelligence brings its own set of threats, which consequently insists on the search for new security measures to counter them. Finally, it should be noted that this guide strongly emphasises privacy issues in the same way as cybersecurity issues, privacy being one of the most important challenges facing society today. Security and privacy are intimately related, but both equally important, and a balance must be made specific to each use. As a result, as seen in this report, efforts to optimise security and privacy can often come at the expense of system performance.

text: See https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges See https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms

text: 1. INTRODUCTION

text: Abuse of Artificial Intelligence (AI), which has developed significantly in recent years, has been identified by ENISA as one of the top emerging threats . By providing new opportunities to solve decision-making problems intelligently and automatically, AI is being applied to more and more business cases in a growing number of sectors. The associated benefits are significant. However, the development of AI is also accompanied by new threats which project teams will have to face.

text: In many projects, it is apparent that these new threats are related to aspects of cybersecurity in addition to privacy, particularly when AI is used for innovative projects whereby processing personal data takes place. In this spirit, on 21 April 2021, the Commission published the AI Act proposal , which sets out requirements (including cybersecurity and privacy) for AI systems deemed to be high-risk (e.g., used in biometric identification or critical infrastructure management and operation) in order to mitigate threats to health, safety, and fundamental rights.

text: To go further into these high-risk AI systems, ENISA proposes this new report: "Cybersecurity and Privacy in AI – Medical Imaging Diagnosis", which builds directly on the work already initiated by ENISA since 2020 on the identification of risks associated with AI. This new report

text: analyses cybersecurity and privacy requirements and measures in use of AI in medical imaging diagnosis of osteoporosis. The report describes the scenario fundamental principles (assets, actors processes etc.), identifies the security and privacy risks it poses, and finally cybersecurity and privacy controls, which counteract the identified risks.

text: 1.1 STUDY OBJECTIVES

text: Findings from the ENISA’s report on securing machine learning algorithms indicate that there is no uniform strategy in applying a specific set of security controls to protect machine learning algorithms and in some cases, deployed security controls may result in trade-offs in security and performance. ENISA therefore recommends that organisations which use AI systems should perform detailed analysis of their own AI systems, and conduct targeted risk assessments to find the appropriate balance between cybersecurity, privacy and performance.

text: The objectives of this publication are then as follows:

text: • Provide a detailed description of a medical imaging diagnosis scenario • Identify AI cybersecurity and privacy measures taking into account requirements, threats and vulnerabilities defined for this scenario and practical guidance on how to implement them • Provide recommendations on how to balance the trade-offs between cybersecurity, privacy, and performance in this scenario

text: 1.2 METHODOLOGY

text: Production of this publication was undertaken in three stages:

text: • Identify and describe in detail the medical imaging diagnosis scenario.

text: • Identify cybersecurity and privacy threats associated with the scenario • Identify relevant cybersecurity and privacy controls

text: 1.2.1 Description of the scenario 1.2.2 Identification of cybersecurity and privacy threats and vulnerabilities

text: This section of the report focusses on the threats and vulnerabilities related to use of AI in medical imaging diagnosis scenario. Based on previously mentioned ENISA work, legislation such as GDPR, and desk research, this report identified the cybersecurity and privacy threats and vulnerabilities that can be exploited. Given the prevalence of machine learning (ML) and ENISA’s past work on the topic, there is more emphasis placed on threats and vulnerabilities related to ML. Nonetheless, wider considerations of AI have been taking into account when identifying security threats and vulnerabilities as well as dedicated to privacy, for which GDPR data protection principles were used as a starting point of our analysis.

text: 1.2.3 Identification of cybersecurity and privacy controls

text: Following the analysis of medical imaging diagnosis scenario, the identification of threats (and their impact), and associated vulnerabilities, this section of the report presents the corresponding cybersecurity and privacy controls that:

text: • Take into account the context of the described scenario • Take into account the security and privacy impact of the threat/vulnerabilities (as described in this report)

text: The specificities of implementation of such controls are described including the possible tradeoffs between cybersecurity, privacy and performance. Each control can be either a cybersecurity control, a privacy control, or a mixture of both, depending on the threats it mitigates and their associated impacts (cybersecurity impacts, privacy impacts, or both). Many of the controls are of technical nature, but when appropriate (e.g., per GDPR requirements), organizational controls have also been identified.

text: 1.3 TARGET AUDIENCE

text: The target audience of this report can be divided into the following categories:

text: • All actors (private or public): to help them in their risk analysis, in the identification of cybersecurity and privacy threats and in the identification of the appropriate security and privacy controls to mitigate the threats related to this scenario.

text: See https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms See https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges See Article 5 of GDPR

text: • AI technical community, AI cybersecurity and privacy experts and AI experts (designers, developers, ML experts, data scientists, etc.) with an interest in developing secure solutions and in integrating security and privacy by design in their solutions. • Cybersecurity and privacy community: to help in identifying cybersecurity and privacy threats related to medical imaging diagnosis and in identifying the appropriate security and privacy controls to mitigate the threats.

text: 1.4 USING THIS DOCUMENT

text: Although we based the scenario close to reality, some assumptions have been to made for the purposes of its analysis. These assumptions should be reassessed by the reader when doing their own analysis. Please note this list is non-exhaustive:

text: • The algorithms chosen for the scenario are based on desk research, and there may be other algorithms that are better suited for it. • Regarding the privacy aspects, we put ourselves in the shoes of the data controller, but only based on the hypothesis that we have made. It is left to the attention of the audience that the privacy requirements as well as the control measures must always be adapted to a context and a situation. • After identifying potential threats, this report identifies security and privacy controls that could be applied to the scenario. However, as is the case for any application of ML, one must also consider traditional security standards (e.g., ISO 27001/2, NIST) because ML applications are also subject to more global threats.

text: It should also be kept in mind that the elements of this report are valid as of the date of publication, and could evolve over time.

text: 2. SCENARIO DESCRIPTION

text: The following figure presents an overview of all topics that will be addressed in this chapter. Figure 1: Scenario overview

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 2.1 PURPOSE AND CONTEXT

text: Osteoporosis is a major bone disease characterised by a reduction in bone mineral density (BMD) and deterioration of bone architecture, leading to an increased risk of fracture, that causes significant morbidity and mortality worldwide. The diagnosis of the disease is mainly through X-ray scans.

text: However, analysis of such scans can be very time consuming. For instance, according to a 2017 report from Royal College of Radiologists , 230,000 patients waited more than a month for X-rays results in the UK NHS in 2016. This is largely due to a large increase in the number of Xrays combined with a plateauing number of radiologists. This resulted in an increase in X-ray analysis time.

text: For this reason, a medical practice specialised in radiology has defined the following purposes: • Reduce the time it takes for a radiologist to analyse an X-ray scan for osteoporosis suspicion • Improve the diagnosis of osteoporosis

text: To reach its objectives, the medical practice seeks to assist radiologists by providing a probability score of presence of osteoporosis in an X-ray scanner. In practical terms, for each patient, this is done with the following three steps: 1. The patient has an X-ray scan with a scanner 2. This X-ray scan is sent back to the radiologist for analysis. Meanwhile, the image is sent to a tool that calculates the probability that the image presented has osteoporosis 3. The radiologist analyses the image together with the probability that the patient has osteoporosis. This probability does not replace the analysis, but it supports the practitioner in their decision, or highlights areas of concern

text: 2.2 HIGH-LEVEL DESCRIPTION

text: As described above, the final goal is to detect the potential presence of osteoporosis by giving the radiologist a probability that the bone contains the disease. To do so, a model of supervised learning is particularly adapted.

text: Technically, the machine learning model can assist the radiologist in the following way:

text: • The scanner sends the image to a computer with a dedicated software to which it is connected by wire. This computer is called X-ray computer-aided diagnostic system. • The radiologist enters the patient's personal information including first name, last name, age, sex, and body mass index (BMI) into the X-ray computer-aided diagnostic system. • The system sends, via API calls, the image, age, sex, and BMI to a cloud server provided by a cloud provider which hosts the machine learning model. • The model returns its diagnosis (probability that the bone contains the disease) to the X-ray computer-aided diagnostic system. • The radiologist makes his diagnosis using the X-ray image, their expertise, and the probability returned by the model.

text: The patient's file (including first name, image, model probability, age, BMI) is then sent to a database in a cloud data lake to be stored there for a pre-defined period. in case of a trial, this period allows, by the relevant prescribing law, audit of the diagnosis made by the radiologist. The images, age, sex, BMI, and model probability (after they have been pseudonymised) are

text: sent to another database, which is accessible to the data scientists and radiologist in charge of the project to improve the model by providing new data.

text: To build this machine learning model, a database containing radiographic images of former patients from the medical practice has been constructed. This database contains both radiographs of people suffering from the disease and other radiographs of healthy people who do not suffer from any other disease. As noted, all images are from a historical library of images from the medical practice, and the models that are trained to ensure the disease detection are housed in the cloud. It should be added that information about the sex, age, and body mass index of the patient from whom the radiograph was taken is associated with this radiograph. This means that for each radiograph, the information on age, sex, and BMI is visible.

text: The major actors involved in this scenario – outside the team of data scientists and data engineers, are the medical staff (in which the key stakeholder is the radiologist) and the patients. The radiologists' expertise is used initially to annotate as many images as possible and to confirm the presence or absence of osteoporosis in each annotated image (it is these annotated images that form the database described above).

text: It is essential to note that this scenario refers to two types of patients. The first group is used to build the image libraries (i.e., historical osteoporosis patients or otherwise healthy). The second group are the future patients who will come to the medical office for a consultation. Although the data manipulated in our case are the same (X-rays of patients), they come from different sources – one source of historical patients whose data is part of the training database, and another source of new patients who require diagnosis:

text: • X-ray images and medical data of historical patients (before the occurrence of the diagnosis) that form the database (these ones are annotated). • X-ray images and medical data of the new patients (during the occurrence of the diagnosis) who come for consultation.

text: It should also be noted that a pre-trained machine learning model is used. Training an accurate neural network is costly - large tech companies train heavy model on very large databases, sometimes with billions of datapoints – as such, most medical practices cannot afford to train their own model. Hence, pre-trained models are made publicly available. These only require a small amount of re-training with the osteoporosis database to transfer the knowledge of the neural network to this scenario (this practice is referred to as Transfer Learning). An additional explanation on why transfer learning is used is provided later in this report, in the model deployment section for this report. Security risks associated with transfer learning are described in detail in subsequent sections of this report.

text: The following sections aim to detail such scenario by describing actors and their roles, processed data, the assets that allow the project to exist, the explanation of the lifecycle, and the privacy and cybersecurity requirements applied to it.

text: Figure 2: High-level description

text: 2.3 ACTORS AND ROLES

text: The following actors are involved in the medical imaging diagnosis scenario.

text: Figure 3: Actors, roles, and their description

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 2.4 PROCESSED DATA

text: As previously mentioned, data in this scenario comes from two sources and its use is detailed in the following figure.

text: Figure 4: Data needed for detecting osteoporosis from panoramic X-ray

text: 2.5 MACHINE LEARNING ALGORITHMS

text: The objective of using machine learning in this scenario is to detect osteoporosis from panoramic X-rays. The ML model used for this purpose is a Convolutional Neural Network (CNN). Besides the fact that this algorithm is one of the best models used for computer vision, its choice is also justified by the fact that it is widely used for scenarios very close to the one at hand, e.g., chest, breast, brain, musculoskeletal system, and abdomen and pelvis X-rays. CNN systems are widely used in the medical classification task. CNN is an excellent feature extractor, therefore utilising it to classify medical images can avoid complicated and expensive feature engineering.

text: Figure 5: Machine learning algorithms used

text: 2.6 ASSETS

text: In addition to the previously described data, the scenario is supported by the following additional assets, which are described in the figure below.

text: Figure 6: asset’s description

text: Models

text: Environment tools premises

text: 2.7 OVERALL PROCESS

text: The purpose of the scenario is predicting the presence or absence of osteoporosis in a patient based on medical imaging data. To do this, Data Engineers, Data Scientists and Radiologists work closely together to enable the system to learn how to detect this bone disease.

text: In the following paragraphs, our scenario is described by incorporating it into the different stages of the machine learning lifecycle:

text: Data collection

text: The first step is to determine how to obtain the data to feed the model. For this, the medical practice already has the necessary medical imaging data (a database of historical patients with or without the disease, with their age, sex, and BMI) which must be annotated. As a pretrained algorithm is used, a relatively small amount of data (reasonably for a medical practice) is needed to re-train the last layers of the neural networks. This is where the expertise of the radiologists is used. Their first role is to annotate many images and for each of them to confirm (or reject) the presence of osteoporosis. Once annotated, these data are routed and stored in a cloud data lake provided by a Cloud Provider and maintained by a Data Engineer and System/Network Administrator. The data thus obtained are images, each with a class describing the state of the bones. The amount of data obtained being small to train a model with satisfactory performance, it is more convenient to use a neural network pre-trained on medical or non-medical imaging to enforce the performance of the model.

text: Data cleaning and data pre-processing

text: The collected data are then cleaned, ensuring that the data is of high quality and structure, by Data Scientists for pre-processing. The pre-processing phase consists of standardising the size of all the panoramic radiographs. For efficient processing in terms of computation and memory requirements, the images are subsampled to a uniform size using bilinear interpolation.

text: Model design and implementation

text: As mentioned earlier, the algorithm used in our scenario is CNN, where personal data such as age, sex or BMI are added just before the affine (fully connected) layer to improve performance. Details on the choice of the algorithm can be found in an earlier section of this report.

text: One aspect that is important regarding the performance of the model is to be sure that both “normal” and “osteoporosis” images are equally represented. It could be necessary to that extend to proceed to data augmentation, transforming real osteoporosis X-rays to create derivative images and increase the number of examples for this label.

text: To build the neural network, a pre-trained CNN is used. Pre-trained CNN are publicly made available by large tech companies, and are trained on billions of images, requiring a huge amount of computing power. For radiologists to benefit from Deep Learning without the huge number of resources needed for training and fine-tuning the model, pre-trained models are essential. According to the amount of labelled data available, one or more layers of the network can be re-trained to adapt better to the use it is destined to. Most of the pre-trained CNN architectures available are proven to produce human-level accuracy of medical imaging diagnosis . At the end of the CNN, an affine layer is added to use the personal information of the patient to evaluate their bone’s health.

text: Figure 7: Synthesis of the CNN structure

text: The input of the network is the grey level of the ROI (Region of Interest) in a defined pixel area and the personal data of the patient (age, sex, BMI, …), and the output of the network is the conditional probability distribution over the two categories, "osteoporosis" and "normal". The network consists of a stack of several convolutional layers, in addition to maximum pooling layers and activation layers.

text: The topmost layer (or output layer) is regularly a SoftMax classifier, that converts a Kdimensional vector of arbitrary real values into a K-dimensional vector of real values in the range (0 ,1) that sum to 1.

text: Model training, model testing and optimisation

text: Regarding the training method phase (if the available data are sufficient and if it is considered as necessary regarding the performance metrics), the training is done in the cloud servers always provided by the same cloud provider. The dataset is divided into training, validation, and test sets. 80% of the X-rays are randomly selected for training, 10% for validation, and the remaining 10% are reserved for testing. Randomness of the split is made with caution: it is important to respect the osteoporosis images ratio among all the sets.

text: Model Evaluation

text: To evaluate the model, regular classification metrics are used: accuracy, recall, precision fscore and AUC score . In fact, radiologists may prefer to detect osteoporosis even if there is no osteoporosis rather than missing this diagnosis – so emphasis is placed on reducing the number of false negatives (i.e., augmenting recall). Even if the false positive rate must be low, a lot of misdiagnoses could be made, incurring high cost.

text: Model Deployment

text: The model is deployed in the cloud by the Developers and the System Network Administrator.

text: As described in the high-level description, the model is called by APIs by the computer aided diagnostic system when required. This is usually as the radiologist uses the scanner with a patient.

text: Monitoring and inference

text: The neural network by itself is not sufficient to make a diagnosis. Verification by an expert remains necessary, but the expert's work is considerably facilitated by the presence of machine learning. The verification of the medical image by a radiologist allows for an expert evaluation. However, automation bias (when the radiologist overly trusts the results and gradually loses his own skill) is a problem that should be taken into account in the medium term.

text: For every patient, the images, age, BMI, model probability and the radiologist’s diagnostic are sent to a specific database in the cloud data lake (different from the one with the historical data). This is accessible to the Data Scientists and Radiologist in charge of the project to improve the model by providing new data when necessary for retraining the model. Performance of the model is continually monitored to keep accuracy above the criteria defined by experts. Once performance drops below that criterion, the model is retrained.

text: Eventually, every patient's file (for instance name, first name, image, model probability, age, BMI) is sent to a database (different from the two above) in the cloud data lake to allow audit of the diagnosis made by the radiologist in case of trial.

text: Figure 8: Synthesis of the ML lifecycle and involved actors

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 2.8 PRIVACY AND CYBERSECURITY REQUIREMENTS

text: Cybersecurity requirements

text: The context information given in the previous section enables evaluation of the application’s cybersecurity and privacy requirements. The following table summarises the cybersecurity requirements.

text: Figure 9: Cybersecurity requirements

text: Privacy requirements

text: On one hand, for the model training, X-rays of medical patients are used. In addition, data indicating the age, sex and BMI of the patients is associated with each radiograph. However, these data are pseudonymised in the medical practice database for the purpose of training machine learning models as outlined as the purpose of this scenario.

text: On the other hand, personal data are processed when patients come to the practice and

text: are diagnosed for osteoporosis, adding information like last name, name, and

text: consultation date in the patient’s file.

text: Because the scenario manipulates personal data, the following privacy requirements and recommendations should be satisfied.

text: Figure 10: GDPR Data protection principles

text: Requirements Explanation

text: Lawfulness,

text: fairness, and

text: transparency

text: Accuracy

text: Security of

text: personal data

text: (Integrity and

text: As a complement to the GDPR data protection principles listed above, other key privacy topics related to AI systems need to be addressed (some of which are specifically mentioned by national data protection supervisory authorities) :

text: Figure 11: Data protection supervisory authorities’ recommendations for AI systems

text: Recommendations Explanation

text: production)

text: Finally, the project requires a data protection impact assessment as it completes at least one criterion in the following table :

text: Figure 12: Is a Data Protection Impact Assessment (DPIA) necessary?

text: To assess the security risks in terms of privacy, personal data processing must consider the following requirements:

text: Figure 13: Privacy requirements regarding data

text: 3. SECURITY AND PRIVACY THREATS AND VULNERABILITIES

text: 3.1 THREAT CONTEXTUALISATION

text: This section highlights the threats applicable in medical imaging scenario. Examples in the context of this scenario are also provided. Also outlined are the impacts that each threat can have in terms of security and privacy as well as their severity in the table.

text: There are two major feared events to consider for this scenario which are the loss of integrity of the data, and the loss of confidentiality of the personal data of the patients.

text: The loss of integrity of the data could lead to reputation degradation and lawsuit to the company and physical and permanent injury for the patient. Indeed, the impact would be serious in both case:

text: • On one hand, a patient suffering from osteoporosis could be wrongly diagnosed and treated incorrectly, • On the other hand, a patient not suffering from osteoporosis could be wrongly diagnosed and treated for the wrong illness.

text: It should be noted, however, that the impact is lessened because the model only assists and does not make decisions directly. When making a diagnosis, decision-makers (the radiologist in this case) take into account not only the ML output but also their own medical knowledge and experience.

text: For the medical practice, the loss of confidentiality of personal data could lead to reputation degradation and a lawsuit, that can result in patients losing trust. For the patients, the loss of confidentiality of personal data could have several impacts. First, the data stolen (X-rays of patients with their name and age for instance) could be used to perform Phishing attempts, targeted advertising. Then, Unique targeted opportunities might not be given like real estate loan, studies, or jobs due to the medical condition. For example, companies could be informed of a patient's state of health and therefore refuse certain requests (e.g., insurance). This could lead to a patient feeling a significant invasion of privacy or discrimination against them. A lack of regard for privacy requirements by the medical practice (such as in the case of unfair processing) could lead to discrimination in treatment, resulting in better diagnosis of osteoporosis for men than for women, for example. The impact on the user could lead to

text: infringement of fundamental rights.

text: Considering these feared events and associated impacts, the following threats are associated with this scenario.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 3.1.1 Compromise of diagnostic system components

text: An attacker could attack the exposed APIs and/or human interfaces during the entire scenario lifecycle. They could, for example, gain access to an account to infiltrate the data lake or the model server. This could be done via brute force password cracking in the case of simple authentication, via a vulnerability exploitation, poor security controls implemented by the cloud provider, credential stuffing, or other password-based attacks. Once access is gained, actions made by the attacker could lead to loss of data integrity (including model parameters) or to loss of data confidentiality (by data leaks). For the medical practice, it could lead to reputation degradation and lawsuit. For the patient, it could lead to a significant feeling of invasion of privacy, phishing attempts, or targeted advertising or the loss of unique targeted

text: opportunities.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 3.1.2 Evasion

text: Direct evasion attacks are possible, but only from an attacker compromising one of the components. An attacker could modify the image capturing devices (i.e., the scanner or the Xray software) to add additional noise to an image, which could trick the model into making incorrect diagnosis. The noise added to the image can be imperceptible for humans and could only influence the output score of the ML program. This could increase the potential diagnostic error of the radiologist if he only relies on the program output. In the long run, the impact could be a misdiagnosis and lead to reputation degradation, lawsuit, and physical and permanent injury for the patients.

text: 3.1.3 Human error

text: A network administrator could incorrectly expose (such as by making public) certain instances of the databases, which would raise the risk profile of the 3 databases. This could leave personal data susceptible to attack, should the database storing personal data and patient files be targeted by attackers. This, in turn, could lead to reputation degradation and lawsuit for the medical practice, significant feeling of invasion of privacy, phishing attempts, or targeted advertising or the loss of unique targeted opportunities for patients.

text: There is a risk that Radiologists may not be careful when handling patient data by bypassing the tool they have been provided with and filling in other media (text files), the creation of which would have very low visibility (shadow IT). This could open a wide, untraceable area of attack and malicious persons could try and get access to the data stored in unprotected/unrestricted applications. This could make personal data leakage easier, leading to reputation degradation for the medical practice, significant feeling of invasion of privacy, phishing attempts, or targeted advertising or even the loss of unique targeted opportunities for patients.

text: 3.1.4 Data disclosure

text: Not all data have the same level of sensitivity. On one hand, there is data that is used by the model (when it is trained, and once in production) which is pseudonymised. It contains only the X-Ray scans, BMI, gender, and age. On the other hand, the data used for the traceability of actions, i.e., the patient's file, is very sensitive because it contains all personal information. This sensitive data is manipulated by the computer-aided system and stored in a database in the data lake.

text: Several scenarios could be conceived:

text: • An attacker could take advantage of the poor exposure of the patient's file database to extract data from it (by usurping an account for example, or by exploiting a security flaw). • An attacker manages, by various means such as phishing, to install spyware on the computer aided system and thus harvest patient data. • The cloud provider does not offer sufficient security guarantees and so an attacker gains access to the data centre and copies the data from the databases because they are not protected (by encryption mechanisms for example)

text: Those 3 examples would cause harm and lead to reputation degradation and lawsuit for the medical practice, serious feeling of invasion of privacy, phishing attempts, targeted advertising or even the loss of unique targeted opportunities for the patients.

text: 3.1.5 Poisoning (by label modification)

text: During the data collection phase, a malicious radiologist could annotate the data of the scanner and force the model to make certain targeted predictions, or to underperform. His motivations could be to discriminate against a certain ethnic, racial, or sexual group, fear of being replaced by AI or financial motivation because he was approached by a competing medical practice.

text: Such actions cause loss of data integrity and lead to reputation degradation and lawsuit for the medical practice and physical and permanent injury for the patient.

text: 3.1.6 Unlawful Processing

text: The data processing associated with AI accelerates and improves the medical practice's ability to detect osteoporosis and accommodate more patients. The legal basis for the data processing is therefore the legitimate interest of the medical practice. However, the practice may not be able to justify that the processing of the data (or even the use of artificial intelligence) is necessary to safeguard these legitimate interests. In this scenario, patients may question the purpose of the use of their personal data and feel a significant feeling of invasion of privacy and a lack of trust in the medical practice.

text: 3.1.7 Unfair processing

text: Even if the medical practice demonstrates that the data processing has a legal basis, it could process the personal data in a way that is unexpected by patients, i.e., instead of detecting osteoporosis equitably for all patients, it could use the data in a way that has an unjustified negative effect on some patients with discriminations created by the treatment such as better diagnosis of osteoporosis for men than for women, for example. Another possible unfair usage of the processing is the case where the radiologist would trust the algorithm too much and spend too little time to examine the radio by himself. This would effectively break the deal made with the patient that the diagnosis is performed by an expert, and assisted by AI, where in reality the diagnosis would mostly be decided by the AI. The impact on the user could lead to a feeling of infringement of fundamental rights.

text: 3.1.8 Lack of transparency

text: The practice may also not be transparent with patients about how their personal data is used and the purpose for which their data is used. The terms governing these aspects may not be included in the information provided to the persons concerned. In addition to general information, individual information may not be provided to patients by radiographers, for example before the start of consultations. The type of information that might not be given to patients is that their data is stored in a database and could be used in case of a lawsuit between a patient and the medical practice. The impact this could have on patients could result in

text: significant feeling of invasion of privacy.

text: 3.1.9 Diversion of purpose

text: The patient data extracted in this scenario is intended only to help diagnostic osteoporosis. However, the medical practice might not respect this principle and use the patient data for ambiguous and non-explicit purposes, even unrelated to the diagnostic of a disease. Indeed, it may be tempting to use these data to offer other services or to resell them to insurers, or for instance, to offer targeted advertisements for medicines. The other impact this could have on patients could result in a significant feeling of invasion of privacy and a loss of unique

text: targeted opportunities.

text: 3.1.10 No respect of data minimisation

text: The medical practice could collect data from the patients’ body other than those solely related to the treatment of osteoporosis. This data could be collected to build other databases for the treatment of other diseases not related to osteoporosis. The impact this could have on patients could lead to a significant feeling of invasion of privacy.

text: 3.1.11 No respect of accuracy

text: Collection of inaccurate data, both in training and in production, can lead to unreliable model behaviour. Inaccurate data could be the result of human error during the data collection phase or by malicious persons who corrupt the data stored in the databases leading a loss of integrity of data. This could lead to temporary or permanent physical injury of the patient.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 3.1.12 No respect of storage limitation

text: This threat takes the form of storing personal data in the medical practice database without relevant data suppression rules. As a reminder, there are three databases our case; one for training data to build the model, another to re-train the model, and a last one to store the patients’ files in case of lawsuit. The Medical Practice may not comply with data retention periods governed by the requirements of local country laws and the recommendations of regulatory authorities. In this case, private data of the patients involved could be exposed. Such a data breach could lead to targeted advertising or the loss of unique targeted

text: opportunities.

text: 3.1.13 No respect of compliance of the training model

text: Medical practices and Data Scientists could develop models not adapted to the problem. The company could develop models that are biased (depending on gender for example). This can be due to insufficiently representative data, malicious acts, or a lack of a testing procedure. Nevertheless, this would lead to reputation degradation and lawsuit. For the patient, it could lead to a significant feeling of invasion of privacy, phishing attempts, or targeted advertising or the loss of unique targeted opportunities.

text: 3.1.14 Synthesis of possible impacts and associated threats

text: The following table sums up the severity of each impact and the associated threats.

text: Figure 1: Synthesis of possible impacts and associated threats

text: Impact Severity Type Associated Threats

text: Compromise of diagnostic system components Evasion Physical and permanent injury Cybersecurity High Poisoning and harm and Privacy No respect of accuracy No respect of compliance of the training model Compromise of diagnostic system components Evasion Human error Lawsuit High Cybersecurity Data disclosure Poisoning No respect of compliance of the training model Compromise of diagnostic system components Evasion Human error Reputation degradation High Cybersecurity Data disclosure Poisoning No respect of compliance of the training model Diversion of purpose Phishing attempts, targeted No respect of storage limitation High Privacy advertising No respect of compliance of the training model Loss of unique targeted Diversion of purpose High Privacy opportunities No respect of storage limitation

text: No respect of compliance of the training model

text: Unlawful processing Diversion of purpose

text: Significant feeling of invasion

text: Moderate Privacy No respect of data minimisation

text: of privacy

text: Unfair processing

text: Feeling of infringement of

text: Moderate Privacy No respect of compliance of the training

text: fundamental rights

text: model

text: 3.2 VULNERABILITIES ASSOCIATED TO THREATS AND AFFECTED ASSETS

text: The table below cross evaluates each threat to a set of associated vulnerabilities. The actors involved and the assets possibly affected by the vulnerabilities are also highlighted:

text: Figure 2: Mapping vulnerabilities to threats and assets/actors on which they rely

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS

text: 4. CYBERSECURITY AND PRIVACY CONTROLS

text: Before expanding on the details of the cybersecurity and privacy controls applied to the scenario, the following figure summarizes all the controls that will be described. Figure 17: Summary of cybersecurity and privacy controls

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 4.1 IMPLEMENT A SECURITY BY DESIGN PROCESS

text: • Compromise of diagnostic system components Cybersecurity Lack of Security by Design • Evasion • Human error • Data disclosure • Poisoning

text: By default, Security by Design is a methodology to strengthen the cybersecurity of the organisation by automating its data security controls and developing a robust IT infrastructure. This approach focuses on implementing the security protocols from the basic building blocks of the entire IT infrastructure design. The goal of such a methodology is to ensure the risks are mitigated before systems go live, and appropriate security controls are implemented.

text: The lack of Security by Design increases the likelihood of all threats related to the scenario. Therefore, the medical practice must ensure, from the development phase, to put in place adequate controls to limit the cybersecurity risk. This starts with a global risk analysis in which all risks associated with all assets are identified. The medical practice will try to reduce the attack surface (for example by auditing its cloud provider on a regular basis), to apply the principle of least privilege (by implementing access rights management), to take care of the confidentiality and integrity of the collected data (by encrypting the patients’ data for example).

text: Conceiving a project following a Security be Design methodology requires less effort than adding security on top of an existing project. However, there could be an impact on the functionality of the scenario if the outcome of the Security by Design methodology would lead to some functionality not being implemented due to the security risk it generates (the installation of unsecure scanners for example).

text: 4.2 DOCUMENT THE DIAGNOSTIC SYSTEM

text: Project and system documentation must be produced to preserve knowledge on the choices made during the project phase, the application architecture, its configuration, its maintenance, how to maintain its effectiveness over time and the assumptions made about the model use.

text: The system is complex with multiple assets (Scanner, X-ray computer-aided diagnostic system, etc…) which enhance the fact that an exhaustive documentation is mandatory, for each of those assets. This documentation should also include the changes that will be applied, including to the documentation throughout the algorithm's life cycle. Therefore, it is necessary that the Data Scientists, Developers and Data Engineers, and the System and Communication Network Administrators work together to create and sustain the documentation. This control does not

text: impact system performance, cybersecurity, or privacy.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 4.3 CHECK THE VULNERABILITIES OF THE COMPONENTS USED AND IMPLEMENT PROCESSES TO MAINTAIN SECURITY LEVELS OF ML COMPONENTS OVER TIME

text: For the data lake, the model server, and the X-ray computer-aided diagnostic system, it is necessary for the company to ensure that regular security audits are carried out to check that there are no vulnerabilities. Regular vulnerability scans and an automatic patch management process should also be implemented to maintain a good security level. The medical practice should also have a remediation plan that can be implemented quickly. Moreover, this plan needs to be reviewed over time.

text: For the data lake and the model server (hosted in a cloud), the company will also have to complete a security questionnaire for the cloud service provider. This questionnaire must cover all aspects of security and each answer must be justified. These answers will be attached to the contract and will have to be reviewed annually by both parties and whenever the contract is modified.

text: To perform audits of scanners, which are IOT devices, specific expertise is required. These devices are connected to the network as regular devices, but due to their scanning functionalities, they are composed of specific hardware and firmware, which can be different from classical IT systems. For the hardware part, the idea is to identify and assess the components that enhance the risk of an attacker gaining administration access to the scanner. To do so, retro engineering techniques such as direct access to electrical components can be used.

text: This control would impact the availability of the system, thus its performance, as it may

text: be audited or even updated. This could lead to unavailability of the system for several hours on a regular basis. However, as the availability need of the system is low, the impact of this unavailability is only moderate. Of greater concern is the risk that updates could cause the system to malfunction, leading to errors in the diagnostic systems which would have a major impact.

text: 4.4 ADD SOME ADVERSARIAL EXAMPLES TO THE DATASET

text: The medical practice must ensure that the model is resilient to evasion attacks to prevent misdiagnosis. To do so, inclusion of adversarial examples to the algorithm's helps enable it to be more resilient to such attacks. Depending on the application domain and ambient conditions, such training could be done continuously. Some examples would be: Adversarial Training, Ensemble Adversarial Training, Cascade Adversarial Training or Principled Adversarial Training. This control does not impact performance or privacy.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 4.5 CHOOSE AND DEFINE A MORE RESILIENT MODEL DESIGN

text: Poor consideration of evasion attacks in the model Cybersecurity Evasion design implementation

text: To avoid evasion attack, the chosen model should be robust against such attacks. For instance, defensive distillation is an adversarial training technique that adds flexibility to an algorithm’s classification process, so the model is less susceptible to exploitation. In distillation training, one model is trained to predict the output probabilities of another model that was trained on an earlier, baseline standard, to emphasise accuracy. This control does not impact

text: performance or privacy.

text: 4.6 INTEGRATE POISONING CONTROL IN THE TRAINING DATASET

text: The DNN algorithm used in this scenario can be checked for poisoning using the STRIP technique. The principle of this technique is to disturb the inputs and observe the randomness of the predictions. For example, the company could intentionally perturb the incoming input, for instance by superimposing various image patterns, and observe the randomness of predicted classes for perturbed inputs from a given deployed model. A low entropy in predicted classes violates the input-dependence property of a benign model and implies the presence of a malicious input. These tests should be performed before the production phase, during the training phase, ensuring that the models used in production are healthy. This control does not

text: 4.7 ENLARGE THE TRAINING DATASET

text: Lack of data for increasing robustness to Cybersecurity Poisoning poisoning

text: A large time frame (multiple years) of medical data can be collected and used to train the algorithm to prevent it from being vulnerable to poisoning attacks, as modifying only small amounts of medical data would have a low overall impact of the prediction. Using a large time frame (multiple years) of scanner image data could also reduce the probability that poisoning attacks on this data have an impact on the prediction.

text: In this case, privacy is negatively impacted because enlarging the data set means taking

text: even more personal data which could be stolen by an attacker. Therefore, it would enlarge the attack surface, and it is a trade-off between retaining personal data and deleting them to comply with the GDPR.

text: 4.8 SECURE THE TRANSIT OF THE COLLECTED DATA CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS

text: Cybersecurity Poor data management Poisoning

text: Considering the possible impacts of Poisoning, the transit of the collected data should be fully protected against a loss of integrity. The medical practice should ensure that the transit of such data is secure and protected against loss of integrity. To do so the company must ensure that the protocols used for the transfer are encrypted, using a secure protocol such as TLS 1.3 for example. As such, an attacker won’t be able to modify the data during its transit phase and won’t be able to poison the model through that vector. Such encryption must be end-to-end, i.e., from the scanner to the model and the data lake. This control won’t have any impact on

text: privacy or performance of the system.

text: 4.9 CONTROL ALL DATA USED BY THE ML MODEL

text: • Use of uncontrolled data Cybersecurity • Lack of detection of abnormal input Poisoning Poor data management

text: Considering the high impacts of Poisoning attacks, several means to control the data must be applied, some specific to data science and others more to business and common sense.

text: During the data collection phase and especially at the data lake level, the data must be controlled to ensure its relevance. A second check should be a more traditional data science check to eliminate data that deviates too much from the “normal”. This would greatly complicate a poisoning attack on the model because attempts using data that deviates from the “normal” would be eliminated by this check. These checks can be done several times including just before training the model. This is to protect against possible data modification in the data lake.

text: This control could add latency to the system in case of many reported anomalies on the data. But this scenario does not have a strong need for availability, and the control is important to reduce the probability of model poisoning. In our case, the checks that are applied could have an impact on privacy, because they would manipulate personal data from patients. To remediate, the systems used (software, scripts, excel sheets, etc.) to perform these checks should be regularly audited, documented, secured by design, and follow access control rules (authentication to use the system, proper access management, etc.).

text: 4.10 IMPLEMENT ACCESS RIGHT MANAGEMENT PROCESS

text: Threats related to poor access rights management, such as data disclosure, can induce high impacts for the company (lawsuits, Reputation degradation) or for the patients (temporary or permanent physical injury and harm). This security measure defines and assigns roles to users to respect the principle of least privilege and thus limits the access scenarios for processes and assets only to those for which the users have a justified reason.

text: For the scanners, the X-ray computer-aided diagnostic system, data lake, model server, integrated development environment, the company should create roles for each user group,

text: define rules, and instigate an access management process. This enables provision of proper accesses only to actors as required, and prevent unauthorised persons gaining access to these systems. In particular, the different user profiles (for instance Radiologist or Data Scientist) need to clearly define a profile that will be able to access specifically the patient's file database, as this is the most sensitive data set in this scenario.

text: However, in the case of an incident or failure, the complexity of the management of these accesses could prevent teams from gaining access to the system to correct a problem which could impact availability of the system, thus its performance.

text: 4.11 ENSURE ALL SYSTEMS AND DEVICES COMPLY WITH AUTHENTICATION, AND ACCESS CONTROL POLICIES

text: For user access in this scenario, the medical practice must ensure that proper user authentication and access control for the Scanner, X-ray computer-aided diagnostic system, data lake, Model server, and Integrated Development Environment is present. This should be managed centrally, with an authentication solution (such as Active Directory for example) linked to all assets. For sensitive assets that manipulate personal data, the medical practice should enforce multi-factor authentication for the Data Scientists and Administrators that should have access to it. The multi-factor authentication solution needs to be linked to the central authentication system.

text: For devices access, the medical practice should ensure that all device-to-device requests (particularly between the scanner and the X-ray computer-aided diagnostic system) are properly authenticated. This can be done using the OAuth 2.0 protocol for all exposed APIs with control via an API gateway. In addition, the certificates for the authorisation server keys should be managed through the company's own Public Key Infrastructure to avoid any identity theft scenario. It is also necessary to respect an authentication policy, based on the current 26 27 recommendations of bodies such as ENISA , NIST . To protect the sensitive data stored in the scanner, it is necessary to secure the scanners by ensuring that the OS version is up-todate, and that non-essential ports are closed.

text: Also, personally authenticating the employees on the system can have an impact on privacy, because personal data (identifier, mail address, etc) would be logged in these authentication systems. These authentication systems must respect the principles of Article 5 of the GDPR (this includes data minimisation etc).

text: 4.12 MONITOR THE PERFORMANCE OF THE MODEL

text: To ensure the reliability of the model, the medical practice must be sure of the intelligence of the model it has developed. To do so, data scientists and data engineers must ensure the quality of

text: the model training by selecting quality data. In addition, the model must be trained continuously and routinely evaluated. This will have an impact on the performance of the model in the

text: sense that by applying this control, the model is always efficient and reliable.

text: 4.13 REDUCE THE AVAILABLE INFORMATION ABOUT THE MODEL

text: This control lowers the likelihood of compromise of the ML application components by limiting the knowledge of a malicious user about his target and consequently making it more difficult to launch a cyberattack. It is important to understand that this security control alone is not enough, as security through obscurity does not provide sufficient protection.

text: Therefore, all documentation concerning the model itself should be protected. It should be encrypted and protected by a Data Right Management mechanism, allowing only the Data Scientists to have access through their computers. Related documentation can be stored in a dedicated file server directory whose rights are regularly reviewed and linked to the access rights management mechanisms defined in 3.2.10 and 3.2.11. This server must be encrypted by default, and located in a network zone dedicated to the company's sensitive resources. In addition, user workstations accessing this file server must be sufficiently protected through encryption of the workstations, a confidentiality filter to be placed on the screen for workstations in a nomadic situation, OS version up-to-date, closure of non-essential ports, amongst other protection mechanisms. To avoid the propagation of documents relating to the model, it is recommended to deploy a data loss protection solution.

text: This control could have an impact on privacy and it may be thought that this contradicts the principle of transparency required by the GDPR. Therefore, it is important to define documentation that can be communicated to external users (i.e., customers), explaining that their data is not directly used by the model. Moreover, this could impact the performance of the system, because in case of an incident or a failure, the complexity of the access management could prevent Data Scientists gaining access to the documentation to correct the problem and have an impact of availability of the system, thus its performance.

text: 4.14 IDENTIFY A DATA CONTROLLER FOR THE MEDICAL DATA PROCESSING

text: Identifying a Data Controller is essential to prevent impact on the company and its employees through unlawful processing, unfair processing, or lack of transparency. Self-identifying the data controller means performing accountability actions (documentation, assessments, etc.) and making sure that the energy consumption anticipation data processing is compliant with personal data protection GDPR obligations and does not infringe on privacy rights of data subjects. Self-identifying as Data Controller means also taking responsibility in case of noncompliance or adverse privacy effects on data subjects. Such a measure does not have a negative impact on this scenario. On the contrary, the control improves the privacy of users

text: without impacting the performance of the model.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 4.15 PSEUDONYMISE DATA COMING FROM THE HISTORICAL PATIENT

text: The reader is reminded that data that could identify patients such as name, surname, age, gender, and body mass index of historical patients are pseudonymised. In this case pseudonymisation requires removing the attributes (“name” and “surname”) from the data, replacing it with random IDs, and keeping the correspondence between these removed attributes and their IDs in a dedicated data base. For instance, instead of storing the patient data with the attributes [first name, surname, gender, age, BMI, X-ray image], the medical practice would store the patient data with the attributes [ID, gender, age, BMI, X-ray image]. A different database would be used to store the correspondence composed of [ID, surname, first name]. Usually, market vendor solutions allow ease and security in this process. This control

text: does not impact on security or performance.

text: 4.16 GENERATE LOGS AND PERFORM INTERNAL AUDIT

text: Generating logs and performing internal audit allows for improvement of supervision of all model assets, to better understand the decisions made in real time by the algorithm, to understand each incident, but also to audit the internal processes to question the processing carried out in the frame of the project. This reduces the likelihood of many threats in this scenario.

text: The process must be auditable in the sense that certain questions must always be answered clearly: Who accessed the data? (Especially before the pseudonymisation of the data) Why did the algorithm make a particular decision? How exactly was the data processed? The technical and organisational parts of the process must generate traces (computer logs, activity reports, etc.) so that they can be audited. These logs should be stored in a dedicated tool, such as a log management solution. The logs provided by the equipment must be signed (using their own private key) and their signature stored by the server that centralises the logs to ensure a principle of non-repudiation. In addition, at each (usually daily) backup of the log server, these backups must also be signed by the log sink to ensure once again the principle of nonrepudiation. Moreover, these logs should be analysed by a SOC and rules should be defined to detect the slightest anomaly such as a high number of calls from a scanner to the X-ray computer-aided diagnostic system, or an abnormal response. In addition, all processes must be regularly audited by the company's internal control teams to ensure compliance with the requirements. As far as privacy is concerned, these audits should at least ensure the lawfulness of the processing, its transparency, its application, the data minimisation, the respect of the storage limitation and the fairness aspect. These checks should be based in part on the logs collected.

text: This control could have an impact on privacy because the generated logs can contain personal data. This log management system should be included in the privacy impact

text: assessment scope of the system. Moreover, the logging system should be carefully implemented to avoid unnecessary overhead due to the extra operation it implies.

text: 4.17 IDENTIFY ALL THE DATA PROCESSORS FOR THE MEDICAL DATA PROCESSING AND PERFORM THE CONTROL ACTIONS NECESSARY TO GIVE REASONABLE ASSURANCE THAT THEY ARE COMPLIANT

text: As the data are manipulated and stored in resources provided by a cloud provider, the medical practice needs to control the cloud provider and its actions.

text: Contractual clauses, internal and external audits, and assessments of the processors in the diagnostic prediction processing all contribute to ensure the compliance of the implied processors.

text: The Medical Practice needs to ensure that the Cloud Provider where the model is trained, and the data stored is also compliant with the security and privacy policy. To do this, the medical practice should have privacy and cybersecurity documents completed to understand the level of security and privacy provided by the cloud provider. For example, the cloud provider will need to explain the processing of data in such documents and the means used to ensure its security. Such documents should be attached to the contract with the cloud provider and should be reviewed annually or whenever the contract is amended by either party. The impact this

text: control could have for the medical practice would be a loss of time and energy spent to formalise documents, and complete the assessments and audits.

text: 4.18 PERFORM A PRIVACY IMPACT ASSESSMENT

text: This privacy measure allows for an in-depth analysis of the impact of the processing on the privacy of users, compliance with the GDPR, and to identify whether the associated risks are well addressed by the proposed privacy and security measures.

text: The privacy impact assessments and general accountability actions described above (in internal audit process control) help ensure that the purpose of the processing is well defined (the diagnostic of osteoporosis) and that the actual use of the data remains within the scope of this documentation. To do so, the medical practice could use the requirements described below, and asses the vulnerabilities of the implemented system to illustrate privacy threats and counter measures. Such analysis may possibly impact the performance of the scenario if it results in one of the functionalities not permitting minimisation of the data protection risks to the rights and freedoms of natural persons.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS 4.19 DEFINE AND IMPLEMENT A DATA RETENTION POLICY

text: This privacy control, in which a storage duration value must be defined for each personal data involved in the processing (scanners images, first name, surname, age, etc) and implementation, minimises the risk of keeping the data longer than is strictly necessary.

text: There are two different situations to address. First, the data used for training can be considered as research data, for which the retention period can be two years. Secondly, the data used for the patient file (which is important in case of legal disputes) for which the retention period can be twenty years. However, in terms of performance, once all that data is deleted it can no longer be used to train models, which is why this control can have an impact on the performance

text: of this scenario.

text: 4.20 STUDY ON DATA FIELDS NECESSITY AND JUSTIFICATION IN THE PRIVACY POLICY

text: A lack of justification on how data are collected correlates with a lack of transparency on the purpose of the processing and on the accuracy of the data collected., This could result in unlawful processing which would have a moderate impact for the patient (significant sense of invasion of privacy). Therefore, the medical practice must provide rationale for the necessity of data fields and justify this in the privacy policy.

text: The personal data used are the age, gender, and body mass index, X-ray images of the patient. The medical practice must be able to explain why it is necessary to use these data specifically for the purpose of this scenario and it must also formalise the explanation and make it available to the patient.

text: Beyond the justification on the use of the data, clear explanation is also very important in the context of this scenario. The medical practice must therefore ensure they are able to explain very clearly why the model it created made the decisions it did (in this case, the detection or not of osteoporosis).

text: Nevertheless, the only data that can be collected is the one with proper justification, purpose and with clear consent of the patient. This practice may have some impact on the relevance of the data collected. Taking the time to justify the need for the data may lead to the abandonment of the collection of certain data for which justification is difficult, which may in turn reduce the quality of the data at hand.

text: 4.21 FORMALIZE A LIA (LEGITIMATE INTEREST ASSESSMENT) CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS

text: • Lack of transparency on the purpose, the exact data that are collected, and how they are processed

text: The Legitimate Interest Impact Assessment (LIA) is used to determine if an organisation can process data using the legitimate interest lawful basis. In this scenario, the legal basis for the use of personal data and images is legitimate interest. If the medical practice does not justify this legal basis, it faces unlawful processing. To justify the legal basis, the medical practice must put in place a LIA. In this LIA, the medical practice must formalise the reflection on how the data processing is necessary, and how it balances with the rights and freedoms of patients.

text: The impact this could have on the medical practice would be a loss of time and energy spent on formalising the LIA.

text: 4.22 MINIMISE DATA AT EACH STEP OF THE PROCESSING; COLLECT ONLY WHAT IS NEEDED WHEN NEEDED

text: Minimisation requires an analysis to correctly identify the subset of fields required, and should be performed as early as possible in the processing. Non-minimisation of data is the lack of selection of necessary, adequate, and relevant data. In this scenario the medical practice gathers data such as age, name, body weight and X-ray images of patients. A study on the necessity of collecting data such as age and body weight must be done, and a proof of its necessity made public. However, as mentioned above, data minimisation also involves data pseudonymisation. In this case, the data is pseudonymised and in the event of a security breach, it may be more difficult to perform a cybersecurity forensic analysis. This has the impact of significantly slowing down the work of those employed to determine the cause of a security issue.

text: 4.23 IMPLEMENT A PRIVACY BY DESIGN PROCESS

text: The lack of Privacy by Design can cause threats related to the privacy aspect of the scenario. Therefore, the medical practice must ensure that a compliance study and privacy risk assessment is formalised in a Privacy Impact Analysis document, and that the identified action plan is implemented before the scenario is put into operation.

text: Conceiving a project following a Privacy by Design methodology requires less effort than adding privacy on top of an existing project. However, there could be an impact on the functionality if

text: the outcome of the Privacy by Design methodology would lead to some functionality not being implemented due to the privacy risk it generates (but in our case, personal data are pseudonymised). The other impact that can be noted is related to security, as it may not be possible to collect some data useful for forensic analysis following a cyber-attack, due to privacy limitations.

text: 4.24 CALL ON ETHICAL COMMITTEE AND EXTERNAL AUDITS

text: The lack of review of treatment by a specialised equity committee can lead to unfair treatment. The Medical Practice can find a solution by employing an ethics committee and external auditors. Therefore, in this scenario, the Medical Practice must involve independent entities to review the inputs and outputs of the processing of data to determine whether the processing is unfair. A review from other Medical Practice and Radiologist using the system with their own images could be a proper way to challenge the model and check his fairness.

text: The impact it could have on the Medical Practice in implementing this measure is the cost of external auditors or the ethics committee.

text: 4.25 DEFINE ACCURACY CRITERIA

text: Privacy Lack of accuracy criteria No respect of accuracy

text: In addition to tracking the correct predictions made by the model (i.e., does the model correctly diagnose a patient in most situations) it is important to assess the accuracy of these predictions. Models provide, in addition to a predictive output, the probability of this prediction which can be interpreted as an “accuracy level”. For each prediction, accuracy criteria (probability score) should be stored to help a Data Scientist to better understand the models and analyse errors.

text: In production, it is recommended to follow these probability scores as they are obtained and to set up indicators of good functioning and alerts. For example, an alert should be raised if several low probability scores are noted for the same patient. With this alert, the data scientists should investigate to find the cause of the poor performance of the model. Such a measure does not affect this scenario. It may require a little time from the Data Scientist teams, but it allows further improvement in the performance of the model.

text: 4.26 ENSURE THAT THE MODEL IS SUFFICIENTLY RESILIENT TO THE ENVIRONMENT IN WHICH IT WILL OPERATE

text: The Medical Practice and Data Scientist teams need to work together to ensure that they build the most resilient model possible. This involves several steps:

text: • Using data from real-life conditions to build the model, i.e., direct patient data. • Testing the model against real-world behaviour to evaluate it • Testing the model under real-life conditions before deploying it with a relevant set of scenarios to test (e.g., a wide variety of patients...)

text: • Investigate and test the impossibility of using the model for purposes other than those intended

text: This control is a compromise between performance and privacy/feasibility. Having as much data as possible from real-life conditions, with different types of scanners, requires a lot of personal data and is not always feasible depending on the equipment available at the time in the medical practice.

text: 4.27 RAISE AWARENESS OF SECURITY AND PRIVACY ISSUES AMONG ALL STAKEHOLDERS

text: This can take the form of regular training sessions or local cyber and privacy risks reporting. This allows the teams to be aware of the issues at stake, and sometimes to find solutions adapted to the specificities of the scenario. For Radiologists, who are not normally trained in computer privacy and security issues, it is necessary to train them in basic computer hygiene rules (handling sensitive data, etc.) and how to respect privacy of personal data they handle.

text: This control does not directly impact performance of the system.

text: 4.28 USE RELIABLE SOURCES TO LABEL DATA

text: In this scenario radiologists label the data, and a label modification could lead to a misdiagnosis, which would have a high impact on the Medical Practice and on the patient. To prevent label modification and misdiagnosis, the medical practice must ensure that this operation is done correctly. To make sure that the model is correctly trained and delivers proper diagnostics with a good accuracy level, images provided for training and testing must be correctly labelled. If not, the model could output incorrect diagnoses with a high accuracy score.

text: The Medical Practice should be responsible for ensuring the labelling of the images used for training is conducted by a reliable Radiologist. Radiologists must be trustworthy, have a proven track record of excellent diagnosis and have the time to do it well. It is also necessary to train and assist them in this task to limit human error in the entry of the diagnosis. This control will

text: have a positive impact on accuracy since the dataset will be correctly labelled.

text: 4.29 ENSURE THAT MODELS ARE UNBIASED CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS

text: It is necessary for the Medical Practice to take various measures to prevent bias in the models. A biased model can cause discrimination, such as gender discrimination, where the model is more effective in producing diagnoses for one sex over another.

text: This can be done by subjecting the models to different techniques (classification parity, calibration, anti-classification, or sample bias, for instance) as well as by monitoring different indicators (e.g., percentage error of the model for people with disabilities compared to the average error) to create non-discriminatory models by preventing potential biases. In addition, the Medical Practice must provide the patients with a means of reporting an incident to allow fair use of the AI by avoiding such a difficulty (re-training the model, for example). This privacy control could improve performance of the system over longer periods, as unbiased models are more accurate.

text: 4.30 SUMMARY

text: The following table summarises every control described in the previous section. Each control is associated with vulnerabilities, mitigated threats, and addressed privacy and security requirements.

text: Figure 18: Summary of controls and mitigated threats

text: Compromise of diagnostic system components Integrity of the data Implement a Evasion Lack of security by Availability of the data security by design design Human error Confidentiality of the data process Traceability of the data Data disclosure

text: Poisoning

text: Integrity of the data Availability of the data Human error Confidentiality of the data Unlawful processing Traceability of the data Lawfulness of the process Unfair processing Fairness of the process Lack of Lack of transparency Transparency of the Document the documentation on process diagnostic system the diagnostic Diversion of purpose Purpose limitation of the system No respect of data process minimisation Data minimisation of the process No respect of storage Accuracy of the data limitation Storage limitation of the data Check the vulnerabilities of Lack of security Compromise of diagnostic the components Integrity of the data process to maintain system components used and a good security level Availability of the data Data disclosure implement of the components of processes to

text: maintain security the diagnostic

text: levels of ML system

text: components over Use of vulnerable time components

text: Disclosure of

text: sensitive data for ML

text: algorithm training

text: Add some Lack of training adversarial based on adversarial Evasion Accuracy of the data examples to the attacks dataset

text: Choose and Poor consideration of

text: define a more evasion attacks in Evasion Accuracy of the data resilient model the model design

text: design implementation

text: No detection of Integrate poisoned samples in poisoning control the training dataset Integrity of the data Poisoning in the training Lack of control for dataset poisoning

text: Lack of data for

text: Enlarge the increasing Integrity of the data Poisoning training dataset robustness to

text: poisoning

text: Secure the transit Poor data Integrity of the data of the collected Poisoning management data

text: Use of uncontrolled

text: data

text: Control all data Lack of detection of Integrity of the data used by the ML Poisoning abnormal inputs Model

text: Poor data

text: management

text: Compromise of diagnostic

text: system components Integrity of the data Implement access Poor access rights Availability of the data right management management Poisoning Confidentiality of the data process process Data disclosure Traceability of the data

text: Human error

text: Ensure all Weak access Compromise of diagnostic systems and protection system components Integrity of the data devices comply mechanisms for Availability of the data Poisoning with machine learning Confidentiality of the data authentication, model components Data disclosure Traceability of the data and access Weak access Human error control policies protection

text: mechanisms of the personal data

text: (encryption, access

text: control mechanism…)

text: Integrity of the data Monitor the Unfair processing Lack of control over Fairness of the process performance of model performance No respect of accuracy the model Purpose limitation of the

text: process

text: Too much

text: information available Reduce the on the model available Compromise of diagnostic Using a widely Confidentiality of data information about system components the model known model allowing the attacker

text: to study it

text: Identify a data Unlawful processing Lawfulness of the process Absence of an controller for the Fairness of the process identified data Unfair processing medical data Transparency of the controller processing Lack of transparency process

text: Lack of Pseudonymise pseudonymisation Confidentiality of data data coming from No respect of data Unprotected Data minimisation of the the historical minimization sensitive data on test process patient environments

text: Lack of auditability of

text: processing Lawfulness of the process Unlawful processing Fairness of the process Lack of traceability of Unfair processing Transparency of the actions and/or process modifications made Lack of transparency Generate Logs Purpose limitation of the to the assets on and perform Diversion of purpose process which rely personal Internal audit Data minimisation of the No respect of data data minimisation process Lack of clear Accuracy of the data No respect of storage explanations and Storage limitation of the limitation traceability of data decisions taken

text: Identify all the Integrity of the data data processors Availability of the data for the medical Lack of control of Confidentiality of the data Data processor data processing Traceability of the data and perform the (including external Diversion of purpose Lawfulness of the process stakeholder) control actions Data disclosure Fairness of the process

text: necessary to give Contract with a low Transparency of the reasonable process security third party assurance that Purpose limitation of the they are compliant process

text: Data minimisation of the process Accuracy of the data

text: Lack of controls to ensure that data is used only for the purposes defined Purpose limitation of the process Lack of controls to Perform a privacy Lawfulness of the process ensure the adequacy Impact Diversion of purpose Data minimisation of the of the purpose and Assessment process its current use Transparency of the Lack of detail on the process purposes and justification for their legitimacy

text: Lack of data deletion Define and mechanisms No respect of storage Storage limitation of the implement a data limitation data Lack of data retention policy retention policy

text: Lack of justification for the collection of individual personal Unlawful processing Lawfulness of the process data collected Study on data Data minimisation of the fields necessity Lack of transparency Lack of transparency process and justification in on the purpose, the No respect of data Transparency of the the privacy policy exact personal and minimisation process medical data that are extracted, and how they are processed.

text: Lack of legal basis related to legitimate interest Lack of practical Formalise a LIA means and (Legitimate justification for the Unlawful processing Lawfulness of the process Interest legal basis Assessment) Lack of transparency on the purpose, the exact data that are collected, and how they are processed

text: Minimise data at Lack of necessary each step of the data selection processing; No respect of data Data minimisation of the collect only what Lack of verification minimisation process is needed when that the data is needed adequate, relevant,

text: and not excessive for

text: the purpose of

text: delivering a

text: diagnostic

text: Lack of measures to

text: prevent further data

text: collection

text: Fairness of the process Unfair processing Transparency of the Implement a Lack of privacy by Lack of transparency process privacy by design design Purpose limitation of the process Diversion of purpose process

text: No respect of data Data minimisation of the

text: minimisation process

text: Lack of review of Call on ethical treatment by a committee and Unfair processing Fairness of the process dedicated committee external audits to check fairness

text: Define accuracy Lack of accuracy No respect of accuracy Accuracy of the data criteria criteria

text: Integrity of the data

text: Availability of the data

text: Confidentiality of the data

text: Ensure that the Lawfulness of the process

text: model is Fairness of the process

text: sufficiently Lack of consideration Transparency of the Failure or malfunction of ML resilient to the of real-life conditions process application environment in in training the model Purpose limitation of the

text: which it will process

text: operate Data minimisation of the

text: process

text: Accuracy of the data

text: Denial of service due to Integrity of the data inconsistent data Availability of the data Poisoning Confidentiality of the data

text: Data disclosure Traceability of the data

text: Raise awareness Lawfulness of the process Lack of consideration Label modification of security and Fairness of the process of attacks to which Unlawful processing privacy issues Transparency of the diagnostic systems among all Unfair processing process could be exposed stakeholders Purpose limitation of the Lack of transparency process

text: Diversion of purpose Data minimisation of the

text: process No respect of data Accuracy of the data minimisation

text: Use reliable Use of unreliable sources to label Label modification Accuracy of the data sources to label data data

text: Unfair processing Ensure that Existing biases in the models are ML model or in the No respect of compliance of Fairness of the process unbiased data the training model

text: 5. CONCLUSION

text: In this report, analysis of medical imaging in osteoporosis diagnosis supported by Artificial Intelligence (AI) is presented. AI is often used as an umbrella term that encompasses the technology behind many smart solutions and devices. It is a constantly evolving field where new innovations appear regularly in different areas of activity.

text: Even though AI can be very beneficial for the industries areas it applies to, it can also have quite a significant impact for security and privacy, especially when these systems have sensitive functionalities, as they are highlighted in this report. Indeed, AI comes with a wide range of privacy and security vulnerabilities causing threats with heavy impacts for organisations.

text: Regardless how AI is being used in support of business functionality, it should not be a surprise that the many identified threats are similar. However, we must be aware of the fact that

text: depending on the context of the scenario, the same threats apply differently and have

text: different levels of impact. Regarding the impact of each threat, it must be also noted that even in the same scenario every instance is unique, and a proper study must be carried out by each company using AI to maintain a proper security and privacy level.

text: This guide helps in the identification and evaluation of threats in a specific business scenario which uses AI, but it is important to remember that while the analyzed scenario is based on real life examples, it includes assumptions which may not match the business context in which other organizations would like to implement in. Therefore, the entire cybersecurity and privacy

text: context (requirements, threats, vulnerabilities, and controls) must be adapted to the

text: context and reality of the individual organization. In addition, the controls proposed in this document are not sufficient on their own, and must be complemented by the standard security measures that already exist.

text: While security and privacy are not necessarily the same, they are intimately related, and equally important. In their management, a balance must be found between the two in the sense that one must always make sure that the regulations and recommendations concerning the two aspects are always respected. Unfortunately, these two parameters are regularly at the expense of performance. It is therefore an equation with three variables, two of which respond to the need for regulation and risk, which need to be correctly balanced to achieve the desired effect.

text: A ANNEX: SECURITY AND PRIVACY SCALES AND REQUIREMENTS

text: A.1 CYBERSECURITY AND PRIVACY SEVERITY SCALES Availability Level Definition

text: Low Service provided must be restored within few days or less. Moderate Service provided must be restored within a day or less. High Service provided must be restored within half a day or less. Critical Service provided must be restored within few hours or less.

text: Integrity Level Definition

text: Low A loss of integrity in the process does not need to be identified or corrected. Any degradation in the process must be identified but not necessarily

text: Moderate

text: corrected. High Any degradation in the process must be identified and corrected. Critical No degradation in the process is tolerated at any time.

text: Confidentiality Level Definition

text: Low Process-related data can be accessed by everyone. Access to process-related data is restricted to internal staff and trusted

text: Moderate

text: partners. Access to process-related data is restricted to employees having an

text: High

text: organisation or functional link with the process. Access to process-related data is restricted to a very limited number of

text: Critical

text: individuals.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS Traceability Level Definition

text: Low The absence of traces of actions on the service provided is acceptable. Actions related to the service provided must be identified. They must be

text: Moderate

text: traced and detected. The actions related to the process and their actors must be identified and

text: High

text: dated. They must be imputable. Service provided actions must be legally enforceable and time stamped.

text: Critical

text: They must have a probative value.

text: A.2 CYBERSECURITY SCALE OF IMPACT Severity Level Definition

text: No operational impact on business performance or on the safety of people 1 - Low and property. The company/entity will overcome the situation without too much difficulty. Degradation of business performance without impact on safety of people and 2 - Moderate property. The company/entity will overcome the situation despite some difficulties (operation in degraded mode). Severe deterioration in the performance of the business, with possible significant impacts on the safety of people and property. The company/entity

text: 3 - High

text: will overcome the situation with serious difficulties (operation in very degraded mode). Inability of the company/entity to carry out all or part of its business, with possible serious impacts on the safety of people and property. The

text: 4 - Critical

text: company/entity is unlikely to overcome the situation (its survival is threatened).

text: A.3 PRIVACY SCALE OF IMPACT Severity Level Definition

text: The persons concerned affected will not be affected or may experience

text: 1 - Low

text: some inconvenience, which they overcome without difficulty. The persons concerned may experience significant inconvenience 2 - Moderate significant inconvenience, which they be able to overcome despite some difficulties. The persons concerned may experience significant consequences, which 3 - High they should be able be able to overcome, but with real and significant difficulties. The people concerned may experience significant consequences, if not

text: 4 - Critical

text: irreparable irremediable consequences, that they may not overcome.

text: CYBERSECURITY AND PRIVACY IN AI – MEDICAL IMAGING DIAGNOSIS A.4 PRIVACY REQUIREMENTS CRITERIA

text: Regarding the privacy, the applied requirements are based on the GDPR data protection principles, which are summarized in the table below.

text: Requirements Article

text: Lawfulness, fairness, and transparency Art. 5.a Purpose limitation Art. 5.b Data minimisation Art. 5.c Accuracy Art. 5.d Storage limitation Art. 5.e Security of personal data (integrity and confidentiality) Art. 5.f

text: Moreover, as a complement to the GDPR requirements listed above, some key privacy topics related to AI systems need to be addressed (some of them are specifically mentioned by national data protection supervisory authorities ). These additional privacy requirements are listed in the next table.

text: Recommendations Details

text: Projects must ensure the compliance of the data collected and injected into the database for training and in production. Moreover, they must

text: Database creation

text: ensure that only legitimate persons have access to the data in the database, prevent data loss and hide personal data. The data collected and the training model must be processed in accordance with the state of the art of Machine Learning development to guarantee control of all processes carried out.

text: Compliance of the training model

text: This consists, for instance, of good justifications for the chosen learning (i.e., before production) method, the reliability of the third-party tools used, paying attention to open-source data, conducting a meticulous training protocol based on the state of the art, and finally verifying the quality of the system once in the experimentation phase.

text: -N -EN 362 3- -2 -03 TP

text: ABOUT ENISA

text: ISBN 978-92-9204-641-5 doi: 10.2824/25285