AI in Healthcare

Many companies strive to disrupt the market with innovation. Artificial Intelligence, like the computer, the internet, and social media before it, is a society-disrupting innovation. The comparison with social media is particularly apt: it has brought many benefits to society but has also negatively affected how we interact with one another. Unfortunately, on social media, factual sources and well-founded arguments often carry less weight than whoever shouts the loudest.

Popular AI chatbots, such as ChatGPT, use our online language as training data. It is impressive what these AI chatbots can generate in response to our questions, but remember: they are trained on how people communicate online. Unfortunately, the internet is rife with misinformation about healthcare and so are the AI chatbots currently in use. Artificial Intelligence is only as good as the dataset it is trained on. A global race is underway to see who can develop the best and largest datasets. Companies and researchers are highly interested in training AI on healthcare data, while many security and privacy questions remain unanswered. For this reason, Z-CERT advises: "Be vigilant with the use of AI in healthcare."
 

Ransomware and data breaches are well-known risks in healthcare. The more data sources we connect (for AI), the greater the impact of ransomware. The more data we store centrally (for AI), the greater the impact of a data breach. The more dependent we become on AI, the greater the impact on healthcare if something goes wrong.
 

The biggest risk of AI in healthcare is that its output may be accepted as truth without expert verification. The loss of transparency, and thus human oversight over healthcare processes, is another major risk. AI assistance, such as through a phone app, is not a substitute for professional medical advice, diagnosis, or treatment. In the Netherlands, you must be authorized to apply medical devices, including software. Additionally, Z-CERT observes an increased indirect impact of ransomware and data breaches when AI is deployed.
 

Opinions on the use of AI vary. To form a sound judgment, it is important to be aware of what others say about AI and to include the necessary pros and cons in your own risk analysis. Below is a brief summary of the different opinions and perspectives on AI deployment, with direct references to in-depth materials.

Patients
When asked what they consider important, patients emphasize that only relevant data should be used (or reused) and that there should be transparency about its use. Patients set specific conditions that strongly depend on the context in which data is (re)used. While digital healthcare is becoming increasingly normalized, many patients are concerned about privacy and the reliability of health information. Patients expect absolute confidentiality in the consulting room.

General Practitioners
In general practices, it is already clear that AI chatbots are replacing doctor’s assistants. Workload is being reduced through assistance in assessing fractures and by decreasing administrative tasks in the consulting room.

Hospitals
In recent years, hospitals have increased their engagement with and knowledge of AI. AI is primarily used as a diagnostic tool. Topics such as integration, user acceptance, and AI validation receive the most attention. Subjects like ethics, privacy, and security are at the bottom of the list and receive the least attention.

University Medical Centers
Erasmus MC, in collaboration with SAS and Microsoft, has developed and implemented data-driven applications in the hospital. They have brought together multiple disciplines, such as IT, data science, and healthcare, to develop an AI algorithm that predicts whether patients can be safely discharged after surgery. Data is used not only for retrospective analysis but also for forward-looking improvements, making daily operations safer and more efficient.
 

Microsoft Copilot is increasingly integrating OpenAI’s chatbot GPT-4 into Microsoft products like Word and Outlook. Electronic Health Record (EHR) provider Epic has partnered with Microsoft to embed the AI chatbot in EHR systems. An example of this is already visible at UMC Groningen, where the AI application in the EHR reads patient questions and provides response suggestions to healthcare providers.

Azure OpenAI
While Microsoft’s Azure OpenAI states that it does not review your data, it does monitor for content related to hate, sexuality, and violence. Azure OpenAI actively monitors all use of the AI system through abuse detection. An opt-out can be requested, but there is no guarantee it will be granted.

Long-term care
In long-term care, AI is being used in supervisory domotics (recognizing sounds to determine whether action by a caregiver is required), social robots, and speech-to-text applications in healthcare apps. Most of these applications are still in an experimental phase.

Dutch government vision on generative AI
The Dutch government has developed a vision on generative AI, describing it as a form of AI where algorithms are used to generate content. The document addresses not only the opportunities and possibilities but also the challenges and risks of deploying generative AI. According to the vision, generative AI has already proven capable of rapidly creating and spreading misinformation and disinformation on a large scale. Additionally, there is a growing dependence on American tech companies. Regarding privacy and data protection, the vision states that, in principle, processing special categories of personal data, such as biometric data and health data, is prohibited unless strict GDPR conditions are met.

Ministry of Health, Welfare and Sport (VWS)
The Dutch Ministry of Health, Welfare and Sport (VWS) believes that applying AI to healthcare data can benefit the healthcare sector. It advocates for a national integrated infrastructure for health data to promote the (re)use of health data for healthcare evaluation, quality improvement, policy, research, and innovation. Within Europe, the European Health Data Space (EHDS) proposal aims to link and share healthcare data broadly, not only for primary purposes (such as providing care), but also for secondary purposes like research and the development of new products. Increasing data availability is essential to keep healthcare accessible and affordable in the future. Trust is a crucial condition for this, as stated in the National Vision on the Health Information System. Governance is needed to build this trust.

Civil rights organizations
According to civil rights organizations EDRi and Privacy First, the European Health Data Space exposes everyone’s medical records to unnecessary security and privacy risks in the name of research and innovation, thereby threatening medical confidentiality. In the negotiations surrounding the EHDS, an agreement has now been reached on an opt-out option for patients who do not want their healthcare data to be shared by default.

The EU AI Act
With the upcoming AI Act, the European Parliament prioritizes ensuring that AI systems used in the EU are safe, transparent, traceable, non-discriminatory, and environmentally friendly. Under the AI Act, AI applications are classified based on their societal risk level. AI applications in medical devices are classified as high-risk under the AI Act. At the request of VWS, research has been conducted into the overlap and inconsistencies between the AI Act and the MDR (Medical Device Regulation) and IVDR (In Vitro Diagnostic Medical Devices Regulation).

ENISA (European Union Agency for Cybersecurity)
The European Union Agency for Cybersecurity (ENISA) has published an extensive report on security and privacy in the application of AI in medical diagnostics. ENISA warns against companies that illegally attempt to obtain and process healthcare data, such as those that are not transparent and do not respect principles like data minimization, data accuracy, and limited data retention.

Health-ISAC (Health Information Sharing and Analysis Center)
The US-based Health Information Sharing and Analysis Center (H-ISAC) advises the healthcare sector to adopt a conservative stance toward AI deployment. Lessons can be learned from the migration to the cloud. Additionally, CISOs should be aware that everyone is still discovering the promises and dangers of AI use.

Rathenau Institute
The Rathenau Institute also emphasizes that the risks of generative AI require caution in its use and that politicians and policymakers must take action to address these risks. It will take time for policies to have an effect.

AIVD (General Intelligence and Security Service of the Netherlands)
In its annual report, the AIVD describes how China has secured a leading position in AI and has a voracious appetite for more data. To strengthen its economic position, China attempts to gain access to more data through collaborations with Western tech companies, universities, and research institutes, as well as through (cyber)espionage. The AIVD has also published a document describing a new attack surface introduced by AI use. AI systems are vulnerable to AI-specific digital attacks, such as data poisoning, deception, backdoors, and the extraction of training data. For example, when loading many AI models, it is possible for malicious actors to execute code remotely.

Science
Within medical science, there is great interest in which AI models can be applied for which medical purposes. However, research from TU Delft shows that there is still little attention paid to the transparency of AI models, and the decisions made by these models are often impossible to trace. The research provides guidelines for applying Explainable Artificial Intelligence (XAI)—AI where you can verify and explain how the AI arrived at a particular decision. Additionally, research has already demonstrated what can go wrong when we give AI control over our inbox. In one experiment, an AI email assistant was convinced to leak information and subsequently spread malicious software to infect others.
 

For now, Z-CERT’s advice is to use AI in healthcare with caution and only when you know what you are doing. We also recommend collaborating on AI projects.

Action Perspective
As described in the sources above, deploying AI increases an organization’s attack surface and makes the IT environment more complex. Having solid basic security measures in place remains crucial.

AI systems cannot be expected to always provide perfect answers. They are based on statistics. No matter how good the training dataset is, there is always a margin of error.

To help you engage in dialogue with your IT supplier and internal data team, we have compiled the following questions. You should expect constructive responses that are verifiable. Any unresolved issues increase the risk of AI deployment:

• How are both staff and patients actively informed about the use of AI? How clear and transparent is the supplier’s information on this?
• How were the training datasets created, and how can they be verified and validated? How do you prevent dependence on external training data that cannot be controlled? How do you prevent training on misinformation? Does the AI solution only use a language model that generates answers, or does it also use a separate dataset with medical information for reference?
• How is the storage of training data limited in terms of volume and duration? How do you prevent data from being linked for different purposes?
  How is it ensured that only healthcare professionals and patients have access to healthcare data? How do you prevent researchers and company employees from accessing healthcare data?
• How is data encryption used? How are Privacy Enhancing Technologies (such as Differential Privacy) applied? How do you prevent healthcare data from being leaked in readable form in the event of a data breach?
• How was the trained data model developed, and how can it be verified and validated? How do you prevent dependence on an external data model that cannot be controlled? How does the model receive updates? How will the supplier communicate changes? What new data has been used to train the model? Has the supplier set up an OTAP (Development, Testing, Acceptance, Production) pipeline to determine whether results are the same, better, or worse?
• How is it arranged for a patient to have their healthcare data removed from the training data and data model? How is the AI system protected against AI-specific attacks? How is abuse through anomalous input or data poisoning prevented? How do you prevent training data from being extracted from the AI system?
• How does the supplier guarantee liability in the event of a data breach caused by incorrect data processing or human error?
   

When using AI, healthcare providers remain responsible for the correct diagnosis, advice, and treatment of their patients. Artificial Intelligence is a society-disrupting innovation. It is essential to remain vigilant in its deployment. Especially in healthcare.