Publications

Over the last decade, the annual Immunorad Conference, held under the joint auspicies of Gustave Roussy (Villejuif, France) and the Weill Cornell Medical College (New-York, USA) has aimed at exploring the latest advancements in the fields of tumor immunology and radiotherapy-immunotherapy combinations for the treatment of cancer. Gathering medical oncologists, radiation oncologists, physicians and researchers with esteemed expertise in these fields, the Immunorad Conference bridges the gap between preclinical outcomes and clinical opportunities. Thus, it paves a promising way toward optimizing radiotherapy-immunotherapy combinations and, from a broader perspective, improving therapeutic strategies for patients with cancer. Herein, we report on the topics developed by key-opinion leaders during the 7th Immunorad Conference held in Paris-Les Cordeliers (France) from September 27th to 29th 2023, and set the stage for the 8th edition of Immunorad which will be held at Weill Cornell Medical College (New-York, USA) in October 2024.

Laurent, Pierre-Antoine, Fabrice André, Alexandre Bobard, Desiree Deandreis, Sandra Demaria, Stephane Depil, Stefan B. Eichmüller et al. Pushing the boundaries of radiotherapy-immunotherapy combinations: highlights from the 7th immunorad conference. OncoImmunology 14, no. 1 (2025): 2432726. doi: https://doi.org/10.1080/2162402X.2024.2432726

Despite major advances in artificial intelligence (AI) research for healthcare, the deployment and adoption of AI technologies remain limited in clinical practice. This paper describes the FUTURE-AI framework, which provides guidance for the development and deployment of trustworthy AI tools in healthcare. The FUTURE-AI Consortium was founded in 2021 and comprises 117 interdisciplinary experts from 50 countries representing all continents, including AI scientists, clinical researchers, biomedical ethicists, and social scientists. Over a two year period, the FUTURE-AI guideline was established through consensus based on six guiding principles—fairness, universality, traceability, usability, robustness, and explainability. To operationalise trustworthy AI in healthcare, a set of 30 best practices were defined, addressing technical, clinical, socioethical, and legal dimensions. The recommendations cover the entire lifecycle of healthcare AI, from design, development, and validation to regulation, deployment, and monitoring.

Summary points

• Despite major advances in medical artificial intelligence (AI) research, clinical adoption of emerging AI solutions remains challenging owing to limited trust and ethical concerns
• The FUTURE-AI Consortium unites 117 experts from 50 countries to define international guidelines for trustworthy healthcare AI
• The FUTURE-AI framework is structured around six guiding principles: fairness, universality, traceability, usability, robustness, and explainability
• The guideline addresses the entire AI lifecycle, from design and development to validation and deployment, ensuring alignment with real world needs and ethical requirements
• The framework includes 30 detailed recommendations for building trustworthy and deployable AI systems, emphasising multistakeholder collaboration
• Continuous risk assessment and mitigation are fundamental, addressing biases, data variations, and evolving challenges during the AI lifecycle
• FUTURE-AI is designed as a dynamic framework, which will evolve with technological advancements and stakeholder feedback

Lekadir, Karim; Feragen, Aasa; Fofanah, Abdul Joseph; Frangi, Alejandro F; Zuluaga, Maria A.; et al. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare BMJ 2025; 388 doi: https://doi.org/10.1136/bmj.r340 (Published 17 February 2025) Cite this as: BMJ 2025;388:r340

Background
Mammographic imaging is essential for breast cancer detection and diagnosis. In addition to masses, calcifications are of concern and the early detection of breast cancer also heavily relies on the correct interpretation of suspicious microcalcification clusters. Even with advances in imaging and the introduction of novel techniques such as digital breast tomosynthesis and contrast-enhanced mammography, a correct interpretation can still be challenging given the subtle nature and large variety of calcifications.

Purpose
Computer simulated lesion models can serve to develop, optimize, or improve imaging techniques. In addition to their use in comparative (virtual clinical trial) detection experiments, these models have potential application in training deep learning models and in the understanding and interpretation of breast lesions. Existing simulation methods, however, often lack the capacity to model the diversity occurring in breast lesions or to generate models relevant for a specific case. This study focuses on clusters of microcalcifications and introduces an automated, flexible toolbox designed to generate microcalcification cluster models customized to specific tasks.

Methods
The toolbox allows users to control a large number of simulation parameters related to model characteristics such as lesion size, calcification shape, or number of microcalcifications per cluster. This leads to the capability of creating models that range from regular to complex clusters. Based on the input parameters, which are either tuned manually or pre-set for a specific clinical type, different sets of models can be simulated depending on the use case. Two lesion generation methods are described. The first method generates three-dimensional microcalcification clusters models based on geometrical shapes and transformations. The second method creates two-dimensional (2D) microcalcification cluster models for a specific 2D mammographic image. This novel method employs radiomics analysis to account for local textures, ensuring the simulated microcalcification cluster is appropriately integrated within the existing breast tissue. The toolbox is implemented in the Python language and can be conveniently run through a Jupyter Notebook interface, openly accessible at https://gitlab.kuleuven.be/medphysqa/deploy/breast-calcifications. Validation studies performed by radiologists assessed the level of malignancy and realism of clusters tuned with specific parameters and inserted in mammographic images.

Results
The flexibility of the toolbox with multiple simulation methods is illustrated, as well as the compatibility with different simulation frameworks and image types. The automation allows for the straightforward and fast generation of diverse microcalcification cluster models. The generated models are most likely applicable for various tasks as they can be configured in a variety of ways and inserted in different types of mammographic images of multiple acquisition systems. Validation studies confirmed the capacity to simulate realistic clusters and capture clinical properties when tuned with appropriate parameter settings.

Conclusion
This simulation toolbox offers a flexible means of simulating microcalcification cluster models with potential use in both technical and clinical research in mammography imaging. The 3D generation methods allow for specifying many characteristics regarding the calcification shape and cluster architecture, and the 2D generation method presents a novel manner to create microcalcification clusters tailored to existing breast textures.

Van Camp A, Punter E, Houbrechts K, et al. An automated toolbox for microcalcification cluster modeling for mammographic imaging. Med Phys. 2024;1-15. https://doi.org/10.1002/mp.17521

The emerging European Health Data Space (EHDS) Regulation opens new prospects for large-scale sharing and re-use of health data. Yet, the proposed regulation suffers from two important limitations: it is designed to benefit the whole population with limited consideration for individuals, and the generation of secondary datasets from heterogeneous, unlinked patient data will remain burdensome. AIDAVA, a Horizon Europe project that started in September 2022, proposes to address both shortcomings by providing patients with an AI-based virtual assistant that maximises automation in the integration and transformation of their health data into an interoperable, longitudinal health record. This personal record can then be used to inform patient-related decisions at the point of care, whether this is the usual point of care or a possible cross-border point of care. The personal record can also be used to generate population datasets for research and policymaking. The proposed solution will enable a much-needed paradigm shift in health data management, implementing a ‘curate once at patient level, use many times’ approach, primarily for the benefit of patients and their care providers, but also for more efficient generation of high-quality secondary datasets. After 15 months, the project shows promising preliminary results in achieving automation in the integration and transformation of heterogeneous data of each individual patient, once the content of the data sources managed by the data holders has been formally described. Additionally, the conceptualization phase of the project identified a set of recommendations for the development of a patient-centric EHDS, significantly facilitating the generation of data for secondary use.

Citation: de Zegher I, Norak K, Steiger D, Müller H, Kalra D, Scheenstra B, Cina I, Schulz S, Uma K, Kalendralis P, Lotman E-M, Benedikt M, Dumontier M and Celebi R (2024) Artificial intelligence based data curation: enabling a patient-centric European health data space. Front. Med. 11:1365501. doi: 10.3389/fmed.2024.1365501 https://www.frontiersin.org/journals/medicine/articles/10.3389/fmed.2024.1365501/full

Objective: To assess the performance of large language models (LLMs) for zero-shot disambiguation of acronyms in clinical narratives.

Materials and Methods: Clinical narratives in English, German, and Portuguese were applied for testing the performance of four LLMs: GPT-3.5, GPT-4, Llama-2-7b-chat, and Llama-2-70b-chat. For English, the anonymized Clinical Abbreviation Sense Inventory (CASI, University of Minnesota) was used. For German and Portuguese, at least 500 text spans were processed. The output of LLM models, prompted with contextual information, was analyzed to compare their acronym disambiguation capability, grouped by document-level metadata, the source language, and the LLM.

Results: On CASI, GPT-3.5 achieved 0.91 in accuracy. GPT-4 outperformed GPT-3.5 across all datasets, reaching 0.98 in accuracy for CASI, 0.86 and 0.65 for two German datasets, and 0.88 for Portuguese. Llama models only reached 0.73 for CASI and failed severely for German and Portuguese. Across LLMs, performance decreased from English to German and Portuguese processing languages. There was no evidence that additional document-level metadata had a significant effect.

Conclusion: For English clinical narratives, acronym resolution by GPT-4 can be recommended to improve readability of clinical text by patients and professionals. For German and Portuguese, better models are needed. Llama models, which are particularly interesting for processing sensitive content on premise, cannot yet be recommended for acronym resolution.

Kugic, A., Schulz, S., & Kreuzthaler, M. (2024). Disambiguation of acronyms in clinical narratives with large language models. Journal of the American Medical Informatics Association, ocae157. https://doi.org/10.1093/jamia/ocae157

In recent years, the analysis of clinical texts has evolved significantly, driven by the emergence of language models like BERT such as PubMedBERT, and ClinicalBERT, which have been tailored for the (bio)medical domain that rely on extensive archives of medical documents. While they boast high accuracy, their lack of interpretability and language transfer limitations restrict their clinical utility. To address this, we propose a new, lightweight graph-based embedding method designed specifically for radiology reports. This approach considers the report’s structure and content, connecting medical terms through the multilingual SNOMED Clinical Terms knowledge base. The resulting graph embedding reveals intricate relationships among clinical terms, enhancing both clinician comprehension and clinical accuracy without the need for large pre-training datasets. Demonstrating the versatility of our method, we apply this embedding to two tasks: disease and image classification in X-ray reports. In disease classification, our model competes effectively with BERT-based approaches, yet it is significantly smaller and requires less training data. Additionally, in image classification, we illustrate the efficacy of the graph embedding by leveraging cross-modal knowledge transfer, highlighting its applicability across diverse languages.

Uma, K., & Moens, M. F. (2024). Unraveling Clinical Insights: A Lightweight and Interpretable Approach for Multimodal and Multilingual Knowledge Integration. In Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health)@ LREC-COLING 2024 (pp. 197-203). https://aclanthology.org/2024.cl4health-1.24

Interoperabilität ist für ein vernetztes Gesundheitssystems unabdingbar. Basierend auf Terminologiestandards wie ICD, LOINC und SNOMED CT erfordert sie eine korrekte Interpretation von Patientendaten in der jeweiligen Anwendungssituation. Dies wird unterstützt durch syntaktische Standards wie FHIR, welche Codes in den patientenspezifischen Kontext einbetten. Um Routinedaten interoperabel zu machen, ist die Kluft zwischen klinischer Sprache und normierter Dokumentation zu überbrücken. Natural Language Processing (NLP) ist hierbei eine Technologie, die sich derzeit im Zeichen der Künstlichen Intelligenz rapide weiterentwickelt. Die Kommunikation mit dem Computer in menschlicher Sprache wird erheblich an Bedeutung gewinnen. Das Kapitel gibt einen Einblick in aktuelle Techniken und Ressourcen zur Unterstützung von Interoperabilität. Dazu kommen Perspektiven der Gesundheitsversorgung, Gesundheitsverwaltung, Wissenschaft, Industrie und Selbstverwaltung zur Sprache.

Daumke, P., Haverkamp, C., Heckmann, S., Kuper, M., Müller, A., Oemig, F., ... & Schulz, S. (2024). Kommunikationsfähigkeit und Interoperabilität von Gesundheitsdaten in einem vernetzten Gesundheitssystem. In Health Data Management: Schlüsselfaktor für erfolgreiche Krankenhäuser (pp. 457-496). Wiesbaden: Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-43236-2_41

The International Statistical Classification of Diseases and Related Health Problems (ICD) is a global standard, a diagnostic tool that is frequently used for endemic research, health management, and clinical diagnosis, and it plays a crucial role in providing shrewd medical treatment. Comparable statistics on the causes of mortality and morbidity across locations and throughout time have been based on the ICD. The traditional procedure of assigning codes is expensive, error-prone and time-consuming, and automated mapping of ICD codes is now a significant area of scholarly research. With the help of statistical modeling, rule-engines, conventional machine learning, and deep learning techniques like graph embedding, attention mechanisms, adversarial learning, and pre-trained language models (PLMs), this paper aims to analyze and document inferences on the evolution of clinical coding automation. We try to summarize with comparative performance analysis various approaches addressed towards codification of free-text clinical narratives on the publicly available Medical Information Mart. This study investigates whether clinicians and researchers could benefit from an adequate interpretation of model predictions from an Explainable Artificial Intelligence (XAI) perspective. Finally, the survey illustrates ICD coding and disease classification applications and its challenges, evaluation metrics, datasets, and directions towards automating explanatory medical code predictions.

Uma, K., Francis, S., Sun, W., Moens, MF. (2024). Towards Explainability in Automated Medical Code Prediction from Clinical Records. In: Arai, K. (eds) Intelligent Systems and Applications. IntelliSys 2023. Lecture Notes in Networks and Systems, vol 825. Springer, Cham. https://doi.org/10.1007/978-3-031-47718-8_40

Despite the increasing availability of ontology-based semantic resources for biomedical content representation, large amounts of clinical data are in narrative form only. Therefore, many clinical information management tasks require information extraction using natural language processing (NLP).

Clinical corpora annotated by humans are crucial resources for this purpose. On the one hand, they are needed to domain-fine-tune language models (LMs) with the purpose to formally represent clinical information extracted from unstructured free-text. On the other hand, annotated corpora are indispensable for assessing the results of information extracting using NLP.

The effectiveness of annotations crucially depends on annotation quality. Detailed annotation guidelines, which define the form that extracted information should take, prevent human annotators from taking erratic annotation decisions and guarantee a good inter-annotator agreement. Our hypothesis is that, to this end, annotations should (i) be based on ontological principles and (ii) be consistent with existing clinical documentation standards.

With the experience of several annotation projects we highlight the need for sophisticated guidelines. We formulate a set of abstract principles on which such guidelines should be based, followed by examples how to keep them, on the one hand, user-friendly and consistent, and on the other hand compatible with the international semantic standards SNOMED CT and FHIR, including their areas of overlap.

We sketch the representation of the resulting representations in a knowledge graph as a state-of-the-art semantic representation paradigm, which can be enriched by additional content on A-Box and T-Box level and on which symbolic and neural reasoning tasks can be applied.

Schulz, S., Del-Pinto, W., Han, L., Kreuzthaler, M., Aghaei, S., & Nenadic, G. (2023). Towards principles of ontology-based annotation of clinical narratives. In Proceedings of the International Conference on Biomedical Ontologies (Vol. 2023). https://ceur-ws.org/Vol-3603/Paper4.pdf

In this paper we investigate automated annotation of tabular data using semantic technologies in combination with neural network embedding. Specifically, we propose an anchoring model in which property and cell types from the data embedding space are aligned with ontology relation and entity types. We show that by combining the power of symbolic reasoning, neural embeddings, and loss function design, a significant performance improvement as high as 86% for column property, 82% for column type, and 87% for column qualifier annotations can be achieved based on DBpedia and Wikidata table extractions.

Shervin Mehryar, Remzi Celebi. Semantic Annotation of Tabular Data for Machine-to-Machine Interoperability via Neuro-Symbolic Anchoring. SemTab’23: Semantic Web Challenge on Tabular Data to Knowledge Graph Matching 2023, co-located with the 22nd. International Semantic Web Conference (ISWC), November 6-10, 2023, Athens, Greece

Due to popular successes (e.g., ChatGPT) Artificial Intelligence (AI) is on everyone's lips today. When advances in biotechnology are combined with advances in AI unprecedented new potential solutions become available. This can help with many global problems and contribute to important Sustainability Development Goals. Current examples include Food Security, Health and Well-being, Clean Water, Clean Energy, Responsible Consumption and Production, Climate Action, Life below Water, or protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss. AI is ubiquitous in the life sciences today. Topics include a wide range from machine learning and Big Data analytics, knowledge discovery and data mining, biomedical ontologies, knowledge-based reasoning, natural language processing, decision support and reasoning under uncertainty, temporal and spatial representation and inference, and methodological aspects of explainable AI (XAI) with applications of biotechnology. In this pre-Editorial paper, we provide an overview of open research issues and challenges for each of the topics addressed in this special issue. Potential authors can directly use this as a guideline for developing their paper.

Holzinger A, Keiblinger K, Holub P, Zatloukal K, Müller H. AI for life: Trends in artificial intelligence for biotechnology. N Biotechnol. 2023;74:16-24. doi: 10.1016/j.nbt.2023.02.001. PMID: 36754147

For many natural language processing systems, the extraction of temporal links and associations from clinical narratives has been a critical challenge. To understand such processes, we must be aware of the occurrences of events and their time or temporal aspect by constructing a chronology for the sequence of events. The primary objective of temporal relation extraction is to identify relationships and correlations between entities, events, and expressions. We propose a novel architecture leveraging Transformer based graph neural network by combining textual data with event graph embeddings for predicting temporal links across events, entities, document creation time and expressions. We demonstrate our preliminary findings on i2b2 temporal relations corpus for predicting BEFORE, AFTER and OVERLAP links with event graph for correct set of relations. Comparison with various Biomedical-BERT embedding types were benchmarked yielding best performance on PubMed BERT with language model masking (LMM) mechanism on our methodology. This illustrates the effectiveness of our proposed strategy.

Uma, K., Francis, S., & Moens, M. F. (2023). Masking Language Model Mechanism with Event-Driven Knowledge Graphs for Temporal Relations Extraction from Clinical Narratives. In International Conference on Complex Networks and Their Applications (pp. 162-174). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-53468-3_14

Artificial intelligence (AI) today is very successful at standard pattern-recognition tasks due to the availability of large amounts of data and advances in statistical data-driven machine learning. However, there is still a large gap between AI pattern recognition and human-level concept learning. Humans can learn amazingly well even under uncertainty from just a few examples and are capable of generalizing these concepts to solve new conceptual problems. The growing interest in explainable machine intelligence requires experimental environments and diagnostic/benchmark datasets to analyze existing approaches and drive progress in pattern analysis and machine intelligence. In this paper, we provide an overview of current AI solutions for benchmarking concept learning, reasoning, and generalization; discuss the state-of-the-art of existing diagnostic/benchmark datasets (such as CLEVR, CLEVRER, CLOSURE, CURI, Bongard-LOGO, V-PROM, RAVEN, Kandinsky Patterns, CLEVR-Humans, CLEVRER-Humans, and their extension containing human language); and provide an outlook of some future research directions in this exciting research domain.

Holzinger A, Saranti A, Angerschmid A, Finzel B, Schmid U, Mueller H. Toward human-level concept learning: Pattern benchmarking for AI algorithms. Patterns (N Y). 2023 Jul 5;4(8):100788. doi: 10.1016/j.patter.2023.100788. PMC: 10435961