Multimodal AI biomarkers: from biology to patient stratification

_{Author: Omar S. M. El Nahhas / CEO & Co-Founder
The content of this article was provided by StratifAI GmbH and reflects the views of the author/company. ESMO does not endorse the content of this publication and does not take any responsibility for the accuracy, completeness or reliability of the information contained in such article.}

Traditional biomarkers such as PD-L1 or HER2, scored manually by pathologists, or molecular alterations like EGFR and ALK mutations detected through sequencing, have long supported precision medicine. However, these methods remain resource-intensive, expensive, and often inaccessible in routine clinical settings.

A new generation of biomarkers is now emerging, built on artificial intelligence. These models analyze data already collected in standard care, including histology slides, genomic and transcriptomic profiles, radiology scans, and clinical records. The goal is to detect complex patterns linked to underlying biology or outcomes. While initial efforts focused on imaging, the field has rapidly expanded toward true multimodal integration, using multiple data types to form more complete models of disease.

Each modality contributes unique information. Tissue morphology offers spatial context. Genomics and transcriptomics provide molecular specificity. Proteomics reveals cellular interactions and the tumor microenvironment. Clinical records capture comorbidities and treatment histories. Integrating these layers enables more accurate predictions and deeper biological insights.

Recent technical advances have made the integration of these modalities more feasible. Self-supervised learning and foundation models enable the extraction of meaningful features from digitized histology samples without requiring labels or annotations. These so-called visual embeddings are then combined with omics and clinical data to predict endpoints such as microsatellite instability, molecular subtype, immune infiltration, or overall survival. Peer-reviewed studies have shown that such models can identify clinically relevant features, even in the absence of explicit annotations. Several are now moving toward clinical validation.

The potential impact is substantial. Once infrastructure is established, these biomarkers can be produced with little added cost and in near real time. Predictions can be available during a patient’s clinic visit, supporting timely decisions. By combining modalities, these models also account for tumor heterogeneity, missing information, and clinical complexity more effectively than single-modality approaches.

Validation remains a critical challenge. Performance must be reproducible across large, diverse cohorts. Regulatory pathways are still developing. Just as important is interpretability. Clinicians need to understand not only whether a prediction works but also why. Hybrid approaches that link AI-derived features with known biological mechanisms may help build confidence and bridge the gap between computational models and clinical reasoning.

Multimodal AI biomarkers are not expected to replace traditional assays. Instead, they will complement them, especially when tissue is scarce or sequencing access is limited. They also offer the potential to enrich clinical trials through deeper biological stratification.

“Progress in this field is driven by collective efforts from academic labs, translational research groups, clinical partners and technology companies.” Omar El Nahhas, CEO and co-founder of StratifAI, one of the companies involved in developing multimodal models, recently noted: “Multimodal AI will unlock a new tier of biomarkers: more affordable, more available, and more biologically complete, by integrating signals across histology, genomics, and clinical data.” Such an approach reflects broader attempts across the field to ensure that these AI-based biomarkers align with clinical needs and capture the complexity inherent in real-world oncology. As AI matures and clinical data becomes more accessible, the future of precision oncology is being rewritten. Multimodal biomarkers will not simply optimize existing decisions; they will redefine what is possible. In the coming years, they could shift from computational prototypes to cornerstones of modern cancer care, transforming diagnosis, treatment, and equity on a global scale.