Harvard boffins build multimodal AI system to predict 14 types of cancer

Many sources become one

Multimodal AI models, trained on numerous types of data, could help doctors screen patients at risk of developing multiple different cancers more accurately..

Researchers from the Brigham and Women's Hospital part of Harvard University's medical school developed a deep learning model capable of identifying 14 types of cancer. Most AI algorithms are trained to spot signs of disease from a single source of data, like medical scans, but this one can take inputs from multiple sources. 

Predicting whether someone is at risk of developing cancer isn't always as straightforward, doctors often have to consult various types of information like a patient's healthcare history or perform other tests to detect genetic biomarkers.

These results can help doctors figure out the best treatment for a patient as they monitor the progression of the disease, but their interpretation of the data can be subjective, Faisal Mahmood, an assistant professor working at the Division of Computational Pathology at the Brigham and Women's Hospital, explained. 

"Experts analyze many pieces of evidence to predict how well a patient may do. These early examinations become the basis of making decisions about enrolling in a clinical trial or specific treatment regimens. But that means that this multimodal prediction happens at the level of the expert. We're trying to address the problem computationally," he said in a statement.

Mahmood and his colleagues described how a single overarching system, made up of numerous deep-learning based algorithms and trained on multiple forms of data, could diagnose up to 14 different cancers. The researchers used training data from The Cancer Genome Atlas (TCGA), a public resource containing data on different types of cancer obtained from over 5,000 real patients, as well as other data sources.

First, microscopic views of cell tissues from whole-slide images (WSIs) and text-based genomics data were used to train two separate models. These were then integrated into a single system to predict whether patients are at high or low risk of developing the different cancer types. The model may even help scientists find or confirm genetic markers associated with certain disease, the researchers claimed. 

"Using deep learning, multimodal fusion of molecular biomarkers and extracted morphological features from WSIs has potential clinical application in not only improving precision in patient risk stratification but could also assist in the discovery and validation of multimodal biomarkers where combinatory effects of histology and genomic biomarkers are not known," the team wrote in a paper published in Cancer Cell on Monday.

Mahmood told The Register the current study was a proof of concept into applying multimodal models to predict cancer risk. "We need to train these models with much more data, test these models on large independent test cohorts and run prospective studies and clinical trials to establish the efficacy of these models in a clinical setting," he concluded. ®