Multi-modal artificial intelligence enhanced systematic colorectal cancer pathology evaluation.

person Po-Jen Lin Western Connecticut Health Network-Danbury Hospital, Danbury, CT info_outline Po-Jen Lin, Tsung-Lu Michael Lee, Cheng-Hsiu Tsai, Xuan Gong, Mu-Hung Tsai, Hsi-Huei Liu, Amy H Huang, Irene Tai-Lin Lee, Ronan Wenhan Hsieh, Jeffrey A. Meyerhardt, Kun-Hsing Yu

Authors person Po-Jen Lin Western Connecticut Health Network-Danbury Hospital, Danbury, CT info_outline Po-Jen Lin, Tsung-Lu Michael Lee, Cheng-Hsiu Tsai, Xuan Gong, Mu-Hung Tsai, Hsi-Huei Liu, Amy H Huang, Irene Tai-Lin Lee, Ronan Wenhan Hsieh, Jeffrey A. Meyerhardt, Kun-Hsing Yu Organizations Western Connecticut Health Network-Danbury Hospital, Danbury, CT, Southern Taiwan University of Science and Technology, Tainan City, Taiwan, Taiwan, Harvard Medical School, Boston, MA, UConn School of Medicine, Farmington, CT, Emory University, Atlanta, GA, University of Washington Fred Hutchinson Cancer Center, Seattle, WA, Department of Medical Oncology, Dana-Farber Cancer Institute, Harvard Medical School, Harvard University, Boston, MA Abstract Disclosures Research Funding No funding sources reported Background: Multi-modal artificial intelligence (AI) approaches have substantial promise for capturing the relationships between pathology images, text descriptions, and patients’ clinical characteristics. Previous attempts showed that multi-modal AI methods classified colorectal cancer (CRC) types with an accuracy of 90%. However, it is unclear if this new AI framework can reliably predict histologic grade and the extent of primary tumor at scale (T stage). To address this research gap, here we developed a multi-modal AI system for detailed CRC pathology analyses, with a focus on histologic factors related to patient prognoses. Methods: We obtained whole-slide pathology images, pathology reports, and detailed clinical information from 441 colorectal adenocarcinoma patients who participated in The Cancer Genome Atlas. These patients were divided into training or testing sets using a 75/25 split. We extracted histology types (invasive adenocarcinoma with or without mucinous features), histology grades (well, moderate, or poorly differentiated), and the extent of the primary tumor (pT2, pT3, or pT4) from the pathology reports in the training subset. We employed the CLIP (Contrastive Language-Image Pre-Training) prefix captioning model as a backbone and developed an AI model for predicting these pathology profiles using whole-slide pathology images. We evaluated the performance (sensitivity, specificity, accuracy, precision, and F1 score) of our model by comparing the model-based prediction in the testing set and the ground-truth information from the pathology reports. Results: Our model successfully predicted histology types with an accuracy of 0.89 and an F1 score of 0.93 (sensitivity = 0.92, specificity = 0.76, and precision = 0.94). In addition, our approach further predicted histology grades with an accuracy of 0.79 and an F1 score of 0.87 (sensitivity = 0.96, specificity = 0.31, and precision = 0.80). Furthermore, our model achieved an accuracy of 0.69 and an F1 score of 0.80 (sensitivity = 0.78, specificity = 0.38, and precision = 0.83) in predicting T stages. Conclusions: Our study suggests that multi-modal AI has the potential to assist with the histopathology diagnosis of CRC tissues. Our approaches performed well in identifying CRC types and could provide second opinions on CRC diagnosis. However, the performance of our model varies across prediction tasks. Extensive external validation using larger and more diverse pathology datasets is crucial to moving our novel AI approaches closer to clinical applications. Category Sensitivity Specificity Accuracy Precision F1 score Histology types 0.92 0.76 0.89 0.94 0.93 Histology grades 0.96 0.31 0.79 0.80 0.87 T stages 0.78 0.38 0.69 0.83 0.80