Title:Automatically Generating Narrative-Style Radiology Reports from Volumetric CT Images; a Proof of Concept

Abstract:The world faces a shortage of radiologists, leading to longer treatment times and increased stress, negatively impacting patient safety and workforce morale. Integrating artificial intelligence to interpret radiographic images and generate descriptive reports offers a promising solution. However, limited research exists on generating natural language descriptions for volumetric medical images. This study introduces a deep learning-based proof of concept model to accurately identify abnormalities in volumetric CT data and generate narrative-style reports. Various encoder-decoder models were assessed for their efficacy in clinically relevant and surrogate tasks. Clinically relevant tasks involved identifying and describing pulmonary nodules and pleural effusions, while surrogate tasks involved recognizing and describing artificial abnormalities such as mirroring, rotation, and lung lobe occlusion. The results show high accuracy in detecting combinations of artificial abnormalities, with the best model achieving a classification accuracy of 0.97 on an independent dataset with a homogeneously distributed 11-class problem. Furthermore, the best model consistently generated coherent radiology reports in natural language, with a next-word prediction accuracy of 0.84. Additionally, 65% of these reports were factually accurate regarding the identified artificial abnormalities. Unfortunately, these models did not replicate this success for clinically relevant tasks. Overall, this study provides a working proof of concept model for a challenge yet to be fully addressed by the scientific community. Given the success on surrogate tasks, the leap to clinically relevant tasks seems feasible. Acquiring a significantly larger high-quality dataset appears to be the most promising path forward, alongside more computational resources for end-to-end model training.