Assessing workflow impact and clinical utility of AI-assisted brain aneurysm detection: a multi-reader study

This repository contains the code for the paper: "Assessing workflow impact and clinical utility of AI-assisted brain aneurysm detection: a multi-reader study".

The goal of the work was to assess the diagnostic performance of two radiologists for the task of brain aneurysm detection under two different settings: 1) Unassisted: normal reading as in clinical routine; 2) AI-assisted: using a CAD support system. Additionally, we investigated how the AI CAD tool impacts the clinical workflow.

Installation/Softwares

The results of the paper were obtained with python 3.9 and a Windows OS. Reproducibility for different configurations is not guaranteed.

For the R scripts, we used RStudio 2022.07.2. For the creation of the overlay dicom series, we used MeVisLab 3.4.2.

Setup venv/conda environment

To run the python scripts:

1. Clone the repository
2. Create a venv/conda environment. If you are not familiar with pip/conda environments, please check out the official documentation. Alternatively, feel free to use your favorite IDE such as PyCharm or Visual Studio Code to set up an environment.
3. Activate your environment:

$ source myenv/bin/activate  # if using venv OR
$ conda activate /miniconda3/envs/myenv  # if using conda or anaconda

4. Install all required packages with:

$ pip install -r requirements.txt

Data

The majority of the dataset used for this study can be downloaded from this OpenNEURO link. The files containing the results of the two readings, both for the junior and senior radiologists, are located inside the directory READINGS.

Usage

Overlay Series Generation

The code used to generate the DICOM overlay series where the segmentations are overlayed on the TOF-MRA volumes is called d20221006_export_fused_images.mlab and is located inside the directory mevislab_overlay.

Sensitivity and Specificity Analyses

To code used to run the McNemar's tests for the sensitivity and specificity analyses presented in the paper is located in the directory sensitivity_specificity_analysis_R

Reading time

The script used to compare the reading times of the two radiologists with and without the assistance of the CAD is called compare_timing_between_readings.py and is located inside the directory reading_time. The files containing the results of the two readings (which include the reading times) are located inside the directory READINGS.

Confidence scores

All the scripts related to the confidence scores are located in the directory confidence_score. To script used to create the barplots that display the confidence scores is d20240916_confidence_scores_barplots.py. To script used to run the XYZ test to compare the distributions of confidence scores is d20240317_compare_confidence_scores.py

How to cite

If you're using our dataset/model, or comparing performances with the ones presented in this work, please cite the two following publications:

    [1] Di Noto, T., Marie, G., Tourbier, S., Alemán-Gómez, Y., Esteban, O., Saliou, G., ... & Richiardi, J. (2023). Towards automated brain aneurysm detection in TOF-MRA: open data, weak labels, and anatomical knowledge. Neuroinformatics, 21(1), 21-34.

and TODO: add (med)-arxiv once it's public