autoPET

Introduction

We invite you to participate in the third autoPET Challenge. The focus of this year's challenge is to further refine the automated segmentation of tumor lesions in Positron Emission Tomography/Computed Tomography (PET/CT) scans in a multitracer multicenter setting. Over the past decades, PET/CT has emerged as a pivotal tool in oncological diagnostics, management and treatment planning. In clinical routine, medical experts typically rely on a qualitative analysis of the PET/CT images, although quantitative analysis would enable more precise and individualized tumor characterization and therapeutic decisions. A major barrier to clinical adoption is lesion segmentation, a necessary step for quantitative image analysis. Performed manually, it's tedious, time-consuming and costly. Machine Learning offers the potential for fast and fully automated quantitative analysis of PET/CT images, as previously demonstrated in the first two autoPET challenges. Building upon the insights gained in these challenges, autoPET III expands the scope to address the critical need for models to generalize across multiple tracers and centers. To this end, we provide participants access to a more diverse PET/CT dataset containing images of two different tracers - Prostate-Specific Membrane Antigen (PSMA) and Fluorodeoxyglucose (FDG) acquired from two different clinical sites (Figure). In this challenge, we give participants the chance to let their models compete in two award categories. In award category one, participants are tasked with developing robust segmentation algorithms applicable to two different tracers. In award category two, the significance of data quality and preprocessing in algorithm performance is addressed. Here, participants are encouraged to enhance our baseline model using innovative data pipelines, fostering advancements in data-centric approaches to automated PET/CT lesion segmentation. Join us in autoPET III to pave the way for robust deep-learning-based medical image analysis in PET/CT, optimizing diagnostics and personalizing therapy guidance in oncology.

AutoPET-III is hosted at the MICCAI 2024:
and supported by the European Society for hybrid, molecular and translational imaging (ESHI). The challenge is the successor of autoPET and autoPET II.

Grand Challenge

Task

Based on the insights of the last two autoPET challenges, we expand the scope of the autoPET III challenge to the primary task of achieving multitracer multicenter generalization of automated lesion segmentation. To this end, we provide participants access to two different PET/CT training datasets: a large, annotated FDG-PET/CT dataset acquired at the University Hospital Tübingen, Germany (UKT), and a large, annotated PSMA-PET/CT dataset acquired at LMU Hospital, LMU Munich, Germany (LMU). The FDG-UKT dataset was already used in the autoPET I and II challenges. The PSMA-LMU dataset is new. It encompasses 597 PET/CT volumes of male patients diagnosed with prostate carcinoma from LMU Hospital and shows a significant domain shift from the UKT training data (different tracers, different PET/CT scanners, and acquisition protocols). Algorithms will be tested on PSMA and FDG data from LMU and UKT, respectively.
In addition, we will have a second award category, where participants are invited to submit our baseline model trained with their advanced data pipelines. This category is motivated by the observation that in autoPET I and II, data quality and handling in pre- and post-processing posed significant bottlenecks. Due to the rarity of publicly available PET data in the medical deep learning community, there is no standardized approach to preprocess these images (normalization, augmentations, etc.). The second award category will thus additionally promote data-centric approaches to automated PET/CT lesion segmentation.
Testing will be performed on 200 studies (held-out test database). Test data will be drawn in part (50%) from the same sources and distributions as the training data. The other part will be drawn crosswise from the other center, i.e. PSMA from Tuebingen (25%) and FDG from LMU (25%).

Database

The challenge cohort consists of patients with histologically proven malignant melanoma, lymphoma or lung cancer as well as negative control patients who were examined by FDG-PET/CT in two large medical centers (University Hospital Tübingen, Germany & University Hospital of the LMU in Munich, Germany).
The FDG cohort comprises 501 patients diagnosed with histologically proven malignant melanoma, lymphoma, or lung cancer, along with 513 negative control patients. The PSMA cohort includes pre- and/or post-therapeutic PET/CT images of male individuals with prostate carcinoma, encompassing images with (537) and without PSMA-avid tumor lesions (60). Notably, the training datasets exhibit distinct age distributions: the FDG UKT cohort spans 570 male patients (mean age: 60; std: 16) and 444 female patients (mean age: 58; std: 16), whereas the PSMA MUC cohort tends to be older, with 378 male patients (mean age: 71; std: 8). Additionally, there are variations in imaging conditions between the FDG Tübingen and PSMA Munich cohorts, particularly regarding the types and number of PET/CT scanners utilized for acquisition. The PSMA Munich dataset was acquired using three different scanner types (Siemens Biograph 64-4R TruePoint, Siemens Biograph mCT Flow 20, and GE Discovery 690), whereas the FDG Tübingen dataset was acquired using a single scanner (Siemens Biograph mCT).

🎥 PET/CT acquisition protocol

FDG dataset: Patients fasted at least 6 h prior to the injection of approximately 350 MBq 18F-FDG. Whole-body PET/CT images were acquired using a Biograph mCT PET/CT scanner (Siemens, Healthcare GmbH, Erlangen, Germany) and were initiated approximately 60 min after intravenous tracer administration. Diagnostic CT scans of the neck, thorax, abdomen, and pelvis (200 reference mAs; 120 kV) were acquired 90 sec after intravenous injection of a contrast agent (90-120 ml Ultravist 370, Bayer AG) or without contrast agent (in case of existing contraindications). PET Images were reconstructed iteratively (three iterations, 21 subsets) with Gaussian post-reconstruction smoothing (2 mm full width at half-maximum). Slice thickness on contrast-enhanced CT was 2 or 3 mm.
PSMA dataset: Examinations were acquired on different PET/CT scanners (Siemens Biograph 64-4R TruePoint, Siemens Biograph mCT Flow 20, and GE Discovery 690). The imaging protocol mainly consisted of a diagnostic CT scan from the skull base to the mid-thigh using the following scan parameters: reference tube current exposure time product of 143 mAs (mean); tube voltage of 100kV or 120 kV for most cases, slice thickness of 3 mm for Biograph 64 and Biograph mCT, and 2.5 mm for GE Discovery 690 (except for 3 cases with 5 mm). Intravenous contrast enhancement was used in most studies (571), except for patients with contraindications (26).
The whole-body PSMA-PET scan was acquired on average around 74 minutes after intravenous injection of 246 MBq 18F-PSMA (mean, 369 studies) or 214 MBq 68Ga-PSMA (mean, 228 studies), respectively. The PET data was reconstructed with attenuation correction derived from corresponding CT data. For GE Discovery 690 the reconstruction process employed a VPFX algorithm with voxel size 2.73 mm × 2.73 mm × 3.27 mm, for Siemens Biograph mCT Flow 20 a PSF+TOF algorithm (2 iterations, 21 subsets) with voxel size 4.07 mm × 4.07 mm × 3.00 mm, and for Siemens Biograph 64-4R TruePoint a PSF algorithm (3 iterations, 21 subsets) with voxel size 4.07 mm × 4.07 mm × 5.00 mm.

⌛ Training and test cohort

Training cases: 1,014 studies (900 patients) and 597 PSMA studies (378 patients)
Test cases (final evaluation): 200 studies (50 FDG LMU, 50 FDG UKT, 50 PSMA LMU, 50 PSMA UKT)
Test cases (preliminary evaluation): 5 studies
A case (training or test case) consists of one 3D whole body FDG-PET volume, one corresponding 3D whole body CT volume and one 3D binary mask of manually segmented tumor lesions on FDG-PET of the size of the PET volume. CT and PET were acquired simultaneously on a single PET/CT scanner in one session; thus PET and CT are anatomically aligned up to minor shifts due to physiological motion.

Training set

Training data consists of 1,014 FDG PET/CT studies and 597 PSMA PET/CT studies. We provide the merged data in nnUNet format. The download will contain the resampled FDG and PSMA data as NiFTI files. It also contains the files obtained by running the nnUNet fingerprint extractor and a splits file which we use to design/train our baselines.
NiFTI:

|--- imagesTr
	|--- tracer_patient1_study1_0000.nii.gz  (CT image resampled to PET)

	|--- tracer_patient1_study1_0001.nii.gz  (PET image in SUV)
	|--- ...
|--- labelsTr
	|--- tracer_patient1_study1.nii.gz       (manual annotations of tumor lesions)

|--- dataset.json                             (nnUNet dataset description)
|--- dataset_fingerprint.json                 (nnUNet dataset fingerprint)

|--- splits_final.json                        (reference 5fold split)

|--- psma_metadata.csv                        (metadata csv for psma)

|--- fdg_metadata.csv                         (original metadata csv for fdg)

FDG PET/CT

FDG PET/CT training data contains 1,014 studies acquired at the University Hospital Tübingen and is made publicly available on TCIA (as DICOM, NiFTI and HDF5 files). After download, you can convert the DICOM files to e.g. the NIfTI format using scripts provided here.

Gatidis S, Kuestner T. A whole-body FDG-PET/CT dataset with manually annotated tumor lesions (FDG-PET-CT-Lesions) [Dataset]. The Cancer Imaging Archive, 2022. DOI: 10.7937/gkr0-xv29

PSMA PET/CT

PSMA PET/CT training data contains 597 studies acquired at the University Hospital Munich and is made publicly available on TCIA. After download, you can convert the DICOM files to e.g. the NIfTI format using scripts provided here.

Jeblick, K., et al. A whole-body PSMA-PET/CT dataset with manually annotated tumor lesions (PSMA-PET-CT-Lesions) (Version 1) [Dataset]. The Cancer Imaging Archive, 2024. DOI: 10.7937/r7ep-3x37

Preliminary test set

For the self-evaluation of participating pipelines, we provide access to a preliminary test set. The preliminary test set uses the same imaging data as the final test set, but consists of 5 studies only.
The access to this preliminary set is restricted and only possible through the docker containers submitted to the challenge, and only available for a limited time during the competition. The purpose of this is that participants can check the sanity of their approaches.

Final test set

The final test set consists of 200 studies.

🗃️ Data pre-processing and structure

In a pre-processing step, the TCIA DICOM files are resampled (CT to PET imaging resolution, i.e. same matrix size) and normalized (PET converted to standardized update values; SUV). For the challenge, the pre-processed data will be provided in NifTI format. PET data is standardized by converting image units from activity counts to standardized uptake values (SUV). We recommend to use the resampled CT (CTres.nii.gz) and the PET in SUV (SUV.nii.gz). The mask (SEG.nii.gz) is binary with 1 indicating the lesion. The training and test database have the following structure:

|--- Patient 1
	|--- Study 1
		 |--- SUV.nii.gz    (PET image in SUV)
		 |--- CTres.nii.gz  (CT image resampled to PET)
		 |--- CT.nii.gz     (Original CT image)
		 |--- SEG.nii.gz    (Manual annotations of tumor lesions)
		 |--- PET.nii.gz    (Original PET image as actictivity counts)
	|--- Study 2            (Potential 2nd visit of same patient)
		 |--- ...
|--- Patient 2
	|--- ...

import nibabel as nib
SUV = nib.load(os.path.join(data_root_path, 'PETCT_0af7ffe12a', '08-12-2005-NA-PET-CT Ganzkoerper  primaer mit KM-96698', 'SUV.nii.gz'))

✒ Annotation

FDG PET/CT training and test data from UKT was annotated by a Radiologist with 10 years of experience in Hybrid Imaging and experience in machine learning research. FDG PET/CT test data from LMU was annotated by a radiologist with 8 years of experience in hybrid imaging. PSMA PET/CT training and test data from LMU as well as PSMA PET/CT test data from UKT was annotated by a single reader and reviewed by a radiologist with 5 years of experience in hybrid imaging. br/> The following annotation protocol was defined:
Step 1: Identification of FDG-avid tumor lesions by visual assessment of PET and CT information together with the clinical examination reports.
Step 2: Manual free-hand segmentation of identified lesions in axial slices.

Evaluation

Evaluation will be performed on held-out test cases of 200 patients. Test data will be drawn in part (50%) from the same sources and distributions as the training data, i.e. 50 PSMA-PET/CT scans from LMU, 50 FDG-PET/CT scans from UKT. The other part will be drawn crosswise from the other center, i.e. 50 PSMA-PET/CT scans from UKT and 50 FDG-PET/CT scans from LMU. A combination of two metrics reflecting the aims and specific challenges for the task of PET lesion segmentation:

1. Foreground Dice score of segmented lesions
2. Volume of false positive connected components that do not overlap with positives (=false positive volume)
3. Volume of positive connected components in the ground truth that do not overlap with the estimated segmentation mask (=false negative volume)

Ranking

The submitted algorithms will be ranked according to:
Step 1: Divide the test dataset into subsets based on center and tracer (i.e., PSMA LMU, PSMA UKT, FDG LMU, FDG UKT) and calculated the average metrics for Dice, FPV, and FNV within each subset.
Step 2: Rank the subset averages across all algorithms. For each metric, we computed an intermediate average rank by averaging the ranks of the subsets.
Step 3: Generate the overall rank by combining the three metric ranks using a weighting factor: Dice (0.5), FPV (0.25), and FNV (0.25). In case of equal ranking, the achieved Dice metric will be used as a tie break.

Award category 1: Modelcentric

Best ranked model wins! (for Dice: higher score = better, for FP or FN: lower volumes = better)

Award category 2: Datacentrism

Best data-handling wins!

Codes and Models

Codes

https://github.com/ClinicalDataScience/autoPETIII

Models

Models and documentation of the submitted challenge algorithms can be found in the Leaderboard.

Leaderboard

Category 1: Modelcentric

Category 2: Datacentric

Organizers

• Clemens Cyran
• Michael Ingrisch
• Jakob Dexl
• Katharina Jeblick
• Balthasar Schachtner
• Andreas Mittermeier
• Anna Theresa Stüber

• Sergios Gatidis
• Thomas Küstner