Toward a standard for the evaluation of PET-Auto-Segmentation methods following the recommendations of AAPM task group No. 211: Requirements and implementation.
- Berthon B Institut Langevin, ESPCI Paris, PSL Research University, CNRS UMR 7587, INSERM U979, Paris, 75012, France.
- Spezi E School of Engineering, Cardiff University, Cardiff, CF24 3AA, United Kingdom.
- Galavis P Department of Radiation Oncology, Langone Medical Center, New York University, New York, NY, 10016, USA.
- Shepherd T Turku PET Centre, Turku University Hospital, Turku, 20521, Finland.
- Apte A Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, 10065, USA.
- Hatt M INSERM, UMR 1101, LaTIM, IBSAM, UBO, UBL, Brest, 29609, France.
- Fayad H INSERM, UMR 1101, LaTIM, IBSAM, UBO, UBL, Brest, 29609, France.
- De Bernardi E Medicine and Surgery Department, University of Milano-Bicocca, Monza, 20900, Italy.
- Soffientini CD Department of Electronics Information and Bioengineering, Politecnico di Milano, Milano, 20133, Italy.
- Ross Schmidtlein C Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, 10065, USA.
- El Naqa I Department of Radiation Oncology, University of Michigan, Ann Arbor, MI, 48103, USA.
- Jeraj R School of Medicine and Public Health, University of Wisconsin, Madison, WI, 53705, USA.
- Lu W Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, 10065, USA.
- Das S Department of Radiation Oncology, University of North Carolina, Chapel Hill, NC, 27599, USA.
- Zaidi H Division of Nuclear Medicine & Molecular Imaging, Geneva University Hospital, Geneva CH-1211, Switzerland.
- Mawlawi OR Department of Imaging Physics, MD Anderson Cancer Center, Houston, TX, 77030, USA.
- Visvikis D INSERM, UMR 1101, LaTIM, IBSAM, UBO, UBL, Brest, 29609, France.
- Lee JA IREC/MIRO, Université catholique de Louvain (IREC/MIRO) & FNRS, Brussels, 1200, Belgium.
- Kirov AS Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, 10065, USA.
- 2017-05-06
Published in:
- Medical physics. - 2017
PET segmentation
PET/CT
conformity index
outlining assessment
Algorithms
Humans
Image Processing, Computer-Assisted
Phantoms, Imaging
Software
Tomography, X-Ray Computed
English
PURPOSE
The aim of this paper is to define the requirements and describe the design and implementation of a standard benchmark tool for evaluation and validation of PET-auto-segmentation (PET-AS) algorithms. This work follows the recommendations of Task Group 211 (TG211) appointed by the American Association of Physicists in Medicine (AAPM).
METHODS
The recommendations published in the AAPM TG211 report were used to derive a set of required features and to guide the design and structure of a benchmarking software tool. These items included the selection of appropriate representative data and reference contours obtained from established approaches and the description of available metrics. The benchmark was designed in a way that it could be extendable by inclusion of bespoke segmentation methods, while maintaining its main purpose of being a standard testing platform for newly developed PET-AS methods. An example of implementation of the proposed framework, named PETASset, was built. In this work, a selection of PET-AS methods representing common approaches to PET image segmentation was evaluated within PETASset for the purpose of testing and demonstrating the capabilities of the software as a benchmark platform.
RESULTS
A selection of clinical, physical, and simulated phantom data, including "best estimates" reference contours from macroscopic specimens, simulation template, and CT scans was built into the PETASset application database. Specific metrics such as Dice Similarity Coefficient (DSC), Positive Predictive Value (PPV), and Sensitivity (S), were included to allow the user to compare the results of any given PET-AS algorithm to the reference contours. In addition, a tool to generate structured reports on the evaluation of the performance of PET-AS algorithms against the reference contours was built. The variation of the metric agreement values with the reference contours across the PET-AS methods evaluated for demonstration were between 0.51 and 0.83, 0.44 and 0.86, and 0.61 and 1.00 for DSC, PPV, and the S metric, respectively. Examples of agreement limits were provided to show how the software could be used to evaluate a new algorithm against the existing state-of-the art.
CONCLUSIONS
PETASset provides a platform that allows standardizing the evaluation and comparison of different PET-AS methods on a wide range of PET datasets. The developed platform will be available to users willing to evaluate their PET-AS methods and contribute with more evaluation datasets.
The aim of this paper is to define the requirements and describe the design and implementation of a standard benchmark tool for evaluation and validation of PET-auto-segmentation (PET-AS) algorithms. This work follows the recommendations of Task Group 211 (TG211) appointed by the American Association of Physicists in Medicine (AAPM).
METHODS
The recommendations published in the AAPM TG211 report were used to derive a set of required features and to guide the design and structure of a benchmarking software tool. These items included the selection of appropriate representative data and reference contours obtained from established approaches and the description of available metrics. The benchmark was designed in a way that it could be extendable by inclusion of bespoke segmentation methods, while maintaining its main purpose of being a standard testing platform for newly developed PET-AS methods. An example of implementation of the proposed framework, named PETASset, was built. In this work, a selection of PET-AS methods representing common approaches to PET image segmentation was evaluated within PETASset for the purpose of testing and demonstrating the capabilities of the software as a benchmark platform.
RESULTS
A selection of clinical, physical, and simulated phantom data, including "best estimates" reference contours from macroscopic specimens, simulation template, and CT scans was built into the PETASset application database. Specific metrics such as Dice Similarity Coefficient (DSC), Positive Predictive Value (PPV), and Sensitivity (S), were included to allow the user to compare the results of any given PET-AS algorithm to the reference contours. In addition, a tool to generate structured reports on the evaluation of the performance of PET-AS algorithms against the reference contours was built. The variation of the metric agreement values with the reference contours across the PET-AS methods evaluated for demonstration were between 0.51 and 0.83, 0.44 and 0.86, and 0.61 and 1.00 for DSC, PPV, and the S metric, respectively. Examples of agreement limits were provided to show how the software could be used to evaluate a new algorithm against the existing state-of-the art.
CONCLUSIONS
PETASset provides a platform that allows standardizing the evaluation and comparison of different PET-AS methods on a wide range of PET datasets. The developed platform will be available to users willing to evaluate their PET-AS methods and contribute with more evaluation datasets.
- Language
-
- English
- Open access status
- hybrid
- Identifiers
-
- DOI 10.1002/mp.12312
- PMID 28474819
- Persistent URL
- https://sonar.ch/global/documents/53219
Statistics
Document views: 48
File downloads:
- Full-text: 0