How are contouring accuracy "scores" calculated?
Method 1: StructSure Method (Most Sensitive)
ProKnow uses the advanced “StructSure” method of scoring (US Patent 8,081,813) to generate the contour accuracy scores for the Contouring Accuracy Program. This patented method was first published in 2012 in the International Journal of Radiation Oncology Biology Physics (i.e. “Red Journal”).
This method generates a score (0 to 100) for the accuracy of the user’s contour when compared to the “gold” or expert contour. It is a 3D algorithm that creates a volume from the user’s contours, a volume from the expert’s contours, and then goes through and finds all errant voxels and penalizes them. An errant voxel can be “missing” (user’s contour is too small, i.e. inside the expert’s contour) or “extra” (user’s contour is too large, i.e. outside the expert’s contour). If an errant voxel is within a “forgiveness region” (usually 1 mm) then there is no penalty. If an errant voxel is outside the forgiveness region, it gets a penalty assigned based on “how wrong it is” (i.e. based on the 3D distance error to find the closest surface of the expert’s contour). The voxel penalty levied per errant voxel is 0.5 voxels per mm. For example, if a user’s contour contains a voxel that is 3 mm outside the expert’s contour, it will get a penalty equivalent to: (0.5 voxels / mm) * (3 mm – 1 mm forgiveness) = 1.0 voxel penalty.
The final score is calculated as: 100 * [ # Expert Voxels – Sum(Penalties, Over All Voxels) ] / [# Expert Voxels]. The max score is 100.0, and minimum is 0.00 (all negative scores will be truncated to 0.00). Even if you have matcing voxels, a score of zero is possible if the number and/or extent of errors is large.
This method is superior (i.e. more sensitive) than more traditional methods such as the Dice Coefficient. However, ProKnow does calculate and display the Dice Coefficient on the results, for those interested. The Dice Coefficient is explained below.
Method 2: Dice Coefficient (Less Sensitive)
The Dice Coefficient can be calculated from two volumes (or in 2D, from two areas). A perfect overlap yields a value of 1.00, while no overlap yields a value of 0.00. The Dice Coefficient is often not sensitive enough for radiation therapy contouring purposes, because a very large voxel error (e.g. 20 mm) is penalized the same as an insignificant error (e.g. 1 mm). The equation is as follows:
Dice Coefficient = [ 2 x | V1 ∩ V2 | ] / [ | V1 | + | V2 | ]
If you have any overlap at all with the expert’s structure volumes, then your Dice Coefficient will be greater than 0.00 (in contrast to the advanced StructSure scoring where your penalties can exceed your matching voxels, thus allowing scores of zero even if there is overlapping volume).
References
Breunig et al. “A system for continual quality improvement of normal tissue delineation for radiation therapy treatment planning.” Int J Radiat Oncol Biol Phys. 2012 Aug 1;83(5).
Rodrigues et al. “Categorizing segmentation quality using a quantitative quality assurance algorithm.” J Med Imaging Radiat Oncol. 2012 Dec;56(6).