The accuracy of a binarization algorithm is often calculated relative to a ground truth image. Except for synthetically generated images, no ground truth image exists. Evaluating binarization on real images is preferred. The ground truthing between and among different operators is compared. Four direct metrics were used. The variability of the results of five different automatic binarization algorithms were compared to that of manual ground truth results. Significant variability in the ground truth results was found.
Barney Smith, Elisa H.. (2010). "An Analysis of Binarization Ground Truthing". DAS '10 Proceedings of the 9th IAPR International Workshop on Document Analysis Systems, .