Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification

Abstract

Computer vision wood identification (CVWID) has focused on laboratory studies reportingconsistently high model accuracies with greatly varying input data quality, data hygiene, andwood identification expertise. Employing examples from published literature, we demonstratethat the highly optimistic model performance in prior works may be attributed to evaluating thewrong functionality—wood specimen identification rather than the desired wood species or genusidentification—using limited datasets with data hygiene practices that violate the requirement of clearseparation between training and evaluation data. Given the lack of a rigorous framework for a validmethodology and its objective evaluation, we present a set of minimal baseline quality standardsfor performing and reporting CVWID research and development that can enable valid, objective,and fair evaluation of current and future developments in this rapidly developing field. To elucidatethe quality standards, we present a critical revisitation of a prior CVWID study of North Americanring-porous woods and an exemplar study incorporating best practices on a new dataset covering thesame set of woods. The proposed baseline quality standards can help translate models with high insilico performance to field-operational CVWID systems and allow stakeholders in research, industry, and government to make informed, evidence-based modality-agnostic decisions.

Citation

Ravindran, Prabu; Wiedenhoeft, Alex C. 2022. Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification. Forests. 13(4): 632. https://doi.org/10.3390/f13040632.