PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of bmcmidmBioMed Centralsearchsubmit a manuscriptregisterthis articleBMC Medical Informatics and Decision Making
 
BMC Med Inform Decis Mak. 2012; 12: 8.
Published online Feb 15, 2012. doi:  10.1186/1472-6947-12-8
PMCID: PMC3307431
Predicting sample size required for classification performance
Rosa L Figueroa,#1 Qing Zeng-Treitler ,corresponding author#2 Sasikiran Kandula,#2 and Long H Ngo#3
1Dep. Ing. Eléctrica, Facultad de Ingeniería, Universidad de Concepción, Concepción, Chile
2Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA
3Department of Medicine, Beth Israel Deaconess Medical Center and Harvard Medical School, Boston, MA, USA
corresponding authorCorresponding author.
#Contributed equally.
Rosa L Figueroa: rosfigue/at/udec.cl; Qing Zeng-Treitler : q.t.zeng/at/utah.edu; Sasikiran Kandula: sasi.kandula/at/utah.edu; Long H Ngo: lngo/at/bidmc.harvard.edu
Received June 30, 2011; Accepted February 15, 2012.
Abstract
Background
Supervised learning methods need annotated data in order to generate efficient models. Annotated data, however, is a relatively scarce resource and can be expensive to obtain. For both passive and active learning methods, there is a need to estimate the size of the annotated sample required to reach a performance target.
Methods
We designed and implemented a method that fits an inverse power law model to points of a given learning curve created using a small annotated training set. Fitting is carried out using nonlinear weighted least squares optimization. The fitted model is then used to predict the classifier's performance and confidence interval for larger sample sizes. For evaluation, the nonlinear weighted curve fitting method was applied to a set of learning curves generated using clinical text and waveform classification tasks with active and passive sampling methods, and predictions were validated using standard goodness of fit measures. As control we used an un-weighted fitting method.
Results
A total of 568 models were fitted and the model predictions were compared with the observed performances. Depending on the data set and sampling method, it took between 80 to 560 annotated samples to achieve mean average and root mean squared error below 0.01. Results also show that our weighted fitting method outperformed the baseline un-weighted method (p < 0.05).
Conclusions
This paper describes a simple and effective sample size prediction algorithm that conducts weighted fitting of learning curves. The algorithm outperformed an un-weighted algorithm described in previous literature. It can help researchers determine annotation sample size for supervised machine learning.
Articles from BMC Medical Informatics and Decision Making are provided here courtesy of
BioMed Central