Search tips
Search criteria 


J Appl Behav Anal. 1977 Spring; 10(1): 141–150.
PMCID: PMC1311161

Artifact, bias, and complexity of assessment: the ABCs of reliability


Interobserver agreement (also referred to here as “reliability”) is influenced by diverse sources of artifact, bias, and complexity of the assessment procedures. The literature on reliability assessment frequently has focused on the different methods of computing reliability and the circumstances under which these methods are appropriate. Yet, the credence accorded estimates of interobserver agreement, computed by any method, presupposes eliminating sources of bias that can spuriously affect agreement. The present paper reviews evidence pertaining to various sources of artifact and bias, as well as characteristics of assessment that influence interpretation of interobserver agreement. These include reactivity of reliability assessment, observer drift, complexity of response codes and behavioral observations, observer expectancies and feedback, and others. Recommendations are provided for eliminating or minimizing the influence of these factors from interobserver agreement.

Full text

Full text is available as a scanned copy of the original print version. Get a printable copy (PDF file) of the complete article (1.5M), or click on a page image below to browse page by page. Links to PubMed are also available for Selected References.

Selected References

These references are in PubMed. This may not be the complete list of references from this article.
  • AZRIN NH, HOLZ W, ULRICH R, GOLDIAMOND I. The control of the content of conversation through reinforcement. J Exp Anal Behav. 1961 Jan;4:25–30. [PMC free article] [PubMed]
  • Cusella-De Angelis MG, Lyons G, Sonnino C, De Angelis L, Vivarelli E, Farmer K, Wright WE, Molinaro M, Bouchè M, Buckingham M, et al. MyoD, myogenin independent differentiation of primordial myoblasts in mouse somites. J Cell Biol. 1992 Mar;116(5):1243–1255. [PMC free article] [PubMed]
  • Mehta S. Anticoagulant drugs and central nerve blockade. Anaesthesia. 1992 Jan;47(1):70–70. [PubMed]
  • Kent RN, O'Leary KD, Diament C, Dietz A. Expectation biases in observational evaluation of therapeutic change. J Consult Clin Psychol. 1974 Dec;42(6):774–780. [PubMed]
  • Bondy C, Werner H, Roberts CT, Jr, LeRoith D. Cellular pattern of type-I insulin-like growth factor receptor gene expression during maturation of the rat brain: comparison with insulin-like growth factors I and II. Neuroscience. 1992;46(4):909–923. [PubMed]
  • Redfield J, Paul GL. Bias in behavioral observation as a function of observer familiarity with subjects and typicality of behavior. J Consult Clin Psychol. 1976 Feb;44(1):156–156. [PubMed]
  • Sheen-Chen SM, Chen YS, Chou FF, Eng HL. Primary squamous cell carcinoma of the breast. South Med J. 1992 Feb;85(2):207–209. [PubMed]

Articles from Journal of Applied Behavior Analysis are provided here courtesy of Society for the Experimental Analysis of Behavior