Our study demonstrates that a parent questionnaire administered immediately after the clinical encounter has acceptable sensitivity in estimating whether discussions on weight, nutrition and physical activity occurred compared to a criterion measure of independent audiotape assessment of visits. However, parents tend to over-report counseling. Electronic health record documentation may not be the most valid method of measuring such counseling, since it resulted in underestimation of some discussions on weight and nutrition but markedly significant overestimation of discussions on screen time and physical activity, which may be related to nuances in EHR templates.
Rigorous assessment of the effectiveness of interventions to increase counseling on weight, nutrition and physical activity requires quantifying baseline levels of counseling and validating counseling measures and instruments. Our findings are applicable to clinical performance measurement and improvement since counseling for nutrition and physical activity are now nationally utilized quality measures.8,9
It is therefore important that these measures accurately reflect what occurs during clinical care. Since EHR documentation underestimates the discussion of many topics related to weight and nutrition, and also has the potential to significantly overestimate counseling on certain topics specific to the design of local EHR templates, analyses based on such data may suffer from information bias. Our study indicates that, overall, parental report is a more sensitive source of information on counseling for nutrition and physical activity in pediatric primary care, when compared to EHR documentation.
Our findings are consistent with other studies that evaluated patient report as a measure of counseling on health behaviors. Pbert et al. found that that patient report correlated well with audiotape assessment of the clinical encounter to evaluate smoking cessation counseling and noted that discrepancies between patient reports and audiotapes were primarily attributable to over-reporting of counseling by patients.24
Wilson et al found patient report to be an acceptable approach for measuring counseling on smoking and alcohol compared to audiotape assessment.28
Specific to counseling topics addressed in our study, Sciamanna et al. demonstrated that adult patients’ report of physical activity counseling immediately following clinical encounters correlated well with audiotape evaluation.29
Similarly, Pill et al found that adult patients’ recall of lifestyle counseling delivered during primary care was reasonably high.30
Our study found that parents generally over-report discussion of topics related to weight, nutrition, and physical activity. It is possible that some parents may have recalled discussions that occurred at prior visits or provided desirable responses to please the interviewer or the interviewer’s perceived employer.
Regarding the accuracy of medical record documentation in measuring counseling, Wilson et al reported that medical record documentation to assess smoking and alcohol counseling delivered to adult patients significantly underestimated counseling.28
Pill et al and DiMatteo et al also showed that medical record documentation significantly underestimated lifestyle counseling delivered to adult patients.30,31
Wilson et al however noted that medical record documentation was an extremely specific measure of counseling, with 0% false positive rate.28
We specifically assessed EHR documentation and found that it underestimated some discussions on weight and nutrition similar to the studies mentioned above, but markedly overestimated discussions on screen time and physical activity. An important issue related to utilizing EHR documentation as a measure of counseling is that nuances of specific systems, such as prompts, shortcuts or documentation templates, may contribute to over-documentation of certain topics. In our study, the low specificity of EHR documentation for screen time and physical activity could be partially explained by the common use of the phrase “television/ reading” in age-specific EHR templates utilized in our clinic that are incorporated into the patient instruction section. In many cases, it was clear from other parts of the visit documentation that reading had been discussed at length in a developmental or educational context and was the specific reason for use of this phrase; however, we could not exclude the possibility that discussions related to television had occurred. In abstracting those medical records, we therefore coded physical activity and screen time (a sub-category of physical activity) as having been discussed. We recommend that future studies utilizing medical record review ensure that EHR templates are designed to minimize overlap or ambiguity related to documentation of specific quality measures.
One limitation of our study that affects generalizability was that it was conducted at one site, namely an academic medical center’s outpatient clinic. Patients in our sample were seen by pediatric residents and faculty and approximately 40% of clinicians were pediatric residents. It is possible that parents at non-academic practices may have inherently different relationships and levels of continuity with their physicians, which may affect their recall of counseling. Our results may be relatively generalizable to other parent populations as there was substantial economic and educational diversity in our sample. However, our study was not powered to find significant variation in the accuracy of parental report and EHR documentation across patients with different characteristics, such as those with family history of obesity-related disease. All physicians in our study worked within one health system that utilized the EHR system, Epic. Documentation practices at clinical locations with non-EHR systems, other EHR systems, or variations in EHR tools and templates may yield different results. For example, it is possible that prompts in our EHR templates for counseling on screen time and exercise made it more likely that physicians in our study documented discussions on these topics.
Despite these limitations, strengths of our study include direct observation of physician-parent interactions, large sample size and specific focus on weight, nutrition and activity discussions during pediatric well child visits. Our study adds to existing knowledge regarding discrepancies between actual discussions occurring during clinical encounters, parental recall of discussions and EHR documentation. Nuances in EHR systems raise the potential for significant variation in documentation across sites. Our findings shed light on increasingly important issues related to how EHRs may contribute to accurate or inaccurate documentation. Since EHR documentation is increasingly being utilized for quality measurement, tools built into EHR systems must be designed to support accurate documentation and counseling consistent with recommended care, and to meet performance requirements for both care delivery and documentation. It is important to acknowledge that while our study primarily focused on comparing strategies to measure counseling, further research to determine the effectiveness of such counseling in improving patient outcomes is required.
In conclusion, EHR documentation may not be the most valid measure for performance assessment of counseling. Parental report using a questionnaire administered immediately after the clinical encounter is a better approach for measuring counseling for weight, nutrition and physical activity in research or quality improvement studies when resources do not allow for direct observation. Errors in parental report may be related to the specific questionnaire utilized as well as the context of administration, but these possibilities will need to be addressed in future research.