PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jfamcommHomeCurrent issueInstructionsSubmit article
 
J Family Community Med. 2012 May-Aug; 19(2): 119–124.
PMCID: PMC3410175

Interpreting the psychometric properties of the components of primary care instrument in an elderly population

Abstract

Objective:

To determine the psychometric properties of the Components of Primary Care Instrument (CPCI) in a patient population aged 65 or older.

Materials and Methods:

795 participants in the OKLAHOMA Studies, a longitudinal population-based study of predominantly Caucasian, elderly patients, completed the CPCI. Reliability analysis and confirmatory factor analysis were done to provide psychometric properties for this elderly sample. Models were constructed and tested to determine the best fit for the data including the addition of a method factor for negatively worded items.

Results:

Cronbach's alphas were comparable to values reported in prior studies. The confirmatory factor analysis with factor inter-correlations and a method factor each improved the fit of the factor model to the data. The combined model's fit approached the level conventionally recognized as adequate.

Conclusion:

CPCI appears to be a reliable tool for describing patient perceptions of the quality of primary care for patients over age 65.

Keywords: Components of primary care instrument, elderly, older patients, primary care, reliability, validity

INTRODUCTION

In a 1996 report, the Institute of Medicine (IOM) updated its 1978 definition of primary care. The new definition states that “primary care is the provision of integrated, accessible health care services by clinicians that are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and community.”[1] Based upon this definition, primary care is a function of the health care system that may be carried out by many types of health care practitioners. The new definition also re-conceptualized the previous standards of comprehensiveness, continuity, and coordination as integration and added that primary care should encompass a sustained partnership and exist in the context of family and community.[2] Numerous studies have shown that primary care results in more effective prevention of sickness and death and is associated with lower health care costs.[3] It is therefore important that we are able to assess the quality of primary care that clinicians provide, especially from the perspective of the patients they serve.

The CPCI was developed for use in the Direct Observation of Primary Care study.[2] It was designed to measure the processes of primary care rather than its structural or systemic aspects such as access to care.[4] The initial CPCI with 20 questions was revised after the initial validation study, increasing the total number of questions to 43 to measure 8 domains of primary care.

Analyses of the revised CPCI showed good internal consistency (Cronbach's alphas range from 0.71 to 0.92).[2] Approximately 15 minutes is required to complete the instrument,[5] which has a Flesch–Kincaid reading grade level of 5.7. Given the overall psychometric properties and content coverage, the CPCI was selected for use in the Oklahoma Longitudinal Assessment of the Health Outcomes of Mature Adults (OKLAHOMA) Studies. However, the CPCI had not been specifically tested in elderly populations. Given that the quality of data differs according to the measurement properties of the instrument,[6,7] it is important to know the psychometric properties of the CPCI when used with an older population. Therefore, the purpose of this study was to determine the usability and psychometric properties of the CPCI as a measure of the functional components of primary care in a population of patients aged 65 or older and specifically, to determine the impact of negatively worded items on the overall data structure.

MATERIALS AND METHODS

The data for this study were obtained from the OKLAHOMA Studies (Oklahoma Longitudinal Assessment of the Health Outcomes of Mature Adults) data set. Previous publications have described the OKLAHOMA Studies data collection methodology in greater detail.[8] Between January 1, 1999, and December 31, 2000, 23 family physician members of the Oklahoma Physicians Resource / Research Network (OKPRN) developed a list of patients 65 years of age and older they had seen in the previous 18 months. The physicians worked in nine different primary care practices within a 50-mile radius of Oklahoma City. Patients were excluded from the lists if they had switched physicians, died, were living in a nursing home, or were felt to be too confused to provide informed consent. Eligible patients were sent a letter by their physician inviting them to participate in the study. Two weeks later, the project coordinator followed up with these patients via telephone.

Those who agreed to participate were asked to complete a questionnaire sent to them 2 weeks prior to their enrolment visit. The questionnaire included demographic information, habits, medical conditions, symptoms, functional status, self-rated health, the CPCI, and three measures of health-related quality-of-life: the Quality of Well-Being Self-administered Scale (QWB-SA), the Health Utilities Index (HUI-3), and the Medical Outcomes Short Form 36 (SF-36). Patients were also asked about the length of their relationship with their physician and when/why they had last changed physicians.

Participating patients were enrolled at their family physician's office by one of two research nurses. The nurses reviewed the study protocol and obtained informed consent from the patients. All participants were invited to re-enroll in each of the following four years. For this study, all analyses used the baseline data. The study was reviewed and approved by the University of Oklahoma Health Sciences Center Institutional Review Board.

Statistical analysis

Sample Description. Characteristics of the sample (number and percentage) were calculated for gender, age, race, education, annual household income, and number of years patients has seen their physician. CPCI subscale scores for all participants were calculated and summary statistics (mean, standard deviation, N of items, min, max) were reported by sub-scale. Cronbach's alphas were calculated for each subscale as an indicator of internal consistency reliability. Cronbach's alphas for each subscale were also reported for the 1999 Flocke sample.[5] These statistical analyses were performed using the Statistical Packages for Social Sciences (SPSS), version 15.[9]

Confirmatory factor analysis

In order to determine the fit of the factor structure to the OKLAHOMA Studies CPCI data, a confirmatory factor analysis was conducted. Recognizing that negatively phrased items might evoke a different type of response, and that such items share some variance distinct from the concepts that the factors measure, we used the Correlated Trait, Correlated Methods Minus One (CT-C(M-1) approach, which is appropriate for data in which items are phrased either positively or negatively.[10] Unlike the CT-CM approach, which would use two method factors, one for the positively worded and one for the negatively worded items, the CT-C(M-1) uses only one method factor, and in this study it was for the negatively phrased items. Interpretation of the method factor involves a comparison with the reference method, i.e., those items that are positively phrased.

The purpose of including the method factor is to capture “reliable residual effects that are specific to the negatively worded items not shared with the positively worded items” (P 51).[10] In this approach, we observe convergent validity when the variance explained by the method factor is small in comparison with the variance accounted for by the trait factor. Advantages of the CT-C (M-1) method are that trait, method and error components are uncorrelated in the model, and that the maximum likelihood model fitting techniques usually succeed at fitting the model.

To express the CT-C (M-1) model, we constructed a confirmatory factor analysis using eight Components of Primary Care Index (CPCI) factors identified by Flocke, and a method factor for the 5 negatively phrased items on the instrument, 4 of which were from the CPCI Communication factor. The model was constructed using the graphics module of the AMOS 16 software[11] and SPSS, Inc., Chicago,[9] analyzing “moment structures” using maximum likelihood techniques. Each factor from Flocke's analysis was constructed as a latent factor, with Flocke's list of items as indicators. In some models, the latent variables were allowed to be correlated with one another [see Model C in Figure 1]. To test the necessity of allowing correlations, in an alternative analysis [Model A of Figure 1], the model was constructed with no curved lines between the latent factor variables, which is equivalent to forcing the value of each inter-correlation to be 0.

Figure 1
CPCI factor loadings for correlated vs. uncorrelated models and method factor vs. no method factor models

The Method factor was constructed by adding one more latent variable, and allowing each negatively phrased item to load on it by drawing an arrow from the factor to each item [Model B of Figure 1]. Thus each of the negatively phrased items was an indicator for two different latent variables, one reflecting the substantive content of a factor of CPCI, and the other being the method factor. The correlation between the method factor and each of the CPCI factors was constrained to be 0, as graphically expressed by the absence of curved double-headed arrows between the factors.

To assess the adequacy of the full model, with factor inter-correlations and with the Negative Items method factor, several of the indices recommended in Appendix C of Arbuckle[10] were used, including Chi-square, Root Mean Square Error of Approximation (RMSEA), Tucker-Lewis Index (TLI), and the modified Expected Cross Validation Index (ECVI). To assess the contribution of the factor inter-correlations, a comparison of the Chi-squared difference, with respect to the difference in the degrees of freedom, was made between the model with only the eight CPCI factors, with inter-correlations (Model C) versus without (Model A). To assess the contribution of the Negative Items Method factor, a comparison of the Chi-squared difference was similarly made between the models with (Model B) and without the method factor (Model A). This was also done for the correlated-factor models (Models D and C). To assess the impact of the negatively worded items upon subscale interpretations, we compared changes in the factor loadings (comparing loadings in Models D and C), while controlling for factor inter-correlations.

RESULTS

Of the 2553 participants eligible for the OKLAHOMA Studies project, 40% declined and 4 attempts to locate 29% proved futile. Seven hundred and ninety-nine patients (31% of eligible participants) completed the initial questionnaire. For these analyses, participants who did not answer more than 50% of the items on the CPCI questionnaire were excluded as per the scoring protocol developed by Flocke[5] that included reverse scoring for 5 items. Four participants were excluded for this reason leaving 795 participants. The scoring protocol also requires a minimum number of items to be answered per subscale. If a participant did not answer the minimum number of items on a particular subscale, the score for that subscale was not calculated. Table 1 shows the demographics of the participants.

Table 1
Descriptive statistics of physician visits and demographics of OKLAHOMA studies year one participants

Internal consistency for each of the CPCI subscales as measured by Cronbach's Alpha are shown in Table 2 along with those reported by Flocke.[5] These range from 0.68 to 0.89 and are comparable to the internal consistency values reported by Flocke. Although these values vary somewhat across subscales, they reflect a reasonably high level of reliability. Item means and standard deviations from the two samples were not dissimilar, although no expectation of comparability was maintained given the differences between the two target populations.

Table 2
Internal consistency reliability values of OKLAHOMA studies CPCI subscales

The confirmatory factor analysis models are illustrated in Figure 1 and their statistics summarized in Table 3. Model A has only the 8 CPCI factors and no method factor. Model B adds the Negatively Worded Item method factor, which allows the algorithm to fit one more latent factor, thereby using an additional 5 degrees of freedom and reducing the Chi-square statistic. This reduction reflects a statistically significant improvement of Model B in explaining the data over Model A. Models C and D show the effect of allowing the CPCI factors to be inter-correlated. Compared to Model A, the inter-correlations in Model C produce a model whose predictions differ from the actual data “moments” less than before, and the Chi-squared difference corresponding to that improvement is statistically significant. This supports Flocke's original decision to use a factor analysis method, which permits the factors to be correlated. The comparison of Model D to Model C shows that the Negatively Worded Items method factor makes a significant improvement when added to the inter-correlated CPCI factors, just as when those factors are forced to be uncorrelated [Table 3].

Table 3
Confirmatory factor analysis model comparison statistics

Model D, the best fitting model, which uses all the theoretically justifiable structures available to us, still has a Chi-square test indicating it is significantly different from the data. The ratio of the Chi-squared of 3314 divided by the 827 degrees of freedom is 4.008, which is still above the 3.84 cut-off below which would no longer be statistically significant. The RMSEA statistic conventionally requires a number less than 0.05 to be considered a good fit. Our best model has an RMSEA of 0.06 (CI 0.058 to 0.062) and so still does not quite attain a satisfactory fit.

Inspection of the normalized loadings of the items on to the factors in Figure 1 shows that the loadings are of roughly similar magnitude within each factor. Factors with fewer items tend to have larger loadings. The similarity of the loadings changes little between the models with the poorest fit (Model A, RMSEA = 0.093) and the best fit (Model D, RMSEA = 0.060), which suggests that our conclusion that equal weights provide an adequate index would not be altered if we could attain an RMSEA of 0.05. The Communication and Advocacy factors, the two factors that include some negatively phrased items, have items with the lowest loadings. The effects of adding the negative item method factor upon the loadings of the Communication and Advocacy scales reduced the loading of each item on its content factor, though only slightly.

DISCUSSION

One potential source of information about the quality of primary care is patient reports and given that the elderly make up 13% of primary care patients,[12] it is important to have an instrument that is reliable for measuring their perceptions. Three patient-completed primary care assessment instruments, the Primary Care Assessment Survey (PCAS), the Primary Care Assessment Tool (PCAT), and the Components of Primary Care Instrument (CPCI), assess the functional components of primary care defined by the IOM from the patient's perspective.[1315]

Each of these instruments approaches primary care in a different way and while all three have published internal consistencies in acceptable ranges, the CPCI is the shortest instrument and therefore, puts the least burden on patients and like the PCAS requires only a 5th grade reading level[13] compared with the high school reading level required by the PCAT.[14]

The reliability of the CPCI was confirmed in this study involving patients over the age of 65. The reliability analysis showed good internal consistency, with Cronbach's alpha values high and similar to those obtained in the primary analysis of the full instrument by Flocke.[5] The normalized parameters reflecting the loadings of the items on the factors were for the most part high and of similar magnitude. The high Cronbach's alpha reliabilities and similar loadings make it reasonable to calculate scale scores using equal weights.

The coordination of care subscale is of concern. A person may choose “does not apply” to three of the six items in that subscale in contrast to other scales. These items address whether a physician helps the patient interpret labs and imaging studies, follows up with, and communicates with other care providers for that patient. If a patient does not use these services, the items are not relevant. Naturally, these items had the highest omission rate of all CPCI items.

Five CPCI items are negatively worded, and 4 of them loaded on a single factor. This raises the concern that participants may have responded differently to these items, particularly if they had trouble shifting the direction of their responses. Since our study population comprised older adults, it is possible that some confusion with the intermittent negatively worded items may have arisen as a result of mental decline. The purpose of including the Negative Items Method Factor was not only to determine if these items cohered and added statistically significantly to the explanation of the data, but also to see if they diminished the meaning of the other intended CPCI factors. Including the Negative Item Method Factor has virtually no effect on the loadings of any indicators on any CPCI factors, except for those items that are negatively phrased.

It can be seen that when the method factor was added, whether in the absence of CPCI factor inter-correlations or with inter-correlations, the CPCI loadings of the Interpersonal Communication factor items that have negative wording decreased slightly. The loadings on the method factor are about the same magnitude as the loadings of the same items on the CPCI factor. The key question is whether the loadings on the Communications factor have been so reduced that there is no longer any justification for their inclusion in that factor, i.e., whether the method factor dominates the meaning factor. The weaker loading factors are reduced from the 0.38 to 0.48 range to the 0.29 to 0.35 range. Those items already had some of the lowest loadings of any of the questionnaire items – probably because of the negative phrasing – even before the Negatively Worded Item Method Factor was added to the analysis. Although adding the Negative Item Method factor highlights this difficulty with the Interpersonal Communication factor, it does not change the character of the loading pattern of the items of that factor; for our present purposes, it still seems reasonable to produce scales using equal weights and use the scales as intended.

For future studies, it may be desirable to spread out negative items more evenly among all the CPCI factors, use only positive items and vary the response set, provide training in how to interpret and respond to negatively worded items, or interview respondents and check their responses for logical inconsistency (rather than have them fill out a paper questionnaire on their own). It is not known whether these measures would be pertinent to elderly respondents only, or would be useful in general.

In summary, the CPCI appears to be a reliable tool for analyzing the quality of primary care a physician provides to elderly patients. Because older patients visit physicians more regularly, their opinions about the primary care a physician provides are particularly valuable. Further analyses using the CPCI might include a more racially diverse population and further evaluation of patient outcomes in relation to CPCI subscale scores.

Footnotes

Source of Support: This study received support from Presbyterian Health Foundation and the University of Oklahoma Department of Family and Preventive Medicine

Conflict of Interest: The authors reported no potential conflict of interest relevant to this article

REFERENCES

1. Donaldson MS, Vanselow NA. The nature of primary care. J Fam Pract. 1996;42:113–6. [PubMed]
2. Flocke SA. Operationalizing primary care concepts. 32nd annual spring conference society of teachers of family medicine. Seattle, Washington: Society of Teachers of Family Medicine; 1999.
3. Starfield B, Shi L, Macinko J. Contribution of primary care to health systems and health. Milbank Q. 2005;83:457–502. [PubMed]
4. Flocke SA. Measuring attributes of primary care: Development of a new instrument. J Fam Psychol. 1997;45:64–74. [PubMed]
5. Flocke SA, Orzano AJ, Selinger HA, Werner JJ, Vorel L, Nutting PA, et al. Does managed care restrictiveness affect the perceived quality of primary care? A report from ASPN. J Fam Pract. 1999;48:762–8. [PubMed]
6. Horan P, DiStefano C, Motl R. Wording effects in self-esteem scales: Methodological artifact or response style? Struct Equ Modeling. 2003;10:444–56.
7. DiStefano C, Motl R. Personality correlates of method effects due to negatively owrded items on the Rosenberg Self-Esteem scale. Pers Individ Dif. 2009;46:309–13.
8. Mold JW, Fryer GE, Roberts AM. When do older patients change primary care physicians? J Am Board Fam Pract. 2004;17:453–60. [PubMed]
9. Version 17.0.1. Chicago: SPSS Inc; 2008. SPSS for Windows [Computer program]
10. Geiser C, Eid M, Nussbeck FW. On the meaning of the latent variables in the CT-C(M-1) model: A comment on Maydeu-Olivares and Coffman (2006) Psychol Methods. 2008;13:49–57. [PubMed]
11. Arbuckle JL. Amos 16.0 User's Guide. Chicago, IL: SPSS, Inc; 2007.
12. U.S. Census Bureau. State and County QuickFacts. State and County QuickFacts. 2012. [Last accessed on 2012 Apr 16]. Available from: http://quickfacts.census.gov/qfd/states/00000.html .
13. Safran DG, Kosinski M, Tarlov AR, Rogers WH, Taira DH, Lieberman N, et al. The primary care assessment survey. Med Care. 1998;36:728–39. [PubMed]
14. Shi L, Starfield B, Xu J. Validating the adult primary care assessment tool. J Fam Pract. 2001;50:161–75.
15. Flocke SA, Stange KC, Fedirko TL. Dissemination of information about the US preventive service task force guidelines. Arch Fam Med. 1994;3:1006–8. [PubMed]

Articles from Journal of Family & Community Medicine are provided here courtesy of Medknow Publications