|Home | About | Journals | Submit | Contact Us | Français|
To develop and psychometrically test an owner self-administered questionnaire designed to assess severity and impact of chronic pain in dogs with osteoarthritis.
70 owners of dogs with osteoarthritis and 50 owners of clinically normal dogs.
Standard methods for the stepwise development and testing of instruments designed to assess subjective states were used. Items were generated through focus groups and an expert panel. Items were tested for readability and ambiguity, and poorly performing items were removed. The reduced set of items was subjected to factor analysis, reliability testing, and validity testing.
Severity of pain and interference with function were 2 factors identified and named on the basis of the items contained in them. Cronbach’s α was 0.93 and 0.89, respectively, suggesting that the items in each factor could be assessed as a group to compute factor scores (ie, severity score and interference score). The test-retest analysis revealed κ values of 0.75 for the severity score and 0.81 for the interference score. Scores correlated moderately well (r = 0.51 and 0.50, respectively) with the overall quality-of-life (QOL) question, such that as severity and interference scores increased, QOL decreased. Clinically normal dogs had significantly lower severity and interference scores than dogs with osteoarthritis.
A psychometrically sound instrument was developed. Responsiveness testing must be conducted to determine whether the questionnaire will be useful in reliably obtaining quantifiable assessments from owners regarding the severity and impact of chronic pain and its treatment on dogs with osteoarthritis.
It is crucial to have quantitative measures of chronic pain that are valid and reliable in clinical patients to enable development and testing of interventions (such as drugs or surgical procedures) designed to reduce such pain. In the past, studies designed to test the efficacy of interventions intended to decrease chronic pain in dogs with osteoarthritis have relied heavily on a veterinarian’s assessment of lameness supported by values generated through gait analysis by use of a force plate. When collected properly, data on gait analysis offer an objective measure that can be reliably monitored over time; however, it can be extremely time consuming, requires specialized equipment, and relies on relatively strict inclusion criteria. In addition, these measures only evaluate an animal at 1 specific point in time, and weight bearing on an affected limb is only 1 part of the much larger picture of chronic pain in dogs with osteoarthritis.1–5
When an owner brings a dog to a veterinarian with a concern that the dog has signs of a chronic painful condition, they report a number of behaviors that they attribute to the painful condition (eg, no longer climbs stairs or jumps onto the bed). In monitoring the progression of the dog’s condition over time, veterinarians rely heavily on an owner’s report of improvement in the dog’s pain-related behaviors following the initiation of treatment (which most often involves a nonsteroi-dal anti-inflammatory drug). When an owner reports improvement in pain-related behaviors without substantial adverse effects, veterinarians continue to offer the treatment. However, when an owner reports no improvement in pain-related behaviors, veterinarians may increase the dose of the nonsteroidal anti-inflammatory drug or switch to another medication. Although an owner’s detailed assessment of chronic pain is routinely relied on in clinical decision making for the management of osteoarthritis pain, it is not used routinely as an outcome measure in clinical trials designed to evaluate interventions for osteoarthritis pain.1–9 This apparently is attributable to the subjective nature of an owner’s assessment; however, sound methods exist for the development and application of tools to assess and quantify peoples’ perception of subjective states.10–12 Although pain behaviors are the result of a complex set of inputs and qualities unique to each animal, appropriate, established principles of questionnaire development10–18 can be used to develop valid and reliable behavior-based assessment instruments for owner appraisal of chronic pain in dogs.
Instrument (questionnaire) development involves the generation of items (questions) that represent theoretic constructs (ie, factors). In the study reported here, the construct we were attempting to assess with the questions posed was an owner’s perception of chronic pain associated with osteoarthritis in their dog. Therefore, the items were generated through focus groups of owners of dogs with osteoarthritis and then reviewed by veterinarians who routinely manage dogs with the disease. Once items are generated, factor analysis can be used to investigate which aspects of chronic pain may be represented by the questions that are asked; then, the instrument is subjected to validity and reliability testing.10,11,19 It must be established that the data-gathering instrument will target the characteristic it is designed to measure (ie, chronic pain in dogs with osteoarthritis), which is defined as the validity of the instrument. In addition, the instrument must measure the characteristic it is designed to measure in a consistent manner. The pattern of consistency is referred to as reliability.10,11,19
The study reported here was performed to develop and then conduct psychometric (validity and reliability) testing of an instrument (ie, the CBPI) designed to measure chronic pain in dogs with osteoarthritis. The CBPI was intended as an owner-completed instrument designed to reliably quantify the owners’ perceptions of the severity and impact of chronic pain on their dogs with osteoarthritis.
Standard methods for the development and psychometric testing of instruments designed to assess subjective states10–13,15,18–37 were used for development of the CBPI. Factor format, item structure, and response scaling of the CBPI were based on the rigorously tested BPI, which is an instrument routinely used to provide a broad picture of the effect of chronic pain on human patients.23,38–41
The BPI is used to assess 2 factors of chronic pain: pain intensity (ie, severity factor) and how the pain interferes with the patient’s function (ie, interference factor). The severity factor is used to assess pain intensity for current pain, worst pain, least pain, and average pain. Because these pain severity items are nonspecific for the BPI, they were maintained unchanged for the CBPI. For the interference factor, the BPI includes items that assess how pain interferes with a human patient’s physical activity, sleep, and social activity. These items had to be adapted for the CBPI to assess behaviors appropriate for dogs. Therefore, interference items were generated through information gathered from focus groups and an expert panel. These items were tested for readability and ambiguity. The items were administered to another population of owners whose dogs had osteoarthritis, and poorly performing items were removed. The reduced set of items was then subjected to factor analysis and testing to determine reliability and validity. This protocol was approved by the Internal Review Board of the University of Pennsylvania.
Items were developed by use of information obtained from key informant interviews with 36 dog owners (ie, focus groups) and an expert panel. The 36 owners comprised 3 focus groups. An open-ended discussion format was used to allow owners the opportunity to characterize their perceptions of the chronic pain in their dogs. Investigators facilitated the discussion to ensure that owners addressed each of the following issues:
On the basis of analysis of these data, items were devised to reflect potentially important clinical signs; 1 global assessment item was also devised. The initial items were then circulated among veterinarians highly experienced in treating dogs with chronic pain, which included 3 general practitioners, 3 orthopedists, 2 oncologists, and 2 neurologists. Individual interviews and group discussions were conducted to further refine and consolidate the questions. For example, it appeared reasonable to consolidate the owners’ descriptions of the difficulties their dogs had playing fetch, chasing a cat, and going for a jog into a single question about interference with the dog’s ability to run. The resulting questions comprised the preliminary instrument.
Statistical analysis of readabilitya (ie, Flesch reading ease score and Flesch-Kincaid grade level) was performed on the preliminary instrument.42 An in-depth interview with 30 owners who were not involved in any of the 3 focus groups was performed immediately after they had completed the preliminary instrument. In that interview, owners were asked for comments regarding ambiguity or uncertainty of the items or response options.
Several hypotheses were tested, which included that the CBPI was a 2-factor questionnaire (Cronbach’s α was > 0.70 for each factor), the arithmetic mean of items in the severity factor (severity score) and interference factor (interference score) had good test-retest reliability (κ > 0.60), the severity and interference scores were moderately correlated (r > 0.40) with the global QOL item, and the severity and interference scores for dogs with osteoarthritis were significantly higher than those obtained for clinically normal dogs. To test these hypotheses, 70 owners of dogs with a medical history, clinical signs, and radiographic evidence consistent with osteoarthritis were recruited by use of e-mail, advertising circulars, and newspaper advertisements. The owners were not involved in the 3 focus groups or the item testing for the preliminary instrument. These owners completed the CBPI twice (baseline and 1 week later). Only owners of dogs that had newly been diagnosed with osteoarthritis and had not yet been managed medically or that had been diagnosed sometime in the past but the owners had opted not to medicate the dogs on a regular basis were recruited to complete the CBPI. Thus, none of the dogs were receiving anti-inflammatory or analgesic medications for at least 1 week preceding completion of the questionnaire by the owners.
In addition, 50 owners of large-breed dogs, with each of the dogs > 5 years old and having no clinical signs of osteoarthritis, were recruited from hospital faculty and staff and veterinary students at our facility via an e-mail announcement. Owners of these clinically normal dogs also completed the CBPI.
Principal factor analysis with subsequent varimax rotationb was used to ascertain whether the underlying factors identified statistically within baseline data collected by use of the instrument were consistent with theoretic factors associated with chronic pain that the study was designed to measure (ie, severity of pain and interference with function). The interitem correlation matrix and item-total correlationsc were used to detect negative correlations and to screen for items that had consistently weak correlations with other items or the scale.
The quadratic weighted κ statisticd was used to assess the instrument for test-retest reliability between the 2 administrations of the instrument (ie, baseline and 1 week later). Pain severity and pain interference scores were correlated with the global QOL score by use of Spearman rank correlations.e Spearman rank correlations were also used to assess correlations between the severity and interference factors. For the extreme groups comparison, the Mann-Whitneyf test was used to compare severity and interference scores between dogs with osteoarthritis and clinically normal dogs. Distribution of data was evaluated for normality by use of statistical software.g Values of P ≤ 0.05 (2-tailed test) were regarded as significant.
Twelve items were generated and grouped on the basis of context into 2 aspects of chronic pain in dogs (4 items were grouped into severity of pain, and 8 items were grouped into interference with function). The 1 item pertaining to owner assessment of overall QOL for each dog was maintained separately. Interference and severity items used an 11-point rating scale (ie, 0 to 10). Severity items ranged from 0 (no pain) to 10 (extreme pain), whereas interference items ranged from 0 (does not interfere) to 10 (completely interferes). The item on overall QOL used a standard 5-point categoric response (1 = poor, 2 = fair, 3 = good, 4 = very good, and 5 = excellent).
Inter-item correlation matrix for the preliminary questionnaire revealed that the 2 items pertaining to pain’s interference with a dog’s ability to sleep and temperament performed poorly, with interitem correlations and item-scale correlations consistently < 0.20. Several owners commented that they did not routinely sleep in the same room with their dog and that the meaning of temperament was not widely or clearly understood. These 2 items were removed, which left 6 items in the interference factor, 4 items in the pain severity factor, and 1 item for QOL for further testing. Results for readability testing (Flesch reading ease score, 83.3; Flesch-Kincaid grade level, 6.6) suggested that the instrument was relatively easy to read.
The completion rate for all items was 100% and the instrument took < 5 minutes to complete in all cases, which suggested ease of use and minimal burden or ambiguity. Ten items (ie, all items except the QOL item) were entered into the orthogonal, varimax-rotated factor analysis. Two factors were identified with an eigenvalue (variance of the factor) > 1.0 (remaining factors had eigenvalues ≤ 0.7). Retention of 2 factors was confirmed by use of a scree plot, which is a graph that plots the eigenvalues against the factor number (Figure 1). The 2 factors in the CBPI accounted for 72% of the variance and were identified on the basis of the items contained in each factor (severity of pain comprised 4 items and had an eigenvalue of 5.8, and interference with function comprised 6 items and had an eigenvalue of 1.4; Table 1).11,29,43–47 Cronbach’s α was 0.92 for the total instrument and 0.93 and 0.89 for severity of pain and interference with function, respectively, which suggested that the items in each of the 2 factors could be assessed as a group to compute factor scores (ie, severity score and interference score). There were no negative values for interitem correlations. Mean interitem correlation was 0.77 for severity of pain and 0.56 for interference with function, which suggested good internal consistency of the factors (ie, each item within a factor was assessing the same attribute). Item-total correlations and communalities were summarized. The test-retest performance of the instrument was 0.75 for the severity score and 0.81 for the interference score, which revealed good stability of the instrument among repeated administrations.
Scores for severity of pain and interference with function were not normally distributed. The severity and interference factors were moderately correlated (r = 0.59). The lack of a high correlation between the 2 factors suggested that the owners were truly evaluating 2 separate aspects of assessment of chronic pain in their dogs and that additional information could be gained by the use of both factors, rather than opting to use only 1 of the factors. For the convergent validity assessment, scores for severity of pain and interference with function were moderately correlated (r = 0.51 and 0.50, respectively) with the overall QOL score in a manner such that as scores for severity of pain and interference with function increased, QOL decreased. In the evaluation of extreme groups, clinically normal dogs had significantly (P < 0.001) lower scores for severity of pain and interference with function, compared with scores for dogs with osteoarthritis (Table 2).
The CBPI is a 2-factor, 11-item questionnaire designed to measure owners’ assessments of the severity and impact of chronic pain on their dogs with osteoarthritis. Methods for the stepwise development of tools to assess subjective states that have been used extensively by researchers in the health sciences11,12,14–19,22–25,27–32,34,36,37 were used in the development of the CBPI. Items were generated through information gathered from focus groups and an expert panel; these items were tested for readability and ambiguity. The items were then administered to another population of dog owners, and poorly performing items were removed. The reduced set of items was subjected to factor analysis, reliability testing, and validity testing. This standard stepwise development resulted in a CBPI with excellent psychometric (reliability and validity) properties. It appears that the CBPI collects information about the owners’ assessments of chronic pain in their dogs with osteoarthritis, and it does so in a consistent manner (ie, it is reliable).
Multiple approaches were used to assess validity of the CBPI (ie, the degree to which the instrument measures the severity and impact of chronic pain). Face and content validity (ie, judgments that a scale appears to be reasonable) consists of an assessment by a handful of experts as to whether the scale appears appropriate for the intended purpose.11 The CBPI was evaluated by 10 veterinarians who regularly treat dogs with chronic pain. They concluded that the instrument was assessing the desired qualities (face validity) and sampling the relevant content (content validity).
The CBPI was then tested for construct validity. Construct validity is evaluated when the attribute being measured cannot be directly observed.11 Chronic pain cannot be seen, but in accordance with theories about chronic pain in companion dogs, behaviors can be observed that result from chronic pain. There is no single experiment or statistic that can unequivocally prove a construct. Multiple analyses and assessments are needed to assess whether a construct appears to be valid.11
The CBPI was evaluated for construct validity in several ways. First, the factors hypothesized a priori were consistent with results determined by use of factor analysis (ie, 2 factors [severity of pain and interference with function], with all of the items predictably incorporating preferentially into 1 of these 2 factors). Second, validation by evaluation of extreme groups revealed that dogs with osteoarthritis had significantly higher scores for severity of pain and interference with function, compared with scores for clinically normal dogs. Third, scores for severity of pain and interference with function were moderately correlated with the overall QOL item in a manner such that as severity and interference scores increased, perceived QOL decreased. This is consistent with the assumption that pain attributable to osteoarthritis will impact the QOL in affected dogs (convergent validity). All of these approaches built the body of evidence that supported the validity of the CBPI as an instrument that measures chronic pain in dogs with osteoarthritis.
In addition to face, content, and construct validity, the CBPI was tested for reliability. An instrument is not useful if it does not yield results in a consistent manner. Internal consistency of the CBPI was excellent (Cronbach’s α was > 0.8 for both factors). This level of internal consistency allows the items in each factor to be assessed as a group to provide a score. In the case of the CBPI, these were scores for severity of pain and interference with function. However, assessment of internal consistency involves only a single administration of the instrument and does not take into account any variation among days, which could lead to an optimistic interpretation of the true reliability of the test.11 Therefore, stability of the CBPI was also evaluated by examining consistency of the responses between 2 administrations of the instrument to the same owners 1 week apart. Stability of the instrument was excellent (≥ 0.75 for both factors, which is much higher than the minimum stability recommendation of 0.50).11
The sound psychometric properties of the CBPI are likely attributable to the strict adherence to the methods that guide instrument development, as well as the use of a factor format, wording structure, and item scaling that has proven to be valid and reliable in the assessment of chronic pain in people through use of the BPI.23,38–41,48 An obvious difference between the BPI and CPBI (other than target species) is that the BPI is a self-assessment, whereas the CBPI is an observer (owner)-completed assessment.
The inability to use self-assessment outcome measures is not exclusive to companion animal studies. Observer (relative or caregiver)-completed assessments are commonly used in pediatric49–59 and cognitively impaired human populations.60–67 Although the subjective worlds of young children, demented adults, and dogs are not directly accessible, readily observable behaviors offer a basis for assessments to be made by individuals who are knowledgeable about the subjects. Development of these tools is based on the concepts that there can be a proxy evaluation of pain intensity via a global assessment based on such things as facial characteristics, body posture, and movement patterns of the subject; pain can interfere with activities of daily living; and a knowledgeable observer can reliably rate the behavior of a subject.68–73 In addition to being applicable to a caregiver’s behavior-based assessment of pain in an adult who is nonverbal as a result of Alzheimer’s disease, these concepts are applicable to an owner’s behavior-based assessment of chronic pain in their dog.
Methods for the development and psychometric testing of instruments are the same regardless of whether the instrument is to be used for self-assessment or proxy reporting.10–13,15,18–37 Although a self-assessment instrument (ie, the BPI) was used as a template for the CBPI, the interference items were generated and the entire instrument was tested as a proxy reporting tool. Testing revealed excellent psychometric properties of the CBPI for owner assessment of dogs with osteoarthritis. However, it must be mentioned that this does not ensure that the CBPI would retain those psychometric properties if applied to dogs with chronic pain attributable to another pathologic condition. For example, the CBPI could be used as an outcome assessment tool for dogs with pain attributable to bone cancer, but psychometric testing for the instrument would first need to be performed in dogs with bone cancer to ensure appropriate levels of validity and reliability.
In addition to being valid and reliable, for an instrument to be useful, it must be able to detect clinically important changes in animals over time (ie, it must be responsive to changes in the health status of an animal). Responsiveness of the CBPI in dogs with osteoarthritis must be evaluated. In addition, the CBPI should be tested concurrently with gait analysis on a force plate to establish criterion validity. Assuming positive results are obtained for responsiveness testing, the CBPI could be used as an outcome assessment tool for dogs with osteoarthritis.
The CBPI is an outcome assessment instrument with sound psychometric properties that is easy to use. Items in the instrument appear reasonable, and construct validity is supported through results of factor analysis, evaluation of extreme groups, and convergent validity analyses. Reliability of the CBPI is excellent. Internal consistency and stability measures are much higher than reported minimums necessary for a group of items to be considered a scale. Assuming responsiveness testing is successful, the CBPI may be useful in reliably obtaining quantifiable assessments from owners regarding the severity and impact of chronic pain and its treatment on dogs with osteoarthritis.
Supported by the National Institutes of Health (grant No. 1-K08-DA-017720-02).
The authors thank Molly Love for technical assistance.
aMicrosoft Word XP Professional, Microsoft Corp, Redmond, Wash.
bFactor, Rotate, Agreigen, Stata Corp, College Station, Tex.
cAlpha, Stata Corp, College Station, Tex.
dKap, Stata Corp, College Station, Tex.
eSpearman, Stata Corp, College Station, Tex.
fRanksum, Stata Corp, College Station, Tex.
gSktest, Stata Corp, College Station, Tex.