PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of hlthqualBioMed CentralBiomed Central Web Sitesearchsubmit a manuscriptregisterthis articleHealth and Quality of Life OutcomesJournal Front Page
 
Health Qual Life Outcomes. 2004; 2: 42.
Published online 2004 August 3. doi:  10.1186/1477-7525-2-42
PMCID: PMC514531

The AMC Linear Disability Score project in a population requiring residential care: psychometric properties

Abstract

Background

Currently there is a lot of interest in the flexible framework offered by item banks for measuring patient relevant outcomes, including functional status. However, there are few item banks, which have been developed to quantify functional status, as expressed by the ability to perform activities of daily life.

Method

This paper examines the psychometric properties of the AMC Linear Disability Score (ALDS) project item bank using an item response theory model and full information factor analysis. Data were collected from 555 respondents on a total of 160 items.

Results

Following the analysis, 79 items remained in the item bank. The remaining 81 items were excluded because of: difficulties in presentation (1 item); low levels of variation in response pattern (28 items); significant differences in measurement characteristics for males and females or for respondents under or over 85 years old (26 items); or lack of model fit to the data at item level (26 items).

Conclusions

It is conceivable that the item bank will have different measurement characteristics for other patient or demographic populations. However, these results indicate that the ALDS item bank has sound psychometric properties for respondents in residential care settings and could form a stable base for measuring functional status in a range of situations, including the implementation of computerised adaptive testing of functional status.

Background

It is now widely accepted that examining quality of life is an important aspect in the treatment and evaluation of many conditions. Functional status is seen as an important determinant of quality of life. A wide variety of instruments have been developed to quantify functional status [1]. These instruments tend to have a fixed length and all items are administered to the whole group of patients under scrutiny. However, currently interest is moving towards the more flexible framework offered by item banks. An item bank is a collection of items, for which the measurement properties of each item are known [2,3]. When using an item bank, it is not essential for all respondents to be examined using all items. This enables the burden of testing to be considerably reduced for both patients and researchers. It is even possible to select the 'best' items for individual patients using computerised adaptive testing algorithms [4]. Furthermore, results from studies using different selections of items from an item bank can be directly compared. Item banks, measuring concepts such as quality of life [2,5], the impact of headaches [6] or functional status [7,8], have been developed.

The AMC Linear Disability Score (ALDS) project item bank was developed to quantify functional status [7,9]. The ALDS item bank covers a large number of activities, which are suitable for assessing respondents with a very wide range of functional status and many types of chronic condition. The item bank is particularly suitable for use in the Netherlands. The ALDS items were obtained from a systematic review of generic and disease specific functional health instruments [1]. Five psychometric aspects of the ALDS item bank need to be considered before it can be implemented. These are: (a) there needs to be enough variation in the response categories used for each item [9]; (b) estimates of the item response theory model parameters should not depend on patient characteristics such as age or gender [10,11]; (c) estimates of the item response theory model parameters, which are stable across different subsets of items from the instrument and based on a sufficiently large sample [12] of respondents, should be available [9]; (d) an examination of the extent to which the ALDS items represent a single construct; and (e) testing whether a simpler item response theory model is suitable for the set of items.

This paper examines these five aspects of the ALDS item bank using the responses given by residents of supported housing schemes, residential care and nursing homes in and around Amsterdam, the Netherlands. This, mainly elderly, population has been chosen because they generally experience some level of functional restriction and consume a large amount of health care services.

Methods

Data collection

This paper considers 160 items, which were considered to be applicable in a residential care setting. Each item has two response categories: 'I could carry out the activity' and 'I could not carry out the activity'. If a respondent had never had the opportunity to experience an activity 'not applicable' was recorded. In the analysis, responses in the category 'not applicable' were treated as if the individual items had not been presented to the individual respondents [13]. It was felt that presenting all 160 items to each respondent would place an unnecessary and unacceptable burden on those responding to the items. Therefore, the data described in this paper were collected using an incomplete, anchored calibration design [7,9,14,15] with four sets of 80 items. Item sets A and B have half their items in common, as do item sets B and C, item sets C and D and item sets A and D. The items in common between two sets of items are known as 'anchors' and allow all items and patients to be calibrated on the same scale. The patterns of missing data in this type of design are, in statistical terms, ignorable [16]. The item sets were administered randomly to 150 respondents (item set A), 143 respondents (item set B), 138 respondents (item set C) and 124 respondents (item set D).

Respondents

A total of 555 residents of supported housing, residential care and nursing homes were interviewed. The median age was 84 years (range 37 to 101 years), while 444 (80%) were female. Since the respondents were interviewed 'at home', accurate data on medical conditions were not available. All respondents gave informed consent. The study was approved by the medical ethics committee in our hospital.

The item response theory models

In this paper the data were analysed using the two-parameter logistic item response theory model [7,9,17,18]. In this model, the probability, Pik, that patient k responds to item i in the category 'can' is modelled using

An external file that holds a picture, illustration, etc.
Object name is 1477-7525-2-42-i1.gif

where θk denotes the ability of patient k to perform activities of daily life. The discrimination parameter (αi) and difficulty parameter (βi) describe the measurement characteristics of item i. The larger the value of βi, the more difficult item i is. In addition, the larger the value of αi, the better an item is a distinguishing between abilities above and below βi. If the values of αi are constrained to be equal for all items, the model in equation 1 becomes the one-parameter logistic item response theory model [19]. The model in equation 1 can be extended to test whether the values of βi for, say males and females, are significantly different. If the values of βi for different groups of respondents are significantly different, then there is evidence of differential item functioning. Full-information factor analysis also uses an extension of the model in equation 1. These approaches are described in mathematical terms in the Appendix. In this paper, estimates of αi and βi were obtained using a marginal maximum likelihood based procedure [20]. This method assumes that the ability parameters (θk) follow a Normal distribution and can account for incomplete designs, as described in the Appendix. Expected a posteriori methods were used to estimate θk [21].

Statistical analysis

To achieve the objectives of this study, there were five steps in the statistical analysis. In step (a), the amount of variation in the response categories used for each item [9] was considered and items demonstrating too little variation were removed. Items were excluded from further analysis if fewer than 10% or more than 90% of the patients responded in the category 'cannot'. In step (b), the items were examined to investigate whether the value of the item difficulty parameter (βi) was similar for male and female patients and for patients younger than 85 years and those aged 85 or older. The model is described in depth in the Appendix. Items were excluded from further analysis if the value of the item difficulty parameter was significantly different (1% level) between gender or aged based groups. In this step, the fit of the model to the data from each item was not assessed. In step (c), estimates of the item parameters (αi and βi) were obtained. The fit of the model to the data from each item was assessed using G2 statistics [22]. Items, for which the fit statistic had a p-value of less than 0.01, were excluded from the item bank. In addition, the stability of the estimates of the item parameters over different sets of items was examined using the model from step (b). Items were excluded from further analysis if the value of the item difficulty parameter was significantly different (1% level) between item sets A and B, B and C, C and D or A and D. Furthermore, a Kolmogornov-Smirnov test was carried out to examine whether the ability parameters (θk) were Normally distributed. In step (d), the dimensionality of the item bank was examined using item response theory based full information factor analysis [18,22,23]. The number of latent roots greater than 1 is regarded as an indicator of the number of factors in the data set. This method is described in more depth in the Appendix. Four exploratory factor analyses were carried out, one on each of the anchors between item sets A and B (293 respondents), B and C (281 respondents), C and D (262 respondents) or A and D (274 respondents). A fifth, confirmatory, factor analysis was carried out on the whole data set (555 respondents). In addition, Cronbach's coefficient alpha was calculated for each anchor and the whole data set [24,25]. In step (e) the one-parameter logistic item response theory model was fitted to the remaining items. The differences between the -2log likelihoods of this model and the two-parameter model fitted in step (c) was tested using a χ2 test. The analysis in steps (a), (b), (c) and (e) was carried out in Bilog, version 3.0 [22]. The analysis in step (d) was carried out using TESTFACT, version 4.0 [22].

Results

Of the 160 items included in the item bank, one was removed because it was worded differently in two different item sets. Of the 159 remaining items, 77 were removed from the item bank. This process is described in Table Table1.1. In step (a), 28 items were excluded from further analysis because fewer than 10% or more than 90% of responses were in the category 'cannot'. In step (b), 26 items were removed because they had significantly different estimates of the item difficulty parameter (βi) for for males and females and/or for younger and older respondents. Of these 26 items, 19 had different measurement characteristics for females and for males, 5 items had different measurement characteristics for those aged under 85 and for those aged 85 or over, and 2 items had different measurement characteristics for both males and females and for older and younger respondents. In step (c), 23 items had an item fit statistic p-value of less than 0.01. In addition, 3 items were excluded from further analysis because the value of the item difficulty parameter (βi) was significantly different between two item sets of items. Hence, 79 psychometrically sound items remained in the item bank. A short description of the content of the 79 items in the final version of the calibrated item bank, together with estimates of the dispersion (α) and difficulty (β) parameters and their standard errors, are given in Tables Tables2a2a and and2b.2b. Following step (c) of the analysis, the anchors between the sets of items contained between 13 and 23 items. In addition, there was no evidence to suggest that estimates of θ do not follow a Normal distribution (Kolmogorov-Smirnov test, p-value = 0.637). In step (d), the full information factor analysis indicated that, for three of the four anchors between the item sets, there was only one latent root of the correlation matrix larger than 1. In the fourth item set, a second latent root was marginally above 1. The percentage of the variance explained by the first factor varied between 67% and 72%. The values of Cronbach's alpha coefficient for the four anchors were between 0.86 and 0.93. The confirmatory factor analysis carried out on the whole data set indicated that 70% of the variance was explained by the first factor. Cronbach's alpha coefficient for the whole data set equalled 0.98. In step (e), the one-parameter logistic item response theory model was fitted to the 79 items remaining after step (c). This model fitted the data significantly less well than the two-parameter model (p-value < 0.0001). For 3 items, the item fit statistic had p-value < 0.01. After removal of these items, the two-parameter model was still significantly better than the one-parameter model (p-value < 0.0001).

Table 1
The number of items proceeding to each step of the analysis The number of and examples of items removed at each stage of the psychometric analysis.
Table 2
The 79 items remaining in the calibrated item bank. The items remaining in the calibrated item bank. The number of respondents, to whom the item was offered (Offered to), the number responding in the category 'not applicable' (NA), the number responding ...

Discussion

In this study, the psychometric properties of the item bank have been examined using a sample of 555 respondents and an incomplete calibration design. Each item was presented to between 262 and 293 respondents. These figures are above the minimum, of 200 respondents, regarded as necessary to implement the models used in this paper [12]. It could be argued that it would have been desirable for all respondents to be presented with all items, but this would have placed an unacceptable burden on the, often frail, population in this study. Incomplete calibration designs are regularly implemented in the development and maintenance of item banks used in educational testing [4,14] and have gained some recognition in health related applications [15]. Developments in psychometric theory mean that it is now possible to perform the same types of analysis on data resulting from incomplete designs, as is performed on data from complete calibration designs [22,23,25]. The number of items in the anchors following the analysis, indicate that the design was still amply linked [9].

One of the major assumptions underlying the use of the item response theory models described in this paper is that the items reflect a single latent trait (θ). This has been examined using item response theory based full-information factor analysis. Part of the full-information factor analysis was performed on sub-sets of the data, as exploratory analyses on incomplete designs may lead to instable results. However, the confirmatory factor analysis was performed on all data. The results, together with the high level of internal consistency, as measured by Cronbach's alpha, and the acceptable fit of the two-parameter logistic item response theory model to the data indicate that the items presented in this paper probably represent a unidimensional construct in a population of respondents requiring residential care.

Another important assumption when using item response theory models in conjunction with marginal maximum likelihood estimation procedures is that the values of the latent trait (θ) follow a pre-specified, usually Normal, distribution. In this study, there was no evidence that these values did not follow a Normal distribution. This is in contrast to many previously published studies into health and quality of life outcomes, where a strongly skewed distribution was found. The authors feel that there are two reasons for this contrast. Firstly, in this study, the respondents all had some level of restriction in their ability to perform activities of daily life. Secondly, the item bank includes items well above and well below the level of functional status enjoyed by the respondents. This means that the item bank did not have a ceiling or floor effect with respect this this population.

In this study, 81 (51%) of 160 items were removed from the item bank because they did not conform to the psychometric standards required of the item bank. This is a much higher level than would be expected in the calibration of an item bank for use in educational measurement. However, when the results are examined more carefully, 28 items were removed because they were too difficult or too easy for the population in this study. In addition, 26 items were removed because they had different item parameters for different groups of respondents. These problems would have been identified much earlier in an educational item bank. Hence, only 26 (25%) of 106 items were removed due to item misfit. The number of items retained in the item bank may have been higher if a more flexible model, based on, for example, non-parametric smoothing techniques had been used [26]. However, this type of model is less suitable as a base for implementing modern testing algorithms, such as computerised adaptive testing. In addition, it is possible that more items could be made available if the items demonstrating differential item functioning were included in the item bank with different item location parameters (βi) for males and females or for younger and older respondents. This may seem complicated, but is straightforward in the framework of a computerised item bank.

This paper has concentrated on the two-parameter logistic item response theory model. However, the one-parameter logistic item response theory model was also fitted to the 79 items remaining in the item bank. This model fitted the data significantly less well than the two-parameter model, even after 3 items demonstrating misfit at the item level were removed. This confirms the choice of the two-parameter model. This model was chosen because it allows the probability of responding in the category 'can' to be modelled more flexibly than when the one-parameter logistic model is used. This enables a more realistic model for the data to be built than when the more restrictive approach associated with the one-parameter model is chosen [18].

This paper has examined the calibration of the ALDS item bank in a population requiring residential care. It has been shown that the item bank has sound psychometric properties and could form a stable base for a wide range of applications. However, it is possible that the items will have different measurement characteristics for patients requiring treatment for specific chronic conditions or in other countries. Hence, it is important that the ALDS item bank is tested carefully before it is used to assess the functional status in other groups of respondents or in other countries.

Conclusions

Now that the measurement properties of the ALDS item bank have been examined carefully, the item bank can be used as a foundation for quantifying functional status. If modern algorithms, such as computerised adaptive testing, are implemented, it will be possible to obtain accurate measurements, whilst keeping the burden of testing on respondents and interviewers to a minimum. Items can be selected for use in further research, for allocation individuals to appropriate care settings and for calculating institutional funding based on the actual care load. It is hoped that the ALDS item bank will play an important part in the implementation of computerised adaptive testing of functional status.

Appendix

Differential item functioning

The model in equation 1 can be extended to test whether different groups of respondents to have different values of βi. This is known as differential item functioning. For instance, if interest is in possible differences in βi between males and females, then the probability, Pik, that patient k responds to item i in the category 'can' is written as

An external file that holds a picture, illustration, etc.
Object name is 1477-7525-2-42-i2.gif

where βiM is the item difficulty for male respondents, βiF-M is the difference between the item difficulty for males and for females and Ik is an indicator variable taking the value 0 if respondent k is male and the value 1 if respondent k is female. The hypothesis H0 : βiF-M = 0 can be tested to examine whether item i has the same measurement characteristics for males and for females.

Item parameter estimation in incomplete designs

In this study, the item parameters (αi and βi) were estimated using marginal maximum likelihood methods. The likelihood, L, over n items and K (K = 555) respondents can be written as

An external file that holds a picture, illustration, etc.
Object name is 1477-7525-2-42-i3.gif

where Iik is an indicator variable taking the value 1 if respondent k was offered item i and the value 0 otherwise and where Jik is an indicator variable taking the value 1 if respondent k responded to item i in the category 'can' and the value 0 otherwise. Furthermore, the probability, Pik, that respondent k responded to item i in the category 'can' is as in equation 1, or, where appropriate, as in equation 2 or 4. In the estimation process, the values of θk or θkm were assumed to follow a Normal distribution with mean equal to 0 and unknown variance, σ2, and were integrated out of the likelihood to obtain the marginal likelihood. The marginal likelihood was maximised using an EM algorithm [20].

Full information factor analysis

Full information factor analysis is a technique based on multidimensional item response theory models where the ability is represented by M variables, denoted θkm where m = 1, 2,..., M [22,23]. The model, in equation 1, for the probability, Pik, that person k responds to item i in the category 'can' can be extended to

An external file that holds a picture, illustration, etc.
Object name is 1477-7525-2-42-i4.gif

where θkm denotes the value of the latent variable θm associated with person k and αim denotes the discrimination parameter for item i with respect to the latent variable θm. Furthermore, δi is a difficulty type parameter. The loading, aim of item i on factor m can be calculated using

An external file that holds a picture, illustration, etc.
Object name is 1477-7525-2-42-i5.gif

The value of the standard difficulty parameter, (βi), can be calculated using

An external file that holds a picture, illustration, etc.
Object name is 1477-7525-2-42-i6.gif

Generally, the parameters αim and δi are estimated using marginal maximum likelihood methods.

Abbreviations

ALDS = AMC Linear Disability Score

Competing interests

None declared.

Funding

RH and RL were supported by a grant from the Anton Meelmeijer fonds, a charity supporting innovative research in the Academic Medical Center, Amsterdam, The Netherlands.

Authors contributions

RL conceived the study and supervised the data collection. RH prepared the first draft and carried out the analyses. RL, MV and RJdH critically reviewed the manuscript. RH prepared the final version.

References

  • Lindeboom R, Vermeulen M, Holman R, de Haan RJ. Activities of daily living instruments in clinical neurology. Optimizing scales for neurologic assessments. Neurology. 2003;60:738–742. [PubMed]
  • Bode RK, Lai JS, Cella D, Heinemann AW. Issues in the development of an item bank. Arch Phys Med Rehabil. 2003;84:S52–60. doi: 10.1053/apmr.2003.50247. [PubMed] [Cross Ref]
  • McHorney CA. Ten recommendations for advancing patient-centered outcomes measurement for older persons. Ann Intern Med. 2003;139:403–409. [PubMed]
  • van der Linden WJ, Glas CAW. Computerized Adaptive Testing Theory and Practice. Kluwer Academic Publishers, Dordrecht, the Netherlands; 2000.
  • http://www.amihealthy.com/static/dynamicsf36info.asp Accessed 8th July 2004.
  • http://www.headachetest.com/ Accessed 8th July 2004.
  • Holman R, Lindeboom R, Vermeulen M, Glas CAW, de Haan RJ. The Amsterdam Linear Disability Score (ALDS) project. The calibration of an item bank to measure functional status using item response theory. Quality of Life Newsletter. 2001;27:4–5.
  • Webster K, Cella D, Yost K. The functional assessment of chronic illness therapy (facit) measurement system: properties, applications, and interpretation. Health Qual Life Outcomes. 2003;1:79. doi: 10.1186/1477-7525-1-79. [PMC free article] [PubMed] [Cross Ref]
  • Holman R, Lindeboom R, Glas CAW, Vermeulen M, de Haan RJ. Constructing an item bank using item response theory: the AMC Linear Disability Score project. Health Services and Outcomes Research Methodology. 2003;4:19–33. doi: 10.1023/A:1025824810390. [Cross Ref]
  • Holman R, Lindeboom R, Vermeulen M, Glas CAW, de Haan RJ. The Amsterdam Linear Disability Score (ALDS) project. Differential item functioning with regard to gender. Quality of Life Newsletter. 2002;29:13–14.
  • Holman R, Lindeboom R, de Haan RJ. Gender and age based differential item functioning in the AMC Linear Disability Score project. Quality of Life Newsletter. 2004;32:1–4.
  • Zimowski MF, Mukari E, Mislevy RJ, Bock RD. BILOG-MG Multiple group IRT analysis and test maintenance for binary items. Scientific Software International, Inc. Chicago, IL; 1996. http://www.ssicentral.com/irt.htm
  • Holman R, Glas CAW, Zwinderman AH, de Haan RJ. Practical methods for dealing with 'not applicable' item responses in the AMC Linear Disability Score project. Health Qual Life Outcomes. 2004;2:29. doi: 10.1186/1477-7525-2-29. [PMC free article] [PubMed] [Cross Ref]
  • Kolen MJ, Brennan RL. Test Equating. Springer, New York; 1995.
  • van Buuren S, Hopman-Rock M. Revision of the ICIDH severity of disabilities scale by data linking and item response theory. Stat Med. 2001;20:1061–1076. doi: 10.1002/sim.723. [PubMed] [Cross Ref]
  • Mislevy RJ, Chang H. Does addaptive testing violate local independence? Psychometrika. 2000;65:149–156.
  • Birnbaum A. Statistical theories of mental test scores. Addison-Wesley, Reading, MA; 1968. Some latent trait models and their use in inferring an examinee's ability. Chapter in Lord FM, Novick MR.
  • Thissen D, Wainer H. Test Scoring. LEA, Mahwah, NJ; 2001.
  • Rasch G. Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research, Copenhagen, Denmark; 1960.
  • Bock RD, Aitkin M. Marginal maximum likelihood estimation of item parameters: An application of the EM algorithm. Psychometrika. 1981;46:443–459.
  • Bock RD, Mislevy RJ. Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement. 1982;6:431–444.
  • du Toit M, editor IRT from SSI: Bilog-MG, Multilog, Parscale, Testfact. Scientific Software International, Inc, Lincolnwood, IL; 2003.
  • Bock RD, Gibbons RD, Muraki E. Full information item factor analysis. Applied Psychological Measurement. 1988;12:261–280.
  • Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334.
  • Harvey WR. Estimation of variance and covariance components in the mixed model. Biometrics. 1970;26:485–504.
  • Ramsay JO. Kernal smoothing approaches to non-parametric item characteristic curve estimation. Psychometrika. 1991;56:611–630.

Articles from Health and Quality of Life Outcomes are provided here courtesy of BioMed Central