This study compared three questionnaires (SDQ, PSC and PSYBOBA) in order to decide which was most suited to improve identification of psychosocial problems among children aged 7 till 12 in community health services. The internal consistency of the overall scales was high. The construct validity of the three questionnaires was highly comparable. All questionnaires had a satisfactory sensitivity, at a specificity of 0.90, for problems defined as a clinical TPS score and a somewhat lower sensitivity for problems defined as a borderline TPS score. All three questionnaires offered substantial added value, improving the identification of children with problems based on readily available health indicators and/or clinical assessment during routine examinations. Due to the simplicity of score calculation, the PSC was rated more favourably by nine PCH professionals, for use in daily practice. Yet, in an overall rating by these professionals the PSC was rated less favourably than the SDQ and PSYBOBA.
The PSC resulted in a higher item non response and one in five parents had some criticisms on the PSC. We know no other studies mentioning such problems with the PSC. Yet, we do not think that these problems are to be explained by our translation. Many remarks concerned ambiguities (e.g. 'Is less interested in school': less than who or when?) and inconsistencies between the questions and the options for answering ('Gets hurt frequently' to be answered by 'never, sometimes or often). These ambiguities and inconsistencies are also part of the original questionnaire.
An essential element of the design of this study is that it used randomisation and aimed at a comparison of three questionnaires. We know of no other studies that used a similar design. Our approach is comparable to what is now rapidly becoming standard in studies assessing effectiveness and economic evaluations of interventions. Such studies do not assess the effectiveness or costs as such, but compare specific interventions with other interventions or usual care. Such an approach is far more helpful in guiding health policy decisions. We feel that such a comparative approach is worthwhile, too, in the evaluation of questionnaires to be used in health care. Only a systematic comparison can guarantee that the best instrument available will indeed be chosen.
Data were collected using a methodology that closely resembles the way Dutch PCH works, i.e. the questionnaires were sent to the parents, together with the invitation for a regular check up of their child. This improves the external validity of our results. However, one caveat is important: the main aim during sampling was to guarantee similarity between the three sub-samples, not an overall representativeness for the Dutch population. This resulted in a clear underrepresentation of ethnic minorities, caused by a higher nonresponse rate and non-participation of PCH services of the three largest cities in the Netherlands,. The results therefore need confirmation among ethnic minorities.
We used the CBCL as criterion. Although the CBCL is one of the best instruments available and is often used for evaluation purposes, it cannot be regarded as the ultimate golden standard. Other studies used psychiatric interviews or assessments by mental health professionals as standard[8
]. Due to financial limitations this was not possible in this study.
The absence of an ultimate golden standard means that the results of the study should be interpreted carefully. The high convergence between the three questionnaires and the CBCL indicates that they can be used as a valuable tool, supporting the professional's assessment and alerting him to probable cases. Yet, it seems unwise to use these short questionnaires as selection tools, limiting further assessment only to those children with elevated scores. The risk of missing serious problems would be too great. The questionnaires are valuable tools to alert PCH professionals on likely cases when they asses psychosocial problems. However, the PCH professional should also include other sources of information: his/her interview with both the parent and the child, the observation of the child and the parent-child interaction, and ratings from teachers whenever possible. Also, the impact of problems on the child's or family's functioning should be taken into account before deciding on further action. The SDQ provides such information in the Impact supplement.
In this study we had to use other cut-off points than those reported in the literature because the latter resulted in large differences regarding the percentage of children with an elevated score. This may raise the question whether these instruments measure the same construct. We think that this is the case indeed, as we found very comparable convergence indices with the CBCL TPS, both for the total scores on each of them and for the proportions with elevated scores based on the adapted cut-off points.
Our data on the validity and reliability of SDQ and the PSC are comparable to those published by other authors[10
]. We found, however, no other studies which question the scale structures of these questionnaires. Probably, our choice for the rigorous SEM as analytical tool instead of the more usual factor analytical approaches is the key factor. The combination of high internal consistencies and the negative SEM results may come as a surprise. What the SEM analyses showed, however, is that the concepts, as implied by the (sub)scale scores, despite the internal consistencies, are an inadequate description of the way the items are related to each other. In other words: the items provide information not covered by the scale scores. Health care providers should therefore not rely on the scale scores alone, but also carefully check the answers on individual items and discuss these answers with the parents.
Few studies assessed the added value of using questionnaires, as we did. The only studies to compare our results with are our own evaluations of two other questionnaires, that assessed the added value of questionnaires, as compared to risk indicators[14
]. In the current study we extended the added value analyses, by also including the signals detected by PCH professionals during routine examinations into the analysis. These extended analyses give a better indication of the real added value, as they compare the quality of questionnaire based detection to what is now standard practice for this age group, at least in the Dutch health care system.
Which of the three questionnaires, then, would be the best choice? The sensitivity of the PSC was somewhat less than that for the other questionnaires. The added value of the PSC in detecting children with clinical TPS was relatively low. One in five of the parents had criticisms on the PSC and item non response was also higher. In the Netherlands, the PSC would therefore be an unlikely choice. The psychometric performance of the PSYBOBA and the SDQ were similar. Although more PCH professionals preferred the SDQ as the instrument to use in the future, it was rated less positively in practical use, mainly because by the relative complexity of calculating the (sub)scale scores. We found little support for the supposed scale structure but also found that the SDQ Total Problems score is a strong indicator of problems. When the primary aim is to make a first distinction between children who probably have problems that need attention and those who do not, the single SDQ Total Problems score suffices. This largely simplifies the use of the SDQ.