In the field of complementary/alternative medicine (CAM), it is recognised that a third attribute of an RCT - its model validity - is also of particular importance [7,8]. Model validity reflects the concordance between the trial study design and "state of the art" practice for the intervention under investigation [4]. In their discussion of RCT methods in CAM, Verhoef and co-authors go farther, advising that model validity should "encompass ... the unique healing theory and therapeutic context of the intervention" [7].

In reviewing RCTs in homeopathy, internal validity has been appraised using either Jadad scores [9] or Cochrane-based criteria [10]. The external validity of this body of literature has also been addressed [11]. However, the 'homeopathicity' [12] of the therapeutic modality and of the clinical outcome measure/s used (i.e. the key characteristics of model validity) have never been formally accounted for in the assessment of trial quality. Some homeopathy trials have been broadly criticised for their lack of acceptable model validity, e.g.: homeopathic complex medication in female infertility [13,14]; individualised homeopathic treatment of childhood asthma [15,16]; isopathic treatment of hay fever [17,18]. Recognising that the characteristics of the homeopathic intervention and outcome measures are important seems a prerequisite for developing the most refined and relevant approaches to systematic review in this field of clinical research.

A set of guidelines (RedHot) [19] has established criteria for reporting data on homeopathic treatments; a similar set of reporting criteria is available for clinical trials of acupuncture (STRICTA) [20]. Recommendations of this nature are important since they require reporting of methodological details not addressed by generic standards such as CONSORT [21]. Nevertheless, the RedHot guidelines for homeopathy do not provide the means to define or assess model validity.

The work of Verhoef and colleagues highlights the importance of model validity, and they offer four broad headings under which assessment of homeopathy RCTs might take place: "representativeness"; "equipoise and credibility"; "model congruity"; "context" [7]. These attributes are a suitable reflection of the key characteristics of homeopathy that must be assessed for model validity, but they do not provide a direct means to undertake a practical formal assessment. The importance of such undertaking has been emphasised in two papers by Bornhöft and colleagues [4,22], who expressed concern that the current research literature - and thus the systematic review of it - may contain high risk of false negative results.

According to accepted guidelines [34,35], the magnitude of the kappa statistic reflects inter-rater agreement as follows: κ = 0.01-0.20, slight, κ = 0.21-0.40, fair; κ = 0.41-0.60, moderate; κ = 0.61-0.80, substantial; κ = 0.81-0.99, almost perfect. Thus, concordance among assessors was 'fair' for domains I, III and VI (κ = 0.42-0.50), 'moderate' for domain II (κ = 0.68), and 'almost perfect' for domains IV and V (κ = 0.92-0.94). It remains to be seen whether disparity of opinion is intrinsic to the complexity of the underlying concepts and/or to any lack of clarity in the particular wording of the domain and criteria descriptions. It is usual, however, to find only limited concordance amongst independent assessors in the judgment of subjectively-based domains [35]. Nevertheless, if ambiguity of wording does become evident, a set of revised descriptions would be drafted and published.

In our use of the method, one trial (Lewith [26]) attained a majority of 'No', in domain I (rationale), which indicates that assessors regarded it unlikely that allergic asthma patients can be benefited by homeopathy per se and/or by the particular homeopathic regimen used in this trial; we would therefore class this trial overall as 'inadequate MVHT'. It is noteworthy that this trial did not fail our MVHT assessment on the basis of domain II and/or domain III (i.e. isopathy using house dust mite is consistent with homeopathic principles, and the study had suitably qualified and experienced homeopathic practitioner input). It was on the basis of perceived deficiencies in all three of the above attributes that led to previous adverse conclusions on this trial using non-formalised criteria for model validity [36]. Two trials (Kainz [27], McCutcheon [28]) were rated as 'unclear' in one or more domains and so may be classed overall as 'unclear MVHT'. The remaining three trials (Bell [29], Robertson [30], Adler [31]) each received a full complement of six 'Yes' majority verdicts and so may be classed overall as 'acceptable MVHT'.

This proposed method for appraising and summarising MVHT harmonises with the Cochrane Collaboration's technique for assessing risk-of-bias for internal validity. In the Cochrane approach, a trial is assessed against seven domains: sequence generation; allocation concealment; blinding of participants and personnel; blinding of outcome assessors; incomplete outcome data; freedom from selective reporting; freedom from other bias. For each domain, risk-of-bias is indicated by the response 'low risk', 'unclear risk' or 'high risk'; an overall classification of risk-of-bias per trial is then identified using the same terminology [3].

The formal appraisal of RCTs, as adopted by the Cochrane Collaboration for risk-of-bias (internal validity), is typically by consensus between or among assessors, not by majority. In our test of the new method, it was not our purpose to seek consensus but to identify a summary verdict per domain and thus to classify MVHT overall for each of six RCTs. In systematic review it is not assumed that each independent assessor will arrive initially at the same judgment per domain; the method for assessing MVHT would operate in the same manner, and we recommend using three assessors to account for diversity of homeopathic opinion. Despite the diverse nature of homeopathic practice, there is little reason to anticipate undue difficulty in the ability of three specialists - even those from differing 'schools' of homeopathic theory and practice - to achieve a reasonable consensus agreement in judging the relevant issues. As was intended in our approach here, the formal appraisal of MVHT would necessarily exclude the consideration of the trial results themselves, and it might be appropriate to conceal the Results and Discussion sections, as well as the authors' names, in the papers that are provided to assessors.

In initial consideration of our criteria for domain III (practitioner), some of our group expressed concern that published RCTs seldom include any account of the homeopathic prescribing rationale or approach to individualisation. After discussion, it was agreed that to insist on full information, such as repertorisation details, exceeds expectation of the current research literature in homeopathy as well as the typical word limits of journal papers. Moreover, we note that the RedHot guidelines include clear recommendations with respect to individualised homeopathic prescribing [19], and we encourage authors of RCTs to implement these high reporting standards. We adopted the more generic domain III judgmental criteria, as given in Table ​Table11.

Additional discussion emerged about whether RCTs of homeopathy adequately take into account the complexity of individuals whose totality of signs and symptoms, rather than a named medical condition per se, is the basis of a homeopathic prescription. While we agreed this is an important facet of future research development, we accepted that the question whether clinical trials of homeopathy should place more emphasis on grouping together individuals with given characteristic signs and/or symptoms, rather than subjects with a specified medical condition, lies outside the terms of reference of the current project.

In judging domains IV-VI (outcome measure, sensitivity, follow-up) solely on a single 'main' outcome, we recognise that alternative outcome measures in the studies might have been selected and might have received different verdicts in our assessment of MVHT. Our approach here is consistent with the fact that a single, pre-determined, primary outcome is an important element in judging the validity of a trial. Our use of the WHO's International Classification of Functioning, Disability and Health, as became necessary for three of the six papers, has at least enabled us to make objective decisions on relative importance of health attributes within a clear hierarchical structure. For domain IV (whether the main outcome measure reflects the main effect expected of the intervention used), it is important that the consideration of 'expected main effect' also embraces the whole-person nature of the intervention and its associated outcome [37] - especially for homeopathy trials that involve individualised patient prescribing. Future assessments of MVHT of published trials should, where appropriate, include such 'whole-person' consideration, which might be achieved by extending the criteria by which 'main outcome measure' is selected [38]. This initiative might, in turn, assist the planning of future homeopathy trials by sharpening researchers' focus on the relevance of the outcome measures used.