The applicability of PLS-PM to coverage needs critical reflection as comparison with existing studies is not possible. The stated constructs can be used for further investigation of linkages between the components of coverage decision-making. The measurement models could be operationalized in a reliable and valid manner. Although the supposed links between observable indicators and constructs could be validated at measurement model level, further exploration is needed at the level of the structural model, especially if the tested hypotheses hold for other technologies. In spite of this, the qualitative interpretations of the results provide insight into whether PLS-PM produces plausible estimates and whether it is a suitable application for hypothesis testing using large data sets.
The case study reveals that the influence from the degree of stakeholder participation on reasonableness is about twice as influential as the degree of transparency in European NBS decisions. Besides, the degree of transparency significantly influences the level of methodological standards for evidence assessments. Thus, the estimation results are capable of demonstrating that the process components of coverage decisions that describe elements of procedural justice and definition of substantive appraisal criteria influence each other. No influence was found for the path between participation and scientific rigour which suggests that NBS technologies were assessed independently from the influence of stakeholders. Also, the R2
for the construct ‘scientific rigour’ was small which suggests that it is not well explained by the exogenous constructs of the SEM. On the contrary, considering the multiplicity of influences on coverage decision-making, the value of R2
of the ‘reasonableness’ construct can be considered acceptable. Examples are institutional configurations such as the level of decision-making or the implementation of the technology within the reimbursement scheme which are not described in this model [44
Some features of NBS in comparison with other health technologies need to be kept in mind when interpreting the results. Although evaluation measures for the construct ‘scientific rigour’ are acceptable, the path coefficient had no significant influence on the degree of ‘reasonableness’. This finding is supported in the literature on evaluation of NBS technologies, which states that cost-effectiveness information have frequently not been considered for appraisal [41
]. This is also in line with the small correlation between the indicator that reflects the scientific rigour of assessing costs/cost-effectiveness and the construct. Compared with technologies such as pharmaceuticals, coverage may not have been regulated as strictly, which is indicated by a low uptake of health technology assessment [37
]. Furthermore, survey respondents stated that funding of the screening tests was frequently negotiated between the payer and service providers, for which processes have not been defined (yet) or did not require disclosure of information. Thus, the degree of stakeholder participation had the strongest influence of the reasonableness of decision-making and was significant. Regarding the construct ‘reasonableness’, the effectiveness in terms of the health gain from testing and the severity of the disease were the indicators that reliably reflected the construct in the selected model estimation. A reason why cost-related aspects were not as meaningful may be the relatively low cost of the screening technologies and, for selected disorders, the high perceived effects from screening of newborns [42
]. Decisions on NBS were often made by institutions that do not typically decide on coverage of health technologies. Especially for pharmaceuticals, criteria such as cost-effectiveness and budget impact are perceived as being more relevant [1
]. Thus, for the construct ‘reasonableness’, all criteria should be used at the start of the analysis if decision-making on health technologies is examined. Also, principal component or factor analysis on all observed indicators could be applied if the sample size is sufficient.
PLS-PM manages to account for the complexity between the components stated in the model. As the goal of PLS-PM is to support the exploration and prediction of models under development, it provides guidance as to which link suggested for coverage decisions can be identified empirically. Before the other hypotheses are ultimately rejected, further evaluation is needed about whether this is also true for other technological areas. The relation between scientific rigour and reasonableness might be significant as these components are tied more closely in other processes, e.g. the technology appraisal by the UK NICE [45
]. Similarly, the relation between the degree of participation and scientific rigour might be meaningful in other technological domains. In many countries, pharmaceutical manufacturers need to submit evidence on their products to obtain coverage [46
]. Typically, this was not the case for NBS technologies. Finally, the model demonstrated that PLS-PM may be applicable for contexts of decision-making where ‘soft’ influences with high complexity and multiple links matter. Besides decision-analytic modelling, such approaches are demanded in healthcare, e.g. shared decisions between patients and physicians [47
]. Nevertheless, a correct specification of the theoretical model is a crucial requirement to accurately interpret the empirical results. The conceptual specification of this SEM needs further elaboration through expert validation and discussion of the theoretical foundations. Furthermore, making confirmatory statements is limited when using PLS-PM. Instead, covariance-based SEM should be used [20
The test of applicability of PLS-PM has some limitations. The estimation was based on a small sample and on decisions made for a very specific technological area. However, the sample size was sufficient according to established rules of thumb for PLS-PM [20
]. The PLS-PM results were not compared with other modelling techniques such as covariance-based SEM or multivariate regression analysis. However, application of these techniques is limited for the reasons for which PLS-PM was considered suitable, namely the capability to account for small sample sizes and no possibility of defining formative measurement models. Omitting possible influences of the survey sample, it was not possible to split the data for the different stages of model development (i.e. model specification, test of significance) because of the small sample size. A split sample design is appropriate for this purpose but was not applied.
Regarding model specification, theory dependency of the results cannot be neglected because the causal dependencies were specified without testing for other possible structures for the network of considered components. Bayesian network analysis would be suited to train and validate the model structure [48
]. However, required data was missing for this purpose. No expert opinion about the possible causal relationships and no information about the probabilistic relationships between constructs and indicators were available. By collection of further information, e.g. through an expert workshop, the model estimation could be used in future studies for validation. However, while theoretical considerations are developing, this study provides a first exploratory estimation of a SEM for coverage decision-making as well as measurement models that can be used for further analysis.
Potential unobserved heterogeneity between decisions has not been accounted for. Decision practices may differ by healthcare system or technological characteristics. However, no distinct explanatory variables have been suggested for coverage decision-making in the literature. To treat heterogeneity, methods for PLS-PM are available to identify plausible clusters ex post. Such techniques have been proved appropriate in marketing research [50
], and similar approaches have been used in other contexts of health economics [51
This study has quantitatively assessed the procedural aspects of decision-making such as stakeholder participation and transparency, which have been claimed as relevant for fair and legitimate decision-making [17
]. The accountability for reasonableness framework has predominantly been evaluated by qualitative approaches for which the evaluation of the effects frequently remains subject to judgements from a few case studies [52
]. Furthermore, the framework neglects appraisal criteria and consensus on adequate assessment methods [53
]. Specification of composites and several endogenous variables allows the combining of both process and appraisal simultaneously.
Compared with existing empirical research, the application of PLS-PM demonstrates that dependencies between several constructs can be tested when using small sample sizes. Previous work focuses on dependencies between the decision outcome and selected appraisal criteria [10
]. Relating to the work of Vuorenkoski et al., the estimation results have reconfirmed the relevance of transparency and stakeholder participation to ensure the quality of decision-making in the case of NBS [9