Demographics and Pilot Participation
There were no significant differences between the intervention and control groups at the beginning of pilot (Time 0 in ) and after 6 months (Time 1) for age, gender, marital status, education, and employment. Of the 46 patients, 25 (54%) were allocated to control group and 21 (46%) to intervention group. The response rate at 6 months (Time 1) was slightly higher in the intervention group (12 out of 21, 57%) versus the control group (10 out of 25, 40%), but the difference was not statistically significant (χ2
1 = 1.34, P = .38).
Group characteristics at Time 0
In the intervention group, the reasons patients gave for dropping out were dissatisfaction with the intervention (1 patient) and early significant clinical improvement (8 patients). For those who said they dropped out because they were much improved, it is unknown whether they were still healthy at Time 1. Reasons for high attrition at Time 1 in the control group were not known.
Medication Adherence, BDI-II Improvement, and Outcome Measures
In the control group, 3 out of 9 (33%) patients were adherent to antidepressants compared with 10 out of 12 patients (83%) in the intervention group (χ2
1 = 5.45, P = .03, odds ratio [OR] = 10.0, 95% confidence interval [CI] = 1.28-78.1).
Both groups demonstrated significant within-group reduction of mean BDI-II score from Time 0 to Time 1 (control group: paired t
9 = 3.95, P = .003, Cohen’s d = 1.23; intervention group: paired t
11 = 7.23, P < .001, Cohen’s d = 2.57), with intervention group seeming to indicate a greater effect size, which is further supported by the between-group comparison shown in .
Between-group comparison of BDI-II for available cases
In outcome sensitivity analysis (), available case analysis and the 3 intention-to-treat (ITT) scenarios with equal risk imputation [32
] resulted in odds ratios in favour of intervention, seeming to indicate an improvement of outcome in the intervention group. The last, pessimistic scenario was insignificantly in favour of the control group. Confidence intervals were wide due to small sample sizes and high dropout ratios.
Outcome measures: available cases and intention-to-treat (ITT) analysis scenarios
In , the Kaplan-Meyer plot depicts the chance of a patient reaching a certain duration of intervention usage. Shown are 2 scenarios: (1) all patients and (2) only patients in need of further intervention where only nonhealthy patients at any given time were taken into account (by treating healthy quitters as censored events). The mean duration of intervention usage by all patients was 107 days (95% CI 90-125 days); for patients in need of further intervention the mean duration was 150 days (95% CI 131-170 days).
Kaplan-Meier survival analysis for use of intervention (dotted vertical line denotes Time 1 at beginning of the 24th week)
The patients submitted a total of 431 questionnaires online of which 198 (46%) were complete. The average number of complete questionnaires per patient was 9.9 (SD 3.35, range 3-14). The remaining 229 submitted questionnaires were only partially completed and were treated as unsuccessful submissions. The patients were required to fill in the missing answers and resubmit the questionnaires.
Of the 21 patients in the intervention group, 6 (29%) required guided registration over the phone by the care manager. Care managers submitted 46 task-resolution reports related to 16 of the 21 patients (76%) (see ).
In addition, care managers reported that 33 assigned tasks were not resolved because the patient did not answer the phone or reply to an email. Of these, 88% (29/33) were requested for patients who dropped out. The average number of tasks performed was 2.2 per patient and 2.9 per patient actually requiring that the case manager undertake a task. In addition, 1 of the 7 physicians involved in the pilot reported patient visits in the ICT system, and none of the 7 physicians performed the e-learning test.
No significant differences were detected between the control group and the intervention group in perception of care quality or accessibility to care and information. Qualitative feedback regarding the intervention provided by patients from the intervention group is shown in . Of the 21 patients, 17 (81%) gave positive feedback whereas 7 (33%) gave negative feedback.
Positive feedback provided by patients in the intervention group: categories and examples
Negative feedback provided by patients in the intervention group: categories and examples
User Experience With the ICT System
Some areas for improvement were identified during the pilot. The following 4 required increased resource utilization and called for a future modification of work processes: (1) Digital certificates (electronic documents required by the ICT system from each user for authentication) required time and were somewhat difficult for both patients and care managers to manage. Further simplifications of certificate handling are necessary and human resources are required to help patients register. In the future, we anticipate that digital certificate “literacy” among users will reduce the importance of this issue. (2) A significant proportion of the care manager workload was due to dropouts not responding to calls and emails. More efficient strategies for interaction with these patients are needed. (3) Physician usage of the ICT system was poor, requiring specific motivational strategies (ie, a reimbursement scheme). (4) Frequently asked questions were available but not used, as the protocol that required the care manager to post these was not strictly enforced.
An additional 4 areas were identified that require changes in the ICT system functionality: (1) The feedback provided when a patient does not complete a questionnaire needs to be improved (ie, that directs users to missing answers). (2) Automated text messages were seen as a disturbance for a minority of patients who experienced fast clinical improvement and wished to finish the intervention early. The solution is for the care manager to have the ability to deactivate these messages for individual patients. (3) The knowledge base of the ICT system needs to be upgraded to increase the pool of available interpretations when questionnaires are submitted. The reason for this is to reduce repetitive answers that are possibly perceived as impersonal by patients. (4) Because the forum was poorly used and the clinical value of forums is unclear [33
], consideration should be given to discontinuing of this function.