The study commenced in November 2001 and concluded in March 2003, with approval from the University of Western Sydney ethics committee. A before and after study design was used. There was no control group. The intervention is described in detail below.
To be included, participants had to be a qualified, employed occupational therapist, working in the state of New South Wales, Australia during the period of the study. An advertisement was distributed by email and post, to a wide range of private and public employers. Therapists were invited to participate, and to encourage junior and senior colleagues, as well as friends to enroll. This method of 'snowball' sampling aimed to recruit at least 100 occupational therapists. No eligibility restrictions were placed on health sector, work location or clinical specialty. A total of 232 therapists expressed interest, with 114 being recruited.
The primary intervention was a 2-day workshop on evidence-based practice, held in February 2002. Of the 114 recruited, 106 attended the workshop. To accommodate the large number of participants, three 2-day workshops were conducted over a month. Each workshop provided the same content but on different weekends over a month. The authors conducted the workshops in a metropolitan city, with the assistance of a health librarian. The authors were considered 'expert' clinicians, each having over 15 years of occupational therapy experience in public and private health sectors and had attended short skills-based courses on evidence-based practice. This experience and knowledge helped us to choose important occupational therapy questions and studies for appraisal during the workshop.
The workshop included lectures, practical sessions and small group discussion focussed around six topics: the process of evidence-based practice; writing focussed clinical questions; searching electronic databases; critical appraisal of qualitative and quantitative research; interpreting statistics in randomised controlled trials; and overcoming barriers/making the change to evidence-based practice.
The workshop used principles of andragogic or adult learning theory, [17
] and social cognitive theory [18
] to help participants engage with the new 'innovation', evidence-based practice. Social cognitive theory aims to promote learning and behaviour change by increasing the self-efficacy of learners (see Bradley and colleagues for an excellent review) [19
]. The format and content of sessions, and clear learning objectives were developed with a steering committee comprising five occupational therapy clinicians, two allied health experts in evidence-based practice, a health librarian and two service users. During the workshop, new skills and knowledge such as writing clinical questions, searching databases, and interpreting statistics were modelled by presenters, using worked examples. Time was set aside after each session for individuals to reflect, consider how they would apply the new skills and knowledge, and write personal learning goals to be achieved post-workshop. During the final workshop session, the change process and potential barriers to adopting evidence-based practice were actively discussed. A short presentation describing Rogers five-stage model of innovation diffusion [20
] and Prochaska and DiClemente's transtheoretical model of change [21
] provided the stimulus for discussion.
To promote post-workshop skill development participants were invited to develop a learning contract which included a critically appraised topic (CAT). These CATs could be completed individually or in pairs as an optional 'assignment'. Participants wrote a clinical question about the effectiveness of an occupational therapy intervention, to focus their CAT. By writing a CAT, it was hoped that participants would develop improved search and appraisal skills, and document the practice implications of research appraised. Although the use of CATs as a learning assignment had not previously been reported in this context, hypothetical assignments have been used [16
The follow-up outreach support consisted of regular email and telephone contact, and an optional workplace visit. Support was provided by an expert occupational therapy practitioner (ML), employed as a research assistant/project manager. The visits (n = 82) provided help with searching and appraisal, and monitored progress with the assignment. An email list was set up to facilitate communication. Information distributed via this list included resources and websites and answers to frequently asked questions. Reminders and individual feedback were provided about the assignment. Between March 2002 and February 2003, approximately 180 email messages were sent to this list, most by the project manager (ML). Approximately 225 email messages, mostly questions, were received from participants. The number of phone calls made and received was not logged. Completed CATs were presented at a one-day conference in February 2003, and uploaded to a new website.
A written questionnaire and the Adapted Fresno Test of competence in evidence-based medicine [23
] were used to measure knowledge, the primary outcome. The questionnaire also captured data on attitudes to evidence-based practice (secondary outcome). A written activity diary and assignment completion were used to measure behaviour (secondary outcome). Copies of the measures are available upon request from the first author (AM).
All measures except the activity diary were collected before, immediately after (i.e. at the end of day two) and 8-months post-workshop. The activity diary was collected prospectively on five occasions: for 3-weeks pre-workshop, then for 8-months during 2002 with bi-monthly collection (March-April; May-June; July-August; September-October).
The 8-page questionnaire was developed from existing instruments [3
]. Pilot-testing was conducted with eight occupational therapists from other states and territories in Australia. Minor changes were made to the content, layout and formatting in response to feedback. Participants completed the questionnaire on three occasions. The first questionnaire was mailed out, and returned on the first morning of the workshop (baseline). The second questionnaire was completed at the end of the workshop (post-workshop). The third and final questionnaire was distributed and returned by mail, approximately eight months after the workshop (follow-up).
The questionnaire contained three sections, and took approximately 20 minutes to complete. Section 1 recorded demographic data, perceived barriers to adopting evidence-based practice, and strategies used to overcome barriers. Section 2 required participants to rank (from one to five) how frequently they relied on different sources of information when making treatment decisions. Section 3 evaluated attitudes, knowledge and skill with regard to evidence-based practice, by asking each participant whether they 'agreed', 'disagreed' or were 'unsure' about specific statements. Examples of statements used were: 'An electronic database such as PubMed can only be accessed from hospital and university libraries' and 'The p value is a measure of reliability'. These questions objectively tested participants' knowledge, and had correct/incorrect answers. Other questions asked about self-reported skills, ability and knowledge. For example, 'I am aware of and have used a range of electronic databases' and 'I feel confident that I can critically appraise research evidence'.
The adapted Fresno test
An adapted version of the Fresno test of competence in evidence-based medicine [23
] was used to objectively measures skills and knowledge. The original Fresno test was designed to evaluate the effectiveness of a university curriculum on evidence-based medicine, and includes 12 short-answer questions, focussed around clinical scenarios relevant to family practice residents. Respondents are asked to write a focussed clinical question based on a scenario, list sources of information that could answer the question (for example, books and electronic databases), then comment on study designs and statistics reported in published papers. Internal consistency for the original Fresno test items, using Cronbach's alpha is 0.88, indicating a satisfactory level of agreement. Inter rater reliability is good to excellent, with correlations ranging from 0.72 to 0.96 for individual test questions, and 0.97 for total test scores [23
For the current study, the original Fresno Test was adapted by AM to include clinical scenarios relevant to occupational therapists. Five of the 12 more advanced statistical questions were removed (for example, those about sensitivity, specificity, numbers needed to treat), since these would not be taught in the workshop curriculum. The new test, referred to hereafter as the 'Adapted Fresno Test', asked participants to choose one of two new scenarios and answer seven related questions (see Table ). The Adapted Fresno Test took about 20 minutes to complete.
Seven questions and clinical scenario from the Adapted Fresno Test
Three sets of different clinical scenarios were written for each test administration (i.e baseline, post-workshop and follow-up), to avoid a practice effect. Diagnoses included in the clinical scenarios were low back pain, traumatic brain injury, occupational overuse syndrome, depression, osteoarthritis and carpal tunnel syndrome. Interventions included transcutaneous electrical nerve stimulation, group education programs, workplace exercises, cognitive behaviour therapy, exercise programs and splinting (see example in Table ). Analysis of internal consistency for the three versions of the Adapted Fresno Test yielded a Cronbach's alpha score ranging from 0.72 to 0.84, indicating an acceptable level of consistency for the adapted instrument [26
The seven questions in the Adapted Fresno Test were scored using standardised grading criteria, similar to those reported by Ramos and colleagues [23
]. The minimum test score was zero, and the maximum 156 for the seven questions (marking criteria available upon request). Each completed test took about 20 minutes to score. Scoring of the Adapted Fresno Test was evaluated for interrater reliability, and involved two raters independently scoring 20 completed tests (10 from pre-workshop and 10 from post-workshop) after receiving a two-hour training session [26
]. Interrater reliability results ranged from poor to excellent depending on the test question being scored. Intraclass correlation coefficients (ICCs) ranged from 0.20 to 0.96 for the pre-workshop test (0.88 for the total score), and 0.41 to 0.92 for the post-workshop test (0.87 for the total score). In the pre-workshop survey, Questions 1, 4 and 5 had an ICC below 0.80 (0.20, 0.23 and 0.53 respectively). In the post-workshop survey, Questions 1 and 3 had an ICC below 0.70 (0.41 and 0.57 respectively). Further refinement of the Adapted Fresno Test scoring system is therefore indicated; implications for study results will be addressed later under 'limitations'.
Behaviour change was measured using a concurrent activity diary, designed by AM for use in the study, and provided in paper or electronic format. Participants were asked to record only those activities that related to evidence-based practice, such as searching, reading research-related articles, critical appraisal and teaching others about evidence-based practice.
The following information was recorded in columns in participants' diaries, then subsequently analysed: date and nature of activity; what prompted the activity; start and finish times; whether the activity was conducted alone or not; if and how practice changed as a result of engaging in the activity. To improve accuracy, participants were asked to complete the activity diary contemporaneously for three weeks before and eight months after the workshop. Diaries were returned by fax, email or post on a bi-monthly basis. A research assistant contacted participants if their bi-monthly activity diary had not been returned.
Completion of the assignment or CAT at the end of 2002 reflected engagement in the first three steps of the process of evidence-based practice (writing a focussed question, searching for evidence, and critically appraising the evidence). This outcome was recorded as completed/not completed.
Descriptive statistics, including means and percentages, were used to compare outcomes.
For the Adapted Fresno Test (the primary outcome measure), and based on statistical advice, paired t-tests were used to evaluate change in objective knowledge. Differences in mean total scores and confidence intervals (95% CI's) were calculated. A power calculation could not conducted in advance because the Adapted Fresno Test had not previously been used as an outcome measure, and expected means and standard deviations were unknown. Rather than performing a post-hoc power analysis, we examined the width of the confidence intervals (95%) for the estimated effects (pre-workshop to post workshop difference). When confidence intervals are reported in this way, a post-hoc power analysis is redundant [27
]. Improvements of 10% (15.6 points) in the mean total score post-workshop, and 15% (23.4 points) at follow-up were considered clinically/educationally important, compared to baseline.
Non-parametric statistics were used for all other measures. Repeated measures were computed for the three occasions, rather than comparing paired samples, to reduce the risk of type 1 errors. Friedman's tests were used to test the hypotheses that over time: (a) more participants would be able to correctly answer the knowledge test questions, (b) more participants would feel confident and able to use published research in their work, (c) fewer participants would report barriers to evidence-based practice, and (d) more participants would conduct searches, read and appraise research than had done so before.
For knowledge test questions, the responses 'agree/disagree/unsure' were recoded as 'correct/incorrect'. For questions related to self-reported skills, confidence and ability, the responses 'agree/disagree/unsure' were recoded as 'yes/no'.
Non-responders were not included in analyses, post-intervention or at follow-up.