|Home | About | Journals | Submit | Contact Us | Français|
To identify literature on:
28 bibliographic databases, research registers, organisational web sites or library catalogues. Reference lists from identified studies. Contact with experts. Current awareness and contents alerting services in the area of learning disabilities.
Inclusion criteria were English language literature from 1980 onwards, relating to people with learning disabilities of any age and all study designs. The main criteria for assessment was relevance to the Guilliford et al. model of access to health care9 which was modified to the special needs of people with learning disabilities. Inclusion criteria focussed on relevance to the model with initial criteria revised in light of literature identified and comments from a consultation exercise with people with learning disabilities, family and paid carers and experts in the field. Data abstraction was completed independently and selected studies were evaluated for scientific rigour and the results synthesised.
2221 items were identified as potentially relevant. 82 studies were fully evaluated.
The process of identifying relevant literature was characterised by a process of clarifying the concept under investigation and sensitive search techniques which led to an initial over-identification of non-relevant records from database searches. Thesaurus terms were of limited value forcing a reliance on using free-text terms and alternative methods of identifying literature to supplement and improve the recall of the database searches. A key enabler in identifying relevant literature was the depth and breadth of knowledge built up by the reviewers whilst engaged in this process.
Librarians carrying out systematic reviews of the evidence in health services research will need to adapt the guidelines used for reviews of treatment evidence and adopt a more inclusive approach. This will have implications for the process and the knowledge needed.
An iterative approach to defining a topic area, developing criteria for inclusion/exclusion and search strategies is required. This will involve allowing extended time for scoping, pilot searches and additional phases of analysing and sifting search results.
Identifying relevant literature for a health services systematic review requires expertise in a wide range of information sources across subjects, to enable searches to be adapted across disciplines and methods.
Health librarians will need to develop their understanding of both the assumptions underlying research evidence and systems for delivering health services to contribute effectively to systematic reviews in this area.
Current health policy emphasises the need for services to be based on sound evidence of what works. There is a need to extend this evidence-base to include the organisation and delivery of health services and increase awareness of the merits of different types of evidence. This is partly reflected in initiatives such as the National Health Service Programme for Service Delivery and Organisation (NHSSDO)1 which aims to improve the evidence base. In June 2002 we were awarded a 12 month contract by NCCSDO to carry out a literature review of access to health care for people with learning disabilities. The aim of this paper, the first of two, is to describe the rationale and methods used to identify the literature. Further details about the methods, results and conclusions of the review can be found in the final report at http://www.sdo.lshtm.ac.uk/sdo232002.html2
There are established guidelines for conducting systematic reviews.3 These methods are designed to identify research evidence where relevance is judged against specific criteria for methodology and quality and include the identification and review of a wide range of study designs, including qualitative research. However, they reflect the state of development of systematic reviewing at that time and therefore more information is provided about reviewing controlled trials than other research designs. Research is being conducted to develop comprehensive methods for reviewing the literature on more diffuse topics4,5,6. Our approach drew on systematic review methods as a source of good practice, but also used methods available for reviewing qualitative data and adapted these to the needs of a diffuse and multi-dimensional topic, in our case, “access”.
Identifying literature relevant to the review involved three phases. These phases are described below.
To carry out the literature review it was necessary to clearly define the subject area. This involved identifying an appropriate theoretical model, making any adaptations required, and consulting stakeholder groups.
The model of access to health care we drew on was that of Gulliford.7 Crucially, this model differentiates between:
The major dimensions of access were defined as:
Recognising that the routes into healthcare for people with learning disabilities are likely to differ from those of the general population the Gulliford model was first expanded to take into account:
The expanded model provided the conceptual framework within which the literature review was located. This ensured we had a guide to underpin the identification of relevant literature.
A consultation exercise was included in the literature review to ensure that the views and experiences of health service users, learning disability practitioners and researchers guided identification of the key issues in access to health care for people with learning disabilities. This formed part of an early “scoping” phase in which we developed our understanding of what issues and areas of literature might be relevant to this topic.
The aim of this was to:
The consultation exercise was carried out between June and November 2002 and comprised three elements
The discussion groups and consultations helped sharpen our inclusion/exclusion criteria by identifying specific issues for people with learning disabilities and their parents or carers.
Searching for literature involved contacting researchers, searching electronic databases, searching personal bibliographies, snowballing and using current awareness services.
We searched the research registers listed in Appendix 1 to identify on-going and completed research projects that might address our topic. The more limited search interfaces provided to electronic research registers meant it was not possible to run a complex search strategy. Therefore, the research registers were interrogated for records with learning disability and its synonyms occurring in the title. These terms are listed in Appendix 2 as Research Register Searches. Terms were identified from the indexing in bibliographic databases, examining literature held by the lead researcher and brainstorming for synonyms. The lead researcher scanned all records retrieved and projects judged potentially relevant to the model were contacted. A research technician sent e-mails or letters providing information about the study, including the model of access to health care, and requested copies of any outputs felt to address the access issues outlined.
Electronic information sources included bibliographic databases, online library catalogues and web sites.
The main method used to identify literature relevant to the model was through searching bibliographic databases. Our search strategy was based on the principle that a good quality literature search should:
The development of our search strategy was an iterative process comprising initial thesaurus searching and main free-text searching. A copy of the search strategies employed is included in Appendix 2.
The thesauri of major medical/health databases were studied to identify appropriate subject terms to search under. Thesaurus searches were carried out linking preferred terms for access to health care with those for learning disabilities. A copy of our initial search criteria is included as Table 1. The terms used to describe people with learning disabilities in the major medical databases are inconsistent. There is overlap in the scope and definition of terms such as “mental retardation”, “learning disabilities” and “learning disorders”. The term “mental retardation” is still used for indexing despite not being widely considered as an acceptable label for people with learning disabilities. Some databases use an educational definition, sometimes based on IQ, whereas in others it is based on American legislation.
As well as identifying relevant papers, the thesaurus searching was used to identify possible terms for natural language searching and identified further inclusion/exclusion criteria. Our initial search criteria defined traditional search limits, for example, the population to be studied, methods, settings and coverage by date and language. Literature identified through the thesaurus searches and the research contacts, together with papers already held by the lead researcher were read. This enabled us to expand the inclusion/exclusion criteria. These are listed in Table 2 and expressed our judgement about the relevance of particular approaches to the subject, from literature identified in the scoping phase and the consultation, to the model of access to health care.
From our initial thesaurus searching, theoretical papers published anywhere in the world were retained. Papers reporting original research findings from the USA were not generally retained because the different systems for delivering health and social services in the USA meant many papers were not relevant. Our main focus was on research from the UK and countries with similar systems of health care, such as Australia and Canada. However, USA papers on areas where we had identified little or no literature from other searches were considered.
Relevant terms to search for were generated by studying the papers retrieved from the thesaurus searches, issues raised in the consultation and a brainstorming session. The terms for the free-text search strategy comprised:
The learning disability terms were combined with either the access terms or the terms for specific conditions or health issues, defined in Appendix 2, and formed the basis of our main free-text search strategy.
Some additional free-text searches were also carried out as part of the main search phase. From information provided by consultation groups, some additional searches were run to identify literature in more specific areas not covered by the initial thesaurus or main free-text searches, but identified by stakeholders as important issues in access to health care for people with learning disabilities. These searches are listed in Appendix 2 as Additional Free-Text searches.
The personal reference collection of the lead researcher, who specialises in the learning disability field, was scanned for relevant records. This comprised over 200 papers in the field of learning disability and health.
New literature appearing during the project was identified by subscribing to the monthly British Institute for Learning Disabilities Current Awareness Bulletin. Owing to the short time available hand searching of key journals was not carried out. Throughout the duration of the project the current contents pages of the main learning disability journals commonly available and used by the learning disability research community in the UK were scanned using the ZETOC Table of Contents Alerting Service. These are listed in Appendix 1.
Relevant records were also identified by checking the citations of papers identified through electronic databases and other methods.
The results of our searching were saved in a single database using Reference Manager bibliographic software, where possible. Our database coverage included some databases which did not have an import facility, for example CareData and AgeInfo. These databases were searched last of all and only unique references were cut and paste into Reference Manager.
As the concept of ‘access’ was diffuse, initial criteria were based on the model identified, population group and language of publication, as listed in Table 1. More exact criteria for inclusion and exclusion of literature emerged throughout the main free-text literature searching and consultation. In addition to identifying the particular research method employed, this involved considering the underlying differences in approach taken to the subject and their relevance to the model of access to health care for people with learning disabilities. The main types of narrative identified are listed in Table 3.
These “points of view” determined the relevance of the publication to the aims of the review, where those addressing the wider experience of people with learning disabilities in relation to accessing health care services were accepted, and those addressing professional issues were rejected.
In this section the results of searching for literature are presented and the productivity of the methods assessed.
A total of 2221 bibliographic records to potentially relevant literature were identified using the various methods. These were assessed by the lead researcher and the librarian individually and designated as either “obtain paper” or “not relevant”. All the records were then checked for agreement between assessors and those where there was disagreement were re-assessed. The level of agreement was high at around 80%. We obtained the full text of articles if we were unable to judge the relevance of the paper from the title and/or abstract alone.
Of the 300 e-mails and letters sent, 57 contacts sent information regarding their research or suggested other contacts. This represented a 19% response rate. The 57 contacts contributed 49 documents, 10 of which were identified as suitable for further assessment. These were all items of “grey literature” including sections of thesis and reports on locally conducted innovations. The role of the consultation was invaluable in illuminating the model of access, identifying gaps in evidence where no research had been identified and identifying attempted innovations.
Electronic database searching was the most productive method for identifying relevant literature. Thesaurus searching was not particularly productive as we had to exclude many papers, for example, those covering children with a purely educational diagnosis of ‘specific learning difficulties’ such as dyslexia. The scope of the thesaurus terms used varied across databases and was narrow in comparison to the dimensions of our model of access. The additional free-text searches identified 52 references which were potentially relevant. For example, searching under the term “overshadowing” identified a number of additional theoretical papers covering this concept where health needs can go untreated as symptoms are attributed to the learning disability.
No unique relevant literature was identified from searches of web sites and on-line library catalogues of organisations.
We had initially planned to conduct a free-text search across all our databases covering the title, abstract and subject heading/keyword fields. However, due to the large number of non-relevant records initially retrieved and the constraints of time available, we limited our final search strategy to terms occurring in the title of the record. At the time the searches were conducted in August 2002 limiting the search to words in title would have reduced the number of hits on Medline from 12,000 to a more manageable 1,000.
It was not possible to execute the full search strategy on all databases, as the facility for wild card searching, or to combine substantial numbers of free text terms, was either not available or not effective. In these cases a database was simply searched in the title field under “learning disability” and its synonyms. This approach was adopted to search the CareData database which forms part of the National Electronic Library for Social Care. At the time the searches were carried out in August 2002 the database did not support the facility to combine multiple free-text terms.
The personal bibliography of the lead research did not contain any unique records that were not identified via the other methods. However, having an initial collection of relevant literature was invaluable in scoping the subject and identifying terms for the database searching.
The use of a specialist current awareness service and the ZETOC Table of Contents Alerts from key journals led to the identification of additional relevant papers which appeared once the literature review had begun. This improved the currency of literature included in the project.
Snowballing contributed a further 51 potentially relevant references. This compensated for the limitations of database searching by producing additional references which had not been identified by the other methods. Snowballing only began in the later stages of the project, once critical appraisal and evaluation had begun and there was only time to obtain the full text of literature which was easily accessible via the John Rylands University of Manchester electronic journals collection. This constituted in the main journal articles from 1998 onwards. The number of potentially relevant references identified as duplicates increased steadily whilst using this method, increasing our confidence that a point was being reached beyond which only marginal gains, in terms of new references, would be made and saturation point was being reached.
In terms of the overall contribution of the various methods used electronic databases were the most productive, however, snowballing, consultation and the use of current awareness scanning all retrieved unique records which would not have been identified through the database searching alone.
Application of the expanded inclusion/exclusion criteria identified 839 records as potentially relevant on the basis of the information available. The full-text of these items were obtained and they went forward to the assessment stage.
The literature searching was not a sequential process and required additional iterations, moving backwards and forwards through the scoping of the initial topic, initial thesaurus searching and a number of free-text searches. This has been identified as a feature of searching in more diffuse subject areas where searching should move through stages of scoping, piloting, refinement and confirmation.4 Given more time we could have used additional methods to “refine” and “confirm” our searches. These could have included hand searching journals most frequently encountered. However, with diffuse topics it is more difficult to identify core journals covering the topic.5 Obtaining CVs of key authors to compare against our search findings, and citation searching for papers referencing those papers selected for inclusion in the final review could also have been used.
We included a wide range of databases and electronic information sources across a variety of disciplines, subject areas, practitioner audiences and organisations. This reflected the diffuse nature of the topic. However, instead of including such a large number of databases more time could have been spent initially assessing their value in terms of relevant records identified, and less productive sources discounted.
Our initial attempt to search across title, keywords and abstract resulted in an unmanageable number of records. Other ways could have been explored to increase the specificity of the search, including a reduction in the number of terms. We attempted to increase recall through snowballing. We could also have used this method to assess recall earlier by examining why papers identified by snowballing had not been picked-up by our search strategy.
We included a set of terms in our search to identify literature relating to key roles in accessing health care and areas of unmet or additional health need for people with learning disabilities. The rationale for this was to identify papers which, though not including access related terms, provided useful literature on how wider issues relating to the health of people with learning disabilities are presented. It also recognised that literature relevant to access to health care for people with learning disabilities may not be presented as an overtly “access” issue. For example, challenging behaviour is a contested label applied to some people with learning disabilities. It has been suggested its existence is in part a reaction by people with learning disabilities to having widespread unmet health and social needs. It was therefore felt it was important to include these terms in the search strategy as well.
To limit the space required on local servers for storing search results hits were downloaded from bibliographic databases into a single Reference Manager database. Where possible search strategies were saved with the database host or as a file. However, it would have improved the accountability and evaluation of the systematic review process if the results of each database search had first been saved as an individual Reference Manager database before merging and de-duplicating into a final database at a later stage. This would have enabled us to evaluate the usefulness of each database individually and ensure we had an audit trail of records eliminated as duplicates or because they did not meet later inclusion criteria.
Where a health services literature review aims to identify gaps in evidence and/or potential innovations, consultation with a wide range of stakeholders is essential. The consultation process takes time and resources and translates into a variety of outcomes. The main value of the discussion groups was in the identification of issues not covered in existing published evidence. This demonstrated that the research evidence was not reflective of many issues of concern to people with learning disabilities and their carers.
In a systematic review it is good practice to attempt to identify research in progress. This can reduce publication bias and missing relevant literature through publication delays.8,9 Several grey-literature reports on innovations were obtained by this method which would not have been identified otherwise. However, the very nature of contract research means that staff move around and this meant that in many cases contacts identified through research registers could not be traced. The National Research Register (NRR) does not include a findings section to alert users to published outputs from the research, this function is served by The Department of Health Electronic Research Findings Register (REFER). The separation of projects from findings complicated the process of identifying potentially relevant published outputs from projects.
It is good practice to include conference abstracts in a systematic review since, owing to publication bias, they are often the only published output from research which returns a negative result.3 Owing to the limitations of time available we decided to exclude conference abstracts from our study. Time needs to be spent during the scoping phase estimating the amount of relevant literature likely to be identified as this affects the time and resources needed to complete the review. For example, we identified 12 theses as potentially relevant for inclusion and decided to obtain copies. However, there were many practical barriers. In particular document supply costs are expensive if a photocopy is available and hard copies can often not be removed from the loaning library. A more practical policy may have been to first identify whether peer-reviewed papers based on the theses existed and discount those without peer-reviewed outputs. However, this would have led to the omission of recent works, and as the fullest documented account of a research project a thesis can provide more information than a journal article. This enables more effective critical appraisal. It might also result in discounting highly rigorous relevant research that the author has been unable to disseminate through peer-reviewed journals.
This literature review was conducted in the context of current health policy towards people in the UK with learning disabilities. One of its purposes was to identify where future research is needed to support this. Therefore, we had to make a judgement on the relevance of research carried out in one health care system to that of other countries. This led to our decision to concentrate on intervention studies based in countries with a similar health care system to the UK, where health services are funded from general taxation and notionally free at the point of care. With hindsight, it was very difficult to translate this criteria into a search strategy as the indexing on most medical databases concentrates on the language or country of publication. To facilitate carrying out systematic reviews in health services research the indexing of literature by broad type of health care system would be useful.
The phases of developing a search strategy and applying inclusion and exclusion criteria to the results were less distinct and more blurred in this type of review. In reviews of this type, search results must be subjected to additional phases of scanning and sifting for relevance, known as “funnelling”10. The first stage of “funnelling” is relevant to literature identification and, in this project, was based on the limited information available in the titles and abstracts of bibliographic records. Abstracts, where available, should assist the user in determining relevance but are of variable quality and this affects their utility.11 Abstracts were generally found to be better in the medical/health disciplines and poorer in social science. Where an abstract was not available, relevance had to be judged on the title alone.
With a diffuse topic it is difficult to devise criteria and a search strategy which is highly specific. What constitutes “relevant” studies emerges via an extended process of searching, scanning for relevance, translation into criteria and further searching. For example, our search strategy identified numerous papers on issues of access to effective treatments, and in particular the over-prescribing of drugs to people with learning disabilities. We judged that if a person was being prescribed medication they had accessed the service and decided to exclude these. That is, we did not include access to treatment within our definition. We recognise that over-prescribing drugs could be interpreted as an outcome of poor on-going access to health care. However, it was necessary to draw a line between access to healthcare services and use of health care services in order to define our topic. The process of decision making is extended in a systematic review of this type and applies to both initial criteria that can be universally applied to the literature, and the development of further criteria related to the nature of the literature, approaches to study, and the variety of narratives revealed.
Documentation and record keeping across all phases and iterations of a search must be thorough and records should be kept of the rationale for decision making. A review of this type involves a series of linked searches, supplemented by other methods of identifying literature, rather than the development of a single search strategy.
If a search strategy cannot be fully defined a priori this has implications for the systematic review process, its accountability and the role of the health librarian. Reviews of diffuse topics benefit from having an underlying model or typology against which to assess search results. Scanning search results for relevance needs to involve all members of the research team to resolve issues of disagreement to develop final inclusion criteria. It is therefore essential to carry out scanning for relevance on initial pilot search results and to proceed in stages, if necessary going back to the literature to conduct further searches once clarification has been reached.
Systematic reviews of health services transcend disciplines, methods and types of evidence. This means that a wide range of electronic sources of information must be searched. Librarians need to develop experience of search interfaces and database operation across a range of providers to ensure search strategies can be translated and adapted and the limitations of resources recognised and recorded.
Health librarians working on such projects should be prepared to participate fully in the development and application of inclusion and exclusion criteria. This requires a substantial amount of additional learning in relation to the topic under investigation to understand the context in which health services are being delivered and the population being studied.
To conduct systematic reviews of diffuse topics in health services requires some adaptation of existing guidelines.3 An approach is required which recognises underlying differences in approach to a topic and can allow for different types of evidence in terms of audience and methods. This paper has tried to identify what some of the implications of this are for the process of systematic reviewing and for health librarians who maybe involved in this type of project.
We thank all those who participated in the consultation exercise, our colleagues Angela Swallow, Caroline Glendinning, the staff of the Document Supply Unit, John Rylands University Library of Manchester and the NCCSDO.
Medline (Index Medicus)
CINAHL (Cumulative Index to Nursing & Allied Health)
HMIC (Health Management Information Consortium)
Social Science Citation Index
ASSIA (Applied Social Science Index)
IBSS (International Bibliography of the Social Sciences)
BEI (British Education Index)
ERIC (Educational Resources Index)
Grey Literature databases
SIGLE (System on Grey Literature in Europe)
ASLIB Index to Thesis
ISI Index to Scientific and Technical Proceedings
On-line library catalogues
Royal College of Nursing
Royal National Institute for the Blind: http://www.rnib.org.uk//library/research/
Web sites of organisations
Down’s Syndrome Association: http://www.dsa-uk.com/
NRR (National Research Register)
CRIB (Current Research in Britain)
REGARD (Register of Economic & Social Research Council)
CORDIS (European Union Research and Development Database)
REFER (DOH Electronic Register of Research Findings)
RCGP Research Intelligence Database (Royal College of General Practitioners)
Journal contents pages scanned using the ZETOC service during the project
American Journal of Mental Retardation
Journal of Intellectual Disability Research
Research in Developmental Disabilities
Journal of Learning Disabilities - New York
Journal of Learning Disabilities - London
Journal of Applied Research in Intellectual Disabilities
Research Register Searches (* indicates the use of a wild card)
Initial Thesaurus searching
The following thesaurus searches were carried out with the terms exploded and applying subheadings where occurring:
Medline & Cinahl
(Learning Disorders or Mental Retardation) AND Health Services Accessibility
(Learning Disorder or Mental Deficiency) AND Health Care Access
(Learning Disorders or Learning Disabilities or Mental Retardation) AND Health Care Utilization
Main Free-text searching
Sample strategy is for OVID databases
Set 1. (learning disab$ or learning disorder$ or developmental$ disab$ or intellectual$ disab$ or mental$ retard$ or learning difficult$ or intellectual$ impair$ or mental$ handicap$ or mental$ subnormal$ or mental$ deficien$).ti
Set 2. (access$ or advoca$ or barrier$ or carer$ or communication$ or information or parent$ or uptake or utili#ation or need$ or provision or consent$ or help seeking or help-seeking).ti
Set 3. (School nurse$ or health visitor$ or abuse or Alzheimer$ or Down$ or autis$ or cerebral palsy or challenging behaviour or dementia or epilepsy or health promotion or hypo-thyroidism or hypo thyroidsim or mental health or prescrip$ or screening or sensory impairment or hearing or vision or sexual health)ti.
Set 4. Set 1 AND Set 2
Set 5. Set 1 AND Set 3
Set 6. Set 4 OR Set 5.
Additional free-text searches (all terms appearing in title only – carried out on conclusion of the consultation exercise
Set 1 AND (Utili#e or Inaccessib$ or Availab$ or Prohibit$ or Affordab$ or Applicab$)
Set 1 AND (Stress or Coping or Diagnos$ or Pain or Overshadow$ or Third Part$ or Health education or Integrat$ or Referral$ or Trust or Confidential$ or Medical record$ or Volunteer$ or Time or Labelling or Partnership$ or Care Pathway$)
Set 1 AND (SEN Statement Review or Disability Discrimination or Children$ Act)
Set 1 AND (Workforce or labo#rforce or staff$ or professional$ or doctor$ or nurs$) AND attitude$
Set 1 AND (Continuity ADJ5 care or Routine ADJ5 care or Care ADJ5 interface)
Set 1 AND (Private Health Care or Pain Clinic$ or Dysphagia or Weight or Dentist$ or dental or Optician$ or optical or eye or NHS Direct or Health Visiting or Health Visitor or Community Nurs$ or Liason Nurs$ or Physiotherapy Receptionist$)
Set 1 AND (Speech ADJ5 Language)
Set 1 AND (Casualty or (Accident ADJ Emergency))
The definitive version is available at: http://www.blackwell-synergy.com/links/doi/10.1111%2Fj.1471-1842.2004.00512.x
Rosalind McNally, National Primary Care Research & Development Centre, 5th Floor, Williamson Building, University of Manchester, Oxford Road, Manchester, M13 9PL ; Email: ku.ca.nam@yllaNcM.dnilasoR.
Alison Alborz, National Primary Care Research & Development Centre, 5th Floor, Williamson Building, University of Manchester, Oxford Road, Manchester, M13 9PL ; Email: ku.ca.nam@zroblA.nosilA.