PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Trends Cogn Sci. Author manuscript; available in PMC 2011 January 1.
Published in final edited form as:
PMCID: PMC2818206
NIHMSID: NIHMS149275

Outsourcing neuroimaging data analysis: Implications for scientific accountability and issues in the public interest

Rodin's statue “The Thinker” depicts a solitary ponderer -- an iconic image of the individual scientist. Historically, this has been a fitting icon for scientific research in cognitive science domains including philosophy and psychology. However, this ideal is increasingly difficult to achieve in many academic disciplines, and particularly in the field of cognitive neuroscience where the mastery of multiple domains often exceeds the ability of any one individual. The result is an increase in collaborative efforts where responsibility is distributed among a research team. There is, however, an emerging development in cognitive neuroscience. In a new form of collaboration, the academic researcher performing functional neuroimaging can outsource data analysis to commercial companies. While outsourcing might be suitable for some analysis situations, for others it raises important considerations for granting agencies and peer-review policies, and in certain instances may be a disservice to the research enterprise.

Although for cognitive neuroscientists outsourcing may be a byproduct of the demands of complex investigations, it is common in other academic fields. Certain biomedical fields, including neuroscience, have outsourced time-consuming procedures that are transparent and/or amenable to standardization (e.g., gene sequencing, tissue analysis, and identification of molecular compounds). In such cases outsourcing enables access to knowledgeable analysts, advanced methodologies, and expensive equipment. This allows the investigator to remain at the forefront of scientific trends, to maintain several different projects in the research pipeline, and to complete projects more quickly.

These benefits are now available to neuroimagers. Outsourcing companies provide an array of neuroimaging services for both academic and for private industry clients (e.g., pharmaceutical companies). These services range from low-level image conversion to intermediate pre-processing steps to complete pre-fabricated analysis pipelines (Table 1). Outsourcing, however, may have negative implications as well. The degree of negative impact depends on the degree to which the researcher transfers control [1], which in turn depends on the nature of the outsourcing relationship. In essence, outsourcing neuroimaging data analysis lies on a continuum from obtaining consultation, to contracting out the research effort itself. The latter obviously represents a loss of control over the analysis process (Box 1).

Box 1The many ways to outsource

There are many possible client-provider relationships, which vary in the degree to which the researcher retains control over the analysis, and consequently in the degree to which they are accountable for the results. We present a selection of examples below.

Example 1. Consulting for analytic knowledge. The researcher approaches a consultant for advice about data analysis. The researcher takes full responsibility for analysis decisions. Authorship for the provider is likely not warranted.

Example 2. Consulting in a collaborative context. The consultant functions as a collaborator, assisting in the study design, construction of detailed analysis plan, and analysis proper. Authorship for the provider is warranted.

Example 3. Outsourcing a pre-determined analysis plan. A researcher limited by computing facilities specifies a workflow using a workflow language [13, 15; also see SPM batch language; http://tinyurl.com/lruyhb, p. 327] that is a complete specification of a desired analysis, and a private company executes this workflow. For the most part the researcher retains control over the analysis workflow, but the degree of control is dependent on the degree of detail of the analysis plan, and on the interactive nature of the outsourcing relationship. Authorship for the provider may or may not be warranted, depending on whether the provider makes a substantial intellectual contribution to the project.

Example 4. Outsourcing the entire analysis. The researcher contracts a private company to complete the analysis of a collected data set in its entirety. In this case, the details of the analysis are unknown to the researcher, and control over the analysis is completely transferred to the provider. This relationship raises fundamental issues about authorship and accountability.

Table 1
Services and policies of some existing neuroimaging providers. (Actual company names have been replaced with letters for confidentiality reasons.)

On this point, lessons can be learned from services offered within academic institutions themselves. Universities have realized that scientists need analysis support in data intensive domains, and several centers provide biostatistical consultancy services (e.g., the Biomedical Research Informatics Core at Beth Israel Deaconess Medical Center; the Center for Biostatistics at The Ohio State University). Although these are “in house” services, their policies highlight the delicate balance between control and delegation of authority that accompanies external assistance. Such centers typically offer two service tracks: (a) general consulting, e.g., low-level analysis with minimal conceptual input, which often do not warrant co-authorship; or (b) support that constitutes “scientific or intellectual contribution” in which case authorship is warranted [2, 3; see sample policies: http://tinyurl.com/nkltdy; http://tinyurl.com/ntkaz7]. Scientific or intellectual contributions include input to the study design or protocol, and planning or execution of analysis [4]. To summarize, “in-house” consultancy carries with it a distribution of authority, which must be adequately communicated to the public by including the consultant as a co-author on published work.

Are these service tracks applicable for neuroimaging? In our experience, neuroimaging analysis decisions are tightly linked to a study's design and goals, and often communicate central aspects of the research question. Further, determining the appropriate analysis requires a solid foundation in statistics, neuroanatomy, and MR physics [5]. A non-expert who wants to outsource cannot know if a “routine” workflow is appropriate since even seemingly simple analysis choices such as determining the size of a spatial smoothing filter [6], correcting for motion in block designs [7], or regressing the global signal [8] can fundamentally influence the quality and interpretation of the results. Indeed, experienced analysts examining the same functional imaging dataset can arrive at strikingly different results [9]. This indicates that neuroimaging data analysis cannot be considered as a “black box” process that can be delegated to a consultant. Analytic choices have genuine implications, and the researcher must be involved in making them. For this reason, we think that a general consulting track, of the sort given by a colleague, will only rarely apply. The second track, where the support by the outsourcing provider constitutes a substantial contribution meriting authorship (Box 1, Examples 2, 3) will be most appropriate.

In cases where some or all of the analysis is outsourced, control is relinquished to the provider (Box 1, Example 4). This situation has potentially negative implications. A primary concern is blurring of accountability for the results, an issue made salient in the wake of recent scandals in biomedical research [10-12]. A researcher who lacks the knowledge to find errors in the workflow, evaluate its relative merit, or describe its provenance [13] cannot be accountable for it. There are also other practical concerns. One issue is the training. Should graduate students outsource data analysis and can they earn advanced degrees based on such analyses? Further, frequent outsourcing can erode the laboratory skill base [1], impacting the development of an adequate training program and future project development.

Outsourcing neuroimaging data analysis also raises general issues related to the public interest in scientific research. First, jurisdiction of university ethical review boards over commercial companies is unclear, and it is uncertain whether privacy regulations could be enforced on an external company. Policies already in place for cross-university collaboration might apply here, but this remains to be determined. A more prominent issue is the appropriateness of using grant funding to subcontract a central aspect of the research process to private for-profit companies. It is well recognized that public funds should benefit public rather than private interests. Is it in the public interest that private companies perform the scientific work rather than the grant recipient? When funds are allocated to investigators, there is an implicit assumption that lessons learned are disseminated and propagated to the public domain. In contrast, companies are motivated to guard their intellectual property, and may or may not disseminate this information. Furthermore, because neuroimaging analysis is such a central part of the research, we believe it merits consideration as a special case. To illustrate, the National Institutes of Health consortium agreements policy states “the grantee must perform a substantive role in the conduct of the planned research and not merely serve as a conduit of funds to another party or parties” (“Consortium Agreements” section in http://tinyurl.com/lthjj2). Outsourcing the bulk of the analysis potentially weakens the substantive role of the investigator. After all, grants are awarded on the basis of research ability, which does not entail the hiring of external parties to conduct the research. Of course it is up to the granting agency to determine whether their policy is violated, but at the very least outsourcing work should be clearly described in grant applications, allowing full transparency and oversight to serve the public interests.

While funding issues can be formally regulated, few ethical policies exist to deal with the issue of accountability. At least one provider we surveyed had no policy in place for how their services should be cited. Editorial policies for journals also fail to provide sufficient protections for accountability. For example, although the journal Nature has recently posted updated policies concerning author responsibility [14], it is unclear how these policies would apply in cases of complete or near-complete transference of control over data analysis. Two additions to Nature's policy now state that (1) authors must report their contributions, and (2) at least one senior author must take responsibility for the contributions of all authors. What do these imply for outsourcing? The first indicates that the provider must be listed as an author because it contributed to the research—a senior author does not have the right to omit mention of the provider if it made a significant contribution. The second indicates that the senior author cannot claim to take responsibility for the work of the outsourcing company if neither they nor any author on the paper could describe the analysis in any level of detail. Importantly, including the provider as a co-author introduces further difficulties. Because acceptance of the paper serves as successful publicity for the company, it has a commercial interest in its acceptance. Thus, the paper must be accompanied by a conflict of interest declaration, which could bear on the evaluation of the work. The academic researcher who wants to outsource must consider these issues.

In summary, we believe there is a place for certain types of outsourcing relationships in neuroimaging. In some cases it is advantageous (e.g., when it accelerates the processing of a prepared analysis). However, currently the main hurdles to its widespread use are the absence of policies regulating its use within grant-funded projects or in the context of journal publications. Of paramount importance here are issues related to accountability. Many of the mentioned problems could be alleviated by transparency in the research process (e.g., by detailed annotation of the analysis steps; by clear author responsibilities; and by full disclosure in grant applications). Other issues are more fundamental (e.g., the potential for conflict of interest in the reporting of results), and will demand detailed positions by journals. In short, while some applications of outsourcing seem in accord with good scientific practice, there are several outstanding issues. The research community should begin to consider these issues in order to be in a better position to address them should outsourcing neuroimaging data analysis become a more common practice in the academic setting.

Acknowledgments

We thank Giovanna Egidi, Dan Llano, Mandy Maguire, Steven Small, and two anonymous reviewers for helpful comments and discussion. We contacted the following companies during the preparation of this manuscript: Cambridge Neuroimaging, Imaging Advantage, Innovative Imaging Ideas, Molecular Neuroimaging, Neuromorphometrics, Predictek, and VirtualScopics.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

1. Pichler FB, Turner SJ. The power and pitfalls of outsourcing. Nat. Biotechnol. 2007;25:1093–1096. [PubMed]
2. Parker RA. Estimating the value of an internal biostatistical consulting service. Stat Med. 2000;19:2131–2145. [PubMed]
3. Guillion CM, Berman N. What statistical consultants do: Report of a survey. Am Statistician. 2006;60:130–138.
4. Mullee MA, et al. Statisticians should be coauthors. Brit Med J. 1995;310:869. [PMC free article] [PubMed]
5. Devlin JT, Poldrack RA. In praise of tedious anatomy. NeuroImage. 2007;37:1033–1058. [PMC free article] [PubMed]
6. Giessler A, et al. Influence of fMRI smoothing procedures on replicability of fine scale motor localization. NeuroImage. 2005;24:323–331. [PubMed]
7. Johnstone T, et al. Motion correction and the use of motion covariates in multiple-subject fMRI analysis. Hum Brain Mapp. 2006;27:779–788. [PubMed]
8. Fox MD, et al. The global signal and observed anticorrelated resting state brain networks. J Neurophysiol. 2009;101:3270–3283. [PubMed]
9. Poline J-B, et al. Motivation and synthesis of the FIAC experiment: Reproducibility of fMRI results across expert analyses. Hum Brain Mapp. 2006;27:351–359. [PubMed]
10. Vogel G. Picking up the pieces after Hwang. Science. 2006;312:516–517. [PubMed]
11. Ross JS, et al. Guest authorship and ghostwriting in publications related to Rofecoxib: A case study of industry documents from Rofecoxib litigation. JAMA. 2008;299:1800–1812. [PubMed]
12. Who is accountable? Nature. 2007;450 doi:10.1038/450001a.
13. MacKenzie-Graham AJ, et al. Provenance in neuroimaging. NeuroImage. 2008;42:178–195. [PMC free article] [PubMed]
14. Authorship policies Nature. 2009;458 doi:10.1038/4581078a.
15. Stef-Praun T, et al. Accelerating medical research using the Swift workflow system. St Heal T. 2007;126:207–216. [PMC free article] [PubMed]