The following sections outline the data structures and analytic tools that form the basis of the XplorSeq workflow.
Data organization and GUI architecture
XplorSeq uses a document-based approach for project data management in which multiple sequences and their associated data are stored and accessed in a single file. As a project evolves, sequences may be added, deleted, amended and analyzed as needed. XplorSeq does not enforce a highly constrained analytic schema and thereby grants the user more autonomy in designing and implementing an analysis plan than typically is possible in a hard-wired software pipeline.
Data are organized as a hierarchy of data objects (Fig ), which represent the model layer in XplorSeq's Model-View-Controller design pattern architecture. Each data object class (e.g. Sequence, Project) is associated with a window (the view layer) that is used to inspect data associated with particular instances of the class.
Organization of basic XplorSeq Data structures.
The top layer data object is the "Project", which stores all other data and is synonymous with the document as a whole. Hence, the main XplorSeq window (Fig ) is the Project Inspector window. Projects organize and manage lists of "Clones", which represent individual cloned genes or PCR amplicons. Clones, in turn, manage groups of "Sequences" which map to unique DNA sequences. Sequences can be imported directly (e.g. as polished GenBank sequences), read from DNA sequencer traces, or assembled from other sequence objects ("contigs"). For each sequence analyzed by BLAST, XplorSeq creates a "BlastInfo" object that summarizes pertinent blast output data: identity and phylogenetic lineage of the sequence's closest BLAST hit, BLAST statistics, etc. Each Clone ranks its constituent Sequence objects based on BLAST bit-score and the "Best Sequence" (i.e. that with the highest bit-score) serves as a proxy for the entire Clone. "Oligo" objects encapsulate data that describe oligonucleotide sequences used in construction of clone libraries.
One of XplorSeq's more important design goals is to provide users with an extremely flexible and simple means of annotating data objects with meta-data consisting of user-specified information. XplorSeq implements meta-data through two strategies: 1) hard-coded data objects (i.e. "BlastInfo" and "Oligo" objects) and 2) a customizable "Metadata" object. Metadata objects are implemented as key-value dictionaries (e.g. hashes, maps) that can be linked to Project, Clone, and Sequence data objects. Users specify keys and values for particular objects through import of tab-delimited spreadsheets in which the first column designates the names of objects (i.e. sequence/clone names) and subsequent columns specify values (keyed by column headings) to attach to the objects. Alternatively, object inspector windows provide GUI-based features in which to display and edit meta-data (c.f., Figs. , , and ).
Figure 3 Clone inspector window. Summarizes information associated with a selected clone, a group of sequences (both individual reads and contigs) from the same amplified or cloned gene. The main window summarizes the top BLAST hit for the clone (i.e. the sequence (more ...)
Figure 4 Sequence inspector window. Displays information associated with a selected sequence. Nucleotides are color-coded to represent the quality scores of individual nucleotides (the legend at the bottom of the window shows the meaning of the colors). Primers (more ...)
Metadata editors. A) Editing all keys for meta-data in project. B) Editing values associated with keys for a particular sequence object.
Data display and control of data processing
A project's Clone, Sequence, and BlastInfo objects are displayed in the Project Inspector window (Fig. ), which functions as the main XplorSeq window. Data are arranged hierarchically to reflect nesting of data structures. For each Clone, a summary of its best blast hit, which includes the taxa name, percent sequence identity and bit-score is displayed in the main XplorSeq window. The phylogenetic lineage of the top blast hit can be imported (through either an entrez idfetch query or import of tab-delimited data) to provide information about the taxonomic placement of a clone.
The user controls all steps of data processing by selecting objects to be acted upon and then choosing a function from menu items presented in the tool panel that extends from the main XplorSeq window (Fig ). Methods to import, export, and analyze data are accessible through these menus. Below the menus lie controls through which oligonucleotides used to generate PCR libraries can be designated, if relevant to the project; entries in the oligo menus can be modified through a preferences dialog.
Project specific meta-data can be recorded in several text fields presented in the Project window, under the "Project Info" tab. An editable text box is presented in which the user can enter comments, for instance details specific to a project (Fig ).
By double-clicking on an entry in the Project window, the user can display and edit more detailed information associated with that entry. For example, the Clone Inspector window (Fig. ) summarizes the content of the Clone Object, including its top BLAST hit sequence and corresponding BlastInfo Object. The phylogenetic lineage and domain of the Clone Object can be set through the controls at the bottom of the window. A panel extending from the Clone Inspector window presents user-specified meta-data associated with the Clone Object.
DNA sequences, including contig sequences, are displayed in a Sequence Inspector window (Fig. ). Individual nucleotides are color-coded to represent quality scores generated by the base-calling software (shades of blue) or trimmed sequences (red). Basic sequence information, such as primer sequences and trimmed sequence length, is displayed in a set of text fields at the bottom of the Sequence Inspector window. Similar to Clone Objects, sequence specific meta-data can be viewed through a panel that extends from the inspector window.
Tools for data import and analysis
Table summarizes the key functions available through XplorSeq. A user can perform base-calling on chromatograms generated by automated DNA sequencers, (through tracetuner [30
] or phred, [31
]); trim away vector, primer, and poor quality sequences; assemble contiguous sequences (TIGR_Assembler [33
] or phrap, [31
]); search for homologous sequences in sequence databases (blastcl3 or blastall; [1
]); format BLAST databases (formatdb), perform multiple sequence alignments (clustalW; [34
]); define O
nits (sortx; D.N. Frank, unpublished); calculate distance matrices (phylip dnadist [37
]); construct phylogenetic trees (clearcut, [38
]; phylip neighbor, [37
]; RAxML, [39
]); and calculate a variety of biodiversity indices (biodiv; D.N. Frank, unpublished). By integrating software tools into a single application, XplorSeq facilitates seamless workflow through the sequence analysis process. Many of the key analytic steps are piped together, thus automating data processing. All data analysis operations may be executed in batch so that multiple sequences can be processed with minimal user intervention.
Summary of XplorSeq functionality
XplorSeq facilitates batch BLAST [1
] analyses of DNA sequences through both networked and local searches of nucleotide databases. Local BLAST searches require properly formatted sequence databases, which may be downloaded from NCBI http://ncbi.nlm.nih.gov
or created by use of the formatdb executable (through XplorSeq or the command line). XplorSeq dispatches sequences to the appropriate client software (blastcl3 or blastall; [1
]) and then parses the resulting output file into BlastInfo Objects.
XplorSeq provides graphical interfaces for several commonly used command-line phylogenetic tools are shown in Fig. . Phylip modules for distance matrix calculation, neighbor-joining/UPGMA trees, data-set bootstrapping, and consensus tree generation are accessible through these interfaces [37
]. Similarly, fast neighbor-joining and maximum-likelihood phylogenetic tree inference are available (clearcut [38
] and RAxML [39
], respectively). In each case, following selection of the appropriate options, sequence data are exported in the required format, the analysis performed, and results imported (if appropriate) into XplorSeq. For tasks that may take prolonged time to execute (e.g. RAxML [39
]), a file listing the command-line invocation of the program is saved along with all data required for the task; these files can be transferred to another computer for subsequent execution of the scripted command.
Figure 6 Analyses of aligned sequences. XplorSeq provides GUI-based access to several command-line programs used for phylogenetic analysis of multiple sequence alignments, including A-D) several commonly used programs from the phylip package ; E) RAxML for (more ...)
OTU clustering is implemented through the program sortx, which was written in tandem with XplorSeq (Fig. ). Sortx uses a fast radial clustering algorithm to bin aligned sequences based on uncorrected pairwise sequence distances (%ID). Clusters can be assembled based on furthest-, mean-, or nearest-neighbor rules. Following cluster formation, sortx selects a representative sequence for each cluster, which maximizes both pairwise similarity to other cluster members and sequence length (simply choosing the sequence with minimum pairwise distance could select for short, but well-conserved sequences, which would not necessarily be representative of the cluster). Finally, the user can select a range of pairwise sequence distance thresholds by which to assemble OTUs in order to create multiple data sets at different phylogenetic depths.
Estimates of biodiversity indices (species richness, diversity, evenness) can be reported through either of two modes. First, the export function OTU Diversity... reports basic calculations of commonly used indices (Sobs
, Good's coverage, Shannon diversity; [40
]) for a set of selected sequences. Alternatively, the same biodiversity estimates can be made in a more thorough manner through execution of the analysis function Biodiversity (biodiv)..., which invokes the program biodiv, a standalone command-line tool built in conjunction with XplorSeq. As shown in Fig. , the user selects OTUs definitions for the selected sequences through choice of meta-data options. To compare indices between different groups of sequences, the user can also select multiple "environments" by which to differentiate the sequence subsets; biodiv then performs separate analyses for each designated environment. Biodiv performs random resampling of OTUs and calculates collector's curves and associated biodiversity indices as a function of sampling effort [41
]. Biodiv also reports rarefied biodiversity indices, based on resampling, with 95% confidence intervals for each type of environment [41
Tools for data export and transformation
XplorSeq provides several features by which to export sequences and data for further analyses. Data reports are generated in tab-delimited format for import into spreadsheets. A particularly useful export feature tabulates sequence abundances and/or prevalences on the basis of user-defined terms that define how data are categorized and enumerated. For instance, as shown in Fig. , the user has opted to create a table in which rows are defined by meta-data that specify OTUs and columns are specified by meta-data that organize samples by the results of a PCR assay; alternatively, similar sequences can be grouped by BLAST results, or phylogenetic lineage. The cells of the output tables enumerate the abundance or presence/absence of sequences for a given row and column (Additional file 1
Figure 7 Tabulation of sequence abundance/prevalence. The Summary Table dialog provides multiple means of tabulating sequence data. In this example, rows are defined by the values of a meta-data key (e.g. 97% OTUs), while columns are defined by the value associated (more ...)
Preparation of sequence data for submission to public databases can be tedious and error-prone, despite the availability of tools such as NCBI's sequin software tool. XplorSeq can facilitate the submission process by automating the organization of sequences and their annotations into a form suitable for input into sequin (Fig. ). The user can modify the output of the sequin script export function to include or exclude particular information and to tailor GenBank descriptors and/or features (e.g. Locus, Definition, Molecule fields).
Sequin script export. Scripted export of sequence data for GenBank submission through Sequin. Data associated with each sequence can be manipulated in order to tailor the level of detail that will go into Sequin.
For detailed phylogenetic studies, sequences assembled by XplorSeq can be exported in formats suitable for input into other software packages, such as ARB [42
], DOTUR [43
], NAST [44
], SILVA [45
], or UniFrac [46
]. Users can choose sequences to be exported by selecting particular objects or through filters for sequence length, bit-score, type of sequence (i.e. contig vs. raw sequence), or other meta-data. Finally, "placeholder" Newick-formatted phylogenetic trees can be exported in order to select specific subsets of sequences in tree visualization software such as ARB. In this format, taxa belonging to user-defined subsets (categorized, for instance, through specific meta-data) are placed in clades and assigned artificial branch-lengths of 0.1; inter-group distances are set to 0.9, which produces a well-demarcated organization of sequences into assigned groups. Import of such a tree into ARB allows rapid, graphical marking of different sets of taxa (Fig. ), the results of which are propagated to other "real" trees in ARB.
Figure 9 Input of placeholder tree into ARB. Sequences were exported from XplorSeq using the "Placeholder Tree" option. Sequences were split into two categories on the basis of associated meta-data (in this example, results of a PCR screen). Here, the user has (more ...)
The data analysis tools employed by XplorSeq (e.g. phred, phrap, blast, sortx, biodiv) are invoked in separate threads, so that multiple tasks can be run in parallel. During these steps, feedback to the GUI is limited in order to minimize the overhead of performing analyses through XplorSeq. Typical latencies associated with XplorSeq contribute no more than an additional ~10% to the total elapsed time required for an analysis, relative to the same task performed through command line execution of the underlying 3rd party software (Table ). This overhead is due primarily to data import/export and parsing of results, which add value to the functionalities provided by command line utilities. Consequently, XplorSeq should not place any undue limitations on the use of command-line software to process and analyze sequence data, beyond those inherent in the computer system and/or software tool being used. In other words, tasks that can be performed on a particular Macintosh system through the command line should also be feasible through XplorSeq.
Execution times of commonly used software: comparison of XplorSeq with command line implementation
The XplorSeq file format, implemented using the Cocoa software framework, does not significantly add to the size of data files. For example, an XplorSeq file containing an alignment of 250,000 rRNA sequences (1585 nucleotides per sequence) requires 396 megabytes (MB) of storage, compared to 381 MB for a fasta formated file storing the same alignment. The addition of metadata, such as blast results, increases file size in roughly linear proportion to the size of parsed input text.
Although the use of a flat file format by XplorSeq greatly simplifies data storage and transfer, it does require that all data be read into memory before being manipulated. For relatively large files, bottlenecks are apparent primarily in tasks requiring import and export of data. Thus, sequence analysis projects are likely to be limited as much by computer hardware (e.g., quantity of random access memory and bus speeds) and the performance of underlying 3rd party software tools as by XplorSeq. However, the XplorSeq environment is scalable from laptops to more advanced workstations (e.g., 8-core/32 GB systems currently available) so its capabilities can be expanded as need arises and hardware evolves.
As a rough guide to system requirements, Table presents benchmark comparisons of common XplorSeq tasks performed on a laptop and a workstation. To open the 250,000 sequence XplorSeq alignment file described above, for example, requires ~130 or ~12 seconds, on the laptop or workstation, respectively. Test datasets of 1 × 106
25-nucleotide sequences can be manipulated with similar lag times. More fully annotated files containing ~50,000 16S rRNA sequences, each of length ~1000 nucleotides, have been used routinely on a laptop with little performance degradation [17
]. However, projects with > 50,000 annotated sequences likely will require higher performance workstations to insure the responsiveness customary to users of GUI-based software. Alternatively, sequence data can be spread across multiple files, each of which maintains information generated from a particular experiment (e.g. a PCR amplicon library). More complex data storage strategies, such as the use of application-specific databases may be implemented in the future if they do not compromise the XplorSeq philosophy of ease of software installation and use.