A hybrid pipeline for reconstruction & analysis
of viral and host genomes at multi-organ level.
TRACESPipe is a next-generation sequencing pipeline for identification, assembly, and analysis of viral and human-host genomes at multi-organ level. The identification and assembly of viral genomes rely on cooperation between three modalities:
- compression-based predictors;
- sequence alignments;
- de-novo assembly.
For the human-host variant call identification, the same procedure is followed although directly starting within the second point, given the use of the same reference (revised Cambridge Reference) to all the cases.
The previous image shows the architecture of TRACESPipe, where the green line stands for the mitochondrial human line. This pipeline has been tested in Illumina HiSeq and NovaSeq platforms. The operating system required to run it is Linux. In windows use cygwin (https://www.cygwin.com/) and make sure that it is included in the installation: cmake, make, zcat, unzip, wget, tr, grep (and any dependencies). If you install the complete cygwin packet then all these will be installed. After, all steps will be the same as in Linux.
The TRACESPipe includes methods for ancient DNA authentication, namely using the quantification of damage (in the tips of the reads) relative to a reference. Other feature is the quantification of y-chromosome presence through compression-based predictors.
Additionally, the TRACESPipe includes read trimming and filtering, PhiX removal, and redundancy controls (at the Database level and for each candidate reference genomes) to improve the consistency and quality of the data.
is needed for installation.
To install Conda use the following steps:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
Additional instructions can be found here:
https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html
To install TRACESPipe, run the following commands in a Linux OS:
git clone https://github.com/viromelab/tracespipe.git
cd tracespipe/src/
chmod +x TRACES*.sh
./TRACESPipe.sh --install
./TRACESPipe.sh --get-all-aux
In the tracespipe/ folder the following structure exists:
tracespipe/
βΒ Β
βββ meta_data/ # information about the filenames in input_data/ and organ names
βΒ Β βββ meta_info.txt # see Configuration section for this file.
βΒ Β
βββ input_data/ # where the NGS reads must be placed (and compressed with gzip)
βΒ Β
βββ output_data/ # where the results will appear using the following subfolders:
β β
βΒ Β βββ TRACES_results/ # where the files regarding the metagenomic
β β # analysis, redundancy (complexity) and control will appear
βΒ Β βββ TRACES_results/profiles/ # where the redundancy (complexity) profiles appear
β β
βΒ Β βββ TRACES_viral_alignments/ # where viral alignments and index will appear
βΒ Β βββ TRACES_viral_consensus/ # where viral consensus (FASTA) will appear
βΒ Β βββ TRACES_viral_bed/ # where viral BED files will appear (SNPs and Coverage)
βΒ Β βββ TRACES_viral_statistics/ # where viral statistics appear (depth/wide coverage)
βΒ Β β
βΒ Β βββ TRACES_mtdna_alignments/ # where mtdna alignments and index will appear
βΒ Β βββ TRACES_mtdna_consensus/ # where mtdna consensus (FASTA) will appear
βΒ Β βββ TRACES_mtdna_bed/ # where mtdna BED files will appear (SNPs and Coverage)
βΒ Β βββ TRACES_mtdna_statistics/ # where mtdna statistics appear (depth/wide coverage)
βΒ Β βββ TRACES_mtdna_authentication/ # where mtdna species and population authentication appears
βΒ Β β
βΒ Β βββ TRACES_cy_alignments/ # where cy alignments and index will appear
βΒ Β βββ TRACES_cy_consensus/ # where cy consensus (FASTA) will appear
βΒ Β βββ TRACES_cy_bed/ # where cy BED files will appear (SNPs and Coverage)
βΒ Β βββ TRACES_cy_statistics/ # where cy statistics appear (depth/wide coverage)
βΒ Β β
βΒ Β βββ TRACES_specific_alignments/ # where specific alignments and index will appear
βΒ Β βββ TRACES_specific_consensus/ # where specific consensus (FASTA) will appear
βΒ Β βββ TRACES_specific_bed/ # where specific BED files will appear
βΒ Β βββ TRACES_specific_statistics/ # where specific statistics appear (depth/wide coverage)
βΒ Β β
βΒ Β βββ TRACES_mtdna_damage_<ORGAN>/ # where the mtdna damage estimation files will appear
βΒ Β β
βΒ Β βββ TRACES_denovo_<ORGAN>/ # where the output of de-novo assembly appears
βΒ Β β
β βββ TRACES_hybrid_alignments/ # where the hybrid data appears
β βββ TRACES_hybrid_consensus/ # where the hybrid data appears
β βββ TRACES_hybrid_bed/ # where the hybrid data appears
βΒ Β β
β βββ TRACES_hybrid_R2_alignments/ # where the second round hybrid data appears
β βββ TRACES_hybrid_R2_consensus/ # where the second round hybrid data appears
β βββ TRACES_hybrid_R2_bed/ # where the second round hybrid data appears
βΒ Β β
β βββ TRACES_hybrid_R3_alignments/ # where the third round hybrid data appears
β βββ TRACES_hybrid_R3_consensus/ # where the third round hybrid data appears
β βββ TRACES_hybrid_R3_bed/ # where the third round hybrid data appears
βΒ Β β
β βββ TRACES_hybrid_R4_alignments/ # where the fourth round hybrid data appears
β βββ TRACES_hybrid_R4_consensus/ # where the fourth round hybrid data appears
β βββ TRACES_hybrid_R4_bed/ # where the fourth round hybrid data appears
βΒ Β β
β βββ TRACES_hybrid_R5_consensus/ # where the automatic choosen hybrid consensus
βΒ Β β # appears (diff will be made using this data)
βΒ Β β
β βββ TRACES_multiorgan_alignments/ # where the multi-organ alignments data appears
β βββ TRACES_multiorgan_consensus/ # where the multi-organ consensus data appears
βΒ Β β
β βββ TRACES_diff/ # where the dnadiff results appear (identity & SNPs)
β βββ TRACES_specific_diff/ # where the dnadiff results appear for specific
βΒ Β β
β βββ TRACES_blasts/ # where the specific blasted results appears
βΒ Β
βββ to_encrypt_data/ # where the NGS files to encrypt must be before encryption
βββ encrypted_data/ # where the encrypted data will appear
βββ decrypted_data/ # where the decrypted data will appear
βΒ Β
βββ logs/ # where the logs (stdout, stderr, and system) will appear
βΒ Β
βββ src/ # where the bash code is and where the commands must be call
βΒ Β
βββ imgs/ # images related with the pipeline
To configure TRACESPipe add your FASTQ files gziped at the folder
input_data/
Then, add a file exclusively with name meta_info.txt at the folder
meta_data/
This file needs to specify the organ type (with a single word name) and the filenames for the paired end reads. An example of the content of meta_info.txt is the following:
skin:V1_S44_R1_001.fastq.gz:V1_S44_R2_001.fastq.gz
brain:V2_S29_R1_001.fastq.gz:V2_S29_R2_001.fastq.gz
colon:V3_S45_R1_001.fastq.gz:V3_S45_R2_001.fastq.gz
Then, at the src/ folder run:
./TRACESPipe.sh --get-all-aux
To run TRACES Pipeline, use the following command:
./TRACESPipe.sh <parameters>
There are many parameters and configurations that can be used.
See the next section for more information about the usage.
./TRACESPipe.sh -h
βββββββββ βββββββ ββββββ βββββββ ββββββββ ββββββββ
βββββββββ ββββββββ ββββββββ ββββββββ ββββββββ ββββββββ
βββ ββββββββ ββββββββ βββ ββββββ ββββββββ
βββ ββββββββ ββββββββ βββ ββββββ ββββββββ
βββ βββ βββ βββ βββ ββββββββ ββββββββ ββββββββ
βββ βββ βββ βββ βββ βββββββ ββββββββ ββββββββ
P I P E L I N E
| A hybrid pipeline for reconstruction & analysis |
| of viral and host genomes at multi-organ level. |
Usage: ./TRACESPipe.sh [options]
-h, --help Show this help message and exit,
-v, --version Show the version and some information,
-f, --force Force running and overwrite of files,
-flog, --flush-logs Flush logs (delete logs),
-fout, --flush-output Flush output data (delete all output_data),
-i, --install Installation of all the tools,
-up, --update Update all the tools in TRACESPipe,
-spv, --show-prog-ver Show included programs versions,
-st, --sample Creates human ref. VDB and sample organ,
-gmt, --get-max-threads Get the number of maximum machine threads,
-t <THREADS>, --threads <THREADS>
Number of threads to use,
-dec, --decrypt Decrypt (all files in ../encrypted_data),
-enc, --encrypt Encrypt (all files in ../to_encrypt_data),
-vdb, --build-viral Build viral database (all) [Recommended],
-vdbr, --build-viral-r Build viral database (references only),
-udb, --build-unviral Build non viral database (control),
-afs <FASTA>, --add-fasta <FASTA>
Add a FASTA sequence to the VDB.fa,
-aes <ID>, --add-extra-seq <ID>
Add extra sequence to the VDB.fa,
-gx, --get-extra-vir Downloads/appends (VDB) extra viral seq,
-gad, --gen-adapters Generate FASTA file with adapters,
-gp, --get-phix Extracts PhiX genomes (Needs viral DB),
-gm, --get-mito Downloads human Mitochondrial genome,
-dwms, --download-mito-species
Downloads the complete NCBI mitogenomes
database containing the existing species,
-dwmp, --download-mito-population
Downloads two complete mitogenome databases
with healthy and pathogenic sequences,
-aums, --auth-mito-species
Autheticate the mitogenome species,
-aump, --auth-mito-population
Authenticate closest population,
-cmt <ID>, --change-mito <ID>
Set any Mitochondrial genome by ID,
-gy, --get-y-chromo Downloads human Y-chromosome,
-gax, --get-all-aux Runs -gad -gp -gm -gy,
-cbn, --create-blast-db It creates a nucleotide blast database,
-ubn, --update-blast-db It updates a nucleotide blast database,
-sfs <FASTA>, --search-blast-db <FASTA>
It blasts the nucleotide (nt) blast DB,
-sfrs <FASTA>, --search-blast-remote-db <FASTA>
It blasts remotly thenucleotide (nt) blast
database (it requires internet connection),
-rdup, --remove-dup Remove duplications (e.g. PCR dup),
-vhs, --very-sensitive Aligns with very high sensitivity (slower),
-gbb, --best-of-bests Identifies the best of bests references
between multiple organs [similar reference],
-iss <SIZE>, --inter-sim-size <SIZE>
Inter-genome similarity top size (control),
-rpro, --run-profiles Run complexity and relative profiles (control),
-rpgi <ID>, --run-gid-complexity-profile <ID>
Run complexity profiles by GID,
-cpwi <VALUE>, --complexity-profile-window <VALUE>
Complexity profile window size,
-cple <VALUE>, --complexity-profile-level <VALUE>
Complexity profile compression level [1;10],
-rm, --run-meta Run viral metagenomic identification,
-ro, --run-meta-nv Run NON-viral metagenomic identification,
-mis <VALUE>, --min-similarity <VALUE>
Minimum similarity value to consider the
sequence for alignment-consensus (filter),
-top <VALUE>, --view-top <VALUE>
Display the top <VALUE> with the highest
similarity (by descending order),
-rava, --run-all-v-alig Run all viral align/sort/consensus seqs
from a specific list,
-rsd <ID>, --run-de-novo-specific <ID/PATTERN>
Run specific alignments of the de-novo
to the reference genome,
-rsr <ID>, --run-specific <ID/PATTERN>
Run specific reference align/consensus,
-rsx <ID>, --run-extreme <ID/PATTERN>
Run specific reference align/consensus
using extreme sensitivity,
-rmt, --run-mito Run Mito align and consensus seq,
-rmtd, --run-mito-dam Run Mito damage only,
-rgid <ID>, --run-gid-damage <ID>
Run damage pattern analysis by GID,
-rya, --run-cy-align Run CY align and consensus seq,
-ryq, --run-cy-quant Estimate the quantity of CY DNA,
-rda, --run-de-novo Run de-novo assembly,
-rhyb, --run-hybrid Run hybrid assembly (align/de-novo),
-rmhc, --run-multiorgan-consensus
Run alignments/consensus between all the
reconstructed organ sequences,
-vis, --visual-align Run Visualization tool for alignments,
-covl, --coverage-latex Run coverage table in Latex format,
-covc, --coverage-csv Run coverage table in CSV format,
-covp <NAME>, --coverage-profile <BED_NAME_FILE>
Run coverage profile for specific BED file,
-cmax <MAX>, --max-coverage <MAX_COVERAGE>
Maximum depth coverage (depth normalization),
-clog <VALUE>, --coverage-log-scale <VALUE>
Coverage profile logarithmic scale VALUE=Base,
-cwis <VALUE>, --coverage-window-size <VALUE>
Coverage window size for low-pass filter,
-cdro <VALUE>, --coverage-drop <VALUE>
Coverage drop size (sampling),
-diff, --run-diff Run diff -> reference and hybrid (ident/SNPs),
-sdiff <V_NAME> <ID/PATTERN>, --run-specific-diff <V_NAME> <ID/PATTERN>
Run specific diff of reconstructed to a virus
pattern of ID. Example: -sdiff B19 AY386330.1,
-brec, --blast-reconstructed
Run local blast over reconstructed genomes,
-ra, --run-analysis Run data analysis (core),
-all, --run-all Run all the options (excluding the specific).
Ex: ./TRACESPipe.sh --flush-output --flush-logs --run-mito --run-meta
--remove-dup --run-de-novo --run-hybrid --min-similarity 1 --run-diff
--very-sensitive --best-of-bests --run-multiorgan-consensus
Add the file meta_info.txt at ../meta_data/ folder. Example:
meta_info.txt -> 'organ:reads_forward.fa.gz:reads_reverse.fa.gz'
The reads must be GZIPed in the ../input_data/ folder.
The output results are at ../output_data/ folder.
Contact: [email protected]
The common use of TRACESPipe as command is:
./TRACESPipe.sh \
--flush-logs \
--run-meta \
--inter-sim-size 2 \
--run-all-v-alig \
--run-mito \
--remove-dup \
--run-de-novo \
--run-hybrid \
--min-similarity 1.5 \
--view-top 5 \
--best-of-bests \
--very-sensitive \
--run-multiorgan-consensus \
--run-diff
From the run all the output is provided at folder output_data and it can be human inspected using IGV.
Nevertheless, for specific runs, below some examples are described.
5.1 Building viral consensus sequences with fixed reference sequence in all organs (if exists in the FASTQ samples):
./TRACESPipe.sh --run-meta --run-all-v-alig --remove-dup --min-similarity 3 --best-of-bests
The output consensus sequence is included at
output_data/TRACES_viral_consensus
while the alignments at
output_data/TRACES_viral_alignments
and the BED files at
output_data/TRACES_viral_bed
./TRACESPipe.sh --run-mito --remove-dup
The output consensus sequence is included at
output_data/TRACES_mtdna_consensus
while the alignments at
output_data/TRACES_mtdna_alignments
and the BED files at
output_data/TRACES_mtdna_bed
TRACESPipe supports secure encryption of genomic data. This allows outsourcing of the sequencing service while maintaining secure transmission and storage of the files.
Place the files from sequencing (e.g. FASTQ gziped files) in the folder to_encrypt_data and, then, run:
./TRACESPipe.sh --encrypt
Insert a strong password.
The encrypted files are in the encrypted_data folder.
Place the encrypted files in the folder encrypted_data and, then, run:
./TRACESPipe.sh --decrypt
Insert the password that has been used in encryption.
The decrypted files are in the decrypted_data folder.
./TRACESPipe.sh --run-meta --run-all-v-alig
The output consensus sequence is included at
output_data/TRACES_viral_consensus
while the alignments at
output_data/TRACES_viral_alignments
and the BED files at
output_data/TRACES_viral_bed
./TRACESPipe.sh --run-cy-quant
The output quantify is included at
output_data/TRACES_results/REP_CY_<organ_name>.txt
./TRACESPipe.sh --run-meta
The output is included at
../output_data/TRACES_results/REPORT_META_VIRAL_ALL.txt
./TRACESPipe.sh --run-meta-nv
The output is included at
../output_data/TRACES_results/REPORT_META_NON_VIRAL_<organ_name>.txt
./TRACESPipe.sh --run-de-novo
The outputs are included at
../output_data/TRACES_denovo_alignments
../output_data/TRACES_denovo_consensus
../output_data/TRACES_denovo_bed
5.9 Run specific viral alignment (AF037218.1) for all organs using extreme sensitivity without duplications:
./TRACESPipe.sh --remove-dup --run-extreme AF037218.1
The output is included at
../output_data/TRACES_specific_alignments
and the depth and breadth coverage values at
cat ../output_data/TRACES_specific_statistics
./TRACESPipe.sh --run-mito-dam
The output is included at
../output_data/TRACES_mtdna_damage_<organ_name>
./TRACESPipe.sh --search-blast-remote-db AF037218.1
The output is included at
../output_data/TRACES_blastn
This approach assumes that the reconstruction has already been processed:
./TRACES_normalized_depth.sh ../output_data/TRACES_viral_bed/B19-coverage-blood.bed 200
The output is provided to the stdout.
TRACES Pipeline uses a combination of the following tools:
If you use this pipeline, please cite:
Pratas, D., Toppinen, M., PyΓΆriΓ€, L., Hedman, K., Sajantila, A. and Perdomo, M.F., 2020.
A hybrid pipeline for reconstruction and analysis of viral genomes at multi-organ level.
GigaScience, 9(8), p.giaa086.
For any issue let us know at issues link.
GPL v3.
For more information see LICENSE file or visit
http://www.gnu.org/licenses/gpl-3.0.html