You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# This workflow will install Python dependencies, run tests and lint with a single version of Python
2
-
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
1
+
# This workflow configures the environment and executes NEAT read-simulator tests using relative paths for a series of configuration files individually
2
+
# For more information on using Python with GitHub Actions, refer to:
Copy file name to clipboardExpand all lines: ChangeLog.md
+10
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,16 @@
1
1
# NEAT has a new home
2
2
NEAT is now a part of the NCSA github and active development will continue here. Please direct issues, comments, and requests to the NCSA issue tracker. Submit pull requests here insead of the old repo.
3
3
4
+
# NEAT v4.2
5
+
- After several bug fixes that constituted release 4.1 and some minor releases, we are ready to release an overhauled vesion of NEAT 4.0.
6
+
- Removed GC bias - it had little to no effect and made implementation nearly impossible
7
+
- Removed fasta creation - we had tweaked this a bit but never got any feedback. It may come back if requested.
8
+
- Improvements/fixes/full implementations of:
9
+
- heterozygosity
10
+
- read creation (now with more reads!)
11
+
- bam alignment/creation
12
+
- bed tool incorporation
13
+
4
14
-Updated "master" branch to "main." - please update your repo accordingly
5
15
# NEAT v4.0
6
16
- Rewritten the models. Models generated on old versions of NEAT will have to be redone, due to the restructuring of the codebase. These new models should be smaller and more efficient. We have replicated the previous default models in the new style. There is no straightforward way to convert between these, unfortuantely.
Copy file name to clipboardExpand all lines: README.md
+28-55
Original file line number
Diff line number
Diff line change
@@ -1,13 +1,13 @@
1
-
# The NEAT Project v4.0
2
-
Welcome to the NEAT project, the NExt-generation sequencing Analysis Toolkit, version 4.0. This is our first (beta) release of the newest version of NEAT. There is still lots of work to be done. See the [ChangeLog](ChangeLog.md) for notes.
1
+
# The NEAT Project v4.2
2
+
Welcome to the NEAT project, the NExt-generation sequencing Analysis Toolkit, version 4.2. This release of NEAT includes several fixes and a little bit of restructuring. There is still lots of work to be done. See the [ChangeLog](ChangeLog.md) for notes. We have discarded the fasta file writing for now and removed that code. We may add that in as a feature in the future, if users call for it. We also removed GC bias for now. It severely complicated implementation, and had very few noticeable effects. After discussing with some people at the Illinois Institute for Genomic Biology, it sounded like GC bias may be a bit of a non-factor with improved chemistries. These will be reintroduced if needed/called for.
3
3
4
4
We are also working on redeveloping NEAT in Rust, a memory and thread safe language that will lend itself well to the way NEAT works, check that out here: https://github.com/ncsa/rusty-neat
5
5
6
6
Stay tuned over the coming weeks for exciting updates to NEAT, and learn how to [contribute](CONTRIBUTING.md) yourself. If you'd like to use some of our code, no problem! Just review the [license](LICENSE.md), first.
7
7
8
8
NEAT's read-simulator is a fine-grained read simulator. It simulates real-looking data using models learned from specific datasets. There are several supporting utilities for generating models used for simulation and for comparing the outputs of alignment and variant calling to the golden BAM and golden VCF produced by NEAT.
9
9
10
-
This is release v4.0 of the software. While it has been tested, it does represent a shift in the software with the introduction of a configuration file. For a stable release using the old command line interface, please see: [NEAT 3.0](https://github.com/ncsa/NEAT/releases/tag/3.3) (or check out older tagged releases)
10
+
This is release v4.2 of the software. While it has been tested, it does represent a shift in the software with the introduction of a configuration file. For a stable release using the old command line interface, please see: [NEAT 3.0](https://github.com/ncsa/NEAT/releases/tag/3.3) (or check out older tagged releases)
11
11
12
12
To cite this work, please use:
13
13
@@ -31,7 +31,6 @@ Table of Contents
31
31
*[Large single end reads](#large-single-end-reads)
## Requirements (the most up-to-date requirements are found in the environment.yml file)
42
41
42
+
* Some version of Anaconda to set up the environment
43
43
* Python == 3.10.*
44
-
* poetry
44
+
* poetry == 1.3.*
45
45
* biopython == 1.79
46
46
* pkginfo
47
47
* matplotlib
@@ -71,13 +71,20 @@ the NEAT repo, after creating the conda environment:
71
71
> poetry install
72
72
```
73
73
74
+
Notes: If any packages are struggling to resolve, check the channels and try to manually pip install the package to see if that helps (but note that NEAT is not tested on the pip versions.)
75
+
74
76
Test your install by running:
75
77
```
76
78
> neat --help
77
79
```
78
80
81
+
You can also try running it using the python command directly:
82
+
```
83
+
> python -m neat --help
84
+
```
85
+
79
86
## Usage
80
-
NEAT's core functionality is invoked using the read-simulator command. Here's the simplest invocation of read-simulator using default parameters. This command produces a single ended fastq file with reads of length 101, ploidy 2, coverage 10X, using the default sequencing substitution, GC% bias, and mutation rate models.
87
+
NEAT's core functionality is invoked using the read-simulator command. Here's the simplest invocation of read-simulator using default parameters. This command produces a single ended fastq file with reads of length 151, ploidy 2, coverage 10X, using the default sequencing substitution, and mutation rate models.
81
88
82
89
Contents of neat_config.yml
83
90
```
@@ -98,7 +105,7 @@ template config file to copy and use for your runs.
98
105
reference: full path to a fasta file to generate reads from
99
106
read_len: The length of the reads for the fastq (if using). Integer value, default 101.
100
107
coverage: desired coverage value. Float or int, default = 10
101
-
ploidy: Desired value for ploidy (# of copies of each chromosome). Default is 2
108
+
ploidy: Desired value for ploidy (# of copies of each chromosome in the organism). Default is 2
102
109
paired_ended: If paired-ended reads are desired, set this to True. Setting this to true requires
103
110
either entering values for fragment_mean and fragment_st_dev or entering the path to a
104
111
valid fragment_model.
@@ -110,7 +117,6 @@ The default is given:
110
117
111
118
produce_bam: False
112
119
produce_vcf: False
113
-
produce_fasta: False
114
120
produce_fastq: True
115
121
116
122
error_model: full path to an error model generated by NEAT. Leave empty to use default model
@@ -119,34 +125,25 @@ mutation_model: full path to a mutation model generated by NEAT. Leave empty to
119
125
model (default model based on human data sequenced by Illumina)
120
126
fragment_model: full path to fragment length model generate by NEAT. Leave empty to use default model
121
127
(default model based on human data sequenced by Illumina)
122
-
gc_model: Full path to model for correlating GC concentration and coverage, produced by NEAT.
123
-
(default model is based on human data, sequenced by Illumina)
124
128
125
-
126
-
partition_mode: by chromosome ("chrom"), or subdivide the chromosomes ("subdivision").
127
-
Note: this feature is not yet fully implemented
128
129
threads: The number of threads for NEAT to use.
129
130
Note: this feature is not yet fully implemented
130
131
avg_seq_error: average sequencing error rate for the sequencing machine. Use to increase or
131
132
decrease the rate of errors in the reads. Float betwoon 0 and 0.3. Default is set by the error model.
132
-
rescale_qualities: rescale the quality scores to reflect the avg_seq_error rate above. Set True to activate.
133
-
include_vcf: full path to list of variants in vcf format to include in the simulation.
134
-
target_bed: full path to list of regions in bed format to target. All areas outside these regions will have
135
-
very low coverage.
136
-
off_target_scalar: manually set the off-target-scalar when using a target bed. Default is 0.02
137
-
(i.e., off target areas will have only 2% of the average coverage)
138
-
discard_offtarget: throws out reads from off-target regions. Regions of overlap may still have reads.
139
-
Set True to activate
133
+
rescale_qualities: rescale the quality scores to reflect the avg_seq_error rate above. Set True to activate if you
134
+
notice issues with the sequencing error rates in your datatset.
135
+
include_vcf: full path to list of variants in vcf format to include in the simulation. These will be inserted as they
136
+
appear in the input VCF into the final VCF, and the corresponding fastq and bam files, if requested.
137
+
target_bed: full path to list of regions in bed format to target.
138
+
All areas outside these regions will have coverage of 0.
140
139
discard_bed: full path to a list of regions to discard, in BED format.
141
-
mutation_rate: Desired rate of mutation for the dataset. Float between 0 and 0.3
140
+
mutation_rate: Desired rate of mutation for the dataset. Float between 0.0 and 0.3
142
141
(default is determined by the mutation model)
143
142
mutation_bed: full path to a list of regions with a column describing the mutation rate of that region,
144
-
as a float with values between 0 and 0.3. The mutation rate must be in the third column.
145
-
no_coverage_bias: Set to true to produce a dataset free of coverage bias
146
-
rng_seed: Manually enter a seed for the random number generator. Used for repeating runs.
147
-
min_mutations: Set the minimum number of mutations that NEAT should add, per contig. Default is 1.
148
-
fasta_per_ploid: Produce one fasta per ploid. Default behavior is to produce
149
-
a single fasta showing all variants. |
143
+
as a float with values between 0 and 0.3. The mutation rate must be in the third column as, e.g., mut_rate=0.00.
144
+
rng_seed: Manually enter a seed for the random number generator. Used for repeating runs. Must be an integer.
145
+
min_mutations: Set the minimum number of mutations that NEAT should add, per contig. Default is 0. We recommend setting
146
+
this to at least one for small chromosomes, so NEAT will produce at least one mutation per contig. |
150
147
151
148
The command line options for NEAT are as follows:
152
149
@@ -155,10 +152,9 @@ Universal options can be applied to any subfunction. The commands should come be
| --log-dir LOG_DIR | Sets the log directory to custom path (default is current working directory |
159
-
| --log-name LOG_NAME | Custom name for log file (default is timestamped) |
155
+
| --log-name LOG_NAME | Custom name for log file, can be a full path (default is current working directory with a name starting with a timestamp)|
160
156
| --log-level VALUE | VALUE must be one of [DEBUG, INFO, WARN, WARNING, ERROR] - sets level of log to display |
161
-
| --log-detal VALUE | VALUE must be one of [LOW, MEDIUM, HIGH] - how much info to write for each log record |
157
+
| --log-detail VALUE | VALUE must be one of [LOW, MEDIUM, HIGH] - how much info to write for each log record |
162
158
| --silent-mode | Writes logs, but suppresses stdout messages |
163
159
164
160
read-simulator command line options
@@ -183,9 +179,8 @@ Features:
183
179
- Can simulate targeted sequencing via BED input specifying regions to sample from
184
180
- Can accurately simulate large, single-end reads with high indel error rates (PacBio-like) given a model
185
181
- Specify simple fragment length model with mean and standard deviation or an empirically learned fragment distribution using utilities/computeFraglen.py
186
-
- Simulates quality scores using either the default model or empirically learned quality scores using utilities/fastq_to_qscoreModel.py
182
+
- Simulates quality scores using either the default model or empirically learned quality scores using `neat gen_mut_model`
187
183
- Introduces sequencing substitution errors using either the default model or empirically learned from utilities/
188
-
- Accounts for GC% coverage bias using model learned from utilities/computeGC.py
189
184
- Output a VCF file with the 'golden' set of true positive variants. These can be compared to bioinformatics workflow output (includes coverage and allele balance information)
190
185
- Output a BAM file with the 'golden' set of aligned reads. These indicate where each read originated and how it should be aligned with the reference
191
186
- Create paired tumour/normal datasets using characteristics learned from real tumour data
@@ -217,7 +212,6 @@ neat read-simulator \
217
212
Simulate a targeted region of a genome, e.g. exome, with a targeted bed:
218
213
219
214
```
220
-
<<<<<<< HEAD
221
215
[contents of neat_config.yml]
222
216
reference: hg19.fa
223
217
read_len: 126
@@ -288,27 +282,6 @@ neat read-simulator \
288
282
# Utilities
289
283
Several scripts are distributed with gen_reads that are used to generate the models used for simulation.
290
284
291
-
## neat compute_gc_bias
292
-
293
-
Computes GC% coverage bias distribution from sample (bedrolls genomecov) data.
294
-
Takes .genomecov files produced by BEDtools genomeCov (with -d option).
295
-
(Not yet implemented in NEAT 4.0)
296
-
297
-
```
298
-
bedtools genomecov
299
-
-d \
300
-
-ibam normal.bam \
301
-
-g reference.fa
302
-
```
303
-
304
-
```
305
-
neat compute_gc_bias \
306
-
-r reference.fa \
307
-
-i genomecovfile \
308
-
-w [sliding window length] \
309
-
-o /path/to/prefix
310
-
```
311
-
312
285
## neat model-fraglen
313
286
314
287
Computes empirical fragment length distribution from sample data.
0 commit comments