Removed GC bias and fasta output as unneeded features, for now. Excised some dead code, which will live on in past versions, if we ever go back to it. Need to reevaluate the models section, but things are working for now.

Author: joshfactorial

README.md

@@ -1,13 +1,13 @@
 # The NEAT Project v4.0
-Welcome to the NEAT project, the NExt-generation sequencing Analysis Toolkit, version 4.0. This is our first (beta) release of the newest version of NEAT. There is still lots of work to be done. See the [ChangeLog](ChangeLog.md) for notes.
+Welcome to the NEAT project, the NExt-generation sequencing Analysis Toolkit, version 4.2. This release of NEAT includes several fixes and a little bit of restructuring. There is still lots of work to be done. See the [ChangeLog](ChangeLog.md) for notes. We have discarded the fasta file writing for now and removed that code. We may add that in as a feature in the future, if users call for it. We also removed GC bias for now. It severely complicated implementation, and had very few noticeable effects. After discussing with some people at the Illinois Institute for Genomic Biology, it sounded like GC bias may be a bit of a non-factor with improved chemistries. These will be reintroduced if needed/called for. 
 
 We are also working on redeveloping NEAT in Rust, a memory and thread safe language that will lend itself well to the way NEAT works, check that out here: https://github.com/ncsa/rusty-neat
 
 Stay tuned over the coming weeks for exciting updates to NEAT, and learn how to [contribute](CONTRIBUTING.md) yourself. If you'd like to use some of our code, no problem! Just review the [license](LICENSE.md), first.
 
 NEAT's read-simulator is a fine-grained read simulator. It simulates real-looking data using models learned from specific datasets. There are several supporting utilities for generating models used for simulation and for comparing the outputs of alignment and variant calling to the golden BAM and golden VCF produced by NEAT.
 
-This is release v4.0 of the software. While it has been tested, it does represent a shift in the software with the introduction of a configuration file. For a stable release using the old command line interface, please see: [NEAT 3.0](https://github.com/ncsa/NEAT/releases/tag/3.3) (or check out older tagged releases)
+This is release v4.2 of the software. While it has been tested, it does represent a shift in the software with the introduction of a configuration file. For a stable release using the old command line interface, please see: [NEAT 3.0](https://github.com/ncsa/NEAT/releases/tag/3.3) (or check out older tagged releases)
 
 To cite this work, please use:
 
@@ -31,7 +31,6 @@ Table of Contents
       * [Large single end reads](#large-single-end-reads)
       * [Parallelizing simulation](#parallelizing-simulation)
   * [Utilities](#utilities)
-    * [compute_gc_bias](#computegcbias)
     * [model_fragment_lengths](#modelfraglen)
     * [gen_mut_model](#genmutmodel)
     * [model_sequencing_error](#modelseqerror)
@@ -40,8 +39,9 @@ Table of Contents
 
 ## Requirements (the most up-to-date requirements are found in the environment.yml file)
 
+* Some version of Anaconda to set up the environment
 * Python == 3.10.*
-* poetry
+* poetry == 1.3.*
 * biopython == 1.79
 * pkginfo
 * matplotlib
@@ -71,13 +71,20 @@ the NEAT repo, after creating the conda environment:
 > poetry install
 ```
 
+Notes: If any packages are struggling to resolve, check the channels and try to manually pip install the package to see if that helps (but note that NEAT is not tested on the pip versions.)
+
 Test your install by running:
 ```
 > neat --help
 ```
 
+You can also try running it using the python command directly:
+```
+> python -m neat --help
+```
+
 ## Usage
-NEAT's core functionality is invoked using the read-simulator command. Here's the simplest invocation of read-simulator using default parameters. This command produces a single ended fastq file with reads of length 101, ploidy 2, coverage 10X, using the default sequencing substitution, GC% bias, and mutation rate models.
+NEAT's core functionality is invoked using the read-simulator command. Here's the simplest invocation of read-simulator using default parameters. This command produces a single ended fastq file with reads of length 151, ploidy 2, coverage 10X, using the default sequencing substitution, and mutation rate models.
 
 Contents of neat_config.yml
 ```
@@ -110,7 +117,6 @@ The default is given:
 
 produce_bam: False
 produce_vcf: False
-produce_fasta: False
 produce_fastq: True
 
 error_model: full path to an error model generated by NEAT. Leave empty to use default model
@@ -119,11 +125,7 @@ mutation_model: full path to a mutation model generated by NEAT. Leave empty to
     model (default model based on human data sequenced by Illumina)
 fragment_model: full path to fragment length model generate by NEAT. Leave empty to use default model
     (default model based on human data sequenced by Illumina)
-gc_model: Full path to model for correlating GC concentration and coverage, produced by NEAT.
-    (default model is based on human data, sequenced by Illumina)
 
-partition_mode: by chromosome ("chrom"), or subdivide the chromosomes ("subdivision").
-    Note: this feature is not yet fully implemented
 threads: The number of threads for NEAT to use.
     Note: this feature is not yet fully implemented
 avg_seq_error: average sequencing error rate for the sequencing machine. Use to increase or
@@ -134,20 +136,14 @@ include_vcf: full path to list of variants in vcf format to include in the simul
     appear in the input VCF into the final VCF, and the corresponding fastq and bam files, if requested.
 target_bed: full path to list of regions in bed format to target. 
     All areas outside these regions will have coverage of 0.
-off_target_scalar: manually set the off-target-scalar when using a target bed (if you want to have some percentage of 
-    reads from outside the targeted regions. Default is 0. (i.e., setting this to 0.02 would mean off-target areas will 
-    have a coverage of ~2% of the total coverage). This is an experimental feature.
 discard_bed: full path to a list of regions to discard, in BED format.
 mutation_rate: Desired rate of mutation for the dataset. Float between 0.0 and 0.3
     (default is determined by the mutation model)
 mutation_bed: full path to a list of regions with a column describing the mutation rate of that region,
     as a float with values between 0 and 0.3. The mutation rate must be in the third column as, e.g., mut_rate=0.00.
-no_coverage_bias: Set to true to produce a dataset free of coverage bias
 rng_seed: Manually enter a seed for the random number generator. Used for repeating runs. Must be an integer.
 min_mutations: Set the minimum number of mutations that NEAT should add, per contig. Default is 0. We recommend setting 
-    this to at least one for small chromosomes, so NEAT will produce at least one mutation per contig.
-fasta_per_ploid: Produce one fasta per ploid. Default behavior is to produce
-    a single fasta showing all variants.                                                                                                                                                                        |
+    this to at least one for small chromosomes, so NEAT will produce at least one mutation per contig. |
 
 The command line options for NEAT are as follows:
 
@@ -156,10 +152,9 @@ Universal options can be applied to any subfunction. The commands should come be
 |---------------------|--------------------------------------|
 | -h, --help          | Displays usage information           |
 | --no-log            | Turn off log file creation           |
-| --log-dir LOG_DIR   | Sets the log directory to custom path (default is current working directory |
-| --log-name LOG_NAME | Custom name for log file (default is timestamped) |
+| --log-name LOG_NAME | Custom name for log file, can be a full path (default is current working directory with a name starting with a timestamp)|
 | --log-level VALUE   | VALUE must be one of [DEBUG, INFO, WARN, WARNING, ERROR] - sets level of log to display |
-| --log-detal VALUE   | VALUE must be one of [LOW, MEDIUM, HIGH] - how much info to write for each log record |
+| --log-detail VALUE   | VALUE must be one of [LOW, MEDIUM, HIGH] - how much info to write for each log record |
 | --silent-mode       | Writes logs, but suppresses stdout messages |
 
 read-simulator command line options
@@ -184,9 +179,8 @@ Features:
 - Can simulate targeted sequencing via BED input specifying regions to sample from
 - Can accurately simulate large, single-end reads with high indel error rates (PacBio-like) given a model
 - Specify simple fragment length model with mean and standard deviation or an empirically learned fragment distribution using utilities/computeFraglen.py
-- Simulates quality scores using either the default model or empirically learned quality scores using utilities/fastq_to_qscoreModel.py
+- Simulates quality scores using either the default model or empirically learned quality scores using `neat gen_mut_model`
 - Introduces sequencing substitution errors using either the default model or empirically learned from utilities/
-- Accounts for GC% coverage bias using model learned from utilities/computeGC.py
 - Output a VCF file with the 'golden' set of true positive variants. These can be compared to bioinformatics workflow output (includes coverage and allele balance information)
 - Output a BAM file with the 'golden' set of aligned reads. These indicate where each read originated and how it should be aligned with the reference
 - Create paired tumour/normal datasets using characteristics learned from real tumour data
@@ -288,27 +282,6 @@ neat read-simulator                 \
 # Utilities	
 Several scripts are distributed with gen_reads that are used to generate the models used for simulation.
 
-## neat compute_gc_bias
-
-Computes GC% coverage bias distribution from sample (bedrolls genomecov) data.
-Takes .genomecov files produced by BEDtools genomeCov (with -d option).
-(Not yet implemented in NEAT 4.0)
-
-```
-bedtools genomecov
-        -d                          \
-        -ibam normal.bam            \
-        -g reference.fa
-```
-
-```
-neat compute_gc_bias                \
-        -r reference.fa             \
-        -i genomecovfile            \
-        -w [sliding window length]  \
-        -o /path/to/prefix
-```
-
 ## neat model-fraglen
 
 Computes empirical fragment length distribution from sample data.

config_template/config_test5.yml

@@ -8,26 +8,20 @@ fragment_st_dev: .
 
 produce_bam: .
 produce_vcf: .
-produce_fasta: .
 produce_fastq: .
 
 error_model: .
 mutation_model: .
 fragment_model: .
-gc_model: .
 
-partition_mode: .
-threads: .
 avg_seq_error: .
 rescale_qualities: .
 quality_offset: .
 include_vcf: .
 target_bed: .
-off_target_scalar: .
 discard_bed: .
 mutation_rate: .
 mutation_bed: .
-no_coverage_bias: .
 rng_seed: .
 min_mutations: 0
 overwrite_output: .