Filter outlier samples in TrainGCNV and GenerateBatchMetrics

[kjaisingh]

This PR addresses Issue #44.

Description

TrainGCNV

• Replaces Array[String]? sample_ids_training_subset with File? outlier_sample_ids, as the former was not being actively used before. The new parameter represents a text file containing outlier samples.
• Creates and calls task Utils.GetFilteredSubsampledIndices that uses outlier_sample_ids and/or outlier_sample_ids to subsample based on the following logic:
  • If only exclude_strings is provided, then only include samples that are not part of the outlier_sample_ids file.
  • If only subset_size is provided, then only include only n_samples_subsample random samples.
  • If both exclude_strings and subset_size are provided, then n_samples_subsample random samples that are not part of the outlier_sample_ids file.
  • If neither exclude_strings nor subset_size are provided, include all samples.
• Trains GCNV model exclusively using the subset of samples produced by Utils.GetFilteredSubsampledIndices.

GenerateBatchMetrics

• Creates new input parameter File? outlier_sample_ids to represent a text file containing outlier samples.
• Modifies existing PE, RD, BAF and SR tests such that they accept parameter--outlier-sample-ids, which provides a path to a file containing outlier samples. Within these tests, outlier samples are excluded from metric calculation for sites with at least 1 non-outlier, though are included if there are no non-outlier samples.
• Finally, we populate the existing column is_outlier_specific in the output VCF with True if it corresponds to a site that only has outlier samples tagged to it, and with False if not.

FilterBatchSites
No changes included - we already filter out sites from the training process using the is_outlier_specific = True tag that we now populate in GenerateBatchMetrics.

Testing

• TrainGCNV: This Terra job passes an outlier file that includes a subset of 6 samples out of a cohort containing 156 samples. We can see in the count_entity_ids parameter for CNVGermlineCohortWorkflow that only 150 samples are passed into the model training step - none of which are listed in the outlier sample file.
• GenerateBatchMetrics: This Terra job passes the same outlier file as above. We can see in the .metrics and .stats files produced across tests that sites tagged to one of those outliers have varying levels of support between these two jobs.
• Validated all WDLs with womtool.

Pre-Merge Changes Required

• Remove changes to .dockstore.yml.
• Output a exclude_sample_ids.txt file from the EvidenceQC notebook - samples part of this should not be dropped from the metrics table though.