Skip to content

More changes for logging and bgen write options #65

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kianfar77 merged 9 commits into from
Nov 8, 2019
Merged

More changes for logging and bgen write options #65

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
921a3c5
minor changes needed for logging
kianfar77 Nov 1, 2019
be0ad9e
Merge branch 'master' into log-fix
kianfar77 Nov 5, 2019
69d4ab2
logging plink blob and minor changes
kianfar77 Nov 5, 2019
877aa8e
bug fix
kianfar77 Nov 5, 2019
097e5a8
moving pipe blob key
kianfar77 Nov 6, 2019
f527101
merge master
kianfar77 Nov 6, 2019
9541245
drop HlsBlobKeys
kianfar77 Nov 6, 2019
c00bb11
Merge branch 'master' into log-fix
kianfar77 Nov 6, 2019
27b227b
changes for logging
kianfar77 Nov 7, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 1 addition & 11 deletions core/src/main/scala/io/projectglow/bgen/BigBgenDatasource.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,17 +39,7 @@ class ComDatabricksBigBgenDatasource extends BigBgenDatasource with ComDatabrick

object BigBgenDatasource extends HlsUsageLogging {

val BITS_PER_PROB_KEY = "bitsPerProbability"
val BITS_PER_PROB_DEFAULT_VALUE = "16"

val MAX_PLOIDY_KEY = "maximumInferredPloidy"
val MAX_PLOIDY_VALUE = "10"

val DEFAULT_PLOIDY_KEY = "defaultInferredPloidy"
val DEFAULT_PLOIDY_VALUE = "2"

val DEFAULT_PHASING_KEY = "defaultInferredPhasing"
val DEFAULT_PHASING_VALUE = "false"
import io.projectglow.common.BgenOptions._

def serializeDataFrame(options: Map[String, String], data: DataFrame): RDD[Array[Byte]] = {

Expand Down
14 changes: 14 additions & 0 deletions core/src/main/scala/io/projectglow/common/datasourceOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,25 @@ object VCFOptions {
}

object BgenOptions {
// Reader options
val IGNORE_EXTENSION_KEY = "ignoreExtension"
val USE_INDEX_KEY = "useBgenIndex"
val SAMPLE_FILE_PATH_OPTION_KEY = "sampleFilePath"
val SAMPLE_ID_COLUMN_OPTION_KEY = "sampleIdColumn"
val SAMPLE_ID_COLUMN_OPTION_DEFAULT_VALUE = "ID_2"

// bigbgen write options
val BITS_PER_PROB_KEY = "bitsPerProbability"
val BITS_PER_PROB_DEFAULT_VALUE = "16"

val MAX_PLOIDY_KEY = "maximumInferredPloidy"
val MAX_PLOIDY_VALUE = "10"

val DEFAULT_PLOIDY_KEY = "defaultInferredPloidy"
val DEFAULT_PLOIDY_VALUE = "2"

val DEFAULT_PHASING_KEY = "defaultInferredPhasing"
val DEFAULT_PHASING_VALUE = "false"
}

object PlinkOptions {
Expand Down
4 changes: 0 additions & 4 deletions core/src/main/scala/io/projectglow/common/logging/HlsUsageLogging.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,3 @@ object HlsTagValues {

val EVENT_PLINK_READ = "plinkRead"
}

object HlsBlobKeys {
val PIPE_CMD_TOOL = "pipeCmdTool"
}
3 changes: 2 additions & 1 deletion core/src/main/scala/io/projectglow/transformers/pipe/PipeTransformer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ class PipeTransformer extends DataFrameTransformer with HlsUsageLogging {
// record the pipe event along with tools of interest which maybe called using it.
// TODO: More tools to be added
val toolInPipe = Map(
HlsBlobKeys.PIPE_CMD_TOOL ->
LOGGING_BLOB_KEY ->
pipeToolSet
.foldLeft(Array[String]())(
(a, b: String) =>
Expand Down Expand Up @@ -141,6 +141,7 @@ object PipeTransformer {
private val ENV_PREFIX = "env_"
private val INPUT_FORMATTER_PREFIX = "in_"
private val OUTPUT_FORMATTER_PREFIX = "out_"
val LOGGING_BLOB_KEY = "pipeCmdTool"
Copy link
Collaborator

@karenfeng karenfeng Nov 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we moving this here?

Copy link
Collaborator Author

@kianfar77 kianfar77 Nov 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all other features, like vcf/bgen/plink read/write and variant normalizer transformer, the options which are defined inside the respective packages are used as keys for the maps in the logging json blobs. I do not want this one to be an exception and be defined outside of pipe transformer. This makes the logging code cleaner on the universe side as well.

Copy link
Collaborator

@karenfeng karenfeng Nov 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Can we move the transformer options to a higher-level object as we did for the datasource options for transparency?

Copy link
Collaborator Author

@kianfar77 kianfar77 Nov 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure I can. But if ok with you let's do it in another PR because I want to get the universe PR review moving froward with data and observability teams and that depends on this PR. Moving options to higher level can be done independently.

Copy link
Collaborator

@karenfeng karenfeng Nov 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, that works.


private def lookupInputFormatterFactory(name: String): Option[InputFormatterFactory] =
synchronized {
Expand Down