Parse valid VCF 4.2 with ##INFO containing Source + Version

[romanzenka]

Description

As per bug #517 - despite VCFv4.2 spec allowing for "Source" and "Version" fields in ##INFO, htsjdk would throw an error. This fixes the issue.

Checklist

• Code compiles correctly
• New tests covering changes and new functionality
• All tests passing
• Extended the README / documentation, if necessary
• Is not backward compatible (breaks binary or source compatibility)

[codecov-io]

Codecov Report

Merging #1248 into master will increase coverage by 0.029%.
The diff coverage is 73.684%.

@@               Coverage Diff               @@
##              master     #1248       +/-   ##
===============================================
+ Coverage     69.324%   69.353%   +0.029%     
- Complexity      8287      8301       +14     
===============================================
  Files            555       555               
  Lines          33075     33116       +41     
  Branches        5561      5572       +11     
===============================================
+ Hits           22929     22967       +38     
- Misses          7884      7890        +6     
+ Partials        2262      2259        -3

Impacted Files	Coverage Δ	Complexity Δ
...n/java/htsjdk/variant/vcf/VCFContigHeaderLine.java	76.471% <100%> (ø)	10 <1> (ø)	⬇️
...rc/main/java/htsjdk/variant/vcf/VCFHeaderLine.java	74.468% <100%> (+1.135%)	22 <0> (+2)	⬆️
...n/java/htsjdk/variant/vcf/VCFFilterHeaderLine.java	100% <100%> (ø)	5 <1> (ø)	⬇️
...main/java/htsjdk/variant/vcf/AbstractVCFCodec.java	73.006% <100%> (ø)	90 <0> (ø)	⬇️
...n/java/htsjdk/variant/vcf/VCFSimpleHeaderLine.java	75.61% <33.333%> (-3.877%)	12 <1> (ø)
...ain/java/htsjdk/variant/vcf/VCFInfoHeaderLine.java	83.333% <50%> (-16.667%)	6 <1> (+1)
...java/htsjdk/variant/vcf/VCFCompoundHeaderLine.java	75.51% <73.333%> (-0.1%)	48 <2> (+7)
...va/htsjdk/variant/vcf/VCFHeaderLineTranslator.java	92.593% <78.571%> (+5.093%)	3 <2> (+1)	⬆️
...sjdk/samtools/util/Md5CalculatingOutputStream.java	84.615% <0%> (+10.256%)	11% <0%> (+3%)	⬆️

[romanzenka]

I will work on getting coverage back in shape

[lbergelson]

@cmnbroad Could you take a look at this? It's one of the tickets that was slated to be closed by your giant refactor branch that never went int.

[lbergelson]

@romanzenka Thanks for doing this! This warning might be too late, but the code coverage is really oversensitive and tends to fail at random due to some non-determinism in the test suite. So don't worry too much about pleasing it.

[romanzenka]

I originally wanted to reuse parts of the original PR, but I could not figure out how it fixes this particular issue (VCF adding optional fields). So I enhanced the interface (retaining backwards compatibility) so now you can specify optional tags.

[cmnbroad]

Yes, the original PR refactored the entire header line class hierarchy, so it would be hard to extract just a piece of it. I'll take a look at the changes here.

[pshapiro4broad]

Many minor comments

[pshapiro4broad]

Thanks for the updates the code looks 👍 to me, except for the issue with the case statement fall-through.

Will also need @lbergelson or @cmnbroad to check this off too, as I'm not a VCF format expert.

[cmnbroad]

@romanzenka Thanks for taking this on - we've been wanting to fix it for a while. It looks you've made changes to make "source" and "version" tolerated on read, but the values aren't retained, accessible to callers, or round-tripped on write. Would you be able to add getters so these values can be retrieved from the header lines, and update the encoder to propagate them on write, with corresponding tests ? Ideally we'd implement generalized attribute support, but thats a lot more work. For now it would be ok to handle these two attributes specifically since they're recommended in the spec, but I'd be reluctant to take a change that accepts and then silently drops them.

[cmnbroad]

Thanks for the last changes - one minor minor change requested.

@lbergelson, except for one last comment on the tests, Source and Version can be round-tripped now. I think this PR strikes as good a balance as we're going to get for that without a major overhaul such as in my other PR. The only remaining question is whether we really want to deprecate the old parseLine in VCFLineParser, and the old constructor in VCFSimpleLineHeader, or just leave them. I think they're harmless to leave, though the new replacements are preferable.

[romanzenka]

I think I addressed everything - the only remaining issue was deprecation of those two constructors (that are not used anymore within the library). I removed the @deprecated for now and replaced it with @see, it can always be changed back later maybe during a larger deprecation activity when you decide to clean the library up.

[cmnbroad]

One hopefully final request and we can get this merged.

[romanzenka]

I added a freebie that you may/may not like - I am tired of the MD5CalculatingOutputStream spuriously messing up my coverage, so I made it deterministic.

[lbergelson]

@romanzenka 👍 to fixing that. Don't worry too much about the coverage though... it's weirdly oversensitive. We need to change the settings to be more lenient.

[romanzenka]

@lbergelson I do not think it is even sensitivity - there is something random about how the coverage gets compared. The latest report makes no sense to me - I have not done anything with those 21 reported files. I suspect the code coverage tool itself might be confused?
What you would want to compare is coverage as it is right now versus coverage as it would be after you merge the PR. I do not think that is happening.

[lbergelson]

Yeah, it choose funny comparisons sometimes and I'm not certain exactly what happens. I think it compares the coverage in the last run of master vs the coverage in the pr build, which isn't necessarily in sync although it should be close.

[romanzenka]

@lbergelson I am simply merging master for now to see what happens... my bad, that's probably best practice before accepting a PR anyway

[romanzenka]

@cmnbroad Do you think this could be merged before Friday 2019-01-25? We are releasing an app and it would be nice to build it on top of an official commit instead of our own branch.
I am not hoping for an official release, although that would of course be awesome!

[cmnbroad]

@lbergelson this looks good to me now (except for the old copyright date), in case you want a final review.

[cmnbroad]

@romanzenka Thanks. Add yes, we should be able to get this merged in the next 24 hours.

[lbergelson]

@cmnbroad @romanzenka 👍

Thanks for working through this one. It turned out to be more work than I expected. I don't know if we'll be able to get another release out by friday. I'll see what I can do...

[romanzenka]

@cmnbroad @romanzenka 👍

Thanks for working through this one. It turned out to be more work than I expected. I don't know if we'll be able to get another release out by friday. I'll see what I can do...

Don't worry about it. Release when it is appropriate to release. We'll build off a particular commit.

Oh and thanks for very thorough and high-quality review!

[cmnbroad]

Thanks again for working through all the issues @romanzenka.