Tribble/Tabix index path support #810

magicDGS · 2017-03-01T17:05:40Z

Description

Motivation

While trying to index a Path (no convertible to File) using the Index interface, the client should:

Create the LittleEndianOutputStreamby its own. This may lead to wrong indexes because they aren't block-compressed.
Get the index file name by its own if it is based on a feature file. This may lead to incorrect index names.
In addition, this is part of a bigger change for include support of writing in whatever filesystem (and also reading).

Changes

Clean up (first commit): removing mentions to vcf in javadoc for Tribble and unused imports.
Path support for Index interface: methods with default values use Path.toFile()
Implementation of the new methods in AbstractIndex and TabixIndex

Checklist

Code compiles correctly
New tests covering changes and new functionality
All tests passing
Extended the README / documentation, if necessary
Is not backward compatible (breaks binary or source compatibility)

lbergelson · 2017-03-01T18:07:33Z

@magicDGS Something went wrong here, tests are failing all over.

magicDGS · 2017-03-01T18:11:23Z

I figured it out already, I'm trying to find the problem.

magicDGS · 2017-03-01T18:30:52Z

Very awful error from my side, sorry. The writeBasedOnFeatureFile was overwriting the input file instead of using the name for making the index.

Now I guess that it's solved. Could you have a quick review, @lbergelson?

codecov-io · 2017-03-01T18:57:33Z

Codecov Report

Merging #810 into master will decrease coverage by 0.021%.
The diff coverage is 62.5%.

@@               Coverage Diff               @@
##              master      #810       +/-   ##
===============================================
- Coverage     64.867%   64.846%   -0.021%     
- Complexity      7175      7187       +12     
===============================================
  Files            526       527        +1     
  Lines          31731     31769       +38     
  Branches        5424      5424               
===============================================
+ Hits           20583     20601       +18     
- Misses          8997      9019       +22     
+ Partials        2151      2149        -2

Impacted Files	Coverage Δ	Complexity Δ
...va/htsjdk/tribble/TribbleIndexedFeatureReader.java	`68.333% <ø> (ø)`	`22 <0> (ø)`	❌
...c/main/java/htsjdk/tribble/TabixFeatureReader.java	`68.657% <ø> (ø)`	`8 <0> (ø)`	❌
src/main/java/htsjdk/tribble/Tribble.java	`77.778% <100%> (+6.349%)`	`7 <2> (+2)`	✅
.../java/htsjdk/tribble/index/linear/LinearIndex.java	`78.363% <100%> (+0.518%)`	`18 <4> (+2)`	✅
...sjdk/tribble/index/interval/IntervalTreeIndex.java	`75.61% <100%> (-1.89%)`	`7 <1> (ø)`
src/main/java/htsjdk/tribble/index/Index.java	`100% <100%> (ø)`	`2 <2> (?)`
.../main/java/htsjdk/tribble/index/AbstractIndex.java	`51.579% <42.857%> (-3.223%)`	`30 <6> (ø)`
...tsjdk/tribble/index/linear/LinearIndexCreator.java	`83.721% <57.143%> (-3.459%)`	`15 <1> (+1)`
...k/tribble/index/interval/IntervalIndexCreator.java	`88.889% <62.5%> (-3.111%)`	`15 <2> (+1)`
...in/java/htsjdk/tribble/index/tabix/TabixIndex.java	`77.108% <62.5%> (-0.669%)`	`38 <3> (+1)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d22658...a6ff57e. Read the comment docs.

magicDGS · 2017-03-02T11:59:46Z

This is still WIP, because I found some other paths in the indexing code that should use Path for fully support. So this should wait for review, @lbergelson.

magicDGS · 2017-03-03T17:04:01Z

This should be ok now, @lbergelson. Previously failing tests are passing in my computer.

magicDGS · 2017-03-07T11:20:52Z

Can you have a look to this one @lbergelson? This is going to be important for my upcoming PR with tribble writing support with indexing on the fly.

Thanks a lot!

lbergelson

@magicDGS This is useful work. Thanks for doing this. I have a a few comments. I think it's probably worth breaking implementations of Index to avoid having weird defaults methods there, but lets see if @droazen thinks otherwise. He's usually more conservative about these things than I am.

Would it be possible for you to add a test that creates an index on a non-file path? There's an example of how to do this using an in memory filesystem called jimfs in AbstractFeatureReaderTest.

The basic idea is that you create a new jimfs filesystem, then you can get paths from that filesystem (ex: "jimfs://filesystemname/path/to/my/in/memory/file"). You may have to copy a file from the local system into jimfs if you want to create an index from it, but that's easy to do using the standard Files operations.

Just be sure that the jimfs filesystem gets closed after your test is done, because it will stay in memory until the jvm shuts down otherwise.

lbergelson · 2017-03-08T19:19:29Z

src/main/java/htsjdk/tribble/Tribble.java

     * @return a non-null File representing the index
     */
    public static File indexFile(final File file) {
        return indexFile(file.getAbsoluteFile(), STANDARD_INDEX_EXTENSION);
    }

    /**
-     * Return the name of the tabix index file for the provided vcf {@code filename}
+     * Return the name of the index file for the provided {@code filename}


forgot to change filename -> path

lbergelson · 2017-03-08T19:21:40Z

src/main/java/htsjdk/tribble/Tribble.java

     * Does not actually create an index
-     * @param filename  name of the vcf file
+     * @param path  name of the path


"name of the path" sounds like it's something distinct from the path itself, which is strange because "name of the file" is fine. Maybe just "the path"

lbergelson · 2017-03-08T20:04:31Z

src/main/java/htsjdk/tribble/index/AbstractIndex.java

@@ -201,7 +213,11 @@ public boolean isCurrentVersion() {
    }

    public File getIndexedFile() {
-        return indexedFile;
+        return getIndexedPath().toFile();


I think we should deprecate this method because it's no longer safe to call in all instances. It also needs a comment saying that it can fail with an exception if the path can't be represented as a file.

lbergelson · 2017-03-08T20:20:43Z

src/main/java/htsjdk/tribble/index/AbstractIndex.java

@@ -251,7 +271,7 @@ private void writeHeader(final LittleEndianOutputStream dos) throws IOException
        dos.writeInt(MAGIC_NUMBER);
        dos.writeInt(getType());
        dos.writeInt(version);
-        dos.writeString(indexedFile.getAbsolutePath());
+        dos.writeString(indexedPath.toString());


I think it's better to use indexedPath.toURI().toString() since it's more likely to roundtrip correctly for non-file paths.

lbergelson · 2017-03-08T20:41:02Z

src/main/java/htsjdk/tribble/index/AbstractIndex.java

@@ -67,7 +70,7 @@
    private final static long NO_TS = -1L;

    protected int version;                    // Our version value
-    protected File indexedFile = null;         // The file we've created this index for
+    protected Path indexedPath = null;         // The file we've created this index for


This is technically a breaking change. I'm going to say it's fine though. I don't believe there are any subclasses of AbstractIndex in the wild that we're going to be breaking.

Oh, I haven't realize that it is breaking compatibility. If we do not want to break compatibility, I can kept the field with the deprecated annotation and set it when the used constructor is a File. Let me know if I should do that.

Yeah, it's tricky. Anything field that's visible to a subclass of a non-final class is technically a breaking change since someone could be relying on it if they implement a subclass. It's why I'm so strongly in favor of making classes final and of making variable private with accessors if they need to be accessed.

lbergelson · 2017-03-08T20:44:36Z

src/main/java/htsjdk/tribble/index/DynamicIndexCreator.java

@@ -56,13 +57,18 @@
    MathUtils.RunningStat stats = new MathUtils.RunningStat();
    long basesSeen = 0;
    Feature lastFeature = null;
-    File inputFile;
+    // TODO: actually this field is not needed
+    Path inputPath;


Also a breaking change, but I don't see the harm in it since I really doubt anyone is subclassing this thing. 👍 to delete the field

In this case, because it is package protected, I do not think that is going to harm at all (unless someone is subclassing in the same package outside HTSJDK).

If we are breaking compatibiity anyway, as it is unused, why don't we just remove the field (no the constructor arg)? It is only used for get the index creators...

Removed the unused field in the commit for breaking compatibility.

lbergelson · 2017-03-08T20:48:58Z

src/main/java/htsjdk/tribble/index/Index.java

+     * @param indexPath Where to write the index.
+     * @throws IOException if the index is unable to write to the specified path.
+     */
+    public default void write(final Path indexPath) throws IOException {


I think it's better idea to have the default throw OperationUnsupportedException than to have it call toFile() which will work most of the time but sometimes fail.

lbergelson · 2017-03-08T20:49:05Z

src/main/java/htsjdk/tribble/index/Index.java

+     *
+     * @param featurePath
+     */
+    public default void writeBasedOnFeaturePath(Path featurePath) throws IOException {


same as above comment

I sort of want to break compatibility and force everyone to implement the Path functions. It would be so much cleaner to have the file versions be default implementations that call the path version. @droazen What do you think?

I suspect no on in the wild implements Index. At least it's not done in gatk/picard/igv.

Using OperationUnsupportedException for now.

Changed the interface in the commit for breaking compatibility.

lbergelson · 2017-03-08T20:59:27Z

src/main/java/htsjdk/tribble/index/tabix/TabixIndex.java

+     *
+     * @param tabixPath Where to write the index.
+     */
+    @Override


make the old file version delegate to this implementation

lbergelson · 2017-03-08T20:59:35Z

src/main/java/htsjdk/tribble/index/tabix/TabixIndex.java

+     * @param featurePath Path being indexed.
+     */
+    @Override
+    public void writeBasedOnFeaturePath(final Path featurePath) throws IOException {


magicDGS · 2017-03-09T07:21:16Z

I added two commits, @lbergelson:

Addressing most of your comments that does not imply backwards incompatibility.
Breaking compatibility for making simpler the interfaces. This could be removed if @droazen thinks that it is better to keep backwards compatibility for subclassing indexes.

Regarding the test, I don't know if I will have time today. I will try, but if not I will add them by tomorrow. You can review the current changes anyway, in case that something else should be changed.

magicDGS · 2017-03-09T07:40:32Z

I've just realized that something is broken with the changes, @lbergelson. Do you have an idea of what is happening? It looks that is breaking indexing on the fly for gzip VCF result (not the indexing itself). The most likely reson is the change to delegate the file constructor to the path one for files, because BlockCompressedOutputStream does not have Path support (see #811) and the filename is not set correctly in the binary codec.

I can add a simple constructor to BlockCompressedOutputStream with a String filename argument, but the change of that class is out of the scope of this commit.

magicDGS · 2017-03-09T08:10:55Z

Ops, I realized that it was an error not related with the BGZ output stream. That's what happens when you program right after waking up... Fixed in the next commit (and removed the commit with the BGZ changes).

magicDGS · 2017-03-09T10:14:21Z

src/main/java/htsjdk/tribble/index/tabix/TabixIndex.java

@@ -219,8 +219,8 @@ public void write(final Path tabixPath) throws IOException {
     */
    @Override
    public void writeBasedOnFeaturePath(final Path featurePath) throws IOException {
-        if (Files.isRegularFile(featurePath)) return;
-        write(IOUtil.getPath(Tribble.tabixIndexPath(featurePath)));
+        if (!Files.isRegularFile(featurePath)) return;


This was the refactoring problem, I overseen this. @lbergelson, I think that this behavior should be documented and/or at least log a warning to avoid getting crazy about what happened with the index. I will rather prefer if it fails with and IO/Tribble exception, but that will break compatibility with public methods...

Ack, good catch. That's terrible behavior. I wonder why it was done that way? Definitely needs to at least log a warning. I would also prefer your suggestion to fail with a TribbleException, but I think we'd need to investigate if that would break existing downstream code somehow. For now, could you have it log a warning, and open a new ticket to change the behavior?

@magicDGS I've opened the issue here: #821

Added warning log here and in AbstractIndex.

lbergelson · 2017-03-09T15:26:20Z

src/main/java/htsjdk/tribble/index/Index.java

@@ -73,41 +73,42 @@
    /**
     * Writes the index into a file.
     *
+     * Default implementation delegates in {@link #write(Path)}


Sorry to nitpick the language, but delegates to instead of delegates in is more idiomatic.

Thanks for this, I really appreciate to have comments to improve my English skills. Done!

lbergelson · 2017-03-09T15:26:39Z

src/main/java/htsjdk/tribble/index/Index.java


    /**
     * Write an appropriately named and located Index file based on the name and location of the featureFile.
     * If featureFile is not a normal file, the index will silently not be written.
+     *
+     * Default implementation delegates in {@link #writeBasedOnFeaturePath(Path)}


same as above

lbergelson · 2017-03-09T15:35:20Z

@magicDGS Looks good. 👍 Once we have the tests for the two index types.

magicDGS · 2017-03-09T16:13:41Z

Three more commits:

Minor changes from review.
A couple of missing constructors for Path (to test)
Small test for using path in writing and loading.

Thanks for reviewing, @lbergelson - Back to you!

lbergelson · 2017-03-10T15:14:22Z

👍

magicDGS · 2017-03-10T16:19:21Z

Thanks!

lbergelson self-assigned this Mar 1, 2017

magicDGS changed the title ~~Tribble/Tabix index path support~~ WIP: Tribble/Tabix index path support Mar 2, 2017

magicDGS mentioned this pull request Mar 2, 2017

Md5CalculatingOutputStream support for Path #814

Merged

5 tasks

magicDGS added 2 commits March 3, 2017 17:50

Some clean up

4303b61

Path support for tribble/tabix index

f293192

magicDGS force-pushed the dgs_tribble_path_support branch from 82a554f to f293192 Compare March 3, 2017 16:55

magicDGS changed the title ~~WIP: Tribble/Tabix index path support~~ Tribble/Tabix index path support Mar 3, 2017

lbergelson requested changes Mar 8, 2017

View reviewed changes

magicDGS added 2 commits March 9, 2017 08:04

Address some comments

97084dd

Breaks compatibility

ce693cd

magicDGS force-pushed the dgs_tribble_path_support branch from 1f23883 to 777d9a2 Compare March 9, 2017 08:12

Fix tabix index

b12c9fa

magicDGS force-pushed the dgs_tribble_path_support branch from 777d9a2 to b12c9fa Compare March 9, 2017 10:09

magicDGS commented Mar 9, 2017

View reviewed changes

lbergelson reviewed Mar 9, 2017

View reviewed changes

magicDGS added 3 commits March 9, 2017 16:50

Address minor comments

f2eda11

Add test for write index into a Path

572958a

Add a couple of missing ctor for Path

4d7dda3

Small test to check if writing/ctor from a Path

a6ff57e

magicDGS mentioned this pull request Mar 10, 2017

Tribble writing support #822

Open

5 tasks

lbergelson approved these changes Mar 10, 2017

View reviewed changes

lbergelson merged commit 0c282b8 into samtools:master Mar 10, 2017

magicDGS deleted the dgs_tribble_path_support branch March 10, 2017 16:19

magicDGS added a commit to bioinformagik/htsjdk that referenced this pull request Mar 10, 2017

Complete path support with samtools#810

c2721e2

magicDGS added a commit to bioinformagik/htsjdk that referenced this pull request Jun 26, 2017

Complete path support with samtools#810

9cd4459

magicDGS added a commit to bioinformagik/htsjdk that referenced this pull request Jun 27, 2017

Complete path support with samtools#810

9234f2c

magicDGS added a commit to bioinformagik/htsjdk that referenced this pull request Aug 25, 2017

Complete path support with samtools#810

95543d9

magicDGS added a commit to bioinformagik/htsjdk that referenced this pull request Jan 25, 2018

Complete path support with samtools#810

f85aab9

magicDGS added a commit to bioinformagik/htsjdk that referenced this pull request Mar 21, 2018

Complete path support with samtools#810

4c67b5e

magicDGS restored the dgs_tribble_path_support branch June 28, 2018 12:27

Tribble/Tabix index path support #810

Tribble/Tabix index path support #810

Conversation

magicDGS commented Mar 1, 2017 • edited by lbergelson Loading

Description

Motivation

Changes

Checklist

lbergelson commented Mar 1, 2017

magicDGS commented Mar 1, 2017

magicDGS commented Mar 1, 2017

codecov-io commented Mar 1, 2017 • edited Loading

Codecov Report

magicDGS commented Mar 2, 2017

magicDGS commented Mar 3, 2017

magicDGS commented Mar 7, 2017

lbergelson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magicDGS commented Mar 9, 2017

magicDGS commented Mar 9, 2017

magicDGS commented Mar 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lbergelson commented Mar 9, 2017

magicDGS commented Mar 9, 2017

lbergelson commented Mar 10, 2017

magicDGS commented Mar 10, 2017

magicDGS commented Mar 1, 2017 •

edited by lbergelson

Loading

codecov-io commented Mar 1, 2017 •

edited

Loading