Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Towards Issue #875, avoid duplicating files added to ipfs #2600

Closed
wants to merge 32 commits into from
Closed

[WIP] Towards Issue #875, avoid duplicating files added to ipfs #2600

wants to merge 32 commits into from

Commits on Apr 20, 2016

  1. Refactor Makefile. 

    Move the go commands that should run under cmd/ipfs in the Makefile in
cmd/ipfs rather than doing a "cd cmd/ipfs && go ..." in the root
Makefile.

The "cd cmd/ipfs && go ..." lines causes problems with GNU Emacs's
compilation mode.  With the current setup Emacs is unable to jump to
the location of the error outputted by go compiler as it can not find
the source file.  The problem is that the embedded "cd" command causes
Emacs's compilation mode to lose track of the current directory and
thus attempts to look for the source file in the wrong directory.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 20, 2016
    Configuration menu
    Copy the full SHA
    6a187c0 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2016

  1. Add Offset() method to files.File and create new AdvReader interface. 

    Required for #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    27ec622 View commit details
    Browse the repository at this point in the history

  2. Add AbsPath() to files.File interface. 

    Also change other paths to be absolute.

Required for #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    c410ea8 View commit details
    Browse the repository at this point in the history

  3. Add extra parameter to indicate how the content should be added. 

    Required for #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    92c13ba View commit details
    Browse the repository at this point in the history

  4. In the measure package don't return ErrInvalidType in batch Put. 

    None of the other methods in the measure package return this error,
instead they only call RecordValue() when the value is []byte.  This
change makes batch Put consistent with the other methods and allows
non []byte data to be passed though the measure datastore.

Required for #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    abfb0a6 View commit details
    Browse the repository at this point in the history

  5. In merkledag.Node and blocks.Block maintain a DataPtr 

    The DataPtr points to the location of the data within a file on the
file system.  It the node is a leaf it also contains an alternative
serialization of the Node or Block that does not contain the data.

Required for #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    e92e9da View commit details
    Browse the repository at this point in the history

  6. New "multi" datastore. 

    The datastore has an optional "advanced" datastore that handles Put
requests for non []byte values, a "normal" datastore that handles all
other put requests, and then any number of other datastore, some of
them that can be designated read-only.  Delete requests are passed on
to all datastore not designed read-only.

For now, querying will only work on a "normal" datastore.

Note: Only tested in the case of just a "normal" datastore and the
case of an "advanced" and "normal" datastore.

Towards #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    d133b68 View commit details
    Browse the repository at this point in the history

  7. Basic implementation of "add --no-copy". 

    This involved:

1) Constructing an alternative data object that instead of raw bytes is
a DataPtr with information on where the data is on the file system and
enough other information in AltData to reconstruct the Merkle-DAG node.

2) A new datastore "filestore" that stores just the information in
DataPtr.  When retrieving blocks the Merkle-DAG node is reconstructed
from combining AltData with the data from the file in the file system.
Because the datastore needs to reconstruct the node it needs access to
the Protocol Buffers for "merkledag" and "unixfs" and thus, for now,
lives in go-ipfs instead of go-datastore.

The filestore uses another datastore to store the protocol buffer
encoded DataPtr.  By default this is the leveldb datastore, as the size
fo the encoded DataPtr is small.

Towards #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    1c91a03 View commit details
    Browse the repository at this point in the history

  8. Add basic tests for "add --no-copy". 

    Note that as per issue #2259 content is not verified for local
retrieval so changing the file content will not always be detected.

Towards #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    2b8704e View commit details
    Browse the repository at this point in the history

  9. When reconstructing block always verify the result. 

    Towards #875.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    f26c2df View commit details
    Browse the repository at this point in the history

  10. Bug fix to commit e92e9da: 

    In merkledag.Node and blocks.Block maintain a DataPtr

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    8ff6185 View commit details
    Browse the repository at this point in the history

  11. Add Basic Query and "Direct" commands to filestore. Needs Testing. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    955e763 View commit details
    Browse the repository at this point in the history

  12. Save ref to Filestore in repo for future direct access. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    14601a2 View commit details
    Browse the repository at this point in the history

  13. Add Self() method to be able to get to FSRepo. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    31e2e21 View commit details
    Browse the repository at this point in the history

  14. Add "filestore" commands to list contents and verify filestore. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 27, 2016
    Configuration menu
    Copy the full SHA
    4ef5531 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2016

  1. Make blocks.Block an interface. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 28, 2016
    Configuration menu
    Copy the full SHA
    a999a72 View commit details
    Browse the repository at this point in the history

  2. Remove addOpts paramater from blockstore methods. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 28, 2016
    Configuration menu
    Copy the full SHA
    7addf3e View commit details
    Browse the repository at this point in the history

  3. Disable failing test. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 28, 2016
    Configuration menu
    Copy the full SHA
    1685cbd View commit details
    Browse the repository at this point in the history

  4. Refactor. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 28, 2016
    Configuration menu
    Copy the full SHA
    12ff7ac View commit details
    Browse the repository at this point in the history

  5. Add temp. utility command to find dangling pins. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 28, 2016
    Configuration menu
    Copy the full SHA
    01ad077 View commit details
    Browse the repository at this point in the history

  6. Add "filestore rm" command. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 28, 2016
    Configuration menu
    Copy the full SHA
    72966dc View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2016

  1. "filestore ls": add "--quiet" option 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    63127e5 View commit details
    Browse the repository at this point in the history

  2. "filestore verify": change "invalid" status to "changed". 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    767d4c8 View commit details
    Browse the repository at this point in the history

  3. Check if the WholeFile flag should be set for leaf nodes in filestore… 

    ….Put.

If the dataObj.Offset is zero and the WholeFile flag is not set, check
if it should be.  That is check if the file size is the same as
dataObj.Size, and if it is, set dataObj.WholeFile.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    437905e View commit details
    Browse the repository at this point in the history

  4. Add "filestore rm-invalid" command. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    a3a2c2a View commit details
    Browse the repository at this point in the history

  5. "filestore ls": add help text and rework output 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    10d724a View commit details
    Browse the repository at this point in the history

  6. "filestore verify": only verify leaf nodes, add help text 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    dd6fbc0 View commit details
    Browse the repository at this point in the history

  7. Add sharness tests for new "filestore" commands. 

    Add tests for:
  filestore ls
  filestore verify
  filestore rm-invalid
  filestore rm

Also rename t0046-add-no-copy.sh to t0260-filestore.sh.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    dd46d8d View commit details
    Browse the repository at this point in the history

  8. Simpilfy files.File and chunk.Splitter interface. 

    Simplify files.File interface by combining Offset() and AbsPath()
methods into one that return a files.ExtraInfo interface that can be
extended with additional information.

Simplify chunk.Splitter by returning a Bytes struct in the NextBytes()
method.  This eliminates the need for the AbsPath() method and the
need to return the data and offset separately

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    13a14a6 View commit details
    Browse the repository at this point in the history

  9. Eliminate separate addOpts parameter by pushing the info into the rea… 

    …der.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 29, 2016
    Configuration menu
    Copy the full SHA
    1e21dbe View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2016

  1. Remove ExtraInfo() from File interface and add SetExtraInfo() to AdvR… 

    …eader.

Remove ExtraInfo() method from the files.File interface as it not
strictly necessary.

Also add SetExtraInfo() to AdvReader to eliminate the need for the
NewReaderWaddOpts wrapper.

License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 30, 2016
    Configuration menu
    Copy the full SHA
    993a6d6 View commit details
    Browse the repository at this point in the history

  2. Report an error when trying to use "--no-copy" when daemon is online. 

    License: MIT
Signed-off-by: Kevin Atkinson <[email protected]>
    @kevina
    kevina committed Apr 30, 2016
    Configuration menu
    Copy the full SHA
    56a6b43 View commit details
    Browse the repository at this point in the history