-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Towards Issue #875, avoid duplicating files added to ipfs #2600
Commits on Apr 20, 2016
-
Move the go commands that should run under cmd/ipfs in the Makefile in cmd/ipfs rather than doing a "cd cmd/ipfs && go ..." in the root Makefile. The "cd cmd/ipfs && go ..." lines causes problems with GNU Emacs's compilation mode. With the current setup Emacs is unable to jump to the location of the error outputted by go compiler as it can not find the source file. The problem is that the embedded "cd" command causes Emacs's compilation mode to lose track of the current directory and thus attempts to look for the source file in the wrong directory. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 6a187c0 - Browse repository at this point
Copy the full SHA 6a187c0View commit details
Commits on Apr 27, 2016
-
Add Offset() method to files.File and create new AdvReader interface.
Required for #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 27ec622 - Browse repository at this point
Copy the full SHA 27ec622View commit details -
Add AbsPath() to files.File interface.
Also change other paths to be absolute. Required for #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for c410ea8 - Browse repository at this point
Copy the full SHA c410ea8View commit details -
Add extra parameter to indicate how the content should be added.
Required for #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 92c13ba - Browse repository at this point
Copy the full SHA 92c13baView commit details -
In the measure package don't return ErrInvalidType in batch Put.
None of the other methods in the measure package return this error, instead they only call RecordValue() when the value is []byte. This change makes batch Put consistent with the other methods and allows non []byte data to be passed though the measure datastore. Required for #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for abfb0a6 - Browse repository at this point
Copy the full SHA abfb0a6View commit details -
In merkledag.Node and blocks.Block maintain a DataPtr
The DataPtr points to the location of the data within a file on the file system. It the node is a leaf it also contains an alternative serialization of the Node or Block that does not contain the data. Required for #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for e92e9da - Browse repository at this point
Copy the full SHA e92e9daView commit details -
The datastore has an optional "advanced" datastore that handles Put requests for non []byte values, a "normal" datastore that handles all other put requests, and then any number of other datastore, some of them that can be designated read-only. Delete requests are passed on to all datastore not designed read-only. For now, querying will only work on a "normal" datastore. Note: Only tested in the case of just a "normal" datastore and the case of an "advanced" and "normal" datastore. Towards #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for d133b68 - Browse repository at this point
Copy the full SHA d133b68View commit details -
Basic implementation of "add --no-copy".
This involved: 1) Constructing an alternative data object that instead of raw bytes is a DataPtr with information on where the data is on the file system and enough other information in AltData to reconstruct the Merkle-DAG node. 2) A new datastore "filestore" that stores just the information in DataPtr. When retrieving blocks the Merkle-DAG node is reconstructed from combining AltData with the data from the file in the file system. Because the datastore needs to reconstruct the node it needs access to the Protocol Buffers for "merkledag" and "unixfs" and thus, for now, lives in go-ipfs instead of go-datastore. The filestore uses another datastore to store the protocol buffer encoded DataPtr. By default this is the leveldb datastore, as the size fo the encoded DataPtr is small. Towards #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 1c91a03 - Browse repository at this point
Copy the full SHA 1c91a03View commit details -
Add basic tests for "add --no-copy".
Note that as per issue #2259 content is not verified for local retrieval so changing the file content will not always be detected. Towards #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 2b8704e - Browse repository at this point
Copy the full SHA 2b8704eView commit details -
When reconstructing block always verify the result.
Towards #875. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for f26c2df - Browse repository at this point
Copy the full SHA f26c2dfView commit details -
In merkledag.Node and blocks.Block maintain a DataPtr License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 8ff6185 - Browse repository at this point
Copy the full SHA 8ff6185View commit details -
Add Basic Query and "Direct" commands to filestore. Needs Testing.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 955e763 - Browse repository at this point
Copy the full SHA 955e763View commit details -
Save ref to Filestore in repo for future direct access.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 14601a2 - Browse repository at this point
Copy the full SHA 14601a2View commit details -
Add Self() method to be able to get to FSRepo.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 31e2e21 - Browse repository at this point
Copy the full SHA 31e2e21View commit details -
Add "filestore" commands to list contents and verify filestore.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 4ef5531 - Browse repository at this point
Copy the full SHA 4ef5531View commit details
Commits on Apr 28, 2016
-
Make blocks.Block an interface.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for a999a72 - Browse repository at this point
Copy the full SHA a999a72View commit details -
Remove addOpts paramater from blockstore methods.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 7addf3e - Browse repository at this point
Copy the full SHA 7addf3eView commit details -
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 1685cbd - Browse repository at this point
Copy the full SHA 1685cbdView commit details -
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 12ff7ac - Browse repository at this point
Copy the full SHA 12ff7acView commit details -
Add temp. utility command to find dangling pins.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 01ad077 - Browse repository at this point
Copy the full SHA 01ad077View commit details -
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 72966dc - Browse repository at this point
Copy the full SHA 72966dcView commit details
Commits on Apr 29, 2016
-
"filestore ls": add "--quiet" option
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 63127e5 - Browse repository at this point
Copy the full SHA 63127e5View commit details -
"filestore verify": change "invalid" status to "changed".
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 767d4c8 - Browse repository at this point
Copy the full SHA 767d4c8View commit details -
Check if the WholeFile flag should be set for leaf nodes in filestore…
….Put. If the dataObj.Offset is zero and the WholeFile flag is not set, check if it should be. That is check if the file size is the same as dataObj.Size, and if it is, set dataObj.WholeFile. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 437905e - Browse repository at this point
Copy the full SHA 437905eView commit details -
Add "filestore rm-invalid" command.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for a3a2c2a - Browse repository at this point
Copy the full SHA a3a2c2aView commit details -
"filestore ls": add help text and rework output
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 10d724a - Browse repository at this point
Copy the full SHA 10d724aView commit details -
"filestore verify": only verify leaf nodes, add help text
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for dd6fbc0 - Browse repository at this point
Copy the full SHA dd6fbc0View commit details -
Add sharness tests for new "filestore" commands.
Add tests for: filestore ls filestore verify filestore rm-invalid filestore rm Also rename t0046-add-no-copy.sh to t0260-filestore.sh. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for dd46d8d - Browse repository at this point
Copy the full SHA dd46d8dView commit details -
Simpilfy files.File and chunk.Splitter interface.
Simplify files.File interface by combining Offset() and AbsPath() methods into one that return a files.ExtraInfo interface that can be extended with additional information. Simplify chunk.Splitter by returning a Bytes struct in the NextBytes() method. This eliminates the need for the AbsPath() method and the need to return the data and offset separately License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 13a14a6 - Browse repository at this point
Copy the full SHA 13a14a6View commit details -
Eliminate separate addOpts parameter by pushing the info into the rea…
…der. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 1e21dbe - Browse repository at this point
Copy the full SHA 1e21dbeView commit details
Commits on Apr 30, 2016
-
Remove ExtraInfo() from File interface and add SetExtraInfo() to AdvR…
…eader. Remove ExtraInfo() method from the files.File interface as it not strictly necessary. Also add SetExtraInfo() to AdvReader to eliminate the need for the NewReaderWaddOpts wrapper. License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 993a6d6 - Browse repository at this point
Copy the full SHA 993a6d6View commit details -
Report an error when trying to use "--no-copy" when daemon is online.
License: MIT Signed-off-by: Kevin Atkinson <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 56a6b43 - Browse repository at this point
Copy the full SHA 56a6b43View commit details