Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raw leaves described by alg != sha2-256/32 do not work ( also generalized test matrix ) #3552

Closed
mib-kd743naq opened this issue Dec 30, 2016 · 15 comments
Labels
help wanted Seeking public contribution on this issue kind/bug A bug in existing code (including security flaws)
Milestone

Comments

@mib-kd743naq
Copy link
Contributor

mib-kd743naq commented Dec 30, 2016

Version information:

go-ipfs version: 0.4.5-dev-99d72c2

Type:

Bug

Priority:

P1

Description:

This is a set of 45 test cases. Table shows what currently works and what doesn't. Description at the end.

Framing Data\Link v0pb-sha2-256-32 v1pb-sha2-256-24 v1pb-sha1-20 v1pb-sha2-512-48 v1pb-sha2-256-32
v0pb-sha2-256-32 ✔ QmXSt...SZy45 ✔ zrkJE...xS9EL ✔ z8bvj...Q2dPd ✔ zM1zo...wYaBs ✔ zdj7W...XkNuC
v1pb-sha2-256-24 ✔ QmNu4...2Xhpa ✔ zrkJE...foNgz ✔ z8bvi...WUNQe ✔ zM1zo...cbmBL ✔ zdj7W...dGsJt
v1pb-sha1-20 ✔ QmPCU...f6DZn ✔ zrkJE...W7gKr ✔ z8bvi...HFE3S ✔ zM1zo...6yrt2 ✔ zdj7W...asVU6
v1pb-sha2-512-48 ✔ Qmasu...R5tzS ✔ zrkJE...Tohnj ✔ z8bvi...zYvF8 ✔ zM1zo...ixMy4 ✔ zdj7W...fr3kd
v1pb-sha2-256-32 ✔ QmYtF...yy61r ✔ zrkJE...18ARP ✔ z8bvi...QszAN ✔ zM1zo...bvmfF ✔ zdj7W...LxkeR
v1raw-sha2-256-24 ✔ QmTZH...1rcBx ✔ zrkJE...Kehpz ✔ z8bvi...fTnTU ✔ zM1zo...ZYJRd ✔ zdj7W...ihtGG
v1raw-sha1-20 ✔ QmQpU...WFdjr ✔ zrkJE...4zJTB ✔ z8bvi...BjrEv ✔ zM1zo...okdG8 ✔ zdj7W...oXWAA
v1raw-sha2-512-48 ✔ QmZf6...QiUfj ✔ zrkJE...ogVJr ✔ z8bvi...3aVtu ✔ zM1zo...riR8P ✔ zdj7W...HDepB
v1raw-sha2-256-32 ✔ QmZr6...mJpNj ✔ zrkJE...YKfbK ✔ z8bvi...7CnXu ✔ zM1zo...4tAgP ✔ zdj7W...R2MyH

The entire set represents the same data:

  • Tested "file" is 7 bytes long, containing exactly:

    123\n45\n
    or
    31 32 33 0A 34 35 0A

  • 4 chunks ( 3 unique ) splitting on the newline ( 0A ):

123
\n
45
\n
  • Organized into a file via 3 links as follows (arrangement is arbitrarily chosen, mostly to hit as many corners as I can think of):
         ___________________
        |                   |
        | FINAL OBJECT LINK |
        |___________________|
         /         \
        /           \
 ===========    ___________________
 || "123" ||   |                   |
 ===========   | INTERMEDIATE LINK |
               |___________________|
                 /     |       \
                /      |        \
      ==========   ==========    ___________________
      || "\n" ||   || "45" ||   |                   |
      ==========   ==========   | INTERMEDIATE LINK |
                                |___________________|
                                       |
                                       |
                                  ==========
                                  || "\n" ||
                                  ==========
  • Each data/link framing is described by <format>-<hashalg>-<truncation>, with the names being self explanatory.

  • Every link omits the optional size field entirely. This is deliberate, ask me for more info if the reasoning is unclear.

  • The entire dataset is available at QmYbb74fCe7pom18nPRw2rKAPMukza63DfQAdHG53WywbW in a prototype of "efficient put format". The same directory contains a (massaged) protoc --decode_raw of each <name>.put for easier reading.

@mib-kd743naq
Copy link
Contributor Author

@jbenet as per your comment this revision of the ticket includes the exact bytes (*.put) that I fed to http://127.0.0.1:5001/api/v0/block/put?format={F}&mhtype={T}&mhlen={L}, including the exact id under which ipfs stored/indexed the block.

@diasdavid @dignifiedquire I have not tested this against js-ipfs, currently do not have it set up. Please feel free to reuse the dataset.

@Kubuxu @whyrusleeping Not sure if this is an 0.4.5 blocker or not...

@Kubuxu
Copy link
Member

Kubuxu commented Dec 30, 2016

@mib-kd743naq I am unable to fetch the dataset, I have set up my server to try to fetch it all the time so when you go online it will be fetched.

@mib-kd743naq
Copy link
Contributor Author

@Kubuxu sorry, only have my laptop node up, and running around the conference doesn't help
Should be avail. for the next ~40 mins

@Kubuxu
Copy link
Member

Kubuxu commented Dec 30, 2016

Got it.

@Kubuxu
Copy link
Member

Kubuxu commented Dec 30, 2016

Typo


Trying to pin it:

ipfs ping add  QmYbb74fCe7pom18nPRw2rKAPMukza63DfQAdHG53WywbW
Error: multihash length inconsistent: &{1  185 [240]}

@mib-kd743naq
Copy link
Contributor Author

@Kubuxu works here... perhaps s/ping/pin/ ?

@Kubuxu
Copy link
Member

Kubuxu commented Dec 30, 2016

Yeah. :|

@mib-kd743naq
Copy link
Contributor Author

@Kubuxu actually I just realized that the individual test objects themselves are not pinned in any way besides my laptop. Could you please:

ipfs pin add QmXStmqqnaBrfP94XjDy6Y39ygYGMEu7wQY1zxACpSZy45
ipfs pin add zrkJEqVXFugrfTfnKouxvqhPYYCEZ8xR6xS9EL
ipfs pin add z8bvj1kMtvw2BBGQNmnrBSbe7rr9Q2dPd
ipfs pin add zM1zoqKTnRq1nDrvRvEtHyY5EVFtqtiU2aSNoEvJ5aBXSryYoqLgx2TRVf8fGb28zRwYaBs
ipfs pin add zdj7WensUcg8BTMCKWay3tqMRFX4EKp7vvG5NhsYPYb8XkNuC
ipfs pin add QmNu4Qr8fx6KGsquhoPxN2r7KhyiEZTDyNMM1Ege42Xhpa
ipfs pin add zrkJEkxWEb91Wqi5MEChEX9K7niexPvwrfoNgz
ipfs pin add z8bvizRrNP6TWeRVFjHj7ppjho5wWUNQe
ipfs pin add zM1zooTWWFb3Qz4o5UPxsTubHjjgUU6XRJ6FfW8oGbcomNHDttmNySGjRP3qa4FHqFcbmBL
ipfs pin add zdj7WjNKJAfstF8sZ24BrzXsATE2j6NJLLHj2dcfMf96dGsJt
ipfs pin add QmPCUSABGvLcbT5mEUke4xnbQtWyYEXFWiYfyVmg6f6DZn
ipfs pin add zrkJEttLb3ZS2ejiCVEABpUS12yALX3HtW7gKr
ipfs pin add z8bvizBNsLQxdjzPwX3diwDRRNuyHFE3S
ipfs pin add zM1zosy8kvPMN224UHPcHS1yG8rkjRrbE1jv7dFmEQd4hrKZyrEePk6pz2RmEP6FMo6yrt2
ipfs pin add zdj7WhJPhomVdpuQfbRXBgAh7fKX5QnPyf4Jx4SvdRbpasVU6
ipfs pin add QmasuD6U3KMh1xk9S7P31FhtmVscgPuLeS7y79fGhR5tzS
ipfs pin add zrkJEkcDgbr8QirPCe1u5NuqanA689TNwTohnj
ipfs pin add z8bvizWcf5ygDhcthcqDncxPcXghzYvF8
ipfs pin add zM1zokDJkyFWUYgHsK3w6gsYojiiVE59WiAwH4cVZ1A64j2G3ZtsPMXe1TovickXiAixMy4
ipfs pin add zdj7WZ26dLvNN8HDb9q97iccbUrpr1SY89cFLCYGye5nfr3kd
ipfs pin add QmYtF44QRxonUCmuTzXzzM6BvZhm31ScNRnAJW8eLyy61r
ipfs pin add zrkJEx6uA5fkVTRzceo8HAfFSkMKjacGD18ARP
ipfs pin add z8bvixjJ5T5i5AuVPvpuWxSdBhfyQszAN
ipfs pin add zM1zotfVBYeLLtZ6FequfE6mtFhpbTC9i1eU9bztGWebLG4CCHzyhVb9mPabJtT17ibvmfF
ipfs pin add zdj7Wf3akXA1wixavzCBuGScQJNDDwXDkFDrSWo76WqqLxkeR
ipfs pin add QmTZHvDKNcteezN4pFuYVxaGsBxdYA7db28y32Fk71rcBx
ipfs pin add zrkJEceiNTnEvAtT1DBB8YYBDUD9xDwD4Kehpz
ipfs pin add z8bvixsxdW29qk86DWuxM1EKqJFqfTnTU
ipfs pin add zM1zokWGSVw4uE7hdYNRJeR5Xuvz6ugCxJyGEu1TfaZLyTegr1hyRaWEZWvXCqwFepZYJRd
ipfs pin add zdj7WVpqrZCizahDsCLGHQ4PthaeURBksvQiLfrYnkJVihtGG
ipfs pin add QmQpUNgBY5zeaxhLJiZegBH5jG9SYn9eEMEQNCTiYWFdjr
ipfs pin add zrkJEyQniyjrFgwVgGeTq6ih6rbUGuZNV4zJTB
ipfs pin add z8bvixjWUf9CMrWPHEPnJfB5UjbjBjrEv
ipfs pin add zM1zomNrhXGxsZFHUsGU8GC8w735DnPj8qndPSF1ThEuwoA652J9SJ6RBRAwaavRjookdG8
ipfs pin add zdj7WfReDohbVfSjnE6b9fVryQCyMPpxX6XWgE9t9cMnoXWAA
ipfs pin add QmZf6rWFuu7yKSfAdRL4ujK82LUejmYVSvxo4n9LAQiUfj
ipfs pin add zrkJEid8DSyZWD8j4S4FEYJEh7D9Ptd92ogVJr
ipfs pin add z8bvizPQpcjusmzKaSZQbCqdTbHD3aVtu
ipfs pin add zM1zopDo3k2mmNi5BPZRBgtAYMpcT8nffmdtqghEV3n1mw3Ko2gGfc4HJoNxkxR7h7riR8P
ipfs pin add zdj7WkpPDQu8S3yVpsM64p9VtrTPrKaYsRkFkKcbpYmnHDepB
ipfs pin add QmZr6fFPus4Djf8fnKC7VdCw2kaeNnHLz7u6hTyNpmJpNj
ipfs pin add zrkJEvoQdqBmpTaZCp8W7CgE2W36BkDgeYKfbK
ipfs pin add z8bviyMwLb6oYLBjWsNTxaBCfwKz7CnXu
ipfs pin add zM1zooVrTECynpfkpXcP7KkJLVmx3PsYiedzap4uQ2zj9rjEkVMag1h7bWFxgkxPGt4tAgP
ipfs pin add zdj7Wdomcy7ZkDTYUSanfJnQ8hxoPiddHR7dmwaYtaZiR2MyH

@whyrusleeping
Copy link
Member

@whyrusleeping whyrusleeping added this to the ipfs 0.4.6 milestone Jan 6, 2017
@whyrusleeping whyrusleeping added the kind/bug A bug in existing code (including security flaws) label Jan 6, 2017
@whyrusleeping whyrusleeping modified the milestones: Ipfs 0.4.7, ipfs 0.4.6 Feb 16, 2017
@whyrusleeping
Copy link
Member

Lets revisit the items in the grid, Its definitely worth trying to make progress here.

@whyrusleeping whyrusleeping modified the milestones: Ipfs 0.4.8, Ipfs 0.4.7 Mar 11, 2017
@whyrusleeping
Copy link
Member

I think with some of the changes we've been making for cidv1 support in ipfs add that these will be a lot better.

Lets try and make a script that permutes all of these and outputs success/failures

@whyrusleeping whyrusleeping modified the milestones: Ipfs 0.4.9, Ipfs 0.4.8 Mar 24, 2017
@whyrusleeping whyrusleeping modified the milestones: Ipfs 0.4.10, Ipfs 0.4.9 May 8, 2017
@whyrusleeping whyrusleeping added the help wanted Seeking public contribution on this issue label May 8, 2017
@mib-kd743naq
Copy link
Contributor Author

w00t! It looks like all the ✘-es in the matrix now properly resolve!

@whyrusleeping / @Kubuxu please add the relevant "fixed in" and close the ticket

@mib-kd743naq
Copy link
Contributor Author

I should probably look into the IPLD / CBOR based stuff, prepare a similar test-matrix...
Is the specification finalized?

@whyrusleeping
Copy link
Member

@mib-kd743naq thats great!! Thanks for reporting back :)

The specification for the cbor stuff is finalized, and its merged into go-ipfs. definitely give all of that a good run over

@magik6k magik6k modified the milestones: Ipfs 0.4.10, Ipfs 0.4.11 Jul 28, 2017
@mib-kd743naq
Copy link
Contributor Author

I re-verified that all the problems are solved, adjusted all the "X"es to ✔s in the original description, and am closing this ticket

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Seeking public contribution on this issue kind/bug A bug in existing code (including security flaws)
Projects
None yet
Development

No branches or pull requests

4 participants