Upload Error: socket hang up

[Barry127]

Uploading a file using storage.upload throws an Error: socket hang up.

fs.createReadStream('./myFile').pipe(s.upload('myFile'));

Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:164:20)
    at endReadableNT (_stream_readable.js:1054:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

I've got the same issue on the original mega repo. I cannot figure it out.

[qgustavor]

From Stack Overflow seems MEGA is closing the connection. Currently uploading isn't being tested automatically. For now I have some test scripts which uploads files to a test account I created. I ran those: upload isn't working.

From what I remember uploading code is the same from the original library (I forgot I put code to upload attributes). Probably MEGA changed how uploading works. I will check official and megatools repositories to check what changed.

Edit 1: both repositories don't have recent changes and megatools is still working (although it showed many warnings I tested not working code on it).

Other possibility is that their server is stricter now. Maybe it's now requiring clients to specify a user-agent for example. I will do some tests on that.

Edit 2: megatools sets up a "content-type" header but the official client doesn't. The official client adds a checksum parameter but megatools doesn't. I tried setting up a user-agent but it don't fixed the issue. Their upload servers use Symantec servers, but Node still don't distrusted those.

Do you have some idea about what happened?

Edit 3: Seems it's somewhat related to #13: when specifying the start byte their server don't hangs up. In the other hand megatools doesn't do that when uploading. I find that strange, is like they're detecting megatools and making their server don't break it.

But it don't fix the issue: when finishing the upload it returns "EAGAIN (-3)" as the file handle (hashes[0].toString()) is set to a empty string.

I found a strange, but probably unrelated, thing: tonistiigi's implementation uses -1 for max speed parameter ("ms"), while official and megatools implementations use 0. I also noticed that the official client uses v: 2 and the base64 of the server response when finishing the upload.

I will keep trying to find a fix for this bug (if my flacky internet connection permits).

Edit 4: the checksum that the webclient does and the start byte in the URL aren't required as if you edit MEGA's official webclient code to not use those it still works. I'm a bit afraid of MEGA having blocked the client (using something like this), even if the chances are too low. Anyway, I will keep trying to find a solution.

[Barry127]

Thanks for all the research. I've been puzzling with it for two days before posting.
I've got no idea what happened, my knowledge about the subject is far less then yours as I conclude by reading your answer.

[qgustavor]

Strange, seems that now it's working: https://directme.ga/view?F!J0VEwBxI!!MkcXwSJJ&c=a.png

Check the date, I just uploaded it: https://mega.nz/#F!J0VEwBxI!DeontologiaPROSOPOPEIA (I use two random words as keys for testing, it's something gfycat-like)

I just don't know yet if it's working because the changes I just did or if MEGA changed something.

Edit: I undoed everything I changed and it still continued working. Seems MEGA fixed the issue.

Anyway in the next version I will fix those cross-library differences.

[Barry127]

It's still not working for me:

fs.createReadStream('./package.json').pipe(s.upload('package.json'))
Stream { ... }
> Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:164:20)
    at endReadableNT (_stream_readable.js:1054:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

I tried both the original mega and megajs.

Any suggestions?

[qgustavor]

Now it's not working for me too. If you need it urgently just download megatools and use require('child_process'): seems it will take some time to fix this bug.

I'm planning to record TCP streams (using pcap) then I will try to make the library behave the most possible similar way as the web client does.

I tried testing uploading using the browser version, but I just discovered that uploading in browser never worked in the first place: I never tested it, I find insecure entering MEGA credentials in a third party library.

[qgustavor]

The server just is being stricter: it's requiring Connection: Keep-Alive to be set. Maybe that's because they expect clients to do parallel uploading, in this case reusing connections is good for them. I just find bad that they just made their server close the connection without returning any error (which would help debugging).

Two files I just uploaded: test 1 and test 2.

You can overwrite request module defaults using the following code
(request.defaults returns a wrapper, so you need to overwrite it):

storage.api.requestModule = storage.api.requestModule.defaults({
  forever: true,
  headers: {connection: 'keep-alive'}
})

I will release a new version that fixes uploading soon, so you don't need to use the above code. Well, it's still useful: I use it to configure user-agent and proxy settings (although I need to document it in wiki).

Edit: by the way, the wiki is a wiki, if edit it documenting how to change request defaults it would be help a lot. I just don't know if I open a new page documenting the api module (gateway and requestModule are meant to be overwriteable if needed) or if I add a new page containing "recipes" containing some code examples (like how to use proxies, how to set a user-agent, how to encrypt and decrypt files "offline", etc).

[qgustavor]

Fixed in 3daa01b