test: regression for fs module monkey-patching

[thefourtheye]

The #2025 breaks graceful-fs module. To catch the accidental breaking like this, this PR introduces a test to make sure that the native fs module is monkey-patchable.

[Fishrock123]

I am really not sure we should add this terrible hack as a test.

[thefourtheye]

@Fishrock123 I think we can avoid surprises like #1898, if we have this test, no?

[brendanashworth]

It really is horrible, but we've already decided to support graceful-fs's horrible way of hacking core 😞, so +1 on adding the test.

[trevnorris]

Not that it's and better, but pretty sure the same functionality can be achieved from the process bindings.

[ChALkeR]

I would recommend against merging this testcase. At least not until #1941 is closed.

The whole situation around graceful-fs is just wrong, there should be a legit (supported in core) way of doing what graceful-fs does, without hacks.

Monkey-patching core and natives-based magic should not be treated as supported, it will never be: it's not possible to support monkey-patching without freezing everything, and supporting the natives magic that graceful-fs currently uses would require to throw out internal/* completely. Supporting that just for fs module is inconsistent.

Once this is declared as supported (which one could assume it would be after merging a dedicated testcase) it would be harder to get this out and solved properly.

[trevnorris]

Agreed. Any short term gain here would be a long term maintenance loss. If this is a feature we should/want to support then let's create a proper API for it.

[vkurchatkin]

This is a temporary thing. We don't want to continue breaking graceful-fs again and again. It doesn't mean that this hack is actually supported.

[bnoordhuis]

we've already decided to support graceful-fs's horrible way of hacking core

We did? What memo did I miss?

[Fishrock123]

we've already decided to support graceful-fs's horrible way of hacking core

We did? What memo did I miss?

I wouldn't use the word support for this.. I can't imagine we would officially support that.

It's more like "ok people use it so we'll delay breaking it".

(That being said I'm still wondering if we couldn't inline internal modules into code returned by process.binding)

[trevnorris]

At what point will the line be drawn to not care that a module has blatantly abused the internal API? It'll get tiresome if every popular module can require us to roll back a change to unsupported API.

[Fishrock123]

 At what point will the line be drawn to not care that a module has blatantly abused the internal API? It'll get tiresome if every popular module can require us to roll back a change to unsupported API.

I'm not sure, maybe this is a case study in that?

This is blatant abuse, but:

1. Functionality cannot be achieved without hacks properly in core with core
2. Broad use because functionality is useful

I think #2 is pretty important also. If this sort of thing is used by less than some amount of people, we should just suggest an older version until #1 is fixed.

[trevnorris]

Those are legitimate points, and for these cases we should propose an officially supported API and write tests for that. My main concern going down this "temporary" path is that afterwards it's likely to just be forgotten. Further solidifying the use of the internal API. There's at least one case in core where this has happened.

As long as an alternative and supported API is added, I don't see the need for a depreciation cycle. Even if it's used, it's still internal API. Anyone who uses it knows what that means. At least they'll have a way to update their code.

[vkurchatkin]

Once again, this is a temporary thing to avoid breakage. Imagine the following workflow:

• someone includes require('internal/') in fs;
• runs tests;
• sees this test failing;
• reads comments, Github issues, etc;
• removes require('internal/');

Once graceful-fs is fixed it would be clear that this test is not relevant anymore and should be deleted.

At what point will the line be drawn to not care that a module has blatantly abused the internal API?

Apparently, never, as it "displays a lack of empathy that is incompetence bordering on malice" according to @isaacs.

[isaacs]

If we are going to make this a viable platform for real world users, then that means not breaking large swaths of user code, whether they're "blatantly abusing internal APIs" or not. What is exposed is external, regardless of what the docs say.

If you want this to be a project for experimentation and making a pretty thing for engineers to adore, then fine, but let's have the TSC formally state that that's the purpose of io.js and it's not intended for production or stability.

"Empathy" isn't about being a nice person. It's about acknowledging the reality of your users' experience. A platform without stability isn't a good platform that people will use, and software without users is a waste.

Give me a better way to build graceful-fs, and I'll gladly take advantage of it. Until then, literally 100% of your users depend on npm continuing to work properly, and ignoring that fact is a failure of platform engineering.

[isaacs]

And yes, it is tiresome. Welcome to the challenges of maintaining a mature open source software platform! The only alternative is obscurity.

[bnoordhuis]

Functionality cannot be achieved properly

I've seen this mentioned a few times but without explanations why. What's so special about the core fs module that graceful-fs can't shim it without eval hacks?

[trevnorris]

What is exposed is external, regardless of what the docs say.

You know that's a non-applicable argument. JS doesn't have the concept of private members. Even if it's hidden behind a Symbol, it's still possible to retrieve and override. So we depend on conventions. Which you have no trouble violating.

If you want this to be a project for experimentation and making a pretty thing for engineers to adore, then fine, but let's have the TSC formally state that that's the purpose of io.js and it's not intended for production or stability.

Comments such as this are not applicable to the discussion and are borderline harassment. Leave them out.

Give me a better way to build graceful-fs, and I'll gladly take advantage of it.

This is exactly as I suggested. Though it seems you are too preoccupied proving a point than having a discussion.

Show me a ticket that requested this functionality be exposed and I will be sympathetic. Otherwise I have no concern for the developer who lazily uses clearly internal methods and expects them to not break in the future.

[vkurchatkin]

In this case it's not actually about private APIs. The only private API used is process.binding and the code can be easily rewritten without it:

var pre = '(function (exports, require, module, __filename, __dirname) { ';
var post = '});';
var src = pre + require('child_process').execSync('curl https://raw.githubusercontent.com/nodejs/io.js/' + process.version + '/lib/fs.js', { encoding: 'utf8' }) + post;
var vm = require('vm');
var fn = vm.runInThisContext(src);
fn(exports, require, module, __filename, __dirname);

Does exactly the same without any private APIs! But this is just crazy. And actual implementation is not much better.

[Fishrock123]

Those are legitimate points, and for these cases we should propose an officially supported API and write tests for that. My main concern going down this "temporary" path is that afterwards it's likely to just be forgotten. Further solidifying the use of the internal API. There's at least one case in core where this has happened.

As long as an alternative and supported API is added, I don't see the need for a depreciation cycle. Even if it's used, it's still internal API. Anyone who uses it knows what that means. At least they'll have a way to update their code.

Agreed, we'd re-implement private modules into fs as soon as this hits. (Might mean floating a patch for deps/npm/(..)/graceful-fs until the updates bubble up.)

"Empathy" isn't about being a nice person. It's about acknowledging the reality of your users' experience. A platform without stability isn't a good platform that people will use, and software without users is a waste.

Give me a better way to build graceful-fs, and I'll gladly take advantage of it. Until then, literally 100% of your users depend on npm continuing to work properly, and ignoring that fact is a failure of platform engineering.

Dropping the top bit, that is what we are attempting to do. Could we please stop firing shots about this?

[ChALkeR]

@vkurchatkin I hope that code is a joke =).

[trevnorris]

@ChALkeR Not really: https://github.com/isaacs/node-graceful-fs/blob/master/fs.js

[ChALkeR]

@trevnorris I saw that. That is bad, but the curl thing above is far worse =).

[ChALkeR]

I want to link this here, it's loosely related: npm/npm#8619 (comment)

[isaacs]

I've seen this mentioned a few times but without explanations why. What's so special about the core fs module that graceful-fs can't shim it without eval hacks?

Try it, and you'll see. File descriptors have to be tracked in a variety of places, and so this means re-implementing fs.ReadStream, fs.readdir, et al, in their entirety. Discussed in more detail in #1941, so I'd rather not rehash it here. See also the entire commit history of https://github.com/isaacs/node-graceful-fs. There have been ample explanations, and I'm not willing to repeat them all on demand.

Since this is functionality that people currently rely on ("correctly" or "lazily" or not), and you want to shift around internals to make the current approach unworkable, either figure out a way to deliver this functionality that allows you to make whatever internal changes you like, without breaking existing userland code. Or decide that user success isn't as important to this project as correctness.

Dropping the top bit, that is what we are attempting to do. Could we please stop firing shots about this?

@Fishrock123 Not everyone is attempting to do that. Thanks for the support, though.

I'm feeling rather exhausted continually trying to make the case for stability. Until there's a clearly defined set of priorities for this project, I'm not comfortable continuing to engage. It's stressful and unproductive.

Send me a PR to make graceful-fs work on io.js, and I'll probably accept it. Or don't.

[bnoordhuis]

File descriptors have to be tracked in a variety of places, and so this means re-implementing fs.ReadStream, fs.readdir, et al, in their entirety. Discussed in more detail in #1941, so I'd rather not rehash it here. [...]

I've read #1941 but I didn't really see where it makes the case for the current graceful-fs implementation, except that any other approach is a lot of work?

[trevnorris]

figure out a way to deliver this functionality that allows you to make whatever internal changes you like

That is completely backwards. We don't need the functionality publicly exposed. You do. So instead of spending so much energy ranting why not open an issue, or probably better a PR, where actual technical discussion can happen.

Or decide that user success isn't as important to this project as correctness.

This type of aggressive rhetoric is unacceptable. Stop with this or leave. It will no longer be tolerated.

I'm feeling rather exhausted continually trying to make the case for stability.

You aren't, but instead fighting for something that has directly affected you. Don't pretend to be a savior for the community. You're belligerent attitude concerning this matter has been in complete contradiction to what you say you're fighting for.

Be like every other developer on this project and create a ticket with a concise explanation of the issue (the one already opened does not meet this criteria) and preferably alternative implementation details that would be helpful.

It is impossible to hide everything in JS. You know that. So we must follow conventions. You broke those conventions and now are expecting us to clean up.

[mikeal]

@trevnorris also, I just realized that this statement in the Stability Policy is from our existing io.js roadmap https://github.com/nodejs/io.js/blob/master/ROADMAP.md#stability-policy which was discussed and ratified all the way back in February https://github.com/nodejs/io.js/blob/master/doc/tc-meetings/2015-02-18.md

[trevnorris]

@mikeal I recall the Stability Policy defined in the roadmap. Which largely deals with native and V8 changes.

As for the vote to joining the foundation, I must have been too involved in the TSC Charter document at the time. Also since the link in that issue pointed to a document that states it is still a work in progress, didn't think too much of it. Thanks for pointing me to the additional information.

As for sending a PR, I'd first like to have a short discussion with the TSC (already opened an issue just for the tsc agenda) and get their thoughts on the matter. Partially because I don't know how the other members feel, and because I suck at writing compared to some other devs like yourself and don't want to accidentally convey the wrong message.

[mikeal]

As for sending a PR, I'd first like to have a short discussion with the TSC (already opened an issue just for the tsc agenda) and get their thoughts on the matter. Partially because I don't know how the other members feel, and because I suck at writing compared to some other devs like yourself and don't want to accidentally convey the wrong message.

Cool, I guess we'll discuss tomorrow :)

[Fishrock123]

@bnoordhuis's re-write has landed in graceful-fs. Closing.

[SimenB]

It was reverted, wasn't it?

isaacs/node-graceful-fs#45 (comment)

[ChALkeR]

@SimenB No, it is merged in graceful-fs 4.x along with a patch that fixes the 0.8 support.

[isaacs]

graceful-fs is now passing tests on latest io.js, and Node >= 0.8

[rvagg]

sweet, good work @bnoordhuis et. al.