Mount a path to a real directory in the file system?

[felixfbecker]

Is it possible to mount a path in the mock-fs to a real directory in the file system?

[felixfbecker]

I'm thinking about a function moch.mirror(path) that returns a mock-directory like mock.directory() does but recursively adds all files and their contents as buffers.

[tschaub]

There is not support for anything like this currently. But I can see how it would be useful.

[JustBlackBird]

It could be useful when a module uses require at run time. In such case we could use node_modules from real fs and mock only directories we need to work with.

[jamestalmage]

👍

As stated, this would be very helpful for runtime requires.

[iliakan]

especially with the often used readable-stream module.
things just fail w/o the ability to exclude certain paths.

[tadatuta]

Hi!

We have a helper to load dir like this: https://github.com/enb-make/enb-stylus/blob/master/test/lib/mock-fs-helper.js which is used this way:

var mockFsHelper = require('mock-fs-helper');
var nodeModules = mockFsHelper.duplicateFSInMemory(path.resolve('node_modules'));
mock({
   node_modules: nodeModules
});

Is it possible to get such or similar implementation to mock-fs?
We'd be happy to send a PR.

// cc @blond

[blond]

I think we can add load method to load dir:

var mock = require('mock-fs');

mock({
  'path/to/fake/dir': mock.load('path/to/real/dir')
});

and to load file:

var mock = require('mock-fs');

mock({
  'path/to/fake.file': mock.load('path/to/real.file')
});

[rom1504]

It would be nice to have some way to have access to the real fs while using mockfs.

[papandreou]

I've made a module that can mount an fs-compatible module inside another fs-compatible module: https://github.com/papandreou/node-mountfs

Unexpected-fs (a plugin for the Unexpected assertion lib) uses a combination of the two to provide a declarative interface for mocking out selected directories. Here's where they're wired together: https://github.com/unexpectedjs/unexpected-fs/blob/master/index.js#L22-L35

[jamestalmage]

@papandreou - nice mountfs seems like the perfect solution.

[jamestalmage]

@tschaub - Would you accept a PR that at least recommended mountfs as a best practice?

---

Ideally, I would like to see mock-fs stop clobbering the entire file system by default. Instead, I think the default behavior should be more like the current mock.fs method, with a mountAt(path) method.

Example usage:

var fs = require('fs');
var assert = require('assert');
var mockfs = require('mock-fs');

var mock = mockfs({
  file1: 'hello',
  file2: 'goodbye'
});


assert(mock.readFileSync('/file1', 'utf8') === 'hello');
assert(mock.readFileSync('/file2', 'utf8') === 'goodbye');

var unmount = mock.mountAt('/foo/bar');

assert(fs.readFileSync('/foo/bar/file1', 'utf8') === 'hello');
assert(fs.readFileSync('/foo/bar/file2', 'utf8') === 'goodbye');

unmount();

---

I'm a maintainer of both AVA (test framework) and nyc (coverage tool). This is causing lots of problems for our users, and I'm spending quite a lot of time responding to GH issues related to it. If there's willingness to make the change, I am sure I can find willing volunteers.

[amwmedia]

@papandreou would you mind releasing this helper as a standalone node module (maybe something like mock-fs-loader)? It would make it easier for other projects to consume.

Or submit a pull request to get this functionality into the mock-fs main codebase 😄

[amwmedia]

I'm looking to work out unit testing for plop and I need a way to pull in the plopfile and templates into the mocked fs, while keeping all changes to the file system isolated during the test.

[gustavnikolaj]

@amwmedia specifically which part of unexpected-fs would you want - and how do you see it becoming part of mock-fs? As documented in the README it's not straight up mock-fs syntax we use. We accept an object that describes the different mock-fs's that we need, and then rewrite that to proper mock-fs input. Other than that, it's mostly just running your async code in a promise after you set up the mock-fs's - and then tearing them down in a finally block...

I'd be willing to make a module out of it, but I'm not sure I see the missing building block. I can't think of changes that would be better than the two modules are on their own - it seems like their interfaces are cut exactly the way they should, and they compose nicely together. Apart from the custom rewriting, it's just instrumenting promises and mock-fs / mountfs. Besides mountfs/mock-fs, it's really the unexpected framework that is the key ingredient.

@jamestalmage Unless I'm misunderstanding you, what you're suggesting is essentially for mock-fs to absorb mountfs? As I said above I think the they work really nicely together, and I don't see the value in merging tools that could be useful on their own. @papandreou wrote mountfs before I found mock-fs and composed the two together - I don't know what mountfs was written for originally, but it was not only intended for mock-fs - and I'm sure that there's uses for mock-fs without mountfs too :-)

[jamestalmage]

Unless I'm misunderstanding you, what you're suggesting is essentially for mock-fs to absorb mountfs

Not absorb it. Just use it as a dependency. mountfs has plenty of uses outside of mock-fs.

I'm sure that there's uses for mock-fs without mountfs too

No, not really. I think if users want the mock to mock away the entire filesystem, then they should be explicit about it:

var mock = mockfs({
  file1: 'hello',
  file2: 'goodbye'
});

// completely replace the whole filesystem. 
mock.mountAt('/');

Mocking the entire file system by default just causes way to many problems. I have spent a lot of time investigating user issues raised in both the nyc and AVA repos and Gitter channels which ended up being directly related to this.

I have considered just writing a simple module that wraps the two of them together with the API described above. However, I think the current API is harmful, and should just be changed.

[amwmedia]

@gustavnikolaj I was actually referencing the helper that @papandreou wrote. Maybe I'm not fully understanding how mountfs works.

For my use case, it would be ideal to create a mock-fs container and then copy some real files and folders into the mock file system. That way my test code can make changes to mocked and "real files" without actually touching the true file system.

The way I understand mountfs to work, it would allow modification of real and mock files/folders, but changes to real files would actually affect the file system. Is this incorrect?

[amwmedia]

Um, my apologies, the helper I was talking about was built by @tadatuta. Very sorry for the confusion!

https://github.com/enb/enb-stylus/blob/master/test/lib/mock-fs-helper.js <== that

[tschaub]

@jamestalmage I'm open to a v4 with different default behavior - or an API that encourages different behavior. But I haven't experienced the problems you are alluding to, so I think I'd need some more detail before understanding your perspective.

I think @felixfbecker original intent was to get support for easily creating a mock filesystem with contents from a directory on the real file system. @amwmedia's comment (#62 (comment)) seems to be in line with this. Wanting a quick way to pull in fixtures and have tests not modify real files.

The recent comments by @jamestalmage (#62 (comment)) seem to be about mounting a set of mock files on the real filesystem.

It might be possible to reconcile the two, but at least to me it seems like people are suggesting different directions.

[jamestalmage]

But I haven't experienced the problems you are alluding to, so I think I'd need some more detail before understanding your perspective

I'll just list some examples of problems I've seen:

1. Calling require (or proxyquire) after the file system has been mocked. Under the hood require does fs.readFileSync().
2. Using source-map-support to remap Error stack traces. This is a very common problem (it's happened to me once, and I've helped two/three AVA users figure it out). When an Error is thrown and source-map-support is installed, it hunts around the filesystem, looking for the the source-map for every module listed in the stack trace.
3. Using nyc for code coverage. nyc writes coverage data to disk when the process exits. If the mock hasn't been unmounted, then that coverage data is lost. AVA allows you to be sloppy with some of this cleanup, since it runs every test file in a new process (and therefore AVA users - including myself - are frequently less careful about cleanup - because it usually doesn't matter). Sloppiness aside, it's still a problem when the process exits because of an error and cleanup hooks (afterEach) are not run.

I think @felixfbecker original intent was to get support for easily creating a mock filesystem with contents from a directory on the real file system

Yes, I believe you are correct. I have no objection to that. I think the best way to handle it would be to expose an API to create a COW filesystem(where writes to the mock filesystem are stored in memory, but never passed to the actual filesystem). I still think those should start as mockFs instances, and not automatically clobber the actual file system unless users explicitly request that:

var mockFs = require('mock-fs');

// cow is a fully functioning `fs` replacement, based on `/some/directory/`
var cow = mockFs.cowFs('/some/directory/');

// reads /some/directory/test.txt - from the actual filesystem.
cow.readFileSync('test.txt');

// behaves like an normal write, but it's only written to memory, not the actual filesystem
cow.writeFileSync('test.txt', 'hello');

// after a write, future reads get the in-memory version, not the actual filesystem version.
cow.readFileSync('test.txt');

// the COW system can be mounted, just like my proposal for mocks:
cow.mountAt('/some/directory/');

// and unmounted
cow.unmount();

// If you want an in-place COW overlay:
var unmount = mockFs.cowOverlay('/some/directory');

// which is equivalent to:
mockFs.cowFs('/some/directory').mountAt('/some/directory');

---

It might be possible to reconcile the two, but at least to me it seems like people are suggesting different directions.

I think it is possible to handle both. My main concern is moving towards an API that encourages users to only mock out the directories they need (since that is what is creating support headaches for me), but I think the COW overlay is pretty valuable as well.

[iliakan]

Bumping up 👍

[tschaub]

This is one thing that could be added:

// load a bunch of fixtures
mock({
  foo: mock.load('path/to/real/foo')
});
// fs functions operate on the in-memory filesystem

mock.restore();
// fs functions operate on the real filesystem

This is another thing that could be added:

// make fs functions operate on the in-memory filesystem for the 'foo' path only
mock.mount({
  foo: 'bar'
});

mock.restore(); 

Those two things can be added in a backwards-compatible way.

Note that as of [email protected], calls to require() should always use the real filesystem (under the hood, require() calls fs functions and process.binding('fs') functions, so behavior was inconsistent before).

[tschaub]

Actually, I think the mock.mount() behavior above would be better implemented as an option. E.g.:

// make it so fs functions operate on the real filesystem except for the 'foo' path
mock({foo: 'bar'}, {overlay: true});

This would make it so the current mock.fs() function could still be used.

// in case you don't want to touch the real fs module
var fs = mock.fs({foo: 'bar'}, {overlay: true});

I've ticketed the overlay option as #142. If this doesn't satisfy the second use case mentioned here, I'd appreciate it if conversation could continue on that ticket (#142). The first use case mentioned in this ticket could be satisfied by a mock.load() function, and I'd like to be able to close this ticket by adding that.

If overlay turns out to be the better behavior, it could become true by default with a major version.

[iliakan]

That ticket is so important and has not yet been adressed :(
Love this module, can't use it.

[frederikheld]

ls there any progress on this issue? I'd like to load prepared data from a file (which is in the real file system) while using mock-fs. Unfortunately this seems to be impossible right now 😕

[timsuchanek]

A workaround (someone could build a real library out of this):

const { patchFs } = require('fs-monkey')
const fs = require('fs')

function mockFs(fileMap) {
  const originalFsRead = fs.readFileSync

  const myFs = {
    readFileSync: (fileName, ...args) => {
      return fileMap[fileName] || originalFsRead.call(fs, fileName, ...args)
    },
  }

  patchFs(myFs)
}

const fileMap = {
  [__dirname + '/my-file.js']: 'console.log("")'
}
mockFs(fileMap)

If you build a proper library, you would also have to mock read, readFile and readSync

[3cp]

mock-fs doesn't need to do anything to support this. Here is a recipe to mock with real directory. If you found this useful, I can raise a PR to update the readme.

mockfs({
  'fake/dir': readTree('real/dir'),
  'fake/dir2': readTree('real/dir2'),
  'fake-file': fs.readFileSync('real-file')
});

function readTree(dir) {
  const filepaths = fs.readdirSync(dir);
  const result = Object.create(null);

  for (const fp of filepaths) {
    const p = path.join(dir, fp);
    const stats = fs.statSync(p);
    if (stats.isFile()) {
      result[fp] = fs.readFileSync(p);
    } else if (stats.isDirectory()) {
      result[fp] = readTree(p);
    }
  }

  return result;
}

[jamestalmage]

@3cp It's a common enough usage case, it should just be covered by the API. Also, your solution would cause performance issues in certain context (let's say someone is testing a module that does something in node_modules - could get expensive).

[3cp]

@jamestalmage I am aware of that. The question is: why user uses mock-fs to mock a part of file system with huge amount of files, isn't the idea that mock is to mock something complex with something much simpler.

BTW, unless I missed something, mock-fs cannot help to mock runtime require(). I tried it before, and tried it again just now, mock a node_modules folder doesn't help delayed require().

Update: ok I do missed the discussion. It's about to retain existing node_modules, not mocking a node_modules folder. I can see this is is painful for runtime require().

[nonara]

Just a heads up - I believe #304 should alleviate people's frustrations.

We add the ability to automatically create dir/files from the filesystem, with options recursive and lazyLoad.
And as a bonus, there's a function to selectively bypass the mock system.

Have a look at the updated readme entries:

https://github.com/nonara/mock-fs#mapping-real-files--directories
https://github.com/nonara/mock-fs#bypassing-the-mock-file-system

Feel free to share your thoughts or questions!