-
Notifications
You must be signed in to change notification settings - Fork 229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Circular dependencies (prevents proper rollup
)
#348
Comments
Do you think this is a rollup problem or a readable-stream one? IMHO, it’s the former, as circular deps are valid cjs. I’m happy to have a look anyway. Can you please produce a project/full example that demonstrate the issue? |
It's kind of both. The problem is not really rollup, as what rollup does is "convert" (through a plugin) into plain ES static imports, which do not support circular dependencies. Circular dependencies are valid in cjs - but are still problematic even there because you have to be really careful about what is really export/imported and when does the circular dependency break it. I'll try to make up a sample project |
Here is an example: You could run Now look at line 1135 - you'll see the compiled As rollup is becoming very popular and big projects are moving there (in my opinion partly because browsers started supporting ES module system and there's much less need for transpilers), it becomes crucial to adjust modules to dump the circular dependencies, even if not adopting ES imports yet. |
Just a note: circular dependencies are valid in ESM code. It's just that they behave differently compared to CJS because files are not executed in the same order. |
This does not look like an easy case to fix:
There is no common dependency that could be factored out. |
Well |
Using webpacks circular dependency plugin also brings this up. |
Instead of this: function ReadableState(options, stream, isDuplex) {
Duplex = Duplex || require('./_stream_duplex');
options = options || {}; // Duplex streams are both readable and writable, but share
// the same options object.
// However, some cases require setting options to different
// values for the readable and the writable sides of the duplex stream.
// These options can be provided separately as readableXXX and writableXXX.
if (typeof isDuplex !== 'boolean') isDuplex = stream instanceof Duplex; // object stream flag. Used to make read(n) ignore n and to You could do this: function ReadableState(options, stream, isDuplex) {
options = options || {}; // Duplex streams are both readable and writable, but share
// the same options object.
// However, some cases require setting options to different
// values for the readable and the writable sides of the duplex stream.
// These options can be provided separately as readableXXX and writableXXX.
if (typeof isDuplex !== 'boolean') {
isDuplex = stream instanceof Readable && typeof stream._write === 'function' && typeof stream._writableState === 'object';
} With NodeJS streams, it's all about duck-typing. In order for something to be a stream it does not need to actually inherit from the base classes, it just needs to conform to a certain protocol that everyone expects. |
Unfortunately we cannot do that, for two reasons:
|
I am aware that there IS support for circular dependencies, but not the way you do them :-) So another compromise which would not change the almost-100% nodejs compatibility, would be a global (private) registry. Something like this: _registry.js:
// An empty file would also do as it has an empty `exports` object, but may behave differently when converted to ES6
const registry = {};
module.exports = registry; _stream_duplex.js:
const Registry = require('_registry');
.
.
.
Registry.Duplex = Duplex; _stream_readable.js:
const Registry = require('_registry');
function ReadableState(options, stream, isDuplex) {
// - Duplex = Duplex || require('./_stream_duplex');
.
.
.
if (typeof isDuplex !== 'boolean') isDuplex = stream instanceof Registry.Duplex; // object stream flag. Used to make read(n) ignore n and to
// make all the buffer merging and length checks go away
.
.
.
}
.
.
.
function Readable(options) {
// - Duplex = Duplex || require('./_stream_duplex');
.
.
.
var isDuplex = this instanceof Registry.Duplex;
.
.
.
}
Registry.Readable = Readable; Executing the |
Would you mind to send a PR with that implementation to https://github.com/nodejs/node? That is a change that we should aim to do there. |
@mcollina I'm not really that this change is relevant to nodejs itself. This is because we never "bundle" node's internal modules. |
We can make that change in Node.js core, as it will simplify the bootstrap phase there as well. Would you like to work on that? |
@mcollina Let me play a little bit with it in a playground, see how this behaves in different rollup/webpack configurations. I want to improve things, not break them :-) |
Any update on this one? |
No progress I'm afraid. Would you like to work on this? |
I ran into this problem and have been catching up on the thread. @mcollina I'm willing to investigate a proposal to change Node.js, but I'm interested to hear from you: if such a change landed in Node, what would the process look like to land a similar change here? |
readable-stream@3 is extracted from the latest Node 10 release. A fix would have to land in Node master, then released in node 12. After that it’s minimum 2 weeks before it hits Node 10. |
Thanks for those details. I'm compiling an issue to present the case for a change in the Node.js project. |
@danielgindi do you find a workaround for rollup/webpack settings? |
Thought the issue should be tied to rollup/rollup#1507 , particularly the workaround in this comment (which has helped our indexeddbshim project be able to switch from browserify+babelify to Rollup). |
Actually this looks good now in Node's core. They took the idea outlined here and implemented it, with It still has a circular dependency, because they did not extract the |
I needed this to work in Vite and didn't have time to wait for the node change to land, so I applied the Registry pattern mentioned above on a fork here: https://github.com/TechInSite/readable-stream#readme And I published it to: https://www.npmjs.com/package/vite-compatible-readable-stream I did not do any testing beyond using this in Vite with React PDF, but it works there, so if it helps you then 👍 . |
thanks @thekevinbrown, I was able to install your package by aliasing the
|
as vite-compatible-readable-stream fork from @thekevinbrown was causing compatiility issues in other modules, I went ahead with a much simpler patch: Which I published here: https://www.npmjs.com/package/readable-stream-no-circular I didn't fix the tests which are directly importing |
@gluck that would be installed with |
I think |
For what it's worth |
Thank you @thekevinbrown ! Adding an alias to I ran import { defineConfig } from "vite";
export default defineConfig({
...
resolve: {
alias: {
"readable-stream": "vite-compatible-readable-stream",
}
}
...
}); BTW my issue came from needing to run simple-peer within a Vite app |
I just wanted to add that I've had issues in my CI pipeline with Here is how I adjusted in
If that's helpful to anyone |
Adds `vite` alongside `react-scripts` as a method for running and building Election Manager. To use it, run `pnpm start:vite` or `pnpm build:vite`. These will soon replace the `react-scripts` method. This took more effort than the equivalent changes for the other frontends. In particular, I had to replace zip-stream with zip.js. I tested it a fair amount, but it's possible there's some difference I haven't accounted for. The reason for the change is that zip-stream uses `readable-streams`, which is a NPM-land implementation of streams from NodeJS. The `readable-streams` package had a circular dependency that rollup chokes on: nodejs/readable-stream#348. While it seems to be fixed in the latest version, the dependency that uses it is not compatible with the latest version so I cannot simply replace all `readable-stream` copies with the newest one. Switching the ZIP library turned out to be simpler.
Adds `vite` alongside `react-scripts` as a method for running and building Election Manager. To use it, run `pnpm start:vite` or `pnpm build:vite`. These will soon replace the `react-scripts` method. This took more effort than the equivalent changes for the other frontends. In particular, I had to replace zip-stream with zip.js. I tested it a fair amount, but it's possible there's some difference I haven't accounted for. The reason for the change is that zip-stream uses `readable-streams`, which is a NPM-land implementation of streams from NodeJS. The `readable-streams` package had a circular dependency that rollup chokes on: nodejs/readable-stream#348. While it seems to be fixed in the latest version, the dependency that uses it is not compatible with the latest version so I cannot simply replace all `readable-stream` copies with the newest one. Switching the ZIP library turned out to be simpler.
Adds `vite` alongside `react-scripts` as a method for running and building Election Manager. To use it, run `pnpm start:vite` or `pnpm build:vite`. These will soon replace the `react-scripts` method. This took more effort than the equivalent changes for the other frontends. In particular, I had to replace zip-stream with zip.js. I tested it a fair amount, but it's possible there's some difference I haven't accounted for. The reason for the change is that zip-stream uses `readable-streams`, which is a NPM-land implementation of streams from NodeJS. The `readable-streams` package had a circular dependency that rollup chokes on: nodejs/readable-stream#348. While it seems to be fixed in the latest version, the dependency that uses it is not compatible with the latest version so I cannot simply replace all `readable-stream` copies with the newest one. Switching the ZIP library turned out to be simpler.
Adds `vite` alongside `react-scripts` as a method for running and building Election Manager. To use it, run `pnpm start:vite` or `pnpm build:vite`. These will soon replace the `react-scripts` method. This took more effort than the equivalent changes for the other frontends. In particular, I had to replace zip-stream with zip.js. I tested it a fair amount, but it's possible there's some difference I haven't accounted for. The reason for the change is that zip-stream uses `readable-streams`, which is a NPM-land implementation of streams from NodeJS. The `readable-streams` package had a circular dependency that rollup chokes on: nodejs/readable-stream#348. While it seems to be fixed in the latest version, the dependency that uses it is not compatible with the latest version so I cannot simply replace all `readable-stream` copies with the newest one. Switching the ZIP library turned out to be simpler.
If one is using pnpm & rollup without vite, one can use pnpm's alias...https://pnpm.io/aliases However, it would be great to have the circular dependency fixed in this package. |
See nodejs/readable-stream#348 and feross/simple-peer#823 This commit will probably be reverted in near future when nodejs readable stream will be compatible with vite
If your code relies on circular dependencies, you can suppress the warnings using // rollup.config.js
export default [
{
// ...
onwarn: (warning, defaultHandler) => {
if (warning.code === "CIRCULAR_DEPENDENCY") {
return;
}
defaultHandler(warning);
},
},
]; |
Rollup spits those out:
It is a huge problem, and will make code unexecutable after "rolling up".
The rollup process will convert commonjs deps into "static imports", which do not support circular dependencies.
Then it will end up with a flat hierarchy where there's a reference to a (yet) undefined variable.
The text was updated successfully, but these errors were encountered: