workers: initial implementation #1159
Conversation
|
What exactly is meant by "Named properties of Arrays?" |
| }, null, WorkerContextBinding.ERROR); | ||
| }; | ||
|
|
||
| require('util')._extend(Worker, EventEmitter.prototype); |
There was a problem hiding this comment.
Can this reuse the existing cached require('util')?
|
Property that is a string or symbol as opposed to an integer (which are called indexed properties) |
|
Epic feature :D. Regarding structured clone, @dslomov-chromium had plans to bring that into the ES spec and presumably also into V8 directly. I am not sure what the status of that is though. It would be good to collaborate to avoid future back-compat breakage if we do ever get the chance to switch to V8's structured clone. |
|
That's the ES7 typed objects strawman, isn't it? |
|
@domenic Yea, the thing should have been implemented in V8 to begin with. Firefox for example implements it in SpiderMonkey, with callbacks for host objects. Another solution can be patching V8 with some API additions that are already possible in v8 but just not done. E.g.:
Also what would be nice to have:
But of course most optimal solution is implementing structured clone in v8, the above additions would just enable avoiding completely ridiculous things like having to call |
You should be able to emulate that like this: Local<Object> object = /* ... */;
Local<String> key = /* ... */;
Local<Value> property = object->GetRealNamedProperty(key);
const bool is_own_property =
!property.IsEmpty() && object->GetRealNamedPropertyInPrototypeChain(key).IsEmpty();
Local<Array> array = /* ... */;
const uint32_t index = /* ... */;
const bool is_from_prototype =
!array->HasRealIndexedProperty(index) &&
!array->HasIndexedLookupInterceptor() &&
array->Has(index);(Or just check HasRealIndexedProperty for each element in the prototype chain.) |
| var context = this._workerContext; | ||
| this._workerContext = null; | ||
| if (typeof callback === 'function') { | ||
| this.on('exit', function(exitCode) { |
There was a problem hiding this comment.
Maybe this.once() so the closure doesn't stay around?
|
@bnoordhuis The check should be much cheaper than emulation, I mean at the cost of calling |
| to_remove = hc; | ||
| break; | ||
| } | ||
| } |
There was a problem hiding this comment.
An O(n) operation per closed handle is kind of unfortunate...
There was a problem hiding this comment.
Yes I thought about that and we should consider changing this to a structure that allows faster removal
There was a problem hiding this comment.
Or if the HandleCleanup object is available to the registerer, removal is constant time
Notable changes * deps: Fixed an out-of-band write in utf8 decoder. This is an important security update as it can be used to cause a denial of service attack.
See next commit for the actual fix. PR-URL: nodejs#1373 Reviewed-By: Fedor Indutny <[email protected]>
Update AUTHORS list using tools/update-authors.sh PR-URL: nodejs#2100 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: James Hartig <[email protected]>
Improve performance by: + Not leaking the `arguments` object! + Getting the last character of a string by index, instead of with `.substr()` or `.slice()` Improve code consistency by: + Using `[]` instead of `.charAt()` where possible + Using a function declaration instead of a var declaration + Using `.slice()` with clearer arguments + Checking if `dir` is truthy in `win32.format` (added tests for this) Improve both by: + Making the reusable `trimArray()` function + Standardizing getting certain path statistics with the new `win32StatPath()` function PR-URL: nodejs#1778 Reviewed-By: Сковорода Никита Андреевич <[email protected]> Reviewed-By: Roman Reiss <[email protected]>
Path functions being benchmarked are: * format * isAbsolute * join * normalize * relative * resolve PR-URL: nodejs#1778 Reviewed-By: Сковорода Никита Андреевич <[email protected]> Reviewed-By: Roman Reiss <[email protected]>
This reverts commit 9436a86.
PR-URL: nodejs#2112 Reviewed-By: Jeremiah Senkpiel <[email protected]>
Every npm version bump requires a few patches to be floated on node-gyp for io.js compatibility. These patches are found in 03d1992, 5de334c, and da730c7. This commit squashes them into a single commit. PR-URL: nodejs#990 Reviewed-By: Ben Noordhuis <[email protected]>
The delay-load hook allows node.exe/iojs.exe to be renamed. See efadffe for more background. PR-URL: nodejs#1433 Reviewed-By: Ben Noordhuis <[email protected]>
PR-URL: nodejs#2099 Author: Jeremiah Senkpiel <[email protected]>
The original test uses a variable to explicitly count how many times the callback is invoked. This patch uses common.mustCall() to track if the callback is called or not. This makes the test more robust, as we don't explicitly hardcode the number of times to be called. PR-URL: nodejs#2122 Reviewed-By: Colin Ihrig <[email protected]>
gdbinit provided by V8 can be very useful for low-level debugging of crashes in node and in binary addons. Most useful commands at 'jst' for JS stack traces and 'job' for printing a heap object. This patch installs the file at $PREFIX/share/doc/node/gdbinit. Reviewed-By: Ben Noordhuis <[email protected]> PR-URL: nodejs#2123
Add a check for crypto before using it, similar to how other tests work. PR-URL: nodejs#2129 Reviewed-By: Shigeki Ohtsu <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]>
|
Can we do anything to help out @petkaantonov? It would be awesome to play around with this! |
|
@alubbe There are still some outstanding issues / comments. I don't think anyone will object if you adopt this PR and make the necessary changes. Just fork Petka's branch and add your changes on top. |
petkaantonov
force-pushed
the
workers-implementation
branch
from
fab6db9
to
5c4dbba
Compare
|
#2133 replaces this PR |
|
For those who want to write Node.js code in multithread program: NW.js implemented this by enabling Node.js in Web Workers: https://nwjs.io/blog/v0.18.4/ |
Workers are light-weight processes that are backed by OS threads.
This is an initial implementation that is behind the flag
--experimental-workers. It should have very little effect on other io.js components, even if the flag is enabled. The module can be required by doingvar Worker = require('worker');- see how the tests do it in test/workers folder.What is currently implemented:
Worker constructor which takes an entry module and an optional options object. Currently (I have some ideas for more) the only option is
keepAlive:The option defaults to
true, which means the worker thread will stay alive even when its event loop is completely empty. This is because you might not haveanything to send to the worker right away when a server is started for instance.
A worker object is an event emitter, with the events
'message','exit', and'error'and public methodspostMessage,terminate,refandunref:'message'event will receive currently a single argument, that is directly the message posted from the worker thread.'exit'event is emitted when the worker thread exits and receives the exit code as argument.'error'event is emitted when there is an uncaught exception in the worker which would abort a normal node process. The argument is the error thrown,if it's an error object, its builtin type, stack, name and message are retained in the copy. It will additionally contain extra properties that were in the
original error, such as
.code.postMessage([message])where message is structured clonable* can be used to send a message to the worker thread. Additionally internal messaging support is implementedthat doesn't need to copy data or go through JS. (* currently implemented as JSON to keep PR small, more details later)
terminate([callback])terminates the worker by using V8 TerminateExecution and uv_async_send, so the only thing thatcan defer a worker's termination is when it is executing C code which shouldn't contain any infinite loop or such :-).
unrefandrefwork like timer's respective methods, however keep in mind that a worker will terminate if its owner terminates.Inside a worker thread, several process-wide features are disabled for example setting the process' current working directory, title, environment variables or umask. Workers can still read these, but only the main thread can change them. Otherwise all core modules and javascript works normally inside a worker, including
console.log.Inside worker thread, the
Workerconstructor object is an event emitter, with the event'message'and the methodpostMessage()to communicate with its owner (see tests for usage). The worker constructor can be used to construct nested workers with the worker being their owner. Passing messages from grand child to grand parent requires the child's parent to pass its message through, however transferable message ports can be implemented if use cases emerge.Nested workers were implemented because Web Workers support them too (except Chrome) and I envision that a setup with ${CPU cores} amount of web servers with N amounts of sub-workers for each will be very useful.
2 new public process read-only properties are introduced
process.isMainInstanceandprocess.isWorkerInstance.Advantages of workers over processes:
and adopted by another.
which is more graceful in high load situations. And if we implement worker-cluster, there shouldn't be the need
to use home-made round-robing scheduler.
Disadvantages:
enables freeing all resources upon Environment destruction is there and
is already used to free all resources that a new worker that doesn't do
anything but exit immediately allocates.
little amount of shared resources as most stuff is tied to a v8 isolate or a uv event loop
which are exclusive to a worker.
Compared to traditional threading:
To keep the PR size small (believe it or not, deep copying will take much more code than workers alone), objects are still copied by JSON serialization. I want to implement something like the structured clone with some differences as the spec has some seriously nasty parts that are not really needed in practice but force the implementation to be so slow as to defeat the whole point of using it in the first place.
So what cannot be implemented from the algorithm:
What could be implemented but don't want to implement from the algorithm:
What I mean by inconsistent is that copying map, set and array properties is inconsistent because the properties of Date, RegExp, Boolean, Number, and String objects are dropped.
To make reviewing this easier, a description of changes in each file follows:
make test-workers. Worker tests are not ran as a part ofmake testSTATIC_ASSERTchar*from js string (TODO)Removemethod toListHeadCallbackInfoclass declaration to a headerALLOC_IDtoenv.hPersistentHandleVisitorthat is used to walk all unfinalized weak persistent handles to release their C resources uponEnvironmentdestructionuv_async_sendunreliability when async handles need to be unreffedContextifyScriptclass declaration to a header. The diff looks horrible but there is no real changesEvalMachinemethodenvironment->CleanupHandles()is actually unused by main instance, but workers use it)ApiLock()is null for main instance but workers need it because module initialization isn't termination safe.MakeCallbacksafe to call inside workers that are being abruptly terminatedexitcalls toenv->Exit()(which only exits the process if called from main thread)process._registerHandleCleanup()atexitfromLoadEnvironmenttoInit(probably not the right place, but it cannot be inLoadEnvironment)CreateEnvironmentoverload to be called from worker threadsInitializeEnvironmentTerminateSubWorkers()before exit event is emitted.ShutdownPlatformanddelete platformat the end ofStartmethod.ClassIdenum which weak persistent wrappers can use so that they can be walked upon Environment destruction and cleaned up for sure.set_using_cares(), which means that in Environment destructordestroy_caresis calledCanCallIntoJs()method: main instance can always call into js but workers can be in the middle of termination in which case one cannot call into js anymoreExitmethod which does process exit if called from main thread and worker termination if called from worker threadInitAresOnceto do one-time process-wide initializationenv->set_using_cares()(see env.h above)MakeCallbackas were made tonode::MakeCallbackEnvironmentis destructed, the handles won't leakEnvironmentis destructed, the handle won't leak. The other handles probably leak too but this was the only one that is caught by current tests. There is currently an assertion in worker termination code for leaked handles.