Roadmap for standardizing on ecma-419?

[tve]

Is there a roadmap for moving Modddable properly to ecma-419? Specifically, moving stuff from examples to modules, updating docs to list implementation enhancements to ecma-419, and perhaps starting a deprecation process for the "old" stuff or marking it as "not recommended for new designs"?

[phoddie]

Sorry to be slow to respond. Busy times.

Agreed. It is time to cut over. We have ECMA-419 IO (and beyond) working well on all supported platforms. It is a clear improvement.

There are quite a few steps, several of which you note.

• Update the manifests for the deprecated modules (pins/digital, etc) with warnings so that developers are aware they should migrate forward
• Update the docs for deprecated modules to indicate they shouldn't be used in new code
• Move the IO implementations in /examples to /modules
• Update manifest_io so that it isn't necessary to include everything. With pins/* you can just include i2c, but with manifest_io.json right now you get everything (i2c, digitalbank, pwm, sockets, etc). For managing firmware image sizes, that needs to be addressed
• Update the build targets (esp32/moddable_two, esp/node_mcu, etc.) to use ECMA-419 instead of pins. That's a a bit of a project, but should be mostly boilerplate after it gets rolling.
• Once IO and network are done, there is still the display drivers. That's a fair amount of work. But, I don't think it can wait too.

We've got several big features for the next couple of months SDK releases, so time is going to be limited for a bit. Maybe we should aim for year's end. In the near term we could take care of some of the easier items, like manifest and doc updates, along with migrating from examples to modules.

[tve]

It is time to cut over.

That's the most significant bit 😄. I thought the same but didn't know whether there was some big roadblock. But yes, it's a lot of work...

[phoddie]

I like the idea of setting a target and mapping out the steps. We can share that with the community, much as we have with ESP-IDF v5 migration. That let's folks prepare, contribute, and help test.

[phoddie]

Just a small update. I've been working on "Update manifest_io so that it isn't necessary to include everything." That's going pretty well. The result is nice. the io/digital/blink example just includes Digital IO.

The biggest challenge is building the host instance provider (e.g. the device global). The tricky bit is that device.io.* contains all the IO constructors. That now needs to built dynamically based on the IO included by the project. I've got a proof-of-concept working, and am working through the details.

There's another item that needs to be addressed from the list above:

• Implement TCP, UDP, and Listener for Windows so network protocols can run in the Windows simulator. (This is already in place for macOS and Linux)

[dtex]

I've got a proof-of-concept working, and am working through the details.

Volunteering my time if you need testers

[phoddie]

Much appreciated! If I'm lucky, I'll have something later this week. In addition to getting it working, it shouldn't break anything. Definitely a multi-step transition project.

[tve]

Something else for the to-do list here: a C interface to low-level portions of io, especially io.Digital

Use-case: I'm writing an HX711 driver, which is a weight scale force sensor ADC chip. It has a clk pin and a data pin. To read the current value one has to produce 25 to 27 clock pulses and read a bit on the data line each time. I'd like to write this loop in C and I now need to refresh my memory on how to pass the io.Digital pin instances to C and then call them from there. Some solutions I see:

• provide an example for doing this so it's an easy copy-paste-edit
• provide some C macros (and type definitions if appropriate) to make it easy & obvious to do all this, for example, modIODigital_read(), modIODigital_write()
• I'm sure there are better ideas!

(Just to be clear: I'm mainly pointing out that there is the need for a canonical solution for writing C code that uses io)

[tve]

Hmm, maybe this is the wrong way to think: calling back into JS from C doesn't sound efficient. Maybe a different approach would be useful, which is for io.Digital (et.al) to have a C interface and a way to query "is this object I got an instance of the built-in io.Digital?". Then:

• JS code can pass io stuff down to C
• C can ask "is this an instance of the built-in class I know?"
• if it is, C can make direct calls to low-level C interfaces
• if it is not, C can raise an error or make indirect calls to the methods of the object passed in

In the end, I believe what matters is for the app code to be able to use device.io to define which pins, etc. to use, and be able to pass that down to a driver, which happens to have functions implemented in C.

But maybe I'm misguided and such drivers should not bother about device.io.XXX and just simply take pin numbers in their constructor.

[phoddie]

I like this topic. There are definitely valid reasons that C could would want to use the IO associated with the ECMA-419 IO objects. How to do that is a little tricky. The IO still needs to be managed by the ECMA-419 IO implementation. If not, chaos will ensue. That basically implies that there is a native C interface ECMA-419 IO. That's out-of-scope for TC53 at the moment, but that doesn't prevent us from doing something.

Moddable did some work on this internally a while back, but never got it all the way to finished. Let me try to explain the approach.

There's already have an object representing the IO -- the JavaScript object. It is entirely responsible for the IO. We need to associate some native APIs with the object. Rather than defining standalone APIs that are statically linked (like doDigitalWrite(obj, 1)) we can add a native dispatch table to the JavaScript object. The native dispatch table will contain function pointers to the native IO methods. Conveniently. it turns out that there's already something of a dispatch table on most IO objects: the XS host hooks used to integrate with the garbage collectors marking phase (details aren't important here). We can extend that dispatch table with additional methods specific to the class. In the case of digital that would be read and write. Given a reference to the JavaScript IO instance, C code can request the dispatch table. The request includes some unique ID (a string likely is enough, like "io/digital"). If it matches, the dispatch table is returned and you can make calls through it.

We should probably dust off that work and get it in place for digital (actually digitalbank.... digital is pure JavaScript built on digitalbank). In fact, if you look at the implementations of digitalbank, you'll see modDigitalBankRead and modDigitalBankWrite which were added to support this feature (they are currently not called anywhere in the source tree). Digital would be used the most for this, I expect. We can expand from there as needed and time permits.

What do you think?

[tve]

Sounds good to me, especially if it's a repeatable pattern, which it sounds like.

I generally like to write the first version of a device driver in JS, and then once it's working I see whether the performance works out. When it doesn't, I really only want to move the critical method(s) into C, not rewrite the whole thing. The approach you describe makes this possible for cases where the issue is a bunch of pin twiddling using io/Digital.

Of course in some cases one just has to bite the bullet and rewrite the whole driver in C, in that case this approach is likely not to help and the C driver would have to call the platform native (not portable) abstractions directly. That's not so great and worse than the current modGPIO situation.

[phoddie]

...especially if it's a repeatable pattern, which it sounds like.

Yes, that's the idea. It is a general-purpose way to associate a C dispatch table with any native host object in XS.

I generally like to write the first version of a device driver in JS, and then once it's working I see whether the performance works out. When it doesn't, I really only want to move the critical method(s) into C, not rewrite the whole thing. T

I do the same. The good news is that it is rare that there is a need to dive into C.

Of course in some cases one just has to bite the bullet and rewrite the whole driver in C, in that case this approach is likely not to help and the C driver would have to call the platform native (not portable) abstractions directly.

If you really need to call the platform APIs directly, that's something else -- there's no portable solution there. That said, the C functions exposed by the JavaScript object should be plenty fast for most cases (just as fast as modGPIO, for example). But, there are exceptions. In that case, you could, for example, still accept the ECMA-419 constructor's options object to configure the IO. From a JavaScript perspective, at least, it would look the same.

I can look at getting this done for DigitalBank. I don't think it should be too big a project. If I do that, can I count on you to give it a try? It really helps to have someone using enhancements like this sooner than later, to help shake out imperfect design decisions and bugs.

[tve]

If I do that, can I count on you to give it a try?

Yup!

[phoddie]

I've got the native API for DigitalBank working on ESP32. It is reasonably straightforward to use. I consider it very experimental at the moment -- everything about it could change with some experience. Here's a simple example of doing a DigitalBank write from C to an ECMA-419 Digital Bank instance:

void xs_blink_write(xsMachine *the)
{
	void *data = xsmcGetHostData(xsArg(0));
	xsDigitalBankHostHooks *hooks = (xsDigitalBankHostHooks *)xsGetHostHooksValidate(xsArg(0), "digitalbank");
	(hooks->doWrite)(data, xsmcToInteger(xsArg(1)));
}

The first line gets the host data to pass as state to the native API. The second line gets the callback table for the Digital Bank. If it isn't a Digital Bank instance, it will throw. The third line does the write, calling through the callback table and passing the host data.

Obviously it is ridiculous to do this for a single write, but for some transaction with several reads and writes it can be useful.

I need to review the code again before committing. But should be soon.