Question: building a production ready Leshan server ?

[jvermillard]

Question

Following the popular topic of reusing the Leshan Demo in a production environment

#1614
#1598
#1260
#926
this is even addressed in the first lines of the F.A.Q.: https://github.com/eclipse-leshan/leshan/wiki/F.A.Q.#is-there-a-rest-api-for-leshan-server-demo-

Rationales

I have been recently contacted by a company that wants to make this a reality: a simple open-source lwm2m server usable in production.

First, after implementing something like that in closed source somewhat four times in the past (for different companies/customers), I find the idea of not re-coding the same thing every time appealing.

Also, there is a problem for people with smaller fleets or with on-premise requirements, because the solutions on the market are quickly quite expensive if you have only a hundred/thousand devices. This is preventing the adoption of LWM2M: it makes using the protocol difficult at a small scale and limits maker-style innovations.

In general, the feedback I got is that LWM2M is complicated, the spec is not light for sure, but building a solution is quite complicated. I think on the embedded device side we start to have more mature open-source solutions, some vendors selling ready-to-use LWM2M sensors, but for the server side, we just have Leshan library or proprietary SaaS solutions.

Ideation

I would limit the effort to an LWM2M device frontend being able to manage device connections and expose a stable API.

The server should be able to be started quickly and be self-contained (no external dependency like installing a database, or message broker) but still be capable of handling 100k of connection with H.A. with the right setup/configuration.

The demo UI would use the API of the server and we should limit it to what the demo is doing: a simple exploration/debugging of the LWM2M protocol. Writting a full-blown device management UI would be a different project than Leshan but the demo UI would give a starting point (and is actually what people are doing by either reusing the code or the UX)

Pro/Cons

In a nutshell, the main advantage would be:

• fostering LWM2M and Leshan adoption by allowing everybody to get to production quicker
• stop having multiple companies building somewhat the same LWM2M device server, and concentrate on building a shared better solution (I would prefer being paid to contribute to open-source than rebuilding the same proprietary code)

Of course, there are some drawbacks:

• the first one is scope creep for the project
• release cycle, which should probably be different than the LWM2M library
• maintenance efforts that need to be funded (I will contribute some, some of my customers probably too)
• business collision with current companies contributing to Leshan (even if it's not the same scope in my mind)

If the idea gathers some interest, I would probably proceed by prototyping and designing the core interface of such a server

[sbernard31]

About :

• "The demo UI would use the API of the server"
• "release cycle, which should probably be different than the LWM2M library"

This leads :

• either a kind of inconsistency
• or it means that Leshan servers demo will also have a different lifecycle then the LWM2M library. (which could be not so good)

[sbernard31]

Is this idea also concerned the LWM2M Bootstrap server ?
(Note that currently server demo and bsserver demo shared some backend and frontend code)

[sbernard31]

maintenance efforts that need to be funded

This is not only about funding the initial development, this is also about maintaining over the time.

[sbernard31]

Another concern is about customization.
Currently Leshan Library follow LWM2M specification as closed as possible.

But often community ask to be able to customize default behavior, because of :

• specific needs (more feature out of LWM2M spec)
• adapt to "bad" device behavior. (by tolerating "invalid" behavior OR even change "valid" LWM2M behavior to make it invalid)

That ☝️ seems to be complicated to satisfy with :

• a simple production ready server.
• How Leshan should behave with Non Compliant Implementations ?

[sbernard31]

If the idea gathers some interest, I would probably proceed by prototyping and designing the core interface of such a server

If needed, we can create a new project in eclipse-leshan GitHub organization.

[jvermillard]

Is this idea also concerned the LWM2M Bootstrap server?

Yes, the main usage would be automatically rotating creds in coordination with the DM server.

This is not only about funding the initial development, this is also about maintaining over time.

For this you need commercial adoption, if it runs in production, then it needs to be maintained

But often community ask to be able to customize default behavior, because of :

specific needs (more feature out of LWM2M spec)
adapt to "bad" device behavior. (by tolerating "invalid" behavior OR even change "valid" LWM2M behavior to make it invalid)

I would keep the strict compliance (maybe with an exception for things where the spec is obviously broken), exception would be allowed by configuration for legitimate cases, or forking since it's open-source.

[boaks]

Interesting "move" or development.

Also, there is a problem for people with smaller fleets or with on-premise requirements, because the solutions on the market are quickly quite expensive if you have only a hundred/thousand devices. This is preventing the adoption of LWM2M: it makes using the protocol difficult at a small scale and limits maker-style innovations.

That's what I also got as feedback in 2022. Therefore I started Californium S3-proxy-server at that time. Today I released the latest version (4.0.0-M2) with extended http-forwarding functions.

I would limit the effort to an LWM2M device frontend being able to manage device connections and expose a stable API.

In the case of the S3-proxy-server in combination with hw-maker even that would cause some invests on their side. In my experience with those hw-maker, they therefore start with other protocols. To lower that initial invest (at least to have a field test with coap) I chose to store the data in S3 and starting with this year to also forward the data via http to an external endpoint. That enables to use "low code services" as tago.io, blync, or datacake without implementing own code.

What stays as pain point is the installation, mainly for the hw-maker. Others, which already run devices with an service in the cloud, feel pretty happy with that solution.

[EmbGangsta]

Hi, that's a good thing that you put the subject up to date and work on this.

In my company I try to promote LWM2M but as you said most of industrial LWM2M platforms are expensive solutions when we have only 10k or just more devices to handle ...

I can participate in testing if you want, my manager recently asked me to reopen the subject on our side to see if we could at least handle LWM2M in our devices as a PoC for device management purpose and FOTA purpose at least as a demo (with /5 , /6 and /3 resources).
Lastly for the client side I used a port of wakaama in C on the device and it was sufficient for demp and cost free for us. We could try some other client solution but maybe you can advice me which C implementation would fit better ?

[jvermillard]

@EmbGangsta if you use Zephyr, you can use the https://github.com/zephyrproject-rtos/zephyr/tree/main/samples/net/lwm2m_client

BTW I would like to understand your use case, do you mind emailing me to [email protected] ?

[jvermillard]

So I plan to start with a single instance LWM2M server (no bootstrap for now) with no or minimal UI for now, and focus on backend integration (Web API + webhook)

Persistence using embedded SQL database (sqlite) with a persistence abstraction front (Repository pattern)

A ‘domain’ model based on Java records, with no logic, services to fill the logic needs for the different frontend pieces (LWM2M, API, UI).

On the LWM2M piece, we should persist the needed DTLS elements for graceful restarts.

Base service with API:
device CRUD (note: will show the current state of device registration)
create/update/delete device with security setting/credentials
get device last data + possible actions, based on the LWM2M model + last known values
LWM2M request sending: read/write/execute at minimum
websocket events (in the style of: https://vermillard.com/post/websocket/)

webhook API will follow to be able to forward events & data to backends

[sbernard31]

You want to use websocket to send backend Event, right ? (let me know if I'm wrong)

I'm not sure why you prefer websocket against Server-Sent-Event.

I read you post and you say :

Though, from a personal perspective, I still find server-sent events to be an elegant solution, especially within the context of HTTP protocols, due to their simplicity and efficient unidirectional data flow.

Additionally, if your interest lies in backend-to-backend solutions, I would generally advise against using WebSockets, but that’s a topic for another discussion.

So I probably misunderstood something 🙂

[jvermillard]

What I mean is Websocket or SSE 😄
It's mainly a question of do you need bi-directional (probably not) and client-side issues: sometimes using WebSocket is simpler than using SSE in your UI framework

[sbernard31]

do you need bi-directional (probably not)

In demo, we don't need it. By the past, I hesitated to migrate from SSE to websocket and finally I decide that SSE was enough.
But if there is good reason to use websocket why not 🙂.

sometimes using WebSocket is simpler than using SSE in your UI framework

In demo, we are using SSE with vue3.
Regarding htmx, it seems there is some SSE support : https://htmx.org/extensions/sse/

[jvermillard]

I committed my prototype start there: https://github.com/clunkymachines/lwm2m-manager