Node-RED - preinstalling nodes does not seem to be working

[DIYtechie]

Just deleted the Node-RED container (and the volumes/nodered folder) and tried installing it again via IOTStack.

I tried pre-installing some nodes from the menu, but the selected notes were not installed.

Regardless of my selection, only pigpiod, Home Assistant and influxdb nodes are pre-installed. Even if I deselect them when installing Node-RED.

[Slyke]

Hey @DIYtechie NodeRed is a special case and requires that the Dockerfile be generated dynamically during build time.

It places that file in the ./.services directory (which it also creates). It might be a bug if the script is not overwriting that file if you are selecting "overwrite everything" when prompted.

But for now, just try deleting that docker file: rm -rf ./.services/nodered and see if that works.

File: https://github.com/SensorsIot/IOTstack/blob/master/.templates/nodered/build.sh#L44
NPM: https://github.com/SensorsIot/IOTstack/blob/master/.templates/nodered/build.sh#L57

[DIYtechie]

Hey @Slyke and thanks for the swift response.

Tried deleting services/nodered as well as volumes/nodered, de-selected Node-RED in the menu.sh, and ran docker-compose up -d --remove-orphans

But when installing Node-RED via the build stack in menu.sh, I still get the same pre-installed nodes regardless of the selection. When the container is already removed, there is of course no option to "overwrite everything / pull full service from template". Also tried deleting the container with portainer + deleted the services/nodered as well as volumes/nodered, but this made no difference.

[Slyke]

Hmmm, that is strange. I will do up a fresh image tomorrow and see if I can reproduce and submit a quick fix. The menu rewrite is about 25% done, and that should also fix it.

[Paraphraser]

I don't know whether this will help but when I want to rebuild Node-Red, I do this:

$ cd ~/IOTstack
$ docker-compose down
$ docker rmi "iotstack_nodered" "nodered/node-red"
$ docker-compose pull
$ docker-compose up --build -d

This has the disadvantage of taking the whole stack down while things are rebuilt but it was the only method I could find that would reliably rebuild Node-Red.

I have not tried just taking Node-Red down because, in my situation, there is little point in the other containers running without Node-Red.

The set of nodes specified via menu.sh get installed as you would expect. The only node I ever select explicitly is sqlite so, together with the ones selected in menu.sh by default, a new image will get:

• node-red-node-pi-gpiod
• node-red-dashboard
• node-red-contrib-influxdb
• node-red-node-sqlite

plus the ones Node-Red brings to the party by default.

Every other node I use I install via the Node-Red graphical user interface.

As far as I can tell, nodes installed via menu.sh do not "appear" in the mapped area of:

~/IOTstack/volumes/nodered/data/node_modules

I surmise this because, using sqlite as an example, it does not appear at the above path but it does appear when you open a shell into the container and examine:

/usr/src/node-red/node_modules

As far as I can tell, rebuilding Node-Red via the method I outlined above does update all the nodes installed via menu.sh but does not update nodes that I have installed via the Node-Red user interface. I update nodes that I install via the Node-Red user interface via that interface.

I have not tried using the Node-Red user interface to update a node that is installed via menu.sh. I have no idea what would happen and, given the way sqlite gets built by being compiled from scratch, I don't think I want to find out.

For the sake of completeness, if I want to update all the containers except Node-Red then I use:

$ cd ~/IOTstack
$ docker-compose pull
$ docker-compose up -d

That, of course, keeps the stack running and does have a tendency to leave orphans behind. I have never tried the "--remove-orphans" flag. I get rid of them manually with a combination of "docker images" and "docker rmi imageid".

[DIYtechie]

I don't know whether this will help but when I want to rebuild Node-Red, I do this:

$ cd ~/IOTstack
$ docker-compose down
$ docker rmi "iotstack_nodered" "nodered/node-red"
$ docker-compose pull
$ docker-compose up --build -d

Hey @Paraphraser Thank you very much for chipping in. I tried using the above method and after some errors I did not understand, I was able to start fresh with IOTstack. With a fresh install, pre-installing nodes works!

However, when opening the menu.sh again, the pre-installed nodes does not appear as selected in the list, and if I select new ones, they are not added by simply running docker-compose up -d.

Since there is no issue the first time Node-RED is built and since pre-installing nodes is not much easier than installing from within Node-RED, don't feel any pressure to spend time on this. Not important to me, just wanted to share my experience because it seemed not to be working at all.

[Paraphraser]

when opening the menu.sh again, the pre-installed nodes does not appear as selected in the list

No. This has always been the case. I have to re-select SQLite every time I rebuild Node-Red. Should it remember user selections? Seems like a good idea that it should. Does it remember? If SQLite is any guide, “no”. Could this be an enhancement? I’ll leave that to Maestro Slyke.

and if I select new ones, they are not added by simply running docker-compose up -d

True. That’s the expected behaviour, because of the way Node-Red is built. In order to get a new node propagated INTO the container, you have to force a complete rebuild. That’s why I explained about also adding nodes via the Node-Red user interface (“manage palette”).

Anything you add via “manage palette” will persist across rebuilds (ie you don’t have to remember to add every node you need each time you run menu.sh). The price for this is you needing to revisit “manage palette” periodically to check for and click “update” on anything that needs updating. That can also involve needing to restart the Node-Red container, which implies needing to re-connect with Node-Red via your browser before getting back into “manage palette” to do the next one if multiple nodes need updating. So, yes, it can get slightly messy but my view is that it’s actually less messy and involves a lot less downtime overall than a full Node-Red rebuild.

My rule of thumb is, if I can install a node via “manage palette” then that’s what I will do. If and only if a node must be installed via menu.sh will I do it that way (and SQLite is the only example I’ve found).

I hope that all makes sense.

[Paraphraser]

And perhaps I should use a bit more emphasis. Once a Node-Red container has been built, it becomes a “sealed unit”. The ONLY way to get a new node type installed via menu.sh is to completely rebuild the Node-Red container.

[DIYtechie]

As far as I can tell, nodes installed via menu.sh do not "appear" in the mapped area of:

~/IOTstack/volumes/nodered/data/node_modules

I surmise this because, using sqlite as an example, it does not appear at the above path but it does appear when you open a shell into the container and examine:

/usr/src/node-red/node_modules

Would it be possible for "pre-installed" and "manage-palette-installed" nodes to behave in the same way (being backed up in the mapped volume)? That way it would be easier to make a clean Node-RED install and also to backup the full configuration.

I also use the manage palette to install most of my nodes, but I always liked the pre-install feature, because once you are done waiting for the containers to be built, you are ready to go.

I do not get, however, why influxdb is selected by default. I would think the best default option was all nodes not selected - or are some nodes not possible to install via "manage palette" later?

Again - not important to me, just some ideas :)

[Paraphraser]

It would probably be better if Graham Garner could answer this question but I think (as opposed to know) that InfluxDB is installed by default because Graham’s starting point was Peter Scargill’s script. I don’t know when Graham actually started work but “second half of 2019“ is probably a good approximation and, at that time, Peter’s script was using InfluxDB and Grafana. If you’re going to install InfluxDB (the container) by default then it probably makes sense to also install the supporting nodes. Maybe, as @Slyke rewrites the menu system in Python, the idea of cross-dependencies could be adopted (eg de-select InfluxDB the container implies de-select of InfluxDB the node, and vice versa).

On the topic of wouldn’t it be nice if all Node-Red nodes persisted in the mapped area where they could be backed-up, restored and everything started up again quickly - couldn’t agree more. I also reckon it would be great if a node added manually through the GUI automatically got updated whenever Node-Red (the container) was rebuilt. That would give you an easy way to bring everything up-to-date and avoid the need to remember, “I installed X via the GUI so I have to check periodically to see if it needs updating via the GUI, and then wear the container restarts that are implied when you do that“.

[DIYtechie]

From a maintenance and avoiding-unexpected-behavior standpoint the best option would perhaps be to simply get rid of the node pre-installation menu entirely. That way all the previously installed nodes would be saved in the volumes folder and be easy to delete or restore. No need to completely rebuild the Node-RED container and no confusion like the one giving rise to this issue. Just an idea.

[Slyke]

Looks like NodeRed is gonna require a deep dive.

The first iteration of the menu will have basic stuff like checking for port conflicts (this is already working, but need to refactor to make it generic and easier to understand for others), and if (optionally) volumes exist. Due to the complexity of the project, each container must have its own separate validations and configuration. Many will have the same, but not all. Due to this, there will basically be a Python script needed in each template. If the script doesn't exist, no checks will be run for that template (but others will still be able to detect port conflicts etc). I'll be making it for all current templates in the project, so there should be tonnes of examples for others if they want to add their own templates (with checks) in the future.

Having a build script and check for each template also allows things like NodeRed to have its own magic, without making other templates have this unnecessary logic affecting them.

[DIYtechie]

Having done a lot of re-installs lately for various reasons, I am pretty sure that the best option would be to have as few pre-configured nodes as possible (despite the fact that I added deCONZ). If none of the nodes have extra dependencies that make it necessary to use the pre-installer, then it should be considered to get rid of the pre-installer all together.

Not having to remember which nodes were pre-installed and not having to check the correct ones on the list just makes it easier to restore a backup without issues. You are of course free to decide what you believe is best, but I think the benefit of having the option to pre-configure nodes is outweighed by the disadvantages. Just something to reconsider :)

[lazyzero]

To overcome the problems remembering which dependencies to click in the menu and coping flows. I switched to use Node-red Projects. I just enable it in the settings.js. Then I am able to load my flows from git(hub) with all the dependencies and flows.
And of cause I can also easily switch the flows of node-red and keep my flows in git up to date.

That makes re-deployment nice and easy.

Best Christian

[DIYtechie]

Thanks for sharing this tip. Sounds interesting. Will definitely look into that. The only caveat I can think of is that Node-RED will be a special case instead of simply part of the complete IOTstack backup, that you want to set up anyway. But nonetheless, it sounds interesting and worth a look. Den tor. 4. jun. 2020 kl. 08.27 skrev Christian Moll < notifications@github.com>:

…

To overcome the problems remembering which dependencies to click in the menu and coping flows. I switched to use Node-red Projects <https://nodered.org/docs/user-guide/projects/>. I just enable it in the settings.js. Then I am able to load my flows from git(hub) with all the dependencies and flows. And of cause I can also easily switch the flows of node-red and keep my flows in git up to date. That makes re-deployment nice and easy. Best Christian — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4BYRMY3YXRJZRRIYWMLMLRU45E3ANCNFSM4MZDSJLA> .

[Paraphraser]

Should probably be considered stale.