Docker image, automatic builds.

[shinebayar-g]

It would be nice to have docker image ready.

I'll try to build docker image for the project and make it build automatically for every commit that passes test (do we have one?).

We've production ready drone setup that we use daily. If you don't mind I'd use that.

[bugy]

Hi, thanks, nice idea. Where this image will be pushed? I don't know much about sharing docker images
Regarding the build tool - at the moment the whole process is in Travis CI: python and js tests, js build, whole package build

[shinebayar-g]

I'll take a look at Travis-ci configuration if possible. Since it's an open source project, travis ci should work. Also since it's an open source project, our docker image should be hosted on docker hub. I want you to make an account on docker hub, create organization and invite me. My docker username is shinebayar. Then our docker image will be available as under that organization's repository.

…

On 01:32, Fri, Jan 25, 2019 Iaroslav Shepilov ***@***.*** wrote: Hi, thanks, nice idea. Where this image will be pushed? I don't know much about sharing docker images Regarding the build tool - at the moment the whole process is in Travis CI: python and js tests, js build, whole package build — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#171 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC8sZgXWOpt7nOp-lisucQthQGgcA2m5ks5vGe46gaJpZM4aRW5o> .

[bugy]

Hi @shinebayar-g after some struggling I created an organization on docker hub and added you to a team. I believe I also have to create a repository and grant you permissions to it, right?

[shinebayar-g]

Hi, I just checked docker repository. Looks like I don't have permission to create repo. Please create repository and add me. I'll test permissions to push, tag and delete.

[bugy]

Added. Also I added automatic builds there: so whenever a commit to master happens, it will build :dev image based on Dockerfile. So there is no need to update any CI tool, just add a Dockerfile

[shinebayar-g]

Great.

[bugy]

Now images are pushed. Still to be done:

• remove dev tagging on before_deploy for non-master branches
• added versioning for stable docker images

[bugy]

Added pushing docker image for stable releases as well + added incremental versioning to them
Also changed base image to slim

To be done:

• fix GitHub release, when merge with fast-forward (in this case dev release overrides stable release)
• add ":major.minor" tag (without .patch) for stable images, e.g. script-server:1.13

[bugy]

Description how to use it

(credits go to @shinebayar-g)

Users should mount config files and scripts into docker image. conf.json file , runners folder , scripts folder.

Example usage:

docker run -d -p 5000:5000 \
           -v /host/path/to/conf.json:/app/conf/conf.json \
           -v /host/path/to/runners/:/app/conf/runners \
           -v /host/path/to/scripts/:/app/scripts/ \
           --name script-server \
           bugy/script-server:latest

Example runner:

$ cat /host/path/to/runners/sample.json
{
  "name": "sample script",
  "description": "sample script description",
  "script_path": "/app/scripts/sample.sh"
}

Example script:

$ cat /host/path/to/scripts/sample.sh
#!/bin/bash
echo "Hello sample script"

Notes: Script-server users should know it cannot access host system processes as it's nature of docker container. If script-server must has access to host system processes then docker image is not suited for this kind of task. (But not saying it's impossible)

[tristanlatr]

Hi,

You forgot to install the required ldap3 python module in the docker image I think.

$ docker logs script-server
2019-04-11 18:47:27,461 [migrations.INFO] Applying migration add_execution_info_to_log_files
2019-04-11 18:47:27,462 [migrations.INFO] Applying migration introduce_access_config
2019-04-11 18:47:27,474 [migrations.INFO] Applying migration add_user_id_to_log_files
2019-04-11 18:47:27,475 [migrations.INFO] Applying migration migrate_output_files_parameters_substitution
Traceback (most recent call last):
  File "launcher.py", line 11, in <module>
    main.main()
  File "src/main.py", line 72, in main
    server_config = server_conf.from_json(SERVER_CONF_PATH, TEMP_FOLDER)
  File "src/model/server_conf.py", line 105, in from_json
    config.authenticator = create_authenticator(auth_config, temp_folder)
  File "src/model/server_conf.py", line 141, in create_authenticator
    from auth.auth_ldap import LdapAuthenticator
  File "src/auth/auth_ldap.py", line 6, in <module>
    from ldap3 import Connection, SIMPLE
ModuleNotFoundError: No module named 'ldap3'

[bugy]

Hi @tristanlatr, thanks for reporting! The thing is, that it's not a required dependency: people can use Script server without auth at all, or with google oauth. And to make the image small, I ignored all optional dependencies

However, I'm thinking about making a "fat" image, in addition to a minimal one. With all the optional dependencies + some linux utilities out of the box. But that's not coming in the nearest week.

[tristanlatr]

Hi, @bugy ,

I see, right..
What's the simplest way to manually install ldap3 module into the docker image afterward then ?
Sorry if it's a stupid question, not very familiar with docker yet.

Thanks,

[bugy]

Hi @tristanlatr, I believe you will have to build an own image:
Create a Dockerfile with

FROM bugy/script-server:latest
RUN pip install ldap3

Then docker build -t my-script-server .

And after that you would be able to run the command above, but instead use your image name (my-script-server in this example)

[MadDud]

In addition to "Description how to use it" for docker installation, you have to create a conf.json with the following content:

{
  "port": 5000,
  "title": "My Script Server",
  "address": "<docker container ip>"
}

You can find IP of your docker container with this command: docker inspect script-server | grep IPAddress, where "script-server" is your container name.

Without this config content the docker wasn't starting up for me.

[bugy]

Hi @MadDud thanks for the info!
For me docker was starting and working fine without any problems. What errors did you have?

PS having explicit port 5000 is not needed

[MadDud]

Hi @bugy, great job on the script. I'm loving it.
I created a conf.json file, and it's empty.
My error message:

Traceback (most recent call last):
  File "launcher.py", line 11, in <module>
    main.main()
  File "src/main.py", line 74, in main
    server_config = server_conf.from_json(SERVER_CONF_PATH, TEMP_FOLDER)
  File "src/model/server_conf.py", line 60, in from_json
    json_object = json.loads(file_content)
  File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Edit: i tested few things and a conf.json file with only {} is enough to start docker.

[bugy]

Yep, empty file is not a valid JSON :)

[MadDud]

@bugy Could you correct the https://hub.docker.com/r/bugy/script-server page, which says:

/host/path/to/conf.json - this should be your server configuration file. Can be empty

[bugy]

Oops, thank for pointing that out!

[bugy]

Since the task is done, I'm closing it. If anyone has questions/problems with running the docker, please create a new git issue.

@aschodinnadcat I moved you comment to a separate issue #228

[shr207-iMerit]

Hey @bugy
I created a new issue for one of my docker problems. Linked it here, please let me know if you need me to create the issue here.
Link to issue:

[hiteshkumarthakur]

I had write a medium article for the installation & usage.
https://medium.com/@hitesh.thakur/script-server-gui-for-scripts-how-to-install-use-it-via-docker-51774d35f6a9

[bugy]

oh nice, thank ou @hiteshkumarthakur !

[SaberStrat]

Hi @tristanlatr, thanks for reporting! The thing is, that it's not a required dependency: people can use Script server without auth at all, or with google oauth. And to make the image small, I ignored all optional dependencies

However, I'm thinking about making a "fat" image, in addition to a minimal one. With all the optional dependencies + some linux utilities out of the box. But that's not coming in the nearest week.

I too was semi-expecting the ldap3 package to be included, since it seemed supported out of the box...and semi-expecting it to be missing since missing packages are a classic Python problem.

I don't think required components for features an app ships with - even if not enabled by default - should be left out of an image. Sure, you can install it in the container (assuming Firewall rules don't make that a hassle) or create a custom image. And it's perfectly normal to expect to install requirements when running script-server 'bare metal'. But with images it's something else imo. Might be a matter of taste, what images should come with - especially where Python is involved and its modular nature. But from experience, unless otherwise noted in the image name or tag, I expect images to be complete for all supported features. If not, then the image is more of a thin or slim image, rather than a full featured image.

My two cents on whether to optimize image sizes or not :)

How many more packages/bytes would need to be included in a full featured image?

[bugy]

Hi @SaberStrat it makes sense :) Either current image can be called slim or there could be another image like all-deps, not a big difference as for me.
I moved it to a separate ticket #637, because I forgot about this already.

[hulleyrob]

would it be possible to add maybe something to the admin page where we can pip install stuff?

otherwise is the only way to run python scripts that require external modules to create a bespoke image?

[MNeill73]

Unless you're running the Script server container, you should be able to just install them at the shell. Since pip no longer has built-in search functionality, it would have to either run as admin, which scriptserver probably isn't, or it would run as the user scriptserver runs as. And in order to develop the script you're running in the first place, you'd have had to already installed the modules before you configured scriptserver to run the script through the GUI. If you really want to pip install things from the GUI, you can probably write a script in and of itself that wraps pip. Or just configure scriptserver to run pip with your preferred arguments.

…

On Tue, Sep 12, 2023, 11:18 AM Rob Hulley ***@***.***> wrote: would it be possible to add maybe something to the admin page where we can pip install stuff? otherwise is the only way to run python scripts that require external modules to create a bespoke image? — Reply to this email directly, view it on GitHub <#171 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3HBI3UIVUOXZG6GXQCFLQLX2B4NVANCNFSM4GSFNZUA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[hulleyrob]

Yeah I'm running it in docker. If running pip from a script works I can work with that for some reason I just thought it wouldn't.

In the meantime I've got a dockefile working with compose to pip install my requirements file during creation but I don't really want to recreate the container every time a script needs a module.

[MNeill73]

You'd probably have to try execing into your container to see if you can install pip and run it from the CLI, but if you can, writing a wrapper script is probably the easiest, if not just invoking pip directly. A wrapper could be smart enough to go parse the includes in your scripts configured under scriptserver (parse the runners directory to find tany invoked python scripts, then grep out includes in those scripts). Then you could just use a single no-parameters script to "go pip-install any includes from any python script I'm running in the container".

…

On Tue, Sep 12, 2023, 5:12 PM Rob Hulley ***@***.***> wrote: Yeah I'm running it in docker. If running pip from a script works I can work with that for some reason I just thought it wouldn't. In the meantime I've got a dockefile working with compose to pip install my requirements file during creation but I don't really want to recreate the container every time a script needs a module. — Reply to this email directly, view it on GitHub <#171 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3HBI3XEYYQBM6ZACI44FP3X2DF4XANCNFSM4GSFNZUA> . You are receiving this because you commented.Message ID: ***@***.***>

[hulleyrob]

Yeah shelling in and doing it works fine that was the first thing I tried just seemed like I was missing something obvious and easier.

[gtuxyco]

Are there plans to make images for ARM?
I was trying to use in a raspberry pi, but after a while I noticed the docker image is amd64

[dreness]

Are there plans to make images for ARM? I was trying to use in a raspberry pi, but after a while I noticed the docker image is amd64

There is an arm64 image currently published under a 'dev' tag, which is built from master - so perhaps not yet 'stable'.
https://hub.docker.com/r/bugy/script-server/tags

[peterpepo]

I would like to run following script.
Do I have to run directly on host, or can I spin up docker image?

jdownloader_enable.sh

sudo systemctl daemon-reload
sudo systemctl start jdownloader.service
sudo systemctl enable jdownloader.service

[ciscoivan]

I'm having some issues with deployment

[root@localhost conf.json]# docker logs bcaeada23707
2024-02-29 03:28:08,088 [root.INFO] Starting Script Server, v1.18.0
2024-02-29 03:28:08,090 [migrations.INFO] Applying migration add_execution_info_to_log_files
2024-02-29 03:28:08,090 [migrations.INFO] Applying migration introduce_access_config
Traceback (most recent call last):
File "/app/launcher.py", line 11, in
main.main()
File "/app/src/main.py", line 86, in main
migrations.migrate.migrate(TEMP_FOLDER, CONFIG_FOLDER, SERVER_CONF_PATH, LOG_FOLDER)
File "/app/src/migrations/migrate.py", line 388, in migrate
migration_descriptor.callable(context)
File "/app/src/migrations/migrate.py", line 210, in __introduce_access_config
content = file_utils.read_file(file_path)
File "/app/src/utils/file_utils.py", line 63, in read_file
with open(path, mode, newline=newline) as f:
IsADirectoryError: [Errno 21] Is a directory: '/app/conf/conf.json'
[root@localhost conf.json]#