Add possibility to dynamically include script parameter section

[bugy]

There should be a possibility to add an "include" section to a script configuration, which would add/fill parameters configuration from another file.
This "include" section can depend on current parameter values, so it will be loaded dynamically while user selects values on UI

There are 2 use cases behind:

1. Sharing parameters in different configurations
2. Depending on some parameter, we could include different corresponding additional parameters

For more details and reasoning please read comments in #119

[bugy]

@yosefy, I've completed this feature. There is a new field include in script configuration section. This could be a static path (e.g. '/home/myuser/included.json') or a dynamic path ('/home/myuser/${parameter name}.json').
If the path is relative, then it will be relative to conf/runners folder.

Included configurations have the same structure as normal configurations. Though only parameters section is supported at the moment (other fields are undocumented/unsupported, so please don't use them).

To avoid loading included files as a standalone configurations, there is a possibility to exclude them with a new field hidden. Any hidden configurations will not be shown to a user and can only be included.

If both base configuration and included configuraition have a parameter with the same name, then included parameter will be ignored.

Example included file (see parameterized.json and included.json from samples for more details):

{
  "hidden": true,
  "parameters": [
    {
      "name": "inc_param1",
      "type": "int"
    },
    {
      "name": "inc_param2",
      "no_value": "true",
      "param": "--inc2"
    }
  ]
}

To support this I had to completely reimplement communication layer when loading script config and starting execution, so it could cause some bugs :(

[yosefy]

thanks a lot!

…

On Sat, Oct 27, 2018 at 12:27 AM Iaroslav Shepilov ***@***.***> wrote: @yosefy <https://github.com/yosefy>, I've completed this feature. There is a new field include in script configuration section. This could be a static path (e.g. '/home/myuser/included.json') or a dynamic path ('/home/myuser/${parameter name}.json'). If the path is relative, then it will be relative to conf/runners folder. Included configurations have the same structure as normal configurations. Though only parameters section is supported at the moment (other fields are undocumented/unsupported, so please don't use them). To avoid loading included files as a standalone configurations, there is a possibility to exclude them with a new field hidden. Any hidden configurations will not be shown to a user and can only be included. If both base configuration and included configuraition have a parameter with the same name, then included parameter will be ignored. Example included file (see parameterized.json and included.json from samples <https://github.com/bugy/script-server/tree/master/samples/configs> for more details): { "hidden": true, "parameters": [ { "name": "inc_param1", "type": "int" }, { "name": "inc_param2", "no_value": "true", "param": "--inc2" } ] } To support this I had to completely reimplement communication layer when loading script config and starting execution, so it could cause some bugs :( — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMVy7dGnXADDwlL0_gYfIiQ_9F3vKW9zks5uo34rgaJpZM4W7c_G> .

[yosefy]

sorry just a question, so i cant use values from included file in script?
so how to use it?

[bugy]

Hi, what do you mean?

[yosefy]

i dont see values form included when echo $@

…

On Mon, Feb 25, 2019 at 2:59 PM Iaroslav Shepilov ***@***.***> wrote: Hi, what do you mean? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMVy7QKpAysd1HPrePzHeWqPvQ260E7wks5vQ948gaJpZM4W7c_G> .

[bugy]

That's extremely strange. Do you see these values on UI? And in the log, before script is called? (or after, in the admin page)

[yosefy]

{
"hidden": false,
"parameters": [
{
"name": "inc_param1",
"type": "list",
"values": [
"aaa",
"bbb"
]
},
{
"name": "inc_param2",
"no_value": "true",
"param": "--inc2"
}
]
}

script is:

args=("$@")
echo $@

[bugy]

Ok, apparently it doesn't work at all :( Will fix it

[yosefy]

@bugy , do you have some estimation of fixing it
blocking from putting in prod :)

thanks

[bugy]

Hi, I'm working on another ticket at the moment (#184), and would need 2-3 days there probably.
The fix here will take ~1 day.

So if it's urgent for you, I could work on it today, otherwise estimation is Friday

[yosefy]

wow if it possible, would be really helpful as i am actually waiting for it now to continue working with script-server thanks

…

On Tue, Feb 26, 2019 at 12:42 PM Iaroslav Shepilov ***@***.***> wrote: Reopened #133 <#133>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMVy7VrfIc0w2AamuYtmDQMxkdibpby9ks5vRQ-jgaJpZM4W7c_G> .

[bugy]

Ok, agreed

[bugy]

Fixed. Should be available in the latest stable release (1.13.2) and dev

[yosefy]

i don't know if it is known but default values don't work in included scripts not a big problem just updating the rest work beautifully thanks

…

On Tue, Feb 26, 2019 at 11:33 PM Iaroslav Shepilov ***@***.***> wrote: Fixed. Should be available in the latest stable release (1.13.2) and dev — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMVy7fP1d8qIGeoOEyj5cSplcKNC5ooSks5vRagagaJpZM4W7c_G> .

[bugy]

Thanks for double-checking and reporting the issue with default values. I moved it to a separate ticket #201

[kbza]

Hello there!
Is there any way of including more than one external included file for script? thinking of one of the following:

• Multiple include directives (I think it only works the latest)
• The possibility of an include directive with array as a type.
• Another include on the included file.

I'm trying to figure out a way of having configuration files that will be the same across installations, some specific to each one and others specific to the servers.
Is there any way of achiving this in the current version? will you consider implementing something like this in the future?

Thanks in advance.

[bugy]

Hi @kbza, I'm afraid it's not possible at the moment, but would be more or less straightforward to implement. Do you want to try and submit a PR?

[kbza]

I don't think I'm confident enough... Many years as a sysadmin only making just some bash scripts, so my coding skills are a bit rusty. Also, I've never done any serious project with python. Nonetheless, I'm gonna give it a try, but I would like you to consider a "plan B" :P.

Anyway, which one of the two last options do you think is the best? I'm thinking more of a "chain" of includes as a better way of structuring... but, on the other hand, the array way could work with multiselect or script/command that finds more than one config file matching a pattern ("script": "ls /path/to/web*_server.json" -> will include web1_server.json, web2_server.json... )

[bugy]

I thought about replacing a single string with an array of strings, but i don't fully know your use case and it might be not enough How dynamic are these values? kbza <[email protected]> schrieb am Mo., 31. Aug. 2020, 19:45:

…

I don't think I'm confident enough... Many years as a sysadmin only making just some bash scripts, so my coding skills are a bit rusty. Also, I've never done any serious project with python. Nonetheless, I'm gonna give it a try, but I would like you to consider a "plan B" :P. Anyway, which one of the two last options do you think is the best? I'm thinking more of a "chain" of includes as a better way of structuring... but, on the other hand, the array way could work with multiselect or script/command that finds more than one config file matching a pattern ("script": "ls /path/to/web*_server.json" -> will include web1_server.json, web2_server.json... ) — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#133 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJXPJNGDGELQC2J7UD43QLSDPOTBANCNFSM4FXNZ7DA> .

[kbza]

This is more or less what I have atm:

{
  "name": "Param Test",
  "script_path": "./scripts/paramTest.sh",
  "include": "./platform_files/${Server Select}",
  "parameters": [
    {
      "name": "Server Type",
      "type": "multiselect",
      "values": [ "app", "db", "web", "lts" ],
      "separator": "|"
    },
    {
      "name": "Server Select",
      "type": "multiselect",
      "values": { "script": "find ./platform_files/ -regex '.*[${Server Type}]+[0-9]+_server.json' -type f -printf \"%f\n\" "}
    }
  ]
}

but only accepts one include file. So something like:

{
  "name": "Param Test",
  "script_path": "./scripts/paramTest.sh",
  "include": [ ${Server Select} ]
  "parameters": [
    {
      "name": "Server Type",
      "type": "multiselect",
      "values": [ "app", "db", "web", "lts" ],
      "separator": "|"
    },
    {
      "name": "Server Select",
      "type": "multiselect",
      "values": { "script": "find ./platform_files/ -regex '.*[${Server Type}]+[0-9]+_server.json' -type f -printf \" \"%f\"\n\" "}
      "separator": ","
    }
  ]
}

Could work with arrays.

[bugy]

Ah, i see, but for this one, i think, there is no need to change syntax in the configuration file The value of multiselect parameter is already an array, so we can just pick every element from the array and include it What do you think? kbza <[email protected]> schrieb am Mo., 31. Aug. 2020, 21:16:

…

This is more or less what I have atm: { "name": "Param Test", "script_path": "./scripts/paramTest.sh", "include": "./platform_files/${Server Select}", "parameters": [ { "name": "Server Type", "type": "multiselect", "values": [ "app", "db", "web", "lts" ], "separator": "|" }, { "name": "Server Select", "type": "multiselect", "values": { "script": "find ./platform_files/ -regex '.*[${Server Type}]+[0-9]+_server.json' -type f -printf \"%f\n\" "} } ] } but only accepts one include file. So something like: { "name": "Param Test", "script_path": "./scripts/paramTest.sh", "include": [ ${Server Select} ] "parameters": [ { "name": "Server Type", "type": "multiselect", "values": [ "app", "db", "web", "lts" ], "separator": "|" }, { "name": "Server Select", "type": "multiselect", "values": { "script": "find ./platform_files/ -regex '.*[${Server Type}]+[0-9]+_server.json' -type f -printf \" \"%f\"\n\" "} "separator": "," } ] } Could work with arrays. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#133 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJXPJMARPTVK2YNGRK25R3SDPZHZANCNFSM4FXNZ7DA> .

[kbza]

Exactly, that will be possible when include directive accepts an array of values/files. I think this is the easiest aproach, though atm, I have no clue how to do it.