Add --api URL support to agama profile

[mvidner]

Problem

agama profile still executes its actions directly from the CLI client, not using the web service. Fix it to make it work like agama --api $URL config ... already does.

• https://trello.com/c/fqPGL2qe

agama profile command summary... it is a diverse bunch! (related, unplanned stub: https://trello.com/c/3aUupc2F/461-unify-config-and-profile-commands-in-the-cli )

profile_evaluate(jsonnet_path) -> json stdout
profile_validate(json_path) -> print whether valid, what problems
profile_autoyast(autoyast_url) -> json stdout
profile_import(url) -> performs config_load
  # oh fun!
  if url ends with .xml OR .erb OR /
    profile_autoyast
  else download the file and analyze content
    if is_script # starts with hash-bang and the program is not "jsonnet"
      EXECVE it # which means replace current process. OMG WTF inconsistent UX
    else if is_jsonnet
      profile_evaluate
    else if is_json
      cat
    else
      return Error
  # at this point we have a JSON file
  profile_validate # NOTE that invalid profile is just a message and we continue
  agama_config_load

Solution

Added these 3 new web API paths corresponding 1-1 to the CLI commands:

• POST /api/profile/validate ?url= ?path= (or body)
• POST /api/profile/evaluate ?url= ?path= (or body)
• POST /api/profile/autoyast ?url=

But agama profile import is notable that it effectively includes an agama config load which is implemented as multiple backend calls, so import stays at the front end and the back end only gets what's needed to make it work,
which is running a script:

• POST /api/profile/execute_script ?url= ?path= (or body)

(TODO explain the rest)

Backend errors now include their causes

Before:

Anyhow(Backend call failed with status 400 and text '{"error":"Error: Could not evaluate the profile"}')

After:

Anyhow(Backend call failed with status 400 and text '{"error":"Error: Could not evaluate the profile: Failed to read system's hardware information: Failed to read agama.libsonnet: No such file or directory (os error 2)"}')

JSON validation errors are more readable now

We used to include a debug representation of the validation error but reading the manual(!) shows there is a way to just point to the offending piece of JSON

$ cat ...
{  "product": {    "ID": "Tumbleweed"  }}
$ agama profile validate rust/agama-lib/share/examples/profile_tw_invalid.json
The profile is not valid. Please, check the following errors:

Before:

• Additional properties are not allowed ('ID' was unexpected). ValidationError { instance: Object {"ID": String("Tumbleweed")}, kind: AdditionalProperties { unexpected: ["ID"] }, instance_path: JSONPointer([Property("product")]), schema_path: JSONPointer([Keyword("properties"), Property("product"), Keyword("additionalProperties")]) }
• "id" is a required property. ValidationError { instance: Object {"ID": String("Tumbleweed")}, kind: Required { property: String("id") }, instance_path: JSONPointer([Property("product")]), schema_path: JSONPointer([Keyword("properties"), Property("product"), Keyword("required")]) }

After:

• Additional properties are not allowed ('ID' was unexpected). /product
• "id" is a required property. /product

Testing

Unit tests: no

Manual tests

...

Added integration tests expecting a running web service.

Run like (part of .github/workflows/ci-integration-tests.yml)

(cd service/; bundle exec rspec --pattern 'test/integration/**_itest.rb')

• agama profile validate
• agama profile evaluate
• agama profile autoyast ... turns out hard to test with the AY-special URLs as none of them work in my container
• agama profile import
• agama --api ... profile ... hitting an opaque SSL handshake error 😭

Screenshots

(textual screenshots coming up)

Documentation

Documented the extended argument handling:

• autoinstallation/README.md adjusted
• agama profile reference updated for the --api URL related changes agama-project.github.io#70

But we are actually still missing documentation for the --api option itself, except for a mention in the blog. IMHO we should add it as part of the config/profile cleanup.

[mvidner]

First commit, client code is missing, server implements only the validation part.

# save auth token
agama auth login
# try what surely works
curl -k -v -H "Authorization: Bearer $(cat $HOME/.local/agama/token)" https://localhost/api/l10n/config
# try out new code
curl -k -H "Authorization: Bearer $(cat $HOME/.local/agama/token)" 'https://localhost/api/profile/validate?path=/etc/os-release'

[coveralls]

Pull Request Test Coverage Report for Build 13897313808

Details

• 1 of 179 (0.56%) changed or added relevant lines in 8 files are covered.
• No unchanged relevant lines lost coverage.
• Overall first build on agama-profile-apiurl at 72.138%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
rust/agama-server/src/web.rs	0	1	0.0%
rust/xtask/src/main.rs	0	1	0.0%
rust/agama-cli/src/lib.rs	0	2	0.0%
rust/agama-server/src/web/docs/profile.rs	0	6	0.0%
rust/agama-lib/src/profile.rs	0	21	0.0%
rust/agama-server/src/profile/web.rs	0	59	0.0%
rust/agama-cli/src/profile.rs	0	88	0.0%

Totals
Change from base Build 13859864963:	72.1%
Covered Lines:	19939
Relevant Lines:	27640

---

💛 - Coveralls

[imobachgs]

I have reviewed the server part now and we are almost there. IMHO the blocking issues are:

• As you mentioned, the execute_script blocks the whole server.
• The progress reporting will not work anymore when using the CLI from a remote system (as it relied on D-Bus).

If you want, we could work on the second problem on a separate PR because it might imply some work.

[imobachgs]

It is fine, but I don't know whether we need this flexibility. In a POST call, I would go with the request body.

[imobachgs]

And that's precisely why I tend to avoid anyhow. Do not get me wrong: anyhow is great. However, I would prefer plain errors for libraries and internal APIs where you might need to do some matching.

[imobachgs]

JFTR, it could be a good idea to offer a way to upload a local profile from the CLI:

agama profile import --api http://server.lan/api --local import my-profile.jsonnet

But, of course, that's totally out of scope for this PR.

[imobachgs]

As discussed in IRC, I am not sure about this behavior. I would expect to have always the same answer, no matter whether the call takes 50ms or 60ms. I would go for:

• Waiting until it finishes but without blocking the server. That's why I mentioned using tokio::process::Command instead (but I have not tried if it works as expected). We could even move the script to a separate Tokio task.
• Or always returning a 200 (if the script was started) and move the script execution to a separate Tokio task.

We even discussed about having an API to run jobs. You would create a job by calling POST /jobs, which would return a job ID. So you can check the current status with GET /jobs/1. Of course, we could integrate this API with the events system too. But I would say it is out of scope.

[mvidner]

Yay, tokio::process::Command simply works: its Child::wait is async so other requests are served