Restructure & handling of the shakyground grid files

[nbrinckm]

This pull request adds the handling of the shakyground grid files & restructures the existing script

The way to handle the grid files is relativly simple:

• We checkout the repo for the grid
• We build it locally
• We checkout shakyground
• And we change the Dockerfile for shakyground, so that we use the image with the grids that we build ourselves.

For the rest, the PR is mainly a bit restructing:

• I have a misses_image function, so that I can test easily with docker image inspect if the image is already there (more robust then having a list & grepping for one entry)
• I structured the build steps in multiple functions
• I added a main function that can be used to run only some parts of the build script (for easier debugging & docker file problem fighting)
• I also added a clean function, so that we can test that all the stuff still works without any effects of our previous steps.

I'm sorry that the restructuring is that large, but I hope I'm not the only one who can see that as at least a bit beneficial.

Things still to do (in other pull requests):

• Building the tsunami service (currently it has an error that it doesn't find some file that it needs) - and the data of cause.
• Building the sysrel service
• Preparing our wps server should be done in an init container within the docker compose (similar to https://github.com/riesgos/async/blob/main/wps/init/init_wps.py). I guess this is what I will work on for the next days (in the hope that this PR here is okayish).

Sorry that it took so much time of the day to reach that stage, but my bash-fu is really limited & the problems I had with the shakyground images were super annoying (not super complicated, but error messages after it downloaded the stuff, so it took way to long to try several things out).

[nbrinckm]

I also think I will have another check with shellCheck. But I will do this tomorrow.

[nbrinckm]

By the way, I changed the name of the script, as I think something like ./build.sh assetmaster or ./build all is better then ./build_all.sh assetmaster.

Still it is possible to just run the script without any parameters (./build.sh) and it will do all the build steps.
(Still feels a little bit like writing half of a Makefile - but (I hope) still a little bit better to handle).

[MichaelLangbein]

Loving it! Nothing to add here. Can be merged!

[MichaelLangbein]

• Building the tsunami service (currently it has an error that it doesn't find some file that it needs) - and the data of cause.

• Building the sysrel service

These are really up to @bpross-52n and @arnevogt - so they shouldn't keep us from merging this.

[MichaelLangbein]

• Preparing our wps server should be done in an init container within the docker compose (similar to https://github.com/riesgos/async/blob/main/wps/init/init_wps.py). I guess this is what I will work on for the next days (in the hope that this PR here is okayish).

Sounds good to me!
For the frontend I'll have a docker-compose file, too, which contains services for middleware, monitor, frontend, and a new version of the frontend. I guess a good part of our work will be in figuring out how to get multiple compose-files to play nicely together - but that's another issue, too.

[nbrinckm]

Unfortunally I can't merge myself (missing permissions). So @MichaelLangbein please merge it today.

[nbrinckm]

One last thing: I did just a little brainstorming regarding shell scripts vs something similar to docker-compose files.

First: I'm not aware if it is actually possible to do all this with docker compose - I just use the subset that I need on a daily basis.

However, if I would introduce configurations similar to the way the docker-compose yml files look like I would endup with a subset like this:

services:
  quakeledger:
    image: "gfzriesgos/quakeledger"
    before_script:
      - "git clone https://github.com/gfzriesgos/quakeledger"
    build:
      context: "quakeledger"
      dockerfile: "metadata/Dockerfile"
      tags:
        - "latest"
    artifacts:
      paths:
        - "metadata/quakeledger.json"
      expose_in: "configs"

  deus:
    image: "gfzriesgos/deus"
    before_script:
      - "git clone https://github.com/gfzriesgos/deus"
    build:
      context: "deus"
      dockerfile: "metadata/Dockerfile"
      tags:
        - "latest"
    artifacts:
      paths:
        - "metadata/deus.json"
        - "metadata/volcanus.json"
        - "metadata/neptunus.json"
      expose_in: "configs"

  shakyground_grid:
    image: "gfzriesgos/shakyground-grid-file"
    before_script:
      - "git clone https://github.com/gfzriesgos/shakyground-grid-file"
    build:
      context: "shakyground-grid-file"
      dockerfile: "Dockerfile"
      tags:
        - "latest"

  shakyground:
    image: "gfzriesgos/shakyground"
    depends_on: shakyground_grid
    before_script:
      - "git clone https://github.com/gfzriesgos/shakyground"
      - "sed -i -e 's/FROM gfzriesgos\/shakyground-grid-file:20211011/FROM gfzriesgos\/skakyground-grid-file:latest/' ./metadata/Dockerfile"
    build:
      context: "shakyground"
      dockerfile: "Dockerfile"
      tags:
        - "latest"
    artifacts:
      paths:
        - "metadata/shakyground.json"
      expose_in: "configs"

While I think this is easier to add/modify for the project partners, I'm not 100% sure if a tool that would be needed to actually do that actions would be reused. (When I currently do something like this, I adjust the dockerfiles - and don't have some before_script, after_script (which I had while brainstorming) or arctifacts.

[nbrinckm]

By the way: I'm not really a fan of yaml - so I would do that only if it would be really beneficial to do.

[nbrinckm]

But if we would go that way overall, we could also set all of that up as an gitlab-ci.yml file too (or maybe similar stuff for the github actions).

[MichaelLangbein]

While I think this is easier to add/modify for the project partners, I'm not 100% sure if a tool that would be needed to actually do that actions would be reused. (When I currently do something like this, I adjust the dockerfiles - and don't have some before_script, after_script (which I had while brainstorming) or arctifacts.

I think that would be pretty nifty, too!
But my understanding is that docker doesn't want to support before_script functionality - possibly because that would open the door to a lot of unpredictable behavior.

Actually, this topic is too interesting to leave in this thread - let's discuss this on the next standup or in mattermost!