add OpenFOAM test case: rotating cylinder

[mschwarzmeier]

Implements OpenFOAMs rotating cylinder tutorial.

---

Open points:

• combine with snakemake
• make it a parameter study (mesh)
  • without pyFoam
  • add postprocessing
• add documentation

---

closes #12

[srosenbu]

Thank you for implementing this:)

Ideally we would now need a parameter file (.json) and a script that inserts the parameters from the json into the appropriate files for OpenFOAM.

[srosenbu]

I guess this a file that can be replaced by the Snakemake file.

[mschwarzmeier]

The Allrun-script is established in OpenFOAM to run simulations, often containing 5 or more steps, which need to be executed (calling applications, moving/deleting files, sometimes receiving input parameters from the CLI). This is the way.

As discussed in the previous comments, we are going to remove the post-processing from this.

[srosenbu]

Is this the file that we would need to change in order to set the mesh-density?

[mschwarzmeier]

Yes

[joergfunger]

How is the planned procedure to this merge request? There are some details missing, including a short documentation, a final integration into the workflows (nextflow and snakemake) as well as an extraction of relevant metrices to compare the performance into a metrics.json file (together with a class that actually provides the analytical solution). @mschwarzmeier or @tmaric do you plan on working on this any further, or should we close the merge request and the corresponding issue?

[mschwarzmeier]

There is one open question, which is whether snakemake or nextflow needs to be used. Have you reached a conclusion in that regard, @joergfunger?
After that, the next step will be to create a parameter study (as mentioned in the to-do list above) without pyFoam. Having looked briefly into what @srosenbu has set up for kratos with snakemake, it seems that the workflow/set-up with snakemake might look very different to its pyFoam equivalent.
In case nextflow should be used, we would need to look into that.

[joergfunger]

@mschwarzmeier thanks for the fast response. We have modified the snakemake and nextflow workflows, so they now use exactly the same structure, i.e. if you have written a snakemake workflow, I guess the transformation into the corresponding nextflow equivalent is relatively straightforward. I will try to shortly describe the current approach, and maybe that than helps to identify/understand the challenges you are facing. We have first defined a simulation problem that has a certain number of parameters, these could be changes in the geometry, mesh, constitutive parameters, bc ... These parameters are stored in a parameter file. Then, there are two options to run the (single simulation) - either the parameter file is passed to a python function as we do in fenics and then use these parameters directly - this is possible if your input is based on a programming language such as python. The other option, as we have done with kratos, is to prepare a a simulation template (e.g. here for kratos that has at all positions, where a parameter is to be changed as placeholder (marked with {{}}). The first step would then be replacing in the template the parameters of the parameter file, then run the simulation as an external tool, and then collect the data into both a metrics file that includes all the relevant output metrics used in a comparison afterwards, usually scalars or vectors and a complete result files are also stored in a zip file (solution_field_data_*), though there is (at least currently) no defined general structure in that zip file. This would then e.g. include result files as vtk. So you would have to define one input file for openfoam, define what the parameters are that you want to make flexible, define these with placeholders and then replace them when running the simulation. We have made the workflows hierarchical, such that some parts (such as mesh generation or generation of the summary file) are done on the top level, and then there are individual steps for each software tool Kratos Snakefile that are included in the top level file. So if you use snakemake, you would essentially have to copy the Snakefiles at the top level and one on the tool level, define the parameter files, adapt the generate_config file and then call the corresponding scripts in the rules. If mesh generation is not to be automated, just provide the meshes in a standard format and then use those as input files rather then generating them in an automatic rule beforehand.

[mschwarzmeier]

Hi @joergfunger,
I have now set up the case as a parameter study for the mesh. As there is only this one software (as of now) running this benchmark, there is no need to have a top level and tool level workflow files. Thus, the file is still rather easily understandable, IMHO.
Next step might be to define the benchmark'ed output measure, then to set up the post-processing accordingly.

[joergfunger]

@mschwarzmeier thanks for writing the snakefile. This is a very good starting point. There are a couple of things that I would suggest:

• I would suggest looping over the parameter files (they could directly incorporate the mesh information as a parameter), but this way it is much easier to document at the end in the summary rule all the inputs and outputs in exactly the same way in all Snakefiles, and that then makes looking into the data once generated much easier (because all the information is in the parameter files - in all benchmarks).
• We would then also need a summary file that outputs both the relevant metrics and the paraview/vtk, .. outputs together with the intermediate steps, here there is still no standard, so a zip file per Parameter file would be great. I would suggest copying the structure from the other benchmark, because at the next iteration I would like to make this summary on another higher level of the hierachy such that we do not have to copy paste biolerplate code (it is the same in all benchmarks)
• As far as I see you implementation, you generate the input file based on a template where you then via sed replace the mesh parameter. This is exactly what we do in Kratos

  NFDI4IngModelValidationPlatform/benchmarks/linear-elastic-plate-with-hole/kratos/create_kratos_input.py

  Line 72 in 607fba1

  with open(kratos_input_template_file) as f:

  ), but in a separate step. This makes it more transparent and in particular allows to store the intermediate files (the system tracks inputs and outputs of all rules). There is no need to generate a script, I would suggest just making this two steps. When there is an error in the compute task, the replacement task is still finished and the correct input file is already stored (and can in a testing setup be used). Another potential challenge is related to how you copy the files. In theory, this is what snakemake is to handle, in particular if you move the execution of the actual job to another cluster. In this case snakemake is taking care of copying all the inputs to the new server, but that requires that all inputs (including what you have copied from Allrun) is actually defined as input.
• in the github actions, the workflow should be executed rather than directly starting the openfoam jobs in the container (there is IMO no need to run the actions in a container, since that container should be executed in snakemake).
  Let me know what you think about these suggestions