Adding BMI to Fortran77 code

[lzhu5]

I'm adding BMI to a model written in Fortran 77 and encountered the following issues so far:

1. Fortran 77 does not support Module, but the current BMI Fortran 90 example uses Module.
   Do you have an example for Fortran 77?
2. My Fortran 77 source code includes a lot of comments and continuation lines that follow the fixed-format layout rules.
   How does BMI's Fortran 77 compatibility help make the code change minimal?

Thanks.

[mdpiper]

Hi @lzhu5,

1. The BMI doesn't require modules. I think, though, that in order to call Fortran 77 functions and subroutines from Fortran 90 or later, they'll need interfaces. I haven't done this myself, but this example appears to cover what's needed.
2. The BMI won't change your code--BMI is just a set of functions that call into your existing code. If you can compile your code and make the functions and subroutines it contains available to other Fortran programs, BMI will work with it.

Also, be sure to look at the documentation and use the latest Fortran BMI specification!

[lzhu5]

Hi @mdpiper,

Thanks for your reply.
I understand that BMI is not supposed to change the code, but the Fortran 77 code that I am working on is somehow different. For simplicity, let me use the bmi-fortran code in https://github.com/csdms/bmi-example-fortran/tree/master/bmi_heat as an example.

For instance, eventually, I want to couple "heat.f" with a python code. I guess I need to wrap "bmi_heat.f90" using babelizer with Python bindings, right?
If so, my question is how to make a Fortran 77 code to something similar to bmi_heat.f90, which is a module.

Another concern is that my Fortran 77 code uses common blocks for global variables, and subroutines use common variables. Please see below a brief structure of the Fortran 77 code:

program MyModel
    common a, b, c, d
    initialize a, b, c, d
    compute: call A, call B 
    finalize
end program

subroutine A()
    common a, c
    compute on a, c
    change a
end subroutine A()

subroutine B()
    common b, d
    compute on b, d
    change b
end subroutine B()

After coupling with the python code, I want to receive values of c & d from the python code, and call subroutines A() and B() to do computation at each time step. My understanding is that the initialize/compute/finalize in Fortran code will be subroutines and be wrapped into a python library that I can call directly in python. If not using module, how to share global variables among initialize/compute/finalize subroutines? Would you please suggest?

Below is the flow that I want to achieve.
Step 1 initialize a, b, c, d

(for time loop, repeat Step 2-4)
Step 2 receive c, d from a python code
Step 3 compute: call A(), call B()
Step 4 send a, b to a python code

Step 5 finalize

Thanks for your help.

[mdpiper]

Hi Ling,

I've been thinking about this problem this weekend. It's still a Fortran problem, not a BMI problem. We need to find a way to call F77 code from F03. A module would be nice, but shouldn't be necessary. Also, the common block won't be a problem for the BMI. (The only issue will be that only one instance of the model can be running at a time; otherwise, the common blocks will conflict.)

Mark

[mdpiper]

Another option, if you have permission to modify the model source code, would be to upgrade it from F77 to F90. The advantage of this is there's a clear path from F90 through BMI to pymt and Python.

[lzhu5]

Hi Mark,

Thanks for your reply.
I agree. We definitely want to avoid the single instance limitation caused by Common Blocks.
What Fortran standard is best supported by Babelizer? F90?

Yes. We are able to adapt F77 code to F90 code. Could you please point me to a full F90 example similar to example-heat-c? (I have already tested wrapping GIPL model from the repository with babelizer, but the final 'make install' step for building the Python bindings cannot go through.) Thanks for your help again.

Best,
Ling

[mdpiper]

Hi Ling,

Just to be careful, what I recommend is to update your F77 model code to at least F90, modularizing it and removing common blocks. This will make creating a BMI for it much easier. You should base your BMI on the current Fortran specification in bmi-fortran, using bmi-example-fortran as a guide. Writing the BMI will take some time, and it can be tricky.

The babelizer uses the current Fortran BMI specification--that's why it didn't work for GIPL, which uses an earlier BMI version, v1.2.

Mark

[lzhu5]

Hi Mark,

I have a question about the bmi-example-fortran. I tried to set the value of a variable in type :: heat_model via BMI's set_value() and found that the value would change after calling the "initialize" function.

For instance,
I added a new integer variable "flag" to type :: heat_model. In bmi_main.f90, before calling BMI's initialize(), I set a value to "flag". However, after calling BMI's initialize(), which doesn't access "flag" at all, the value of "flag" changed. This is strange to me.

Also, valgrind detected data loss for bmi-example-fortran. I am wondering whether the data leak is related to the value change. Would you please shed light on this issue? Thanks for your help.

The modified code can be found here.

Best,
Ling

[mdpiper]

Hi Ling,

This is by design--model variables that are exposed through the BMI (we sometimes call them "exchange items") can't be get/set before calling the initialize method.

Thank you for the tip about running valgrind; I hadn't thought of that. While it may be a consequence of accessing the "flag" variable, there may very well be a problem with the way I've written the example. I'll look into it.

Mark

[lzhu5]

Hi Mark,

Thank you for explaining the order of calling BMI functions. I guess this order is enforced by "intent(out) :: this” in heat_initialize function in bmi_heat.f90?

In our F77 model’s own initialization subroutine, the input file read and disk writes are blended together. We intend to enable an optional disk write turn-on and keep F77 model’s own initialization subroutine unchanged. That is why we need to set the value of a logical parameter, which controls the disk writes, before calling F77 model’s own initialization subroutine.

We figured out a workaround to this issue. Please advise whether our method is compatible with the babelizer and PyMT users’ coding conventions.

Our workaround (use bmi-fortran-example as an example)

---

(1) In “type :: heat_model” (in heat.f90), add variables: “enable_write”, “model_initialized”, and “config_file_path”.
(2) In function heat_initialize (in bmi_heat.f90),
- save “config_file” in variable “config_file_path”,
- set “enable_write” to False, and set “model_initialized” to False.
- move the actual initialization steps to a new function internal_initialize in bmi_heat.f90.
(3) In functions heat_update and heat_update_until (in bmi_heat.f90), prepend the following
if (model_initialized==False) call internal_initialize()
(4) In the new function internal_initialize(), initialize variables in heat_model and set “model_initialized” to True on success.

---

In short, we still call BMI’s initialize() function as the first step, but we do actual initialization in BMI’s update() or update_until() function. This modified bmi_heat model can be used as below:

---

bmi_model%initialize()
bmi_model%set_value('enable_write_name’, 1) ! optional
bmi_model%update()
bmi_model%finalize()

---

Would this modified bmi_heat model be compatible with the babelizer?

BTW, we updated our compiler to gcc 10.2.0, and valgrind stopped complaining about the data loss.

Thanks again for your help.

Best,
Ling

[mdpiper]

Hi Ling,

Yes, your solution should work with the CSDMS Workbench. I've seen other models with multi-step initialization, and we even have an open issue to determine a better way to address it.

I wonder if it might be easier to pass in these parameter values through the configuration file? This is just a thought, since you already have a working solution. It would still require some refactoring, but perhaps less than what you have done.

Thank you for the update about the possible data loss. I need to include a check for this in the continuous integration.

Mark

[lzhu5]

Hi Mark,

Thank you for your suggestion.

I have two more questions:

1. I want to make a bmi function return an integer different from 0 and 1. For instance, I want to use different integers (for instance, -1 or 100) to represent different return statuses. When a python program calls this bmi function through pymt, can the python program receive that integer? I just want to confirm that babelizer would not treat any non-zero return integer as 'true'.
2. Can I remove some of the bmi functions from bmi.f90? Would babelizer support that?

Thanks.

Best,
Ling

[mdpiper]

Hi Ling,

The BMI functions return zero for success and nonzero for failure, so you can use different status codes for different errors. This isn't really clear in the documentation, so I'll add language to improve it. The status code isn't available in Python through pymt--any nonzero (not success) value gets swallowed by the babelizer. The way we handle errors from wrapped libraries is something that could be improved. If you have suggestions from your work, please feel free to create issues in the BMI repo or the BMI Fortran repo.

By definition, all BMI functions have to be implemented. However, not all of them are always used. For example, in bmi-example-fortran, the Heat model is solved on a uniform rectilinear grid, so grid functions like get_grid_face_count aren't used. In this case, you can just set them to fail. For example:

  ! Get the number of faces in an unstructured grid.
  function heat_grid_face_count(this, grid, count) result(bmi_status)
    class(bmi_heat), intent(in) :: this
    integer, intent(in) :: grid
    integer, intent(out) :: count
    integer :: bmi_status


    count = -1
    bmi_status = BMI_FAILURE
  end function heat_grid_face_count

Thank you for your questions, Ling. You're helping me think about BMI, and how to improve the documentation and testing.

Mark

[mdpiper]

Hi Ling,

I made a mistake. The nonzero status code is returned through a RuntimeError in pymt. Here's the relevant code in the babelizer:

def ok_or_raise(status):
    if status != 0:
        raise RuntimeError('error code {status}'.format(status=status))

Mark

[lzhu5]

Hi Mark,

Thanks for your reply. It is very helpful.

I want to elaborate on my second question. I actually want to delete un-used functions from bmi.f90 so that I don't need to implement those functions in bmi_heat.f90. Does babelizer support fortran libraries using the modified bmi.f90? Thanks.

Ling

[mdpiper]

Hi Ling,

No—by the definition of an interface, you can't leave out or remove functions from it.

Mark

[lzhu5]

Hi Mark,

Thanks again for your help.

I successfully converted my model to bmi, tested the bmi-model through bmi_main.f90, and built Python bindings for the model through babelizer.

However, I encountered an error when I tried to import my model in Python. The error message says:
ImportError: path/to/model.cpython-39-x86_64-linux-gnu.so : cannot map zero-fill pages: Cannot allocate memory

Many posts online say the error is caused by insufficient memory. However, my machine has 40G memory. Would you please provide some suggestions? Thanks.

Best,
Ling

[mdpiper]

Hi Ling,

Congratulations on adding a BMI to your model and babelizing it! These aren't easy tasks, so it's great that you've accomplished them.

It's interesting that the exception is raised on import, and not on instantiation or when initialize is called. When are you allocating arrays for your model, and how large are they? Arrays occupy contiguous memory on the system, and depending on what other processes are running, Python won't have access to the full 40 GB of memory.

Mark

[lzhu5]

Hi Mark,

Thanks for your reply. Your help and time are greatly appreciated.

We strictly follow the structure of "bmi-fortran-example" to implement the BMI in our Fortran program. Still use "bmi-fortran-example" as an example. We put all global variables (i.e., variables in common blocks) in "heat_model" module in heat.f90. The "heat_model" is instantiated within bmi_heat.

All global arrays are statically declared. There are around 50 two-dimensional arrays with the size of 5000*100. There are other arrays with relatively smaller sizes.

If we run "bmi_heat" through a Fortran main program (bmi_main.f90), the program uses 388MB of memory and Valgrinds reports no errors.
By trial and error, I found that changing the 2D array size to 5000*45 allows me to import pymt_heat and call initialize(). The corresponding memory is 313 MB.

Would you please suggest how to make Python access more memory? Thanks.

Best,
Ling

[mcflugen]

@lzhu5 I would like to try to reproduce this error. Could you please point me to the code so I can try to install it on my machine and see if I get the same error?

Could you also verify that if you simply open python and try to allocate that same amount of memory, you can do so without a problem? So maybe something like,

>>> import numpy as np
>>> x = np.array((50, 5000, 100))

[lzhu5]

@mcflugen Please find the code here.

The code was downloaded from bmi_example_fortran. I made the following changes to "heat.f90":

1. adding global variables at Line 11-68. The global arrays are statically declared using parameters declared at Line 6.
2. adding subroutine compute at Line 97-109.

With NL=100 (Line 6 in heat.f90), I encountered ImportError (ImportError: ~/BMI/build_bmi_fortran_heat/pymt_heat/pymt_heat/lib/heatmodel.cpython-39-x86_64-linux-gnu.so: cannot map zero-fill pages: Cannot allocate memory) while loading the babelized "Heatmodel" in Python.

After changing NL to 30, I could successfully import "Heatmodel".

Yes. I could do x = np.array((50, 5000, 100)) in Python without any problem.

Best,
Ling

[mcflugen]

@lzhu5 Thank you! I was able to download your code, build it, and import it into Python without a problem. I'm on a Mac. It looks like you're also on a Mac, is that right? Are these more or less the steps you followed? From what you say, it sounds like your error is occurring after you run from pymt_heat import Heatmodel from within Python, correct?

First, I built and installed the bmi-fortran-heat example:

$ cd build_bmi_fortran_heat
$ conda create -n heat python fortran-compiler cmake bmi-fortran babelizer -c conda-forge
$ conda activate heat
$ cd bmi-example-fortran
$ mkdir _build && cd _build
$ export BMIF_VERSION=2.0
$ cmake ../ -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX
$ make all install

Second, I babelized the library,

$ cd build_bmi_fortran_heat
$ babelize init babel_heat.toml
$ cd pymt_heat
$ pip install -e .

I was then able to import your wrapped model in Python,

>>> from pymt_heat import Heatmodel
>>> heat = Heatmodel()
>>> heat.get_component_name()
'The 2D Heat Equation'

[lzhu5]

@mcflugen Yes. I followed these steps to build and import the code.

I ran the model on HPC (centOS). The ImportError seems like an HPC-related issue now. I will reach out to the HPC help desk for their advice on the memory issue. Thanks again.

Best,
Ling

[lzhu5]

@mcflugen I still cannot resolve the ImportError. I tried on a local Linux OS (ubuntu 18.04) and two different HPC systems (using centOS and RedHat Enterprise Linux, respectively). I got the same ImportError. I started from scratch (i.e., installing anaconda3) in all tries.

The code I tested is still the version that works on your Mac, and I followed exactly the steps you listed in your previous reply.

I am wondering would you please provide suggestions? Thanks a lot.

Best,
Ling

[mdpiper]

@lzhu5 Following @mcflugen's instructions above, I was able to build your example in a Docker container based on the condaforge/miniforge3 image (which is built on Debian Linux), then run it in Python. I'll set up a Dockerfile and post an image on Docker Hub so that you can try it locally.

I was surprised that this worked because I think I've spotted a problem in the Fortran template code I wrote for the Babelizer. I'll keep digging to try to reproduce the error you're seeing.

[lzhu5]

@mdpiper I am a bit confused. Do you mean that you don't have the "ImportError" in the example built in the Docker container?

[mdpiper]

@lzhu5 Yes, that's correct.

I still need to build and post an image, but I should be able to finish that tomorrow.

[mdpiper]

I think I've got it.

As I mentioned above, I was able to build, install, and run the modified Fortran BMI example through Docker using a Debian Linux base image on my iMac. A Dockerfile and everything needed to reproduce this is in this repo. The README has build and run instructions.

I tried the same on a test Linux machine I have. I was able to build the Docker image, but running it threw a familiar error:

$ docker run --rm help-desk-104
Traceback (most recent call last):
  File "/opt/run.py", line 1, in <module>
    from pymt_heat import Heatmodel
  File "/opt/build_bmi_fortran_heat/pymt_heat/pymt_heat/__init__.py", line 7, in <module>
    from .bmi import Heatmodel
  File "/opt/build_bmi_fortran_heat/pymt_heat/pymt_heat/bmi.py", line 3, in <module>
    from .lib import Heatmodel
  File "/opt/build_bmi_fortran_heat/pymt_heat/pymt_heat/lib/__init__.py", line 3, in <module>
    from .heatmodel import Heatmodel
ImportError: /opt/build_bmi_fortran_heat/pymt_heat/pymt_heat/lib/heatmodel.cpython-39-x86_64-linux-gnu.so: cannot map zero-fill pages

(I don't know lots about Docker, but it's interesting that a Docker Linux image would run on macOS, but fail on Linux.)

This is where I'd thought I'd try the hunch I mentioned earlier. I modified the Babelizer, removing the ability to run multiple instances of the same Fortran model simultaneously (this feature works fine for dynamically allocated variables in a model, but not for static variables). In a branch of my Docker repo, I built this version of the Babelizer before using it on the modified Fortran BMI example. Building and running the Docker image now works on both my iMac and my Linux machine:

$ docker run --rm help-desk-104
The 2D Heat Equation

If you'd like, what you can do is follow my steps to build the Babelizer locally from the mdpiper/help-desk-104 branch, then use this modified Babelizer to build your example. I think this should work.

What I'll do is add a new option to the Babelizer babel.toml file that allows a user to specify the number of model instances that can run simultaneously, with a default of 1.