Skip to content

Inconsistent arg types (integer kind) across subroutine call#1300

Merged
davegill merged 7 commits intowrf-model:release-v4.2.2from
davegill:raspberry_pi
Oct 27, 2020
Merged

Inconsistent arg types (integer kind) across subroutine call#1300
davegill merged 7 commits intowrf-model:release-v4.2.2from
davegill:raspberry_pi

Conversation

@davegill
Copy link
Contributor

@davegill davegill commented Oct 13, 2020

TYPE: bug fix

KEYWORDS: integer kind

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
The nine calls to io_int_loc had inconsistent kinds of integer for the actual and dummy arguments. On every machine and
compiler previously, the integer kinds for MPI_offset and c_int64_t were identical. On the raspberry pi (using the
default apt-get openmpi and gnu fortran), these are not identical.

Solution:
As the integers are being shared through a C interface, that integer kind was selected.

LIST OF MODIFIED FILES:
modified: external/io_int/module_io_int_read.F90

TESTS CONDUCTED:

  1. Without the mods, the WRF model did not build on a raspberry pi. With the mods, the code builds and runs. Here testing
    on a raspberry pi includes the distributed memory, so the MPI_offset variable would be in use.
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 1 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            0  of            1
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.79946 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.79981 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.80066 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.79946 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.80392 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.80236 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.80006 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.79996 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.80133 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 2 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            1  of            2
 starting wrf task            0  of            2
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.53475 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.53516 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.53412 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.53324 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.53357 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.53547 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.53507 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.53406 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.53477 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 3 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            0  of            3
 starting wrf task            1  of            3
 starting wrf task            2  of            3
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.48709 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.48533 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.48523 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.48553 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.48532 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.48635 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.48445 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.48409 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.48494 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF

pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 4 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            3  of            4
 starting wrf task            0  of            4
 starting wrf task            1  of            4
 starting wrf task            2  of            4
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.65011 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.64889 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.64934 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.64886 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.65009 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.64658 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.64700 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.64807 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.64591 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
  1. Jenkins testing is all PASS.

RELEASE NOTES: An inconsistency in the declaration of integer kinds was discovered in the seldom-used routine that processes binary formatted data. In one routine the values were declared as an MPI type (and integer kind mpi_offset_type). That argument was passed to a routine that declared the integers as c_int64_t (which is part of the standard technique now to be used when working with C). These two integer kinds must have been the same 8-byte size for all of the x86_64 Linux machines that we have used. These integer kinds are however different for at least some gfortran / openmpi combinations on ARM processors (specifically found on a raspberry pi). There is no impact to existing users. If the code previously worked (built), this changes nothing.

…call

TYPE: bug fix

KEYWORDS: raspberry pi, arm

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
1. On a Raspberry Pi 4, the WRF code needed a target in the "uname -a".
2. The nine calls to io_int_loc had inconsistent kinds of integer for the actual and dummy arguments.

Solution:
1. The first GNU stanza now includes the string `armv7l`.
2. As the integers are being shared through a C interface, that integer kind was selected.

LIST OF MODIFIED FILES:
modified: arch/configure.defaults
modified: external/io_int/module_io_int_read.F90

TESTS CONDUCTED: 1. Without the mods, the WRF model did not build. With the mods, the code builds and runs.

```
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 1 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            0  of            1
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.79946 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.79981 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.80066 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.79946 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.80392 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.80236 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.80006 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.79996 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.80133 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
```
```
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 2 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            1  of            2
 starting wrf task            0  of            2
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.53475 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.53516 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.53412 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.53324 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.53357 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.53547 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.53507 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.53406 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.53477 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
```
```
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 3 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            0  of            3
 starting wrf task            1  of            3
 starting wrf task            2  of            3
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.48709 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.48533 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.48523 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.48553 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.48532 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.48635 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.48445 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.48409 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.48494 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF

```
```
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 4 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            3  of            4
 starting wrf task            0  of            4
 starting wrf task            1  of            4
 starting wrf task            2  of            4
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.65011 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.64889 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.64934 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.64886 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.65009 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.64658 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.64700 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.64807 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.64591 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
```
2. We will see what happens with jenkins

RELEASE NOTES: Modifications to to the WRF build system and small source fixes now all the WRF model to be built on a Raspberry Pi.
@davegill
Copy link
Contributor Author

Please find result of the WRF regression test cases in the attachment. This build is for Commit ID: 9252bb8, requested by: davegill for PR: #1300. For any query please send e-mail to David Gill.

    Test Type              | Expected  | Received |  Failed
    = = = = = = = = = = = = = = = = = = = = = = = =  = = = =
    Number of Tests        : 19           18
    Number of Builds       : 48           46
    Number of Simulations  : 166           164        0
    Number of Comparisons  : 105           104        0

    Failed Simulations are: 
    None
    Which comparisons are not bit-for-bit: 
    None

@davegill davegill changed the title Raspberry Pi: build stanza, inconsistent arg types across subroutine call Inconsistent arg types (integer kind) across subroutine call Oct 13, 2020
@davegill
Copy link
Contributor Author

This PR uncovered an inconsistency in the declaration of integers. In one routine the values were declared as an MPI type (and integer kind mpi_offset_type). That argument was passed to a routine that declared the integers as c_int64_t (which is part of the standard technique now to be used when working with C. These two integer kinds must have been the same 8-byte size for the x86_64 Linux machines that we used. They are different for at least some gfortran / openmpi combinations with ARM processors. There is no impact to existing users. If the code previously worked, this changes nothing.

@smileMchen
Copy link
Collaborator

Dave,
I just want to confirm that the integer kinds of MPI_offset and c_int64_t for all machines and compilers should be:
integer(c_int64_t) :: offset
integer(c_int32_t) :: count
They are specified to the values above by default in machines we use previously, but need to be explicitly declared when using raspberry pi.
Thanks.

@davegill
Copy link
Contributor Author

@smileMchen
Ming,
The default integer for fortran is 4-bytes (32 bit), and the default integer used for a pointer is twice that big (without a sign bit). All of the compilers we have ever used on all of the architectures have assumed these values. We simply found an example where a variable declared as an MPI_offset_type was NOT the same length as a c_int64_t. This was unexpected. The solution was simply to declare that the data going in and out of a C function was actually using C-based integer kind types.

For all of our previously tested existing compilers and architectures, these two integer kinds are already the same. Those existing users of standard compilers and architectures will see no impact at all.

@davegill davegill merged commit 76943c1 into wrf-model:release-v4.2.2 Oct 27, 2020
davegill added a commit that referenced this pull request Dec 30, 2020
…1301)

TYPE: new feature
    
KEYWORDS: raspberry pi
    
SOURCE: Daniel Zurawski and internal
  
DESCRIPTION OF CHANGES:
Additional strings, `armv7l` and `aarch64`, have been added to the first gfortran configuration stanza
so that these may be keywords for search from Raspberry Pi's `uname -a` output. No new
stanza was added for the gfortran target, so no re-numbering is required  for existing scripts that 
are reliant on such things as "32 = Gfortran for Linux with a serial build".

Associated with this PR, #1300 (Inconsistent arg types (integer kind) across subroutine call) is
also required to get the Raspberry Pi to build successfully.
 
LIST OF MODIFIED FILES:
modified:   arch/configure.defaults
    
TESTS CONDUCTED:
1. Previously, a Raspberry Pi build returned this:
``` 
*********************************************************
***              ERROR ERROR ERROR ERROR              ***
***                                                   ***
*** Configuration not found in configure.defaults     *** 
*********************************************************
``` 
2. After this PR, the configure command is able to find a target (raspian buster):
``` 
-----------------------------------------------------------------------
Please select from among the following Linux armv7l options:
    
  1. (serial)   2. (smpar)   3. (dmpar)   4. (dm+sm)   GNU (gfortran/gcc)
``` 
3. After this PR, the configure command is able to find a target (ubuntu mate):
```
------------------------------------------------------------------------
Please select from among the following Linux aarch64 options:

  1. (serial)   2. (smpar)   3. (dmpar)   4. (dm+sm)   GNU (gfortran/gcc)
```
4. Jenkins testing is all pass.
    
RELEASE NOTES: Modifications to the WRF code now permit the model to be built on a Raspberry Pi with the GNU/8 compiler.
vlakshmanan-scala pushed a commit to scala-computing/WRF that referenced this pull request Apr 4, 2024
…el#1300)

TYPE: bug fix

KEYWORDS: integer kind

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
The nine calls to io_int_loc had inconsistent kinds of integer for the actual and dummy arguments. On every machine and
compiler previously, the integer kinds for MPI_offset and c_int64_t were identical. On the raspberry pi (using the 
default apt-get openmpi and gnu fortran), these are not identical.

Solution:
As the integers are being shared through a C interface, that integer kind was selected.

LIST OF MODIFIED FILES:
modified: external/io_int/module_io_int_read.F90

TESTS CONDUCTED: 
1. Without the mods, the WRF model did not build on a raspberry pi. With the mods, the code builds and runs. Here testing
on a raspberry pi includes the distributed memory, so the MPI_offset variable would be in use.

```
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 1 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            0  of            1
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.79946 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.79981 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.80066 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.79946 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.80392 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.80236 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.80006 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.79996 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.80133 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
```
```
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 2 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            1  of            2
 starting wrf task            0  of            2
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.53475 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.53516 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.53412 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.53324 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.53357 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.53547 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.53507 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.53406 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.53477 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
```
```
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 3 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            0  of            3
 starting wrf task            1  of            3
 starting wrf task            2  of            3
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.48709 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.48533 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.48523 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.48553 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.48532 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.48635 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.48445 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.48409 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.48494 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF

```
```
pi@raspberrypi:/media/pi/gill/WRF/test/em_quarter_ss $ mpirun -np 4 ./wrf.exe ; tail rsl.out.0000
 starting wrf task            3  of            4
 starting wrf task            0  of            4
 starting wrf task            1  of            4
 starting wrf task            2  of            4
Timing for main: time 0001-01-01_00:00:24 on domain   1:    0.65011 elapsed seconds
Timing for main: time 0001-01-01_00:00:36 on domain   1:    0.64889 elapsed seconds
Timing for main: time 0001-01-01_00:00:48 on domain   1:    0.64934 elapsed seconds
Timing for main: time 0001-01-01_00:01:00 on domain   1:    0.64886 elapsed seconds
Timing for main: time 0001-01-01_00:01:12 on domain   1:    0.65009 elapsed seconds
Timing for main: time 0001-01-01_00:01:24 on domain   1:    0.64658 elapsed seconds
Timing for main: time 0001-01-01_00:01:36 on domain   1:    0.64700 elapsed seconds
Timing for main: time 0001-01-01_00:01:48 on domain   1:    0.64807 elapsed seconds
Timing for main: time 0001-01-01_00:02:00 on domain   1:    0.64591 elapsed seconds
d01 0001-01-01_00:02:00 wrf: SUCCESS COMPLETE WRF
```
2. Jenkins testing is all PASS.

RELEASE NOTES: An inconsistency in the declaration of integer kinds was discovered in the seldom-used routine that processes binary formatted data. In one routine the values were declared as an MPI type (and integer kind mpi_offset_type). That argument was passed to a routine that declared the integers as c_int64_t (which is part of the standard technique now to be used when working with C). These two integer kinds must have been the same 8-byte size for all of the x86_64 Linux machines that we have used. These integer kinds are however different for at least some gfortran / openmpi combinations on ARM processors (specifically found on a raspberry pi). There is no impact to existing users. If the code previously worked (built), this changes nothing.
vlakshmanan-scala pushed a commit to scala-computing/WRF that referenced this pull request Apr 4, 2024
…rf-model#1301)

TYPE: new feature
    
KEYWORDS: raspberry pi
    
SOURCE: Daniel Zurawski and internal
  
DESCRIPTION OF CHANGES:
Additional strings, `armv7l` and `aarch64`, have been added to the first gfortran configuration stanza
so that these may be keywords for search from Raspberry Pi's `uname -a` output. No new
stanza was added for the gfortran target, so no re-numbering is required  for existing scripts that 
are reliant on such things as "32 = Gfortran for Linux with a serial build".

Associated with this PR, wrf-model#1300 (Inconsistent arg types (integer kind) across subroutine call) is
also required to get the Raspberry Pi to build successfully.
 
LIST OF MODIFIED FILES:
modified:   arch/configure.defaults
    
TESTS CONDUCTED:
1. Previously, a Raspberry Pi build returned this:
``` 
*********************************************************
***              ERROR ERROR ERROR ERROR              ***
***                                                   ***
*** Configuration not found in configure.defaults     *** 
*********************************************************
``` 
2. After this PR, the configure command is able to find a target (raspian buster):
``` 
-----------------------------------------------------------------------
Please select from among the following Linux armv7l options:
    
  1. (serial)   2. (smpar)   3. (dmpar)   4. (dm+sm)   GNU (gfortran/gcc)
``` 
3. After this PR, the configure command is able to find a target (ubuntu mate):
```
------------------------------------------------------------------------
Please select from among the following Linux aarch64 options:

  1. (serial)   2. (smpar)   3. (dmpar)   4. (dm+sm)   GNU (gfortran/gcc)
```
4. Jenkins testing is all pass.
    
RELEASE NOTES: Modifications to the WRF code now permit the model to be built on a Raspberry Pi with the GNU/8 compiler.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants