Merging feature/sharedobject to develop

[flampouris]

For different applications, the ww3 library is needed as a shared object; therefore, the "make" scripts were updated accordingly. The main modification is the addition of -fPIC flag for compiling the code. The shared object, named libww3.so, has been added as a target at the makefile, it is created similar to the other executables and libraries of WW3 (./w3_make libww3.so); the "libww3.so" resides at the [...]/obj folder.

The code has been tested with intel17 and gnu7.3.

[ajhenrique]

Hi @flampouris, we've found that building shared libraries has doubled the compilation time for some of the programs in our package (eg, using the -fPIC option instead of -ip). Having the option to build shared libraries is a wanted feature, would it be possible for you to build the shared library option without affecting the compilation time? This is critical for us in running multiple regression tests. We are reverting your changes to the NOAA-EMC/WW3 repo, and will be ready to reintegrate when you have a solution for reducing compile time. Thanks!

[flampouris]

Hi @ajhenrique,

No, it is impossible to build shared objects "without affecting the compilation time," especially if the rest of the compilation flags remain the same.

I do not understand the comment about the "-ip". The -ip is not related to shared objects, and according to the control version, I did not touch it.

[ajhenrique]

Hey @flampouris, thanks for the reply. I'm sure about that, and compile time is really important in our case because when pushing code from feature branches to the master/develop branches we run a matrix of regtests with more than 500 cases, the increase in compilation time while building shared objects was adding > 16h to the process, which made it impractical. Unfortunately, when your change was submitted our regtests matrix did not have a way to monitor effectively compile time, and I was unable to determine how much additional time this added to the build, so this went unnoticed (I am now adding a compile time tracker so we can make this part of our certification process).

To fix the issue, I suggest making the build of shared objects an option, which we can easily incorporate into cmplr.env by adding the intel_so label. I'll clone your branch and make the changes, if you agree that is an option.

Sorry about the -ip option comment, I was under the wrong impression that you had been replaced it by -fPIC. by accident.

[JessicaMeixner-NOAA]

@ajhenrique I agree total compile time is important and adding an "intel.so" option to the cmplr.env could be an option. How can we improve our regression testing so that we catch this sooner the next time?

@flampouris another thought of a fix: If the goal for the so is for JEDI, do you use cmake/ecBuild? And if so, can you pass the opt, and other things that are going into cmplr.env from the top level build to WW3? We'd have to maybe update cmplr.env to accept predefined values, but this could be a nice feature for anyone who wants to pass this information (compiler and options) down from an overall build system. I can't remember if @mickaelaccensi already added that feature or not?

[flampouris]

hello @ajhenrique and @JessicaMeixner-NOAA

The modification of the cmpl.env is not the issue. The problem was the way that the design of the compiling procedure, and specifically the w3_make, and in a smaller degree, the list of progs at the make_makefile.sh.

I solved the problem by "grepping" the fPIC flag from the comp; in principle, I violated the design of the w3_make. See the pull request for details.

BTW, I ran a couple of experiments, the difference between the master and the branch with shared objects, is about 0.2 seconds, so I don't think that the problem was the compilation time but the running time.

About the cmake: I am not a guru, but the best option is to forget the recycling of the w3_make_family_scripts and the customized preprocessor and start from scratch. In my case, I just created a "poor man's" CMakeLists.txt based on "execute_process" that serves my development.

[ajhenrique]

@flampouris thanks for the feedback. We (@aliabdolali + myself) inspected the ~500 cases output from the regtests matrix, and in all of them the run time was not significantly different between the master and develop branch which had included the fPIC option: in our experiments the time burden was indeed in the code compilation. The latter was confirmed by running w3_make for individual programs: compilation times were significantly reduced (by > 2x) by removing the fPIC option. It would be interesting to compare notes in person, since we sit not that far from each other. We can take that chance to discuss options to move forward, and clarify issues that may not seem evident to each other. Thanks!

[mickaelaccensi]

Hi all,

I would rather prefer to add it as a prefix so_intel to keep it consistent with options added in specific case or for specfic machine as it was done for zeus and datarmor.
It should be also done for the mpt and pgi compilers.

Mickael

[ukmo-ccbunney]

@ajhenrique When compiling for regtests, what level of compile time optimization do you set?

I ask because I found that I was able to dramatically decrease the total time required for regtests by using a reduced optimization level. I wonder if this would help mitigate the longer compile times you are experiencing when copiling with the -fPIC flag?

For our system (Cray XC), I found that using -O1 gave the best results: the compilation is significantly quicker than our operational settings (-O3) with only a small impact on the runtime.

Using no optimization (-O0) resulting in really quick compile times, but the runtime suffered dramatically (compilers are very good at optimizing code!) Level 1 optimzations gave the best balance...

Just a thought....

Chris

[ajhenrique]

Thanks @ukmo-ccbunney this is good to know. The added option to use or not the fPIC does the job of reducing compile time (which seems to be a platform-dependent issue as well). We are trying to keep the compile flags as close to our operational config as possible, but perhaps in some cases if we can fit matrix runtime under 8h (our wall-time limit), using -O1 may come in handy. I'll run some tests to figure if this is the case. Cheers, Henrique.

[ajhenrique]

Thanks @ukmo-ccbunney this is good to know. The added option to use or not the fPIC does the job of reducing compile time (which seems to be a platform-dependent issue as well). We are trying to keep the compile flags as close to our operational config as possible, but perhaps in some cases if we can fit matrix runtime under 8h (our wall-time limit), using -O1 may come in handy. I'll run some tests to figure if this is the case. Cheers, Henrique.

[ajhenrique]

For completion, I just finished updating the run_test script to display metrics indicating compile and run time for all programs under any restest. I tested the new run_test with the code added by @flampouris in #29 and without the fPIC flag. The modules loaded on NCEP's R&D machine Theia were:

1. intel/14.0.2 2) imagemagick/7.0.5 3) grads/2.0.1a 4) matlab/R2011b 5) wgrib2/0.1.9.5.1 6) netcdf/4.3.0

Results below indicate a substantial increase in compile tie particularly for ww3_grid. For the first regtest below (ww3_tp1.1), ww3_grid compilation went from 60s to 100s. For the second used in these examples. ww3_grid compile time went from 56s tp 102s. Other codes in these cases were not affected. However, the differences noted explain the significant impact of using fPIC with the above modules while running our regression tests matrix. Please note that this issue has now a proposed solution in pull request #47 by @flampouris .

Results for test

run_test -c Intel -S -s PR1 -T -w work_PR1 -- ../model ww3_tp1.1

A) Without fPIC
ww3_grid
60.06 sec compile time
0.04 sec run time
ww3_strt
3.04 sec compile time
0.26 sec run time
ww3_shel
53.13 sec compile time
ww3_outf
27.01 sec compile time
0.01 sec run time
ww3_outp
4.07 sec compile time
0.02 sec run time
Total compile time: 147.31 sec
Total run time : 0.33 sec

B) With fPIC
ww3_grid
100.19 sec compile time
0.04 sec run time
ww3_strt
2.65 sec compile time
0.01 sec run time
ww3_shel
49.76 sec compile time
ww3_outf
26.58 sec compile time
0.01 sec run time
ww3_outp
4.67 sec compile time
0.02 sec run time
Total compile time: 183.85 sec
Total run time : 0.08 sec

Results for test

run_test -c Intel -f -S -s PR1 -T -w work_PR1 -- ../model ww3_tp1.5

A) Without fPIC
ww3_grid
56.01 sec compile time
0.03 sec run time
ww3_strt
2.63 sec compile time
0.02 sec run time
ww3_shel
50.86 sec compile time
ww3_outf
25.79 sec compile time
0.01 sec run time
ww3_outp
4.03 sec compile time
0.02 sec run time
Total compile time: 139.32 sec
Total run time : 0.12 sec

B) With fPIC
ww3_grid
102.02 sec compile time
0.04 sec run time
ww3_strt
3.34 sec compile time
0.02 sec run time
ww3_shel
52.19 sec compile time
ww3_outf
28.36 sec compile time
0.02 sec run time
ww3_outp
4.89 sec compile time
0.02 sec run time
Total compile time: 190.80 sec
Total run time : 0.15