[flang-rt] Implement iso module in C++ in the runtime

[jhuber6]

Summary:
Right now we have a very heavy dependency on flang for a single module
file. The flang build will handle creating the actual module file, we
just need to export the symbols. This PR ports the source file from the
flang .f90 to equivalent C++. This allows us to build flang-rt
standalone without bringing in the entire flang infrastructure, which is
by far the heaviest build in the entire LLVM project.

I verified the symbols and values against a previous build, so they are
identical.

[jhuber6]

libc++ test failure is unrelated, everything is green.

[clementval]

Will the dependency come back anyway once @Meinersbur move the module files to the runtime?

[jhuber6]

Will the dependency come back anyway once @Meinersbur move the module files to the runtime?

You are correct, so I have removed the part of the code that removes the flang setup and dependency. However, I believe this PR is still valuable. The incoming changes in #171515 will move module generation here, but this file is different as it creates real symbols. With this PR, we can logically split flang-rt compilation and Flang module generation. That means we can compile the entire runtime stack without a fortran compiler. @Meinersbur's PR already sets up a separate target for module file generation.

The main motivating factor here is that internally at AMD we are struggling with the extremely heavy resource requirements for building flang, so the effort is to contain it as much as possible. We are going to limit the dependency graph such that we can build as much as possible without flang so we can do it at the end or separately.

[Meinersbur]

Will the dependency come back anyway once @Meinersbur move the module files to the runtime?

I was actually trying to go the other way. It seems obvious to be able to implement (parts of) a language's runtime in the language itself, in the sense that I don't know any other language that forces its runtime to be entirely written in another language. There have been several attemps already: #169525 (since the reason why iso_fortran_env.f90 and iso_fortran_env_impl.f90 were split have been removed), #174474 (changed during review to a C++ implementation), #170374 (openmp; solves a problem that some OpenMP functions/routines have the same name in C/Fortran, but different signatures).

With iso_fortran_env_impl.f90 converted to C++, flang-rt could be split into two parts: Compilation of the machine code libraries, and build the module files. With #171515 building is modular: FLANG_RT_ENABLE_STATIC, FLANG_RT_ENABLE_SHARED can both be off, but still builds the module file. An option FLANG_RT_ENABLE_MODULES did not make sense so far because iso_fortran_env(_impl).f90 was needed to be compiled in any case, and its module is a byproduct of it, but I would add such an option if the PR lands. So with FLANG_RT_ENABLE_MODULES=OFF there would be no dependency on Flang. We could even go so far to make flang-rt-mod a sibling "runtime" next to flang-rt.

Of course, a complete Fortan toolchain, including running check-flang will still require the modules as well.

[jhuber6]

With iso_fortran_env_impl.f90 converted to C++, flang-rt could be split into two parts: Compilation of the machine code libraries, and build the module files. With #171515 building is modular: FLANG_RT_ENABLE_STATIC, FLANG_RT_ENABLE_SHARED can both be off, but still builds the module file. An option FLANG_RT_ENABLE_MODULES did not make sense so far because iso_fortran_env(_impl).f90 was needed to be compiled in any case, and its module is a side-output of it, but I would add such an option if the PR lands. So with FLANG_RT_ENABLE_MODULES=OFF there would be no dependency on Flang. We could even go so far to make flang-rt-mod a sibling "runtime" next to flang-rt.

This is my plan, basically. The changes in #198205 set up the dependencies so that we only need to build flang for -mod* runtime targets. I tested this in practice and I was able to set up the entire runtimes stack and then separately install the module files. That's pretty much all that's required to let us delay building flang only until users actually need it. The NFC in #198111 is also helpful as it brings flang-rt into a reasonable build time so I don't need to worry about building it as much.

[vzakhari]

I verified the symbols and values against a previous build, so they are identical.

Is there a way to verify that the C++ implementation and the Fortran implementation of the module provide "the same" binary automatically at least in some flang/flang-rt builds? I am worried about future changes causing misalignment between the two implementations.

[jhuber6]

Is there a way to verify that the C++ implementation and the Fortran implementation of the module provide "the same" binary automatically at least in some flang/flang-rt builds? I am worried about future changes causing misalignment between the two implementations.

There's not an easy way to do this as far as I'm aware. We'd need a separate test that compiles the .f90 file with flang and then compares the values in flang-rt. It could be done, but it would require a lot of extra harness for CMake. The test I have now is a smoke test, and practically I expect the comments I added to hopefully be the real guard for this. This split will also be more obvious when the module files get moved into flang-rt I believe. Let me know if you think this is a blocker and I can try to figure something out, but I'd rather not.

[vzakhari]

Let me know if you think this is a blocker and I can try to figure something out, but I'd rather not.

I am okay with using the C++ implementation for the development purposes, but I think it will be better if we use the actual Fortran module for the complete toolchain build (i.e. the one that will probably be used for the official release builds). Will it be possible to only use the C++ implementation under an explicit CMake option?

[jhuber6]

I am okay with using the C++ implementation for the development purposes, but I think it will be better if we use the actual Fortran module for the complete toolchain build (i.e. the one that will probably be used for the official release builds). Will it be possible to only use the C++ implementation under an explicit CMake option?

That doesn't really solve my problem. For context, we at AMD are dealing with several issues that stem from the flang project's long compile times and massive RAM usage. The purpose of this PR and #198205 together is to let us defer the flang dependency as long as possible. These two PRs are my attempt to make this work for us without AMD taking the nuclear option and removing the flang build entirely.

We will still have the module file step, and #171515 will move that here as well (which will also help keep these in sync). It's just that the linkable symbols are in flang-rt which now becomes a C++ only library. I think this simplifies things greatly.

[vzakhari]

Isn't it possible to have dependency on flang and the Fortran module in one configuration and do not have it in another? What I am suggesting is to add a CMake option like FLANG_RT_BUILD_WITHOUT_FLANG: the configuration step will not use the Fortran module and will not set dependency on flang, if this option is set. Otherwise, we will keep using the real Fortran module built by flang to get the corresponding binary file.

What am I missing?

[jhuber6]

Isn't it possible to have dependency on flang and the Fortran module in one configuration and do not have it in another? What I am suggesting is to add a CMake option like FLANG_RT_BUILD_WITHOUT_FLANG: the configuration step will not use the Fortran module and will not set dependency on flang, if this option is set. Otherwise, we will keep using the real Fortran module built by flang to get the corresponding binary file.

What am I missing?

Is the suggestion that we have two paths? I don't think it is worth the complexity. I will try to get a comprehensive test working so that we can prove the symbols always match up. I think that will ease a lot of anxiety.

[Meinersbur]

Isn't it possible to have dependency on flang and the Fortran module in one configuration and do not have it in another? What I am suggesting is to add a CMake option like FLANG_RT_BUILD_WITHOUT_FLANG: the configuration step will not use the Fortran module and will not set dependency on flang, if this option is set. Otherwise, we will keep using the real Fortran module built by flang to get the corresponding binary file.

I am not looking forward into maintaining two redundant versions. Generally, contributors also rarely think about updating configurations that they do not use. By experience, those go out of maintanance/grow inconsistencies over time.

For iso_fortran_env in particular, if this is about consistency with Flang's understanding of these kinds, I think a unittest would be the best approach.

[jhuber6]

Added a test, it should fail if these every get out of sync.

[vzakhari]

I am not looking forward into maintaining two redundant versions. Generally, contributors also rarely think about updating configurations that they do not use. By experience, those go out of maintanance/grow inconsistencies over time.

Right, I have the same concern about the two implementations of the module symbols. Thank you for adding the test! LGTM

[jhuber6]

Going to go ahead and merge this. I don't think this precludes us from having Fortran sources in the future but I think we should think explicitly about how they're partitioned. This makes it easier to make the 99.9% C++ file 100% C++.

[akuhlens]

I am getting the following failure from this PR:

# RUN: at line 13
diff ...
# .---command stdout------------
# | *** compare_iso_fortran_env_symbols.f90.tmp.f90.syms
# | --- compare_iso_fortran_env_symbols.f90.tmp.cpp.syms
# | ***************
# | *** 1,6 ****
# |   _QMiso_fortran_env_implEC__builtin_integer_kinds 14
# |   _QMiso_fortran_env_implEC__builtin_logical_kinds 10
# | ! _QMiso_fortran_env_implEC__builtin_real_kinds 18
# |   _QMiso_fortran_env_implECbfloat16 4
# |   _QMiso_fortran_env_implECint128 4
# |   _QMiso_fortran_env_implECint16 4
# | --- 1,6 ----
# |   _QMiso_fortran_env_implEC__builtin_integer_kinds 14
# |   _QMiso_fortran_env_implEC__builtin_logical_kinds 10
# | ! _QMiso_fortran_env_implEC__builtin_real_kinds 14
# |   _QMiso_fortran_env_implECbfloat16 4
# |   _QMiso_fortran_env_implECint128 4
# |   _QMiso_fortran_env_implECint16 4
# `-----------------------------
# error: command failed with exit status: 1

[jhuber6]

Ah, sorry, what's your platform? At least the test is doing its job.

[jhuber6]

My guess is that it might be a platform without 128bit float and for some reason the original ISO file says it's supported.

[vzakhari]

Our flang-rt is built with -DFLANG_RT_SUPPORTS_REAL16=1, so it contains the real-kind-16 in the array. The test's clang command does not specify this macro, so the real-kind-16 is missing from the *.cpp.o file.

[vzakhari]

I wonder if it is possible to compare the module compilation with the real iso_fortran_env_impl.cpp.o that was built during the library build.

[vzakhari]

llvm-nm --defined-only --format=posix --print-size ./lib/clang/23/lib/x86_64-unknown-linux-gnu/libflang_rt.runtime.* | grep iso_fortran_env_impl seems to work for shared and static libs

[jhuber6]

That's what I was thinking, but if we build with both it would double up all the symbols. Maybe I can just run it through uniq?

[jhuber6]

Let me know if #198922 fixes it

[vzakhari]

I guess we can check for flang_rt.runtime, flang_rt.runtime.static and flang_rt.runtime.shared targets, then find the output file for them and pass it to the LIT test from CMake, but uniq should probably also work.

[vzakhari]

Let me know if #198922 fixes it

It works for me. Thanks!