posix: deprecate CONFIG_POSIX_API and remove zephyr/posix include prefix

[cfriedt]

Deprecation of CONFIG_POSIX_API

Deprecate CONFIG_POSIX_API.

Users may choose one of the pre-defined POSIX subprofiles, for simplicity.

• CONFIG_POSIX_AEP_CHOICE_BASE
• CONFIG_POSIX_AEP_CHOICE_PSE51
• CONFIG_POSIX_AEP_CHOICE_PSE52
• CONFIG_POSIX_AEP_CHOICE_PSE53

Libraries should depend on CONFIG_POSIX_SYSTEM_INTERFACES as well as other POSIX Option Groups, as needed.

Custom Configurations

Of course, instead of, or in addition to, using a pre-defined POSIX subprofile, users can still fully customize their POSIX configuration via Kconfig.

Zephyr's POSIX Kconfig option names correspond 1:1 with official POSIX Option Group names. Simply add a CONFIG_ at the beginning of an Option Group to get the corresponding Kconfig option.

For more information on POSIX Subprofiling Option Groups, please see
https://pubs.opengroup.org/onlinepubs/9799919799/xrat/V4_subprofiles.html

Removal of zephyr/posix/ prefix for standard includes

The majority of the changes in this category (~156 files) are mechanical and are contained in the commit posix: remove zephyr/posix in header prefixes. They are approximately equivalent to running this command (with some adjustments).

FILE_LIST="$(grep "^#include.*zephyr/posix/" $(find * -name '*.c' -o -name '*.h') | awk '{split($0,a,":"); print a[1];}' | sort -u)"
sed -i -e 's|zephyr/posix/||g' $FILE_LIST

Originally from #97152, but included here for simplicity, these changes mean that we no longer need to have include/zephyr/posix in the library search path for C libraries that have (at least partially) conformant POSIX headers. In other words, we use the C library's POSIX headers.

• Eliminates conflicts between Zephyr's POSIX headers and the libc headers
  • C library headers and types used as-is whenever possible
  • Still possible to "fix up" libc headers, as needed, via #include_next <foo.h>
  • It allows non-POSIX C libraries to also have POSIX support in Zephyr
• Ensures that POSIX applications and libraries are actually portable at the source level and, in many cases, at the binary level
• Paves the way for Zephyr's POSIX implementation to become a module

Originally added to mitigate C-library conflicts, the <zephyr/posix/...h> prefix on standard includes has been a thorn in our sides for several years. Since these changes remove the conflicts, opting to use C-library headers (if they exist), we no longer need to namespace those in Zephyr.

Depends on

• [noup] pull in standard headers for POSIX types and functions hostap#108
• simplelink: remove zephyr/posix prefix and add posix feature test macros hal_ti#72
• scripts: coccinelle: add fork as a reserved name #98509

Note

Compliance issues are false positives

Fixes #97949
Fixes #98661

SUMMARY

Summary:
@nordicjm

• remove the last of the global defines
  • agreed to a follow-up PR for _POSIX_C_SOURCE and _XOPEN_SOURCE since the last commit was not exactly trivial

@aescolar

• remove ifdef config arch posix
• libcl3 pedantic
  • happy to work with Alberto to resolve concerns in the stabilization period
  • also note, modules/liblc3/CMakeLists.txt has lots of cmake formatting issues, unrelated to this PR
  • this is generally not a blocking concern, according to @nashif
  • used suggestion made, and it caused a cascade of failures in CI
  • logically, it is not feasible to use the provided suggestion, again going with @nashif here in that it is not a blocking concern
• use _GNU_SOURCE instead of __GNU_SOURCE
• create a bug report for tracking
• correct commit message terminology
• squash removal of posix api commits
  • as mentioned, one commit was basically string replacement and the other was code. they were squashed
• lots of file changes
  • i guess a little bit of confusion. but also covered in the PR descriptions(s), one of the most important aspects of this pr was to use standard, non-prefixed POSIX header files
• fork to philosopher_fork
  • scripts: coccinelle: add fork as a reserved name #98509
• _GNU_SOURCE instead of __GNU_SOURCE
• investigate MBEDTLS_TEST_SW_INET_PTON
  • agree, I think this would be great to fix in a better way, if possible, in the stabilization period.
  • this is generally not a blocking concern according to @nashif
• remove unnecessary change re asctime_r() tests
• debug and fix workaround for qemu_leon
• remove addition of standard GETENTROPY_MAX
• include non-portable / non-standard pthread apis
  • posix: threads: np: create a kconfig option and testsuite for non-portable pthread APIs #98536
  • my preference would be to do this properly with a Kconfig and separate testsuite either in the stabilization period or next release cycle
• add missing CONFIG_
• fix include reordering
  • this was brought up again although there are no reordered includes
  • yes, the necessary, standard header files were added
  • literally nothing in the files specified seems out of the ordinary, in any way whatsoever
• create a tracking issue
  • posix: remove workarounds for missint pid_t in picolibc sdk-ng#1035
• drop last commit from this pr
• apply fixup to previous commit
• define _GNU_SOURCE before other includes
• create enhancement issue for imacros and add disclaimer to posix_features.h
  • posix: investigate an alternative for -imacros ../posix_features.h #98525
• do not use CONFIG_ARCH_POSIX as conditional in vlan.c
  • used CONFIG_NATIVE_LIBC instead
  • conditional will be fixed by @jukkar as a follow-up item
  • further posix / networking issues tracked in
    • Networking subsystem should not (re)define C library types, macros or functions #97050
    • Networking subsystem should not use extensions to the C library beyond the allowed subset #98212
• limit issue number 2 from this comment
  • issue number 1 is factually incorrect, and is spreading fear, uncertainty, and doubt (FUD)

@nashif

• create a bug report for the qemu_leon3 issue
  • boards: qemu: leon3: fault unique to this platform #98258
• nashif: make a pr to add fork() to scripts/coccinelle/symbols.txt
  • scripts: coccinelle: add fork as a reserved name #98509
• make the shim header include path optional in lib/libc/picolibc/CMakeLists.txt

@jhedberg

• unrelated ci error in tests/arch/arm/arm_user_stack_test
  • tests/arch/arm/arm_user_stack_test/arch.arm.user.stack failure on mps2_an385 #98494
  • tests: arm: exclude mps2/an385 on user stack test #98518

@carlescufi

• is it necessary to defer the POSIX change to fix the libc and / or network subsys?
  • it is not, as demonstrated by the last month or so of successful CI runs, and since libc and network issues have been converted into much smaller, separable, problems that may be solved independently.

[github-actions]

The following west manifest projects have changed revision in this Pull Request:

Name	Old Revision	New Revision	Diff
hal_ti	zephyrproject-rtos/hal_ti@cc04902	zephyrproject-rtos/hal_ti@33dae8f (master)	zephyrproject-rtos/[email protected]
hostap	zephyrproject-rtos/hostap@cf05f33	zephyrproject-rtos/hostap@ab7630c	zephyrproject-rtos/[email protected]

✅ All manifest checks OK

Note: This message is automatically posted and updated by the Manifest GitHub Action.

[cfriedt]

• rebased

[cfriedt]

I guess if the shims are only needed when posix is needed, we could just guard this in cmake and do the following, no?

--- a/lib/libc/picolibc/CMakeLists.txt
+++ b/lib/libc/picolibc/CMakeLists.txt
@@ -10,7 +10,9 @@ zephyr_library_sources(
   locks.c
   stdio.c
   )
+if(CONFIG_POSIX_WHATEVER)
 zephyr_include_directories(include)
+endif()
 if(CONFIG_PICOLIBC_USE_MODULE)
   # temporary workaround for https://github.com/picolibc/picolibc/pull/1079
   zephyr_compile_options(-Dpid_t=__pid_t)

It wasn't quite that easy, but this seems to have worked and given me the opportunity to fix something that was missed up until this point as well. Namely, that net/if_arp.h is not a POSIX header file, so that is moved back to include/zephyr/net and a few unnecessary shim headers were removed.

There are additional workarounds needed in order to avoid using shims.

1. No preprocessor guards exist around the declaration of struct timespec in Picolibc. Removed inclusion of the POSIX header <sys/types.h> from lib/os/printk.c and lib/os/cbprintf_complete.c to remove build errors due to re-declaration due to aforementioned lack of preprocessor guards, and since those files do not rely on POSIX types.
2. Manually declare inet_pton() as well when Picolibc is used since it is missing the declaration in <arpa/inet.h>.

This should be sufficient to satisfy one of Alberto's pending comments mentioned here. I will not be shuffling headers around to some unrelated folder at this time, and I do not see the need to do so in the future. However, headers required as workarounds for issues in Picolibc can be removed as Picolibc is improved over time.

Changes are here
https://github.com/zephyrproject-rtos/zephyr/compare/e68fe2eeae59d9d24f2c80c704e66da204c5afba..191fb795b609b57a9022f2226a07524c1c6e0738

[cfriedt]

@nashif @jhedberg @ycsin @jukkar - just a re-ping here since the PR was updated. All tests are still passing.

[cfriedt]

Hey @carlescufi

The CMakeLists.txt suggestion worked with some extra effort.

I completely agree with you, but then my question would be why not first fix the issues with the networking subsystem that you mention (I can work with @jukkar and @rlubos to make this happen) which to me seems a prerequisite to being able to clean up the POSIX subsystem

It's not really a prerequisite, as demonstrated by the last month of successful CI runs. As long as there are non-broken headers for the network stack to use, everything is fine, and it works identically.

What's nicely demonstrated here, is that it's quite straight-forward to make the problem of refactoring the network stack a separable one, due to the well-established interface. I.e. as long as the interface is consistent and functionality is the same, the implementation may be modified.

Since reworking the network stack has been demonstrated to be a separable problem, there is no need to defer this change to pay off that technical debt.

[aescolar]

Thanks, this is better.
But note we'd want to do the same for the other libCs too, not just picolibc.

Note that placing these new headers in a separate folders and only including them in a more fine grained way than CONFIG_POSIX_BASE_DEFINITIONS OR CONFIG_EVENTFD would help limiting side-effects as described in #99075.

[cfriedt]

But note we'd want to do the same for the other libCs too, not just picolibc.

Disagree. The only reason that including the path was made conditional was because it was your preference, which, in my opinion, is not sufficient technical justification.

I called it a workaround for the workaround, because the original workaround was to correct inconsistencies in C libraries, but then the workaround for the workaround was for you, since you seem to prefer broken headers, which introduces more inconsistence and tech debt.

For a technical justification, we have to consider why the path is included in the first place. It's included to provide a way to supply POSIX headers when Picolibc (and other C libraries) does not (either partially or fully) provide the header or required definitions (i.e. to have a consistent compilation environment).

Of course, this is only necessary when POSIX is configured, due to work done over the last few years.

I should take a step back - that's only true until some source file includes a POSIX header without configuring POSIX, which is historically extremely likely in the Zephyr project. So having shim files available unconditionally would be more consistent, just like they are unconditionally available from the C library (even if only partially complete as provided by the C library).

Now, inconsistence aside, assuming that it is possible to fix the C library over time to be more conformant, then shim headers can be removed over time as well (as well as the path to include shim headers).

If that assumption does not hold, which is the case for (at least Zephyr's SDK) newlib, minimal libc, IAR, ARCMWDT, most commercial toolchains, etc), then shim headers, and the include paths, should remain as long as they are required.

Note that placing these new headers in a separate folders and only including them in a more fine grained way than CONFIG_POSIX_BASE_DEFINITIONS OR CONFIG_EVENTFD would help limiting side-effects as described in #99075.

What side effects? There are none described in #99075.

If "side effects" mean having functional code, that is, by definition, not a side effect, but an intentional and direct effect.

The conditions for including the shim header path are by no means meant to be permanent. Personally, I would have not added them at all due to aforementioned concerns with inconsistence, had it not been for @nashif's suggestion, and @carlescufi's follow-up, squarely meant to satisfy you.

So from that perspective, you seem to want the rest of the Zephyr community to suffer with additional technical debt beyond the release cycle due to your preference.

So no, "we" do not want to do the same for the other libCs, and I still question the value in making the shim header path conditional for Picolibc as well.

[sonarqubecloud]

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
3.9% Duplication on New Code

See analysis details on SonarQube Cloud

[aescolar]

I created #99075 to facilitate discussing the more fundamental issues. As there seems to be a disagreement about how the POSIX_API should relate to the C library, its integration, and how optional or not it should be.