Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement libc in zig #514

Closed
andrewrk opened this issue Oct 1, 2017 · 16 comments
Closed

implement libc in zig #514

andrewrk opened this issue Oct 1, 2017 · 16 comments
Labels
contributor friendly This issue is limited in scope and/or knowledge of Zig internals. enhancement Solving this issue will likely involve adding new logic or components to the codebase.
Milestone

Comments

@andrewrk
Copy link
Member

andrewrk commented Oct 1, 2017

If Zig ships with a libc implementation, it seals the deal for streamlined cross platform compilation. It improves the situation for native compilation as well. When a zig project depends on a C library such as GLFW, one could package it up with zig build system to build with zig (See #490). This accomplishes several things:

  • A single zig build from the root project on any platform will fetch the project's dependencies and perform a deterministic build using only zig and no system dependencies.
  • Zig can be intelligent about how it links dependencies together. It can link a bunch of .o files together instead of doing dynamic linking, where only 1 of the .o files has the libc implementation in it. So any code shared by zig std library and the libc implementation is actually shared in the binary, including stdio buffers and such. The default allocator can be shared between zig std lib and malloc/free.
  • This build would have no native dependencies, so cross compiling works perfectly out-of-the-box, even though you have C library dependencies that depend on libc.

Our goal is for developers to reach for zig instead of gcc, clang, or msvc, even if they intend to use it only to compile C code.

For linux, we can port musl. A lot of our stdlib is ported from there anyway - a lot of this code would be shared between zig standard library and the libc implementation. We should also look at glibc, for example for the getpwnam function, which in musl reads /etc/passwd but in glibc has an extension for a more advanced user account system.

For windows it might be worth looking into newlib-cygwin, reactos libc, and wine libc.

For macos, we always link dynamically against libSystem which has libc in it because this is the stable syscall interface. When cross compiling we allow functions to be unresolved that come from libc. We still need to ship .h files with appropriate constant values, however. I think the .h files from apple might be BSD licensed and thus we could redistribute those.

@andrewrk andrewrk added enhancement Solving this issue will likely involve adding new logic or components to the codebase. contributor friendly This issue is limited in scope and/or knowledge of Zig internals. labels Oct 1, 2017
@andrewrk andrewrk added this to the 0.3.0 milestone Oct 1, 2017
@pluto439
Copy link

On windows, from what part of libc should people start?

@andrewrk
Copy link
Member Author

Can you elaborate on the question?

@pluto439
Copy link

Do you need all libc, or only some parts of it? Which parts are more important right now?

@andrewrk
Copy link
Member Author

The goal is to provide API compatibility for existing C projects. We can let that drive the use case. So to figure out what is more important, we would try to, for example, build ffmpeg against a zig-libc.

@pluto439
Copy link

I'd start with libjpeg, but I need exceptions for that. Or at least setjmp/longjmp.

@andrewrk
Copy link
Member Author

You can implement setjmp/longjmp with inline assembly.

@tiehuis
Copy link
Member

tiehuis commented Mar 22, 2018

For anyone interested in this, I've started a simple repository which should give you some ideas here. Anyone reading, feel free to use this as a base or as inspiration.
https://github.com/tiehuis/zligc

I won't be doing much implementation here (too many things I already have to work on) but I may use this to figure out some zig -> c questions and some missing build system details needed in zig to accomodate the required goals.

This was referenced Mar 27, 2018
andrewrk added a commit that referenced this issue Apr 11, 2018
@suirad
Copy link
Contributor

suirad commented Oct 6, 2018

@andrewrk would you mind making a zlibc repo under the ziglang account that people could coordinate on to accomplish this? You could probably start it with a fork of @tiehuis' repo if you want.
I know you probably already have plenty to manage with zig itself, so you could probably recruit a mod or two to manage the repo.

@andrewrk
Copy link
Member Author

@suirad sorry, I missed your comment above. The libc code is going to live in this repository itself; contributions can be done via pull requests. I'll get the effort started so that people can see the pattern.

Now that we have #490 done, this issue of having libc is even more valuable, and easier to implement. How it's going to work is that Zig will have a set of supported libcs corresponding to a target.

  • --target-os linux --target-environ gnu (or other gnu environs) will be glibc, and it will be a dynamic dependency, the only kind supported by glibc. Zig will be able to build glibc's startup files from source (crtn.o, crtbegin.o, etc) and provide .h files. Zig does not need implementations of libc functions since it is dynamically loaded.
  • --target-os linux --target-environ musl (or other musl environs) will be musl's .h files. It will be a static dependency. Some of musl's source code can be imported directly such as their startup files. Other implementations of libc functions will be shared with the Zig standard library - this will prevent code bloat. For example the math functions will share implementations.
  • --target-os windows --target-environ msvc I think for this one we will use newlib/wine source code. It will be a static dependency.
  • --target-os windows --target-environ gnu (or other gnu environs) - this is mingw libc. Same idea as musl. Static dependency. Build from source.
  • --target-os freebsd (not sure what the default target environ is) - this is similar to glibc for linux. Zig will be able to build the startup files from source, but the implementations will be resolved at runtime.
  • --target-os macosx -libSystem actually does not even require any startup files. Brilliant.
  • --target-os netbsd same idea as freebsd

Once this is done, it raises the bar for Tier 1. A target can only be Tier 1 if Zig can cross-compile link against its libc.

Zig will no longer look for the native libc and try to build against it. zig libc will be the only way to do that. And unless a --libc foo.txt argument is provided to use a custom libc, Zig will use one from the bulleted list above; never the native one. This makes Zig 100% free of system dependencies, which makes it a true cross compiler.

andrewrk added a commit that referenced this issue Mar 7, 2019
@andrewrk
Copy link
Member Author

andrewrk commented Mar 8, 2019

I've done a proof of concept of this with glibc and it's merged into master. (The CI build is broken but I have a fix coming shortly)

There's a new tool libc/process_headers.zig and a wiki page documenting how to maintain the various libcs.

Here are some screenshots on Windows, Linux, macOS:

cross-glibc-windows
cross-glibc-macos
cross-glibc-linux

So at this point it's a checklist:

  • musl libc - this one we will do the headers thing, but try to implement most of the functions in Zig. Some of the functions we can build from C source - that's fine. We can always do a piecemeal port.
  • Windows libc (newlib? wine?) - same idea as musl - get the headers from somewhere, but the implementation should be shared with musl implementation, with perhaps some .c code to augment it.
  • mingw libc / cygwin libc - these will probably be their own issue. Probably low priority. Why target these when you can target Windows directly?
  • macosx libc - we don't need any startup files for this one, but we do need headers.
  • FreeBSD libc - follow glibc's example
  • NetBSD libc - follow glibc's example
  • OpenBSD libc - follow glibc's example

@suirad
Copy link
Contributor

suirad commented Mar 8, 2019

There probably be a new builtin.<libc-implementation> or something exposed to allow switching implementations at compile time. That should be added to the checklist.

The reason why it should be exposed and not assumed based on OS, is because some platforms are able to use multiple types, such as linux with musl & glibc.

@andrewrk
Copy link
Member Author

andrewrk commented Mar 8, 2019

There probably be a new builtin.<libc-implementation> or something exposed to allow switching implementations at compile time.

That's @import("builtin").abi, which is specified at compile time with -target <arch><subarch>-<os>-<abi>.

andrewrk added a commit that referenced this issue Mar 12, 2019
bundles musl 1.1.21

See #514
@andrewrk
Copy link
Member Author

Musl is done
Screenshot_2019-03-12_17-24-17

Repeating this here:
Currently zig builds musl from source with minimal modifications. The next step will be to also include std/special/libc.zig (which I have not started yet) as the root source file when creating libc.a for musl. And then slowly, we can start deleting C source files from the musl that zig bundles, and porting the code to zig instead.

Another nice optimization/research area would be to not have a separate .a file for libc, but actually export some additional functions from the same object file as the root zig source file. In this way the libc and the zig code could share implementations and shave off bloat. This would probably be helped a lot by an incremental compiler.

@andrewrk
Copy link
Member Author

For Windows, someone pointed me to MidiPix which could be another source of inspiration.

andrewrk added a commit that referenced this issue May 16, 2019
 * introduce wasm32-freestanding-musl .h files to fix
   conflicts with stddef.h and errno.h
 * fix an issue with zig build system regarding installation of
   webassembly libraries
 * add implementations to zig's libc:
   - strcmp
   - strncmp
   - strerror
   - strlen

See #514
andrewrk added a commit that referenced this issue May 30, 2019
 * introduce wasm32-freestanding-musl .h files to fix
   conflicts with stddef.h and errno.h
 * fix an issue with zig build system regarding installation of
   webassembly libraries
 * add implementations to zig's libc:
   - strcmp
   - strncmp
   - strerror
   - strlen

See #514
andrewrk added a commit that referenced this issue Jul 10, 2019
@andrewrk
Copy link
Member Author

#2868 has been merged and now we have a libc for Windows.

I think the proof-of-concept is done and now this issue can be split into separate issues:

Future work will be consolidating common functionality, e.g. the math functions, using one canonical implementation. See #2879 for that.

@wakaztahir
Copy link

wakaztahir commented Mar 16, 2024

@andrewrk Hi, I am building a small programming language, I can see that zig can compile libc, I've found the sources for libc's that zig compiles in the zig-bootstrap repository, Can you tell me which command is used, how are these libc's compiled, I am trying to implement cross compilation in my llvm programming language

https://github.com/Qinetik/chemical

The problem is that when I use lld to link a single object file that I created from llvm::Module, it gives error for unresolved printf symbols, Now I need to link against standard library using -lc but that also gives error that library not found, In this case this is a problem because I don't want my compiler to fail because lld can't find standard lib, the right way probably is to just compile libc from sources and link against it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contributor friendly This issue is limited in scope and/or knowledge of Zig internals. enhancement Solving this issue will likely involve adding new logic or components to the codebase.
Projects
None yet
Development

No branches or pull requests

5 participants