Skip to content

0001: Enhanced static compilation and C interface

Jump to bottom
Jameson Nash edited this page Feb 8, 2018 · 4 revisions

Primary issue

The behavior of ccall depends on the ability of codegen to recurse back into Julia code. This makes it's actual behavior slightly unpredictable. It also means the static compilation semantics are subtly and unpredictably different from the JIT compiled semantics. This in turn severely limits the ability of the generic function cache logic to coalesce calls, as it doesn't know if this might affect the meaning of a ccall.

Summary of solution

Remove the special handling of ccall in the parser. Make the special-case lowering code more similar to other macro-expansion. Remove the Intrinsics.ccall function. Introduce a new Expr(:foreigncall) to represent the primitive foreign function invocation.

Later, add new features to Expr(:foreigncall) that the old structure couldn't easily represent, including: integrated handling of llvmcall functions and arbitrary function and argument attributes.

Goals

  • Perfectly statically compilable (use declarative syntax rather than runtime discovery)
  • Ability to split declaration and usage sites
  • Zero Intrinsic functions
  • Break runtime dependence on requiring LLVM as a backend
  • Support all current features
    • Parameterized return types for Ptr, Array, Ref
    • Call by value or name
    • Runtime/lazy memoized dlsym
    • Explicit library linking
    • Calls to LLVM intrinsics and LLVM modules
  • Add support for attributes (pure, fortran, compiler target)
  • Add support for calling convention on cfunction
  • Add support for backend-managed LLVM modules from other frontends (e.g. Cxx.jl),including for precompilation & serialization
  • Less usage of @static? (fewer invalid syntax issues)
  • More predictable behavior of the type parameters to ccall
    • Makes it clear that this is called in the global scope
    • Don't permit dependancies on the later definition of other functions, late binding resolution, etc.
    • Don't require the evaluation of arbitrary code to compile the function body
    • Don't permit uncomputable dependencies on the static parameters of the function
  • Permit dependencies on the static parameters of the enclosing function that resolve to simple UnionAll bounds and don't impact the ABI of the type (all pointers are equivalent, but other values may not be handled identically across all ABIs).

Non-goals

  • Supporting va_args more (left as an exercise for future PR)
  • Supporting changing the library name/path during load time (some thoughts for doing this was to make the library search function and DL_LOAD_PATH either be a special type or accept a thunk, like ifunc)
  • Preserving support for & argument syntax (although it is not actually changed here)

Implementation details

  • Remove special parsing rules for ccall (and thus removes Expr(:ccall)), fixes #18687
  • Make Expr(:call, :ccall) special in lowering (effectively, reserving ccall as a macro name)
  • Use Expr(:foreigncall) to represent the lowered ffi call (instead of expanding to Expr(:call, Core.Intrinsics.ccall)
  • Adds support to ccall for calling llvm intrinsics directly (via a "llvmcall" calling convention)
  • Evaluate the global arguments to the ccall macro during method definition time.
    • Eliminates the call back into the runtime from inside codegen (yay!)
    • Makes its semantics independent of runtime values (so that its behavior is statically compilable)

In this scheme, a call of the form ccall(to, :attrs, return_type, arg_types, args...) would effectively lower to:

roots = cconvert.(arg_types, args)
args = unsafe_convert.(arg_types, roots)
Expr(:foreigncall, to, $return_type, $arg_types, zip(args, roots)..., Expr(:meta, :attrs, ...))

Additionally, static parameters from the containing function can be used in the arg_types and return_type, provided that they don't impact the layout of the type (as seen by the C ABI). For example, f{T}(x::T) = ccall(:valid, Ptr{T}, (Ptr{T},), x) is valid, since Ptr is always a word-size bitstype. But, g{T}(x::T) = ccall(:notvalid, T, (T,), x) is not valid, since the type layout of T is not known statically.

Packages could provide alternative macros to configure the behavior in other ways, if desired. For example, we might imagine the creation of a @extern macro which converts extern declarations of C functions and wraps them in the appropriate callable type.

const library = "libraryname"
@extern name(ArgumentType,)::ReturnType in library # ccall to ("name", "libraryname")
@extern name(ArgumentType,)::ReturnType in library, pure, stdcall # ccall with some attributes
@extern name{T}(ArgumentType{T},)::ReturnType{T} in "libraryname" # ccall with a type parameter
@extern jlname = name(ArgumentType,)::ReturnType in library # ccall to "name" from `Module.jlname`
@extern jlname = (ArgumentType,)::ReturnType # ccall-by-pointer

Or we might use a ccall macro that just arranges the syntax more clearly:

const library = "libraryname"
ret = @ccall name(a::ArgumentType,)::ReturnType in library # ccall to ("name", "libraryname")
ret = @ccall name(a::ArgumentType,)::ReturnType in library, pure, stdcall # ccall with some attributes
ret = @ccall name(a::ArgumentType{T},)::ReturnType{T} in "libraryname" # ccall with a type parameter
ret = @ccall $julia_ptr(a::ArgumentType,)::ReturnType # ccall-by-pointer to julia_ptr::Ptr{Cvoid}

Attributes

Arbitrary attributes would be handled via syntactic symbols specified before the return type. A tuple of symbols would be interpreted to be specifying the attributes for the arguments, indexed by position.

ccall((:name, "library"),
  (:attr_for_function, :attr_for_return_value,
   (:attr_for_arg1, (:attr_for_arg2, :another_attr_for_arg2), (#=no attribute for arg 3=#), :arg4_attribute)))
  ReturnType,
  (ArgType1, ArgType2, ArgType3, ArgType4, ArgType5),
  arg1,
  arg2,
  arg3,
  arg4,
  arg5)

For attributes that take integer or string arguments, those value(s) should appear immediately after it in the list. For example, __attribute__((aligned (4))) would be expressed as ..., :aligned, 4, ....

If interpolation is desired, that must be done manually via @eval:

const attrs = (:(:fastcall), :(:pure), #=(:arg_attrs, :if_we_had_arguments)=#)
@eval myfun() = ccall((:myfun, "mylib"), $(attrs...), Void, ())

No validation on attributes will be performed by the frontend, but would simply move them all into an Expr(:meta) argument appended to the end of the Expr(:foreigncall) argument list. The lowering for ccall may choose to do some validation and call emit_error or directly print a warning for invalid or unhandled attributes.

Changes

2/8/18 vtjnash: added example @ccall syntax proposal demonstration

Clone this wiki locally