Julep: Internal fields and functions

[c42f]

(Moved to JuliaLang/Juleps#54)

The problem

Julia APIs use a mixture of public and internal fields so it's never syntactically obvious whether a given field access x.a violates API boundaries. This makes it hard to detect these problems when reviewing code and hard to write a linter which can tell the difference. That is, a local reading of the code can't tell the story; you must read the documentation of whatever package you're using.

Goals

This Julep aims to make the access of internal fields syntactically obvious. Ideally this is just ugly enough to make you think twice but not so bad that it becomes difficult to use the internal structure of types inside internal functions. Some specific goals:

1. Accessing internal fields should be syntactically obvious.
2. Declaring internal fields in structs should be syntactically obvious and consistent with the notation for access.
3. For use in the originating module it would be nice to have short syntax for getfield.
4. Avoid adding syntax and consuming precious ascii characters if possible.
5. Respect existing naming conventions in the ecosystem and aesthetic qualities of the language.

Prior discussion

• See extended discussion of binary $ as notation for getfield over at Support dict.field syntax for Dict{Symbol, <:Any} #25750.
• Binary @ also appears to be available and was suggested in the discussion of RFC/Julep: Introduce getproperty on Array for built-in data tables. #30646; some of the somewhat off topic musings there led to this julep.
• The meaning of export with regard to public symbols has been discussed in many places.

[c42f]

[edit: See Proposal 2 below, which I think is much better]

Proposal 1

A possible way forward is to change the lowering of x._a to a new function
getproperty_(x,:_a) rather than getproperty(x,:_a), and to define

getproperty_(x,s) = getfield(x,s)

This makes the underscore prefix into a convention for internal fields which can still be flexibly overridden in unusual cases.

I think the goals listed above are met, with the exception of 5 which is a bit of a mixed bag. Many packages use underscore prefixed function names for internal functions. Some packages (for example, ZipFile) also apply the underscore prefix to field names though there is no ecosystem wide consistency about this. Then again that's not to be expected as the very point of this Julep is to discuss possible conventions.

Compatibility

The definition above would technically be a breaking change with respect to existing code which overrides getproperty and also expects to consume symbols with leading underscores. If it's felt that this is too breaking for the 1.x series, we may of course define

getproperty_(x,s) = getproperty(x,s)

and document that getproperty_ will default to getfield in version 2.0.

[c42f]

I edited this a bit to emphasize what I see as the problem and goals, rather than my solution in "Proposal 1" which probably fails on aesthetic grounds.

I don't think the problem and goals are precisely stated yet and syntax may only be one facet of the larger problem of "why can't I know whether I'm using internal detail of an API by local inspection of the code".

[c42f]

Here's a counter proposal which attacks the problem from a very different angle.

Proposal 2: Module-local lowering of field access

Change the lowering of x.a to a module local shim function getprop(x, :a), which defaults to getproperty for all types, but automatically gets a method which calls getfield for every new type introduced into the module.

I think this does much better on several of the stated goals; it kind of "dissolves" goals 1-2, and does far better on goal 5, as it doesn't uglify the implementation details of a module.

Implementation sketch:

module Mod1

export Private, Public, use_field

# @private struct Private ...
struct Private
    a::Int
end

struct Public
    a::Int
end

# use_field(x) = x.a
use_field(x) = Mod1.getprop(x, :a)  # New lowering for x.a

## The following are auto generated
# Each module gets a default `getprop`
getprop(x, s) = getproperty(x, s) 
# Each type gets a method of the module-local `getprop`
getprop(x::Private, s) = getfield(x, s)
getprop(x::Public, s) = getfield(x, s) 
# Invocations of `@private struct ...` generate something like
Base.getproperty(x::Private, s::Symbol) = error("`Private.$x` is an internal field")

end


module Mod2
getprop(x, s) = getproperty(x, s)  # Auto generated

using ..Mod1

# use_field(x) = x.a
use_field(x) = Mod2.getprop(x, :a)

end

Mod1 is allowed to use the fields of types defined inside it with agreeable syntax:

julia> Mod1.use_field(Mod1.Public(1))
1
julia> Mod1.use_field(Mod1.Private(1))
1

Mod2 is not allowed to use the nice syntax to access the internal field of Mod1.Private. It would have to use getfield for that:

julia> Mod2.use_field(Mod1.Public(1))
1
julia> Mod2.use_field(Mod1.Private(1))
ERROR: `Private.Main.Mod1.Private(1)` is an internal field

What about cases where implementation details are shared between modules? For example, between several closely cooperating submodules within a project? This could be handled by explicitly importing getprop so that the method table is shared between modules.

Compatibility

Introducing the above is fairly compatible with the existing mostly getfield-based ecosystem, as only types marked with @private would disallow module-external access to the fields. However, it breaks packages which define getproperty and use it internally within the same module via the x.a syntax.

A transition plan during the 1.x timeframe could be to make this lowering opt-in on a per-module basis, with a compiler meta attached to the module AST. Then switch the lowering completely in 2.0.

[c42f]

I keep wanting to edit the description... I think perhaps I'll move this to the juleps repo instead.

See JuliaLang/Juleps#54