Setting an output directory

[richburdon]

"python" below sets the python_out flag; but how do you assign a value?
e.g., python_out=$buildDir/generated-sources/main/python

sourceSets {
  main {
    proto {
      builtins {
        python {
        }
      }
    }
  }
}

[zhangkun83]

The value should have been set to the default output directory, which is $buildDir/generated-sources/main/python. Do you want to set it to somewhere else? Why do you want to do that?

[richburdon]

No, g-s/main/python would be ideal, but currently both my java and python are dumped into g-s/main

Also, is there a standard for building a python package? I don't want to put my setup.py in build/... and if I specify package_dir to the generated directory then the packaged files have the generated-sources prefix.

[richburdon]

See https://github.com/google/protobuf-gradle-plugin/blob/master/src/main/groovy/com/google/protobuf/gradle/ProtobufCompile.groovy#L112

[zhangkun83]

Sorry I was wrong. The default output directory is in fact $buildDir/generated-sources/main. The plugin was originally created for java projects only. Extending its ability to build non-java project is an interesting topic. For now, it might not hurt to provide an option for output directory in the builtins.

[richburdon]

Yes -- one of the many purposes of protobuf is to be cross-platform/language so, that would be good. I think the right behavior would be to append $builtin.name (i.e., to mirror src/main/{java|python}), but I guess that would change the current behavior -- unless that only happens if multiple languages are detected?

Also, do you have any thoughts on the packaging problem? I.e., setup.py having to be in a different directory and no init.py being generated?

[0o-de-lally]

+1 for @richburdon 's issue. Would be nice to deal with hybrid environments better.

[zhangkun83]

Also, do you have any thoughts on the packaging problem? I.e., setup.py having to be in a different directory and no init.py being generated?

@haberman - since you work on Python Protobuf, could you help on answering this question?

[richburdon]

@zhangkun83 any thoughts here? I can patch to correct this, but would appreciate some thoughts on how to deal with legacy issues. The build tools should assume a standard file layout -- especially in the case of cross-platform technologies like protobuf.

My proposal: each xxx_out should be able to specify a different output directory.

[zhangkun83]

@richburdon I am current working on a redesign of the plugin DSL for 0.5.0, which will support the kind of configuration you have proposed.

[zhangkun83]

TL; DR: I am working on the new version of the plugin. I thought I would be able to allow customizing the directory for each builtin/plugin, but it turns out almost impossible due to constraints from Android build model. I am thinking about separating the outputs per builtin/plugin as previously suggested, but not allowing customizing them. Would that suffice?

The full story: Android projects are compiled per variant (an Android concept), not per sourceSet. In order to work with Android projects, the Protobuf plugin has to have a protoc run per variant, thus the protoc options have to be configured per variant. So we are moving the configuration knobs to the generateProto tasks, because we will have one generateProto task for each variant.

Variants are generated in afterEvaluate phase, thus generateProto tasks are generated after that. At the time a generateProto task is generated, we register it and its output dir to the corresponding variant (so that the generated files can be compiled by the Java compilation task). Only after that can the build author configure the tasks. If the build author then changes the output dir, we will have trouble updating it on the Android variant because Android plugin doesn't seem to be an API for that. Ideally this can be resolved by delaying registering generateProto tasks to Android variants to a point after the build author has done configuring the tasks, but there doesn't seem to be a notification or hook for that, other than asking the build author to explicitly call something, which is also undesirable.

[richburdon]

I think the approach you're intending is optimal (i.e., fixed outputs per
language: java/python, etc.)

When can I try it? :)

On Fri, Jun 19, 2015 at 2:44 PM Kun Zhang [email protected] wrote:

TL; DR: I am working on the new version of the plugin. I thought I would
be able to allow customizing the directory for each builtin/plugin, but it
turns out almost impossible due to constraints from Android build model. I
am thinking about separating the outputs per builtin/plugin as previously
suggested, but not allowing customizing them. Would that suffice?

The full story: Android projects are compiled per variant (an Android
concept), not per sourceSet. In order to work with Android projects, the
Protobuf plugin has to have a protoc run per variant, thus the protoc
options have to be configured per variant. So we are moving the
configuration knobs to the generateProto tasks, because we will have one
generateProto task for each variant.

Variants are generated in afterEvaluate phase, thus generateProto tasks
are generated after that. At the time a generateProto task is generated,
we register it and its output dir to the corresponding variant (so that the
generated files can be compiled by the Java compilation task). Only after
that can the build author configure the tasks. If the build author then
changes the output dir, we will have trouble updating it on the Android
variant because Android plugin doesn't seem to be an API for that. Ideally
this can be resolved by delaying registering generateProto tasks to
Android variants to a point after the build author has done configuring the
tasks, but there doesn't seem to be a notification or hook for that, other
than asking the build author to explicitly call something, which is also
undesirable.

—
Reply to this email directly or view it on GitHub
#13 (comment)
.

[zhangkun83]

I am currently coding it :). Hopefully I can check it in in a week or two.

[zhangkun83]

@richburdon @Xorlev I have created branch for the next version v0.5.x. It puts the generated code under "$buildDir/generated/source/proto/$sourceSet/$name".

It has not been released to repositories, so you have to build it and install it locally. Please follow the README on how to do it. Feedback is appreciated.

[zhangkun83]

0.5.0 has been released. Please refer to the README on the master branch.

[Xorlev]

Apologies for not seeing this earlier. This looks great to me and I'll give it a spin tonight. Thanks @zhangkun83!