cmd/go: drop support for binary-only packages #28152
Comments
|
I'd like to see them removed, mostly for code simplicity. For the professor use case, perhaps source obfuscation of some form would be sufficient. |
|
Looking for people who use binary-only-package and might be able to explain what they use it for. @ianlancetaylor filed #23473 (but presumably passing along another report). /cc @dneil @dsnet for knowledge of any potential Google-internal usage. |
|
I'm not aware of anyone using binary-only packages inside Google. Doesn't mean someone isn't, of course. |
|
I'm in favor of the proposal, nonetheless I guess the important property of binary packages is they make some closed source package vendors feel "safe" about their IP. |
That or plugins need to not panic (when loading) and complain about different versions of the same package and also offer unloading support (i.e: find a way to sever the link with the runtime and free evreything). Vendoring binary packages is useful if you're selling an SDK or toolkit. |
|
Hi there, thank you for considering us!, In our case we use it as part of our offering to Enterprise customers. Being able to provide non-source distributions and as a way to hide some of our I.P clears us with the IT departments of such companies. Our biggest use is being able to provide a static library (the BOP package) to third parties without providing our source, we provide an SDK that should work without any connection to our Servers, hence we package the SDK as binary-only-package that includes some proprietary I.P. that is licensed. While that approach is not the most secure way to protect our source, it is the right balance right now, we just don't include more sensitive functionality in what we distribute. Another important characteristic is that being able to have a higher level of friction of preventing modification to what we distribute; i.e. even if we provide non-IP that has to function in a specific way, not having BOP's would make easier for third parties to modify the SDK. I'm not entirely in favor of dropping support for binary-only-packages because I believe that many other languages support a way of distributing packages that don't have the original source in them:
|
|
#23473 was for an internal Google use case (internal reference number 72160357) that was resolved by changing to not use binary packages. |
|
Posted to golang-nuts to cast a wider net for use cases. |
|
In general I'm in favour of giving incentive to provide source. But there are legitimate binary-only cases related to not only IP but security. What relationship would removing support for BOPs this have to plugins and related build modes (shared/c-shared/c-archive)? Could one still provide a binary in any of these forms and be able to use that? If so, then a simple solution for at least some BOP use cases would be to use a wrapper package that loads a plugin, or uses -linkshared or a c-archive via cgo and the plugin or c-archive would be outside of go get/modules and distributed separately as binary. I have no plans to distribute such things, but would like to be able offer the capacity to others in some form. If something like the above workarounds worked, then that would provide a way to drop BOP At the same time I understand there are still some issues with these other binary distribution mechanisms and modules. |
|
Thanks for asking about plugins. There are no plans to remove plugin support, and suggesting the use of plugins as binary blobs instead of binary-only packages makes sense to me. Plugins have many of the same concerns, but they already have a proper check for mismatched API (avoiding the silent memory corruption problem) and it's always better to have one mechanism than two. |
|
Are there any performance concerns for plugins? E.g., I would imagine that you can certain link-time optimizations with binary-only packages that aren't available for dynamically loaded code? |
|
I agree that binary-only-packages are problematic and should be removed. |
|
@joshlf It's hard for me to imagine any real case in which plugins are noticeably less efficient than binary-only packages. It's possible to construct an artificial case, but I doubt that would match any real implementation. |
Great to hear. It seems to me it wouldn't be too much to adapt the source stubs from a BOP to wrap a plugin and an init() to load it from somewhere, and well worth it to both BOP users and Go maintenance. |
|
Will leave this open for another week but it sounds like we can retire binary-only packages in favor of using plugins. We could advertise in the Go 1.12 release notes that it will be the last release to support binary-only packages and they will be removed in Go 1.13. |
|
Accepted since we waited a week and nothing came up/no objections were raised. Per discussion with @golang/proposal-review |
andybons
changed the title
|
As a corporate programmer, the closed-source use case is very important. A language that is so RMS idealism friendly that it doesn't even allow one to use closed-source dependencies is a very hard sell in many corporate environments. |
|
@edburns see the discussion above; using compiled binaries will still be possible via plugins. binary-only packages are harder to develop, maintain, and use, as can be seen above. The later stages of this thread were about finding any reason why plugins aren't a good replacement for binary-only packages, to which there wasn't a clear answer. |
|
@edburns Thanks for the comment. Do you have direct experience of this? How do those corporate environments handle languages like Python or Javascript? |
|
Consider the case when a corporation is using a licensed closed-source software package as a dependency on an internal project. Right now, if the vendor of a closed-source package wanted to make their licensed package useful to go programmers, their only option is to author it as a plugin. As stated elsewhere, this has drawbacks: complexity of .so linking, lack of non Linux support, etc. |
I'd argue that using a binary-only package was never really an option. The binary would require an exact version of Go, as well as an exact version of all the packages it depends on. It's certainly possible, but I wouldn't say it's ever been a good or easy way to distribute software. The intended purpose, the professor scenario, is a much more restricted use case.
Please expand on this. If you believe there's a bug or something to improve in the plugin package or the go tool, you should open a separate issue.
What particular platform is noticeably missing plugin support? Assuming you mean Windows, that's being tracked in #19282. I believe effort would be better invested in issues like that one, instead of on maintaining binary-only package support. |
|
@edburns Thanks. Personally I am more interested in direct experience than in hypotheticals. We can construct a number of hypothetical situations, but in the end they won't tell us what to do. |
|
It is possible to use vendoring with plugins (without set package import path)? The use of multiple plugins is hard because they need to use exactly the same version of common dependencies. Would it be possible to use the latest minor version for each dependency otherwise? |
|
Again,
I’m very grateful for the consideration
|
|
@cpipero and @aaveidt, from a "code hiding" point of view, the compiled .a file has a lot of source information embedded in it. A LOT. It might be equally practically effective just to run a code obfuscator on your code and distribute source. Perhaps much better, since it would hide more of the internal names. If such a code obfuscator tool existed, would that work as a replacement for binary-only packages for you? |
I don't know about the current cases, but the malware community would be very happy with such a tool to make reverse engineers' tasks slightly more difficult. cf. https://blog.malwarebytes.com/threat-analysis/2019/02/new-golang-brute-forcer-discovered-amid-rise-e-commerce-attacks/ |
|
We ran that option to the customers (two) and both legal teams came back with “no”.
On Feb 27, 2019, at 2:53 PM, mirtchovski <[email protected]<mailto:[email protected]>> wrote:
If such a code obfuscator tool existed, would that work as a replacement for binary-only packages for you?
I don't know about the current cases, but the malware community would be very happy with such a tool to make reverse engineers' tasks slightly more difficult. cf. https://blog.malwarebytes.com/threat-analysis/2019/02/new-golang-brute-forcer-discovered-amid-rise-e-commerce-attacks/
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#28152 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABwUtxNqITLOdHZt5-6U23TDQ_as8-jcks5vRuI_gaJpZM4XXyuW>.
|
|
@cpipero, you might need to help explain things to your legal team, then. You can't expect them to know how all (or any) compilers work. |
|
@rsc, I'm supportive of exploration of Go obfuscation in general, but curious if it isn't trending a more broadly applicable solution near @Fabian-F and @cpipero's comments regarding auto-generation of plugin scaffolding. I spent years reverse engineering obfuscators for MMORPGs who used a wide variety of Java obfuscators, open source and commercial alike. Bytecode is easily represented in source - so in effect, even the best Java obfuscator is basically what you're proposing - a source obfuscator. Reversing these obfuscation techniques were common practice by myself and other teenagers. Also, at the end of the day, much of the obfuscation techniques that work well will diminish the optimization strategies of the compiler, as well as negatively effect performance of the compiled binary. This is now how most enterprise mobile obfuscators work today. Nowadays, I work in information security and actively research and develop techniques of binary obfuscation using Go. I've spent a lot o time leveraging the power of I still believe the stability of the language ecosystem and compiler requires BOPs to be deprecated, but I do think there is room to innovate something on behalf of those who have use cases. How do you feel @rsc and @ianlancetaylor about the concepts of some sort of plugin scaffolding generator? I don't know where this lives, but I see the feature working something like this:
What happened? The compiler walked the AST looking for plugin exports, then generated a scaffolded export that implements those exported types and function calls using a standardized entrypoint. This go code, along with the plugin, could then be distributed to developers. Thoughts? [edits for grammer/spelling] |
|
Not my legal team: the customer’s.
You base your project on a set of documented features that keep all their IP concerns at bay. Then you go to them one year later saying “we can do two out three but we can’t keep the code in binary form as previously stated “.
Believe me, at those companies the last thing you want to do is to school their lawyers on computer science.
But we are getting off track. I don’t think people who are pushing back are questioning the technical reason for the decision. More than anything we are hoping for a more structured approach to change. And I have seen a few good ideas in the last hour or so.
Success is often a compromise
On Feb 27, 2019, at 3:15 PM, Brad Fitzpatrick <[email protected]<mailto:[email protected]>> wrote:
@cpipero<https://github.com/cpipero>, you might need to help explain things to your legal team, then. You can't expect them to know how all (or any) compilers work.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#28152 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABwUt1djKESR4JKOt4xDAAEEiXhGlhKzks5vRudogaJpZM4XXyuW>.
|
@mirtchovski, I would encourage you to try and separate the deficiencies of the security industry from the needs of software engineering as a whole. The information security industry is a billion dollar industry that by in large, is filled with vaporware. The cognitive dissidence is on full display when technology is discouraged due to "what it will mean in the hands of hackers". This same mindset fueled anti-encryption sentiment in the 80s and 90s, and continues today around the need to invade user privacy more and more. Hackers do not need this hypothetical obfuscator to lay waste to even the most sophisticated anti-malware systems. Those system's are ineffective, not because the hackers have magic tools, but because they are attempting to solve a non-sequiter. They have to be flexible enough to not cause false positive detection/prevention, but also detect nefarious behavior. Since the possible permutations of this is different not just from company to company, but from employee to employee, and even day to day per employee, heuristics based detection will never be the primary defensive strategy. Good security is about empowering the user to be inherently secure. Go does a great job of this in their Sorry for the long position, but as a security professional and an avid Go user, I've got an interest in not letting security theater discourage innovation. [edit to add the forgotten mention to reply] |
|
@gen0cide I believe this code generator idea to be a good workaround, and was my plan for when this change goes ahead anyway. It being part of Go's core tools would definitely soften the impact of losing BOPs. However, plugins need to work on all main platforms before they can ever be considered an alternative. Also aside from trimpath and |
|
I think we should focus on the technical arguments against adopting plugins. So far, it seems like the only missing critical features in plugins are Windows support and the ability to expose type definitions. Is that correct? If so, perhaps we should move the discussion towards looking at those issues, and perhaps even blocking this proposal until they've been considered. I don't think verbosity is nearly as important as the other issues, nor do I think it's a stopgap. Having to write an extra 50 lines is certainly not ideal, but there are many factors at play here. Perhaps Go2 type parameters would allow getting rid of some of the reflect boilerplate. |
|
@mvdan I agree. Plugins combined with a command line tool could replace BOPs, if plugins have the OS limitation fixed, support type definitions, and provide an option to export constant values (although the command line tool could simply copy constant definitions to the shared code). All of the boilerplate code is so generic and repetitive that it could be generated. |
|
I'm not terribly worried about the "malware community" one way or another. In the page you linked, the binary hadn't even been stripped. |
|
The fact is binary-only packages do not work with Go 1.12 except in limited circumstances. (And I understand that some people are working within those limited circumstances successfully, if luckily.) In Go 1.13 they won't work at all. The limited cases that do work in Go 1.12 will continue to work in Go 1.12 - we aren't going to retroactively remove them from Go 1.12. People who need extra time to move to what's next can keep using Go 1.12 as long as they like, of course. It seems like there are two paths forward. First, people could work on an obfuscator of sorts. Second, people could work on improving plugin support. |
|
@rsc Why not wait until plugins are at a sensible point before deprecating a solution that does work? |
|
@Fabian-F The unfortunate fact is that it doesn't work today, as described earlier. And it's going to break even worse with future changes such as module support. |
|
Change https://golang.org/cl/165746 mentions this issue: |
|
@ianlancetaylor But you've heard from 3 different people who have clients who successfully use them in a production environment, so it does work, today. |
|
@Fabian-F There are several options that the client could pass to |
|
Reading through all of these comments, it seems the question has evolved to "Can plugins replace binary-only packages?" and "What is needed to make binary-only packages viable?" Given that, I am hoping my comments are not off-topic, but if they are please forgive me and feel free to mark this comment as off-topic like one of the ones above. We are not actually using binary-only packages but very recently explored using plugins to allow a 3rd party extension mechanism for a product we are building that would be marketed to people who are not-Go developers. IOW, we are looking for an extensibility mechanism that would allow 3rd parties to build and deliver binary "add-ons" for our product to our mutual customers. Unfortunately we found plugins lacking, for two reasons:
So, I understand that plugins possibly — by design — will always require version parity between the main app and the plugin(s). But if that is the case then I think Go really needs a standardized extensibility mechanism that allows Go apps to load 3rd party extensions that were compiled with any compiler, maybe with communication over named pipes — can you do (something like) that in Windows? — or via protobuf over TCP/IP? We can, of course, develop our own, but I think this really should be a widely applicable use-case — imaging 3rd party extension for Docker? — and I doubt our team and many others would have the expertise and experience to envision all the edge cases that might be found by users, so I am asking that the Go team consider creating a "blessed" standard package for extensibility that does not have the limitations that I understand plugins have. That said, if I am misunderstanding anything about how plugins work, please forgive me for that as well. I have only been actively coding in Go for less than 6 months. |
|
I prefer not remove the feature, for protect some source code. |
|
@shuxiao9058 Protect source code from what? |
|
I thought this whole issue was closed and the decision was made to just drop the binary packages, and maybe, possibly, hopefully improve the plugins to replace with the same value proposition (that is, it solves a hairy business issue not a technical one -- and no, I cannot force legal teams to go through sufficient computer science training to change their ways). I understand your motivation and support your decision. We accepted this fact, but we also accepted the fact that basically the binary-only package option was a golang failure (among many successes of course). Our solution was to go back to good old C++ because of their stricter governance that prevents failures like this to have such short term impact. Of course, we will use and enjoy golang whenever the customer IP is not involved in the deliverable agreement. |
|
I think a big difference between binary only packages and plugins are the structs themselves. With BOP you can hide package level details in the struct, when you move to plugins, you need to create public only structs and then copy and extend to the internal structs. This is a lot of work. |
and others
Binary-only packages are increasingly hard to support safely. There is no guarantee that the compilation of the binary-only package used the same versions of the dependencies that the final link does (and it would probably be too onerous to insist on that). As a result, the binary-only package may have been compiled using escape analysis results or inline function bodies for dependencies that are no longer accurate. The result can be silent memory corruption.
My memory is that we added binary-only packages originally so that a professor could hand out binary solution sets so that a student who didn't finish lab 2 could still move on to lab 3. I don't know why else anyone uses them anymore, but they're probably going to break more and more as the compiler gets more sophisticated.
Should we just remove them? If not, why not? What do people need them for?
See also #28146 (less severe).
/cc @bcmills
The text was updated successfully, but these errors were encountered: