Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release asset naming #43

Open
sagikazarmark opened this issue Jun 25, 2018 · 15 comments
Open

Release asset naming #43

sagikazarmark opened this issue Jun 25, 2018 · 15 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@sagikazarmark
Copy link

Currently release asset names look like this: protolock.20180621T191216Z.darwin-amd64.tgz

It's quite hard to build tooling around this naming convention. It would be nice to have the version in the file name instead of the build date.

WDYT?
@nilslice
Copy link
Owner

@sagikazarmark - good thinking.. I would be happy to consider any changes you propose. Unless the timestamp is specifically needed by others, I think a version number would suffice in its place.

cc/ @noel-yap for guidance & recommendation as well.

@noel-yap
Copy link
Contributor

AFAIK, there have been no actual releases but now it occurs to me that I may not have been looking in the right place for any released artifacts. Are there publicly published released artifacts? If so, where are they?

If there are, IMO, it would make sense to follow https://semver.org/ and even https://keepachangelog.com/en/1.0.0/ and to document the release process (including publishing to some public repository).

And having said that, I may still have some invalid assumptions as to how Golang projects work WRT releases and published artifacts (my more recent experience has been in the Java world) so please call out these assumptions if this is the case.

@nilslice
Copy link
Owner

@noel-yap I believe this is in reference to the way the distributed pre-built binaries are named:

https://github.com/nilslice/protolock/releases

@noel-yap
Copy link
Contributor

Yes, I understand that. What are the steps for those binaries to be published there? Can, for example, someone insert malicious source, build it, then publish the binaries to that location without committing the code?

Is there something like Maven Central or Bitbucket for Golang? Currently, we're building the binaries on premises and publishing to a private Artifactory instance.

I didn't get much sleep last night so I'm probably not communicating well my concerns and how they relate to the versioning scheme. I'll try to break it down more for my sake so I'm hoping no one takes it as me talking down to them. Also, my background having been in release engineering might help put things into context.

Ideally, versions mean something (eg SemVer) and represent the exact same bits to everyone. This means that one can checkout a git tag or branch representing the release (eg release/v0.1.0) and produce the "same" binary as what's been published. This is why I consider it important to document the release process and to have more confidence that it can't be easily subverted.

When I see timestamps instead of version numbers, it's an indication to me that the inputs to the release can change (ie they're not hermetic) such that checking out a specific release branch and building may not actually reproduce what's been published publicly. The timestamps also, at least in theory, can be be used to figure out the actual inputs to the build (including the dependencies that were used).

If I'm still not communicating clearly enough my concerns, I'll try to get better sleep tonight and try again tomorrow.

@sagikazarmark , what sorts of issues are you running into WRT tooling?

@nilslice
Copy link
Owner

@noel-yap this is great feedback -- and it's fantastic to have someone with release engineering experience contributing to the project (I admittedly have very little personal experience in that domain).

What are the steps for those binaries to be published there? ...

Currently, the only process is manually running the release.sh script locally to cross compile. I have been stringent to use the same commit to build the binaries as the tag (version used for the release) is pointing to.

Is there something like Maven Central or Bitbucket for Golang? Currently, we're building the binaries on premises and publishing to a private Artifactory instance.

GitHub often is used as a place to host and serve downloads of pre-built binaries, but I am not familiar with any CI/build process that would integrate directly with it in an automated, remote "test -> build -> release" cycle.

As per a request from the @envoyproxy team: envoyproxy/envoy#3703 (comment), a SHA verification process may end up helping solve some of the issues you bring up. I'm not familiar with this process though, do you have any suggestions?

@noel-yap
Copy link
Contributor

I had thought of generating a SHA file but it occurred to me that it has the same issue. One can make a file change, build the artifact and the SHA file, publish them, and not commit/push the code change.

I want to be clear that I'm not implying you can't be trusted (I don't think you're reading into this this way but want to be sure). It sounds to me that you understand these processes are put in place also to guard against mistakes (eg one could forget to commit a code change, build out of the wrong area, etc).

There are a few things that need beefing up, IMO. You had mentioned automated CI and release. That's some of them. Another is ensuring the build is repeatable. This means not pulling dependencies at HEAD. I've heard other Golang projects would include the source of their dependencies as part of the project so that that wouldn't happen. I would rather find some way to use git submodules for this but I've heard it's not trivial.

If those things were put in place, I'd be all for using actual versions for this project. I'd investigate it myself but currently I have other priorities. @sagikazarmark , since you're feeling the most pain with the consequences of this, would you be able to do some legwork?

@nilslice
Copy link
Owner

@noel-yap this might be a solution for the automated build and release: https://goreleaser.com/

@sagikazarmark
Copy link
Author

what sorts of issues are you running into WRT tooling?

Knowing the version number and the URL scheme is usually enough to just upgrade the version in a script or a dockerfile. Having a timestamp in the URL makes that impossible without manually checking the releases page every time.

This means not pulling dependencies at HEAD. I've heard other Golang projects would include the source of their dependencies as part of the project so that that wouldn't happen. I would rather find some way to use git submodules for this

Golang has nice dependency management solutions that can lock to exact commits of other dependencies. Personally I don't like including the dependencies in the VCS when we have the tools to properly manage the dependencies. Since they are mostly third-parties anyway, I don't see the reason not to trust them when installing dependencies. If someone tampers with the version tags the dependency tool will either install it's locked version or complain that it's missing.

Goreleaser is a popular tool for building Golang CLI tools, I used it a few times.

since you're feeling the most pain with the consequences of this, would you be able to do some legwork?

What legwork do you exactly mean?

Here are a few examples for automated, pre-built binary releases:

https://github.com/golang/dep/releases
https://github.com/twirphp/twirp/releases

@noel-yap
Copy link
Contributor

noel-yap commented Jun 29, 2018 via email

@nilslice
Copy link
Owner

nilslice commented Jun 30, 2018 •
edited
Loading

I'd like to investigate GoReleaser for the automated releases. I just need to ensure that there is a checksum option to satisfy the needs of this PR: envoyproxy/envoy#3703 (comment)

@sagikazarmark - if you have some tips on integration, please let me know. Also, to get back to your original question, I think that if we just replace the timestamp with the release tag, it will get you what you want while still maintaining the immutability promise of the release (semi-provided by the timestamp). Do you agree? Please let me know if there are other suggestions you would make. @noel-yap if you disagree with this, let me know as well. I think it would be reasonable to support others desire to build tooling around the ability to download a release by using a URL template of: $GITHUB/nilslice/protolock/releases/download/$TAG/protolock.$TAG.$OS-$ARCH.tgz

What do you think?

@noel-yap
Copy link
Contributor

I'm OK with this so long as there's an audit trail from the binary back to the source.

@noel-yap
Copy link
Contributor

noel-yap commented Jul 12, 2018 via email

@sagikazarmark
Copy link
Author

Sorry, I lost in the jungle of Github emails.

Having the tag in the file name is okay, actually a good practice. The proposed format looks good to me. 👍

@sagikazarmark
Copy link
Author

This should work for CircleCI and Goreleaser:

https://github.com/banzaicloud/banzai-cli/blob/master/.circleci/config.yml
https://github.com/banzaicloud/banzai-cli/blob/master/.goreleaser.yml

@nilslice
Copy link
Owner

Awesome -- will dig in once I'm back from vacation. Thanks!

@nilslice nilslice added enhancement New feature or request help wanted Extra attention is needed labels Feb 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sagikazarmark @noel-yap @nilslice