add brotli to PyPI repository

[anthrotype]

It would be nice to add Brotli to the official Python Package Index, so that users can download it with a simple pip install brotli.

We could add just the sdist tarball, or also some pre-compiled wheel packages for Windows and Mac platforms, maybe built automatically via Travis and/or AppVeyor -- like here

/cc @khaledhosny

[khaledhosny]

We probably need a release of some kind to by able to identify which version have been uploaded?

[anthrotype]

@szabadka would you be ready to tag a release?

[szabadka]

There will be some more improvements to the encoder in the coming weeks and I would prefer cutting a release after that.

[anthrotype]

Ok, fair enough. Looking forward to those.
Thank you,
C.

[szabadka]

I have one quick question about the tests in the python subdirectory. Is there any easy way to run them on Linux in a self-contained manner, i.e. without needing to install anything outside the root directory in the repository. Ideally the following would work:

$ python setup.py build_ext
$ python setup.py test

The first command succeeds, but when I run the second, I get 'ImportError: No module named brotli', which suggests to me that it is looking for brotli in a system-wide lib and not in the created build/ directory.

[anthrotype]

you're right, it shouldn't do that.
I'll try to fix it, thanks.

[anthrotype]

@szabadka this works: #80

[szabadka]

Thanks, I confirmed that it works on Linux.

[anthrotype]

Good! I can confirm it works on OSX and Windows too.

[khaledhosny]

Before the release we would need also to make sure the Python module exposes all features of the brotli library (like the new encoder params).

[CoolOppo]

I'm not one to really judge here, but if you guys are deciding on a versioning scheme, take a look at Semantic Versioning! This project would benefit from it over other versioning schemes because it makes it easy to determine when breaking changes have been made to the API, which is obviously a concern considering the last comment that @khaledhosny made in this thread.

[anthrotype]

Before the release we would need also to make sure the Python module exposes all features of the brotli library (like the new encoder params).

Yes, we should add the encoder's newly introduced BrotliParams.

What other feature would you like to expose in the Python module?
Currently, the module's compress function calls BrotliCompressBuffer, whereas the bro tool is using the new BrotliCompress with in/out callbacks.
Shall we expose BrotliCompress in the Python module as well?

[khaledhosny]

I think the parameters would be enough, unless we can come up with a compelling use case where Python users would benefit from the granularity of the API.

[anthrotype]

do you think brotli is now ready to be uploaded to the Python Package Index (PyPI)?

[szabadka]

We just pushed a new version of the encoder and decoder, which is ready to be uploaded to PyPI after we fix the issue you pointed out.

There is one more thing that could be changed in the python interface. In the latest version of the encoder, the advanced fields in BrotliParams were deprecated and are ignored (these are enable_dictionary, enable_transforms, greedy_block_split, enable_context_modeling).
I think it would make sense to remove them from the python command line interface as well.

[anthrotype]

@szabadka very good!

I have a question: those fields are not exposed in the command line script (bro.py) already. I gather you meant we need to remove them also from the brotlimodule.cc extension module.
I'll do that shortly in a new PR.

[szabadka]

Yes, I meant the brotlimodule.cc

[anthrotype]

do we also need to bump the Python module's version in the setup.py before uploading it to PyPI?
currently it's 0.1, but 1.0 would look nicer ;)

[szabadka]

Actually, we are planning to tag the current version of brotli as v0.1.0

[anthrotype]

Ok, that's fine.
Could you hold on an hour or so before actually tagging the release?
I'd like to submit one more PR that fixes building the Python extension on Windows Python 2.7 when installing via pip. Thanks

[szabadka]

Sure, let me know when the python part is ready.

[szabadka]

I configured the Travis deployment in PR #147.

@anthrotype Is this all we need to do? Will every release automatically uploaded to PyPI now?

[anthrotype]

sorry for the late reply, I was on vacation.

You have set up automatic "Github Releases" deployment for Travis. The OSX compiled wheels will now be uploaded to GitHub every time you push a new tag (e.g. see v0.2.0). This is cool, thanks for that! (By the way, Travis currently builds for OS X only; it would be nice to similarly configure AppVeyor with "Github Releases", so that the Windows wheels are also uploaded to GitHub...).

However, the Github Releases and PyPI are two different things which need to be configured separately.

Travis has some built-in support for automatic PyPI deployment -- see help docs:
http://docs.travis-ci.com/user/deployment/pypi/

For AppVeyor, you can define a custom deploy_script that calls the
twine command to securely authenticate with PyPI and upload the compiled wheels.

I'd say, let's first have both OSX and Windows wheels auto uploaded to GitHub upon tagging. Then we could see how to have them uploaded to PyPI as well. In the meantime, we could just upload them to PyPI manually.

[cicku]

I just read news about this. I think it's great (better than zopfli).

It'd be better to use it from pypi.

[Lukasa]

FWIW, my CFFI-based Python wrapper for Brotli is available from PyPI, but I very deliberately did not register the name brotli. If you are still planning to ship the C-based bindings to PyPI, you should make sure you register that name as soon as possible to prevent anyone else from accidentally stepping on it.

[TheNain38]

Any news on this?

[bsergean]

+1 for this. Great to know that @Lukasa has published a wrapper but it would probably be better to have the "legit" module on PyPI.

[anthrotype]

what is the status of this? Is there something I could do to help speeding this up?

Basically, all is left to do is register with the PyPI server and upload the binary wheels from 0.3.0 release, along with the source distribution (i'd say both .tar.gz and .zip formats, as in python setup.py sdist --formats=gztar,zip).

http://python-packaging-user-guide.readthedocs.org/en/latest/distributing/#uploading-your-project-to-pypi

Setting up automatic PyPI deployment upon tagging from Travis and Appveyor is a little tricky, but it can be done if you're interested in doing that too.

[eustas]

Hello. Let's revive this effort.

What needs to be done to proceed with publishing PyPI module?

[anthrotype]

It's all covered in the PyPA Packaging User Guide linked above. You need to create an account on PyPI, make a .pypirc file and use twine to register the new project and upload the built artifacts.
You can optionally have the CI deploy to PyPI on tag releases, in which case you need to encrypt the password.
https://docs.travis-ci.com/user/deployment/pypi

[anthrotype]

I can help you with the PyPI thing. Setting up the CI is possible, though I would need the PyPI account credentials to set that up.
I don't know if it's feasible, but I was thinking I could maintain the PyPI account for you, and manually upload the wheels on every new tag.

[Lukasa]

While this is still ongoing, I'll continue to remind people that there is still a CFFI-based wrapper published on PyPI under the name brotlipy. The update to v0.5.1 of brotli should be coming along shortly.

[anthrotype]

@Lukasa thanks for the reminder. I haven't tried it yet, but it would be nice if the two wrappers had the same API so they could be used interchangeably.

[Lukasa]

@anthrotype Agreed. I think it'd be better for the core brotli implementation to choose what that API looks like though. =) At that point, I'd consider brotlipy to basically be a wrapper that is primarily useful as a drop-in replacement for people using PyPy.

[anthrotype]

So, any news on the PyPI front?

At the very least, it would be already something if someone could upload to PyPI the wheels and sdist from the current 0.5.2 release.

All it takes is:

1. create a PyPI account at https://pypi.python.org/pypi?%3Aaction=register_form
2. install Twine tool, required to securely upload packages to PyPI

pip install twine

3. run twine upload Brotli-0.5.2*.whl Brotli-0.5.2.zip

I could do that myself, but would be better if the official owners/maintainers register it first.
If you like, you can add me as collaborator on the newly registered brotli project on PyPI, and then I could help you setting up automatic deployment from Travis/Appveyor.

But for now, uploading manually the artifacts that are already published on Github Releases would be good.

Thanks.

[anthrotype]

@eustas sorry to ping you (I see you are back online). Any plans on pushing the wheels to PyPI?

[eustas]

Hi. Actually, thanks for pinging! Going to do it now.

[eustas]

Hey! That happened! Wasn't that scary as I thought =)

Yup, making releases automated is a nice idea. But to add you as a maintainer I need to know your username =)

[anthrotype]

AWESOME!!! Big thanks!
https://pypi.python.org/pypi/Brotli
my username is anthrotype, of course ;)

[tripulse]

On Windows, this fails to load with ModuleNotFoundError. brotli._brotli seems somekind of CFFI binding to the underlying library that does the compression (as of sourcecode it's written in C++).

ModuleNotFoundError: No module named 'brotli._brotli'