Regenerate C files with Cython 0.29 #90
Conversation
Guessing we don't need this in the source as HTML files are not generated for the other Cython modules, so dropping this in an effort to tidy things up here.
Use the latest Cython to regenerate C files for distribution. These should include support for Python 3.7 as well, which will be useful as we add Python 3.7 support to Numcodecs generally.
|
Thanks @jakirkham. This would probably be a good time to also upgrade the packages specified in requirements_dev.txt, so versions used in our dev and CI environment are in line with version of cython used to generate the C sources, and bring everything else up to date too. When I upgraded requirements_dev.txt recently in the zarr repo, I started from a blank environment, then used pip to install the latest version of all the packages that are imported anywhere in the zarr codebase, then did a pip freeze and dumped out to requirements_dev.txt, then moved lmdb and bsddb3 to requirement_dev_optional.txt (because these are only run under linux CI). Probably also a good idea if pinned versions of packages in requirements_dev.txt are the same between zarr and numcodecs projects, although the set of packages required is not exactly the same (i.e., some packages are needed for zarr and not strictly needed for numcodecs). Not sure what is the best way to achieve that. There is also a security warning currently regarding the version of the requests package in the requirements_dev.txt on the zarr repo. Doesn't look serious but would be nice to make it go away. Maybe we should update requirements_dev.txt on both repos at the same time? Any other suggestions for how to manage the process of upgrading the pinned dev/CI requirements? |
|
Updating the requirements sounds like a good idea. Would recommend enlisting a bot to do this for us to keep the maintenance simple. Basically these send PRs periodically to update any dependencies that are out-of-date. One of the common ones along these lines I've seen is PyUp. Looks like GitHub's Marketplace has a few other options with a couple of them noting Python support. WDYT? |
|
A bot sounds cool, but what's a bit complicated is that requirements_dev.txt currently pins everything in the environment, and some of those are indirect dependencies, i.e., not directly imported by zarr or numcodecs. In upgrading a direct dependency, that package may add or drop completely an indirect dependency, and I don't think a bot would be able to capture that type of change. Also if a bot is just bumping version numbers, but there are some incompatibilities expressed in min or max versions for other packages, that would not get captured. What I think we need is a way to specify the direct dependencies, then a way to generate a complete list of pinned packages that would be obtained by installing the latest versions of direct dependencies into an environment. Then redo that procedure periodically when new versions of direct dependencies become available. Btw I think it is a good thing to be listing and pinning everything in requirements_dev.txt, not just direct dependencies, so we get fully reproducible builds. I guess the alternative would be to list only direct dependencies in requirements_dev.txt, and let indirect dependencies float free. Then we could use a bot like PyUp. |
Just to add this makes me slightly nervous, but would make maintenance much simpler, especially if we can use a bot. Tempted. WDYT? |
|
Yeah I think you're right that the bot probably won't pick up indirect dependencies. Historically PyPI hasn't provided enough data to make this reasonable (mostly because there are old packages that don't supply this information either and still are on PyPI). Incompatibilities are of course also hard. Though even Yeah, can definitely see the value in having a full Dropping indirect dependencies is certainly another option. There's at least one case where this has been helpful for us. ( zarr-developers/zarr-python#126 ) So maybe this is a good option after all. On a totally unrelated note, we could set up daily cron jobs of the CIs to be notified when anything goes off the rails. Maybe this helps alleviate one underlying concern motivating this discussion. |
|
At least as far as this PR is concerned, we probably don't need to address all of these issues. Though I agree they are things we should address and ideally soon. For now, have bumped the Cython dependency to match what I used to regenerate the C files. |
|
On Thu, 1 Nov 2018, 10:23 jakirkham ***@***.*** wrote:
At least as far as this PR is concerned, we probably don't need to address
all of these issues. Though I agree they are things we should address and
ideally soon. For now, have bumped the Cython dependency to match what I
used to regenerate the C files.
Ok, cool, good idea. Let's deal with upgrading other requirements
separately.
—
… You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<#90 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAq8QoTSZFW4fAWaCeWumUcm-XlXlQLRks5uqxHigaJpZM4YElOa>
.
|
|
Have broken out the requirement refreshing ( #91 ) and using a bot to manage requirements ( zarr-developers/zarr-python#322 ) as issues. |
…9_regenerate_c_files
2 participants
Fixes #86
Went ahead and used the latest release of Cython (0.29) to regenerate the C files here. Not sure if there is a version preference, but happy to update if needed. Should move us toward Python 3.7 support.
Also there was a
vlen.htmlfile in here, which I went ahead and dropped as we didn't have HTML files for the other Cython sources. Updated.gitignoreto keep these from sneaking back in again. ;) Though if I missed something and this needs to be reverted, happy to do so.TODO:
tox -e py36passes locallytox -e py27passes locallytox -e docspasses locally