Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add date and time of pip-compile to output #306

Closed
wants to merge 1 commit into from
Closed

Add date and time of pip-compile to output #306

wants to merge 1 commit into from

Conversation

khwilson
Copy link

Hi again!

This whole toolchain is great and we've incorporated it into our workflow, but we've found that people sometimes forget to recompile their requirements.txt files when they change their requirements.in files. It's easy enough to ask git for the list of files that changed in a commit, but even if the requirements.in changed, it's not necessarily true that the requirements.txt changed (e.g., a higher lower bound has a decent chance of not changing the version that pip-compile chooses).

So to make our check easier to implement (and much faster and lightweight) we wanted every requirements.txt file to have something slightly different in it, and the most convenient such thing is the date.

This PR adds a flag to pip-compile to include the date in the comments of the output requirements.txt. It does not change the default behavior.

Best and thanks again,

Kevin

Our use case involves checking to make sure that after someone
has changed requirements.in that they have also made sure to
rerun pip-compile.

Does not change default behavior, by adding --date flag will the
additional line be output.
@@ -27,6 +29,8 @@ def write_header(self):
if self.emit_header:
yield comment('#')
yield comment('# This file is autogenerated by pip-compile')
if self.write_date:
yield comment('# On {} UTC'.format(datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree UTC is unambiguous, but I'm not sure how useful it is. I think I'd prefer the local date for the user that last generated it. This is for human consumption after all.

@blueyed
Copy link
Contributor

blueyed commented Jan 27, 2016

While I can see that having a timestamp might be useful in general, I cannot see why it helps in your use case.
It sounds like you want to force a changed .txt if the .in was changed? Why isn't it enough to only commit the .in then, without the unchanged .txt?

btw: you can regenerate .txt files for .in files in make with this:

$(PIP_REQUIREMENTS_DIR)/%.txt: $(PIP_REQUIREMENTS_DIR)/%.in
    pip-compile $< > $@

@nvie
Copy link
Member

nvie commented Jan 27, 2016

I agree with @blueyed here. You typically also check in the requirements.txt into version control, which should reflect the last time the file changed anyway.

@khwilson
Copy link
Author

Thanks for taking the time to make suggestions! I really appreciate it.

The problem is that if you have a requirements.in change that doesn't affect your requirements.txt, then there is no way to verify that someone regenerated the file without regenerating the file yourself.

An example of this occurring happens when, say, you are using boto and you really cared about a feature from 2.8, so your requirements.in says boto>=2.8,<3. Then you realize that Amazon changed the way they handle credentials files, and they incorporated this into boto at 2.29, so you change your requirements.in to boto>=2.29,<3. However, as there is boto==2.38.0 available, nothing will change in your requirements.txt.

Now I would like to check that if requirements.in changed, the user did, indeed, regenerate their requirements.txt. Most of the time that's easy (just check with git --diff --name-only HEAD^ and make sure that either both files are present or neither file is present), but in the above scenario, your test becomes a bit more complicated. In particular, you have to regenerate the requirements.txt file and diff.

There are two major issues with this strategy: First, pip-compile takes a while, so your unit tests job can slow down briefly. Not the worst, but it garners complaints.

Second, and more nefariously, you get things like #277 cropping up, where your CI infrastructure is on a different platform (e.g., standard Amazon Ubuntu AMI) than your development boxes (e.g., standard Mac OS X).

Anyway, that's the whole motivating story. Perhaps there's an easier solution that I'm not seeing?

And local time makes sense, but our problem is that we have devs in a couple time zones, so it becomes a bit of a game either way. Perhaps default to local time but have date take an optional argument for a UTC offset? So you can always stipulate --date 0 if you want UTC?

@blueyed
Copy link
Contributor

blueyed commented Jan 27, 2016

no way to verify that someone regenerated the file without regenerating the file yourself

But the file being changed does not mean that it's correct?!
For that you would need to regenerate it anyway, and that might include some updates then? (there's a new flag regarding that, but still).

You might want to add a check to your pipeline that does the real check, but possibly in a subtask on Travis/CI instead of in the regular developer's test suite.
Given caches etc (and not too many VCS requirements), it should be pretty fast. With Travis for example you should cache ~/.cache, which pip uses by default.

@nvie
Copy link
Member

nvie commented Apr 6, 2016

Closing this in favor of using file modification dates and/or VCS systems for tracking these dates. Thanks for the work/initiative though!

@nvie nvie closed this Apr 6, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants