Unreachable pyup.io leads to failing pipenv check

[l0b0]

I'm working on a project which has been running a lot of pipenv checks (possibly on the order of hundreds of requests per day). I think we've been throttled at the network level, because this command now results in messages like this:

Connection to pyup.io timed out. (connect timeout=5)

curl pyup.io can't reach the site, but torify curl pyup.io (that is, via a proxy) works, and dig pyup.io returns the same IP as SSL Labs detects, so it's definitely restricted to somewhere between this network and that site.

I've changed our process to run pipenv check much less often, but would it be possible to detect or avoid this issue in pipenv? Ideas:

• Configuring our own API key so we can reliably stay below any limits.
• Suggesting solutions when the connection times out.
• Configurable caching of the security issues database. I realize this is probably a very risky move since security issues may have to be fixed within hours, but it could be useful for people who for process reasons end up running a lot of checks.

PS: The recommended forum for these requests returns HTTP 404.

[uranusjr]

Allowing user configuration may be the simplest way to go. You can set it to something else (default to our built-in key if not set), or (maybe) a special value to disable pyup checks completely. At least the first part would be fairly straightforward, and contributions would be welcomed :)

[techalchemy]

Since we have merged #2835 I believe we can close this one out, thanks for the report / hope the fix works for your use case!

[leotrubach]

Currently pyup.io gives 500 error and we need to distinguish the following situations:

1. site is unreachable (okay, we can skip in this case)
2. there are outdated packages (we need to update packages)

This could be done by returning different error codes
@techalchemy could you please reopen this issue?

[leotrubach]

As a workaround this one could be used:

pipenv check || if [ $(curl --write-out %{http_code} --silent --output /dev/null pyup.io) -eq 500 ]; then echo "Site unreachable. Skipping check"; else false; fi

[leotrubach]

Hmm... I couldn't really figure out what's wrong. But when running pipenv check it fails but safety check succeeds

[leotrubach]

Strace output

Checking installed package safety…
read(9, "", 50000)                      = 0
read(5, "Unable to load vulnerability database\n", 32768) = 38
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2189, si_uid=1001, si_status=255, si_utime=0, si_stime=0} ---
read(5, "", 32768)                      = 0
read(7, "", 32768)                      = 0
An error occurred:

[mcallaghan-bsm]

As a workaround this one could be used:

pipenv check || if [ $(curl --write-out %{http_code} --silent --output /dev/null pyup.io) -eq 500 ]; then echo "Site unreachable. Skipping check"; else false; fi

Is this really a good workaround?
You don't want your pipeline security checks "false passing".

(of note, our workaround is that we have our CI jobs "allow fail = true" which converts the gitlab job into a WARNING instead, therefore not blocking the entirety of the pipeline -- this way the TRUTH is reported still, but does not block development during outages)