CURL_CA_BUNDLE= disables certificate verification

[owtaylor]

I'm not the first to notice this, see:

https://stackoverflow.com/questions/48391750/disable-python-requests-ssl-validation-for-an-imported-module

Which implies people have even relied on the current behavior as a hack ... but I think it's pretty clear that the current behavior is an accidental bug, which should be fixed (for requests 3?)

Vaguely related to #3829

Expected Result

An empty-string CURL_CA_BUNDLE should use default system verification, the same way as:

• An unset CURL_CA_BUNDLE
• An empty-string or unset REQUESTS_CA_BUNDLE
• Behavior of curl/libcurl with an empty-string or unset CURL_CA_BUNDLE

Actual Result

Empty CURL_CA_BUNDLE disables certificate verification

Reproduction Steps

• Set CURL_CA_BUNDLE to an empty value, try to fetch a self-signed or invalid HTTPS endpoint => success

[nateprewitt]

Hi @owtaylor,

Thanks for bringing this to our attention! This was definitely not intended behavior and we agree it's a bug. We've pushed up a PR (#6074) to fix the issue in the next minor release of Requests. While this may be breaking for the workflow discussed in Stackoverflow, that was never a supported method for disabling verification.

Please let us know if you have any other concerns or comments on the PR. Thanks!

[moshekaplan]

@nateprewitt : It pains me to request this, but given how popular the hack is, do you think it might make sense to create an environment variable for globally disabling SSL validation in requests?

[sigmavirus24]

given how popular the hack is,

Can you give evidence of this being popular?

[moshekaplan]

given how popular the hack is,

Can you give evidence of this being popular?

https://stackoverflow.com/questions/48391750/disable-python-requests-ssl-validation-for-an-imported-module has approximately 50 upvotes and 29k views.

[sigmavirus24]

given how popular the hack is,

Can you give evidence of this being popular?

https://stackoverflow.com/questions/48391750/disable-python-requests-ssl-validation-for-an-imported-module has approximately 50 upvotes and 29k views.

Yeah, we're not interested in giving people a global hammer to misuse based off of view numbers that don't reflect how widely used that was. I have viewed stack overflow answers that were useless because a search engine thought them relevant. You're qualification for popular is deeply flawed

[nateprewitt]

do you think it might make sense to create an environment variable for globally disabling SSL validation in requests?

While we take breakages very seriously in Requests due its wide usage, I think this falls far outside the realm of intended behavior. I don't believe we have interest in adding an alternative environment variable either. Requests has historically limited usage of env vars because of their "magic" behavior between systems.

The main concern is once this is set, it's very easy to forget, leading to unintended consequences. Globally disabling cert verification not only affects direct Requests invocations but any tool using Requests, which is not something we want to support.

If tools using Requests are working against endpoints without an accessible cert bundle, they should either expose a configurable endpoint or method to disable verification which is then passed to requests verify argument.

[jaklan]

@nateprewitt they should either expose a configurable endpoint or method to disable verification which is then passed to requests verify argument - maybe they should, but they don't always do, so having any option to disable the verification after removing the above hack would be quite important. Cannot verify be also configured with some envar like REQUESTS_SSL_VERIFY=[true|false]?

[sigmavirus24]

It can not, no. We don't support that. We shouldn't support that. To quote myself:

Yeah, we're not interested in giving people a global hammer

And quoting Nate,

The main concern is once this is set, it's very easy to forget, leading to unintended consequences. Globally disabling cert verification not only affects direct Requests invocations but any tool using Requests, which is not something we want to support.

This isn't going to happen.

[jaklan]

@sigmavirus24 / @nateprewitt - so what would be your "unofficial" recommendation to disable SSL verification for requests calls in 3rd party libraries (other than waiting for maintainers to address the issue / forking the repo to add verify=False)? Ofc the best solution would be to provide custom CA_BUNDLE and pass the verification, but sometimes it's problematic to figure out the proper one if the internal networking setup is messy, and you need a way to bypass that in the meantime to allow people do their job.

[sigmavirus24]

you need a way to bypass that in the meantime to allow people do their job.

Pin/Cap the version of Requests you use and use the old bypass. We can't and won't support 100% of possible users. We have to do the best to secure as many users as possible and prevent them from accidentally injuring themselves with a large and dangerous weapon.

[sigmavirus24]

Also if your jobs are so dependent upon this software, your employers should be sponsoring the PSF

[jaklan]

@sigmavirus24 I didn't mean to convince you to implement the workaround, I see your point, I just wanted to ask if there were some non-obvious, probably no less dirty than the old one, workarounds which can be used in such situation which you were aware of.

I heard about PYTHONHTTPSVERIFY=0, but it seems it doesn't have any effect on requests, right?

Also if your jobs are so dependent upon this software, your employers should be sponsoring the PSF

If developers had impact on corporate budgets, especially in non-IT-first companies... Book-size topic.

[nateprewitt]

@89z it's highly unlikely this issue is affecting your usage, unless you've installed Requests directly from this Github repository. The change is not publicly available on PyPI.

[ryshoooo]

Just an FYI guys, this is actually really widely used out there. This is going to break quite a lot CICD pipelines, applications and whatnot, especially in a corporate setting where private CAs are a common thing. I understand your reasoning and I sort of agree. But you should likely prepare for a bunch of complaints :) for most that have to deal with private CAs, this is not a bug, it's a feature.

[sigmavirus24]

Just an FYI guys, this is actually really widely used out there. This is going to break quite a lot CICD pipelines, applications and whatnot, especially in a corporate setting where private CAs are a common thing. I understand your reasoning and I sort of agree. But you should likely prepare for a bunch of complaints :) for most that have to deal with private CAs, this is not a bug, it's a feature.

For the security people trying to ensure a company has secure CI/CD pipelines and can trust what you're doing, this is a significant improvement. For management that may have to deal with regulatory fines as a result of an engineer bypassing internal attempts at securing the software pipeline that can be undermined so easily, this is an improvement.

Requests bears a lot of responsibility. People are constantly trying to gain access to my accounts here and elsewhere ostensibly to take malicious control of this and other packages. Requests is so widely used that the slightest thing can be leveraged wide and far. This seems like such a tiny thing, but so many large scale attacks are combinations of things deemed too small to be worth fixing.

When we fix it we're demonized by developers for making their lives harder. When we're one link in the chain when a company's data or network is compromised, we're demonized for not having been proactive enough. Personally, I would rather do everything I can to protect anyone relying on this library whether they know it or not than I would rather protect someone's ability to bypass internal measures they feel are too difficult to comply with.

[nateprewitt]

@89z, we'd ask you participate in the conversation transparently and constructively. Retroactively deleting your comments and misrepresenting others statements doesn't move the conversation forward. You can find the code of conduct here if you have questions.

In regards to the issue at hand, we have patched a security issue that was never intended to exist in the library. What is being asked for is a feature addition that we don't plan to add. As stated above, we acknowledge the immediate transition may be difficult for some user, but is the best long term decision. Fixing this in other tools is a one line change and will give users more fine grained control if they do intend to disable verification. That's the safest path forward for the community.

[sigmavirus24]

Retroactively deleting your comments

@nateprewitt Since youre not my employer, I dont think you get to dictate how I use the site.

misrepresenting others statements

So if @sigmavirus24 misrepresents developer actions, then its fine, but if I call him on it, then its not?

You can find the code of conduct here if you have questions.

No thanks. @sigmavirus24 started the off topic comments, not me. So point your links to them instead.

I was responding to someone else, not to you but if you feel those statements apply to you, perhaps that's not a problem with what I said.

If you think what I have said violates the code of conduct, please do report me.

[icovada]

Requests bears a lot of responsibility

I know it does, and everyone is thankful for the work you are doing. On the other hand, I see it as a big limitation of software freedom to prevent people to do whatever they like on their own stuff, and the responsibility for what they do should never fall onto the library developer

[sigmavirus24]

Requests bears a lot of responsibility

I know it does, and everyone is thankful for the work you are doing. On the other hand, I see it as a big limitation of software freedom to prevent people to do whatever they like on their own stuff, and the responsibility for what they do should never fall onto the library developer

You fundamentally misunderstand software freedom. It's not the freedom to have the software be preprogrammed to do whatever you wish, no matter how inadvisable and insecure. It's the ability to read and modify the source to fix issues and then release those for others. You still have software freedom.

All software has politics. This project's politics is that it shouldn't be easy to lose one's life savings or a cool hundred million because a large caliber pre-loaded firearm is pointed at every user's (& users of dependent software) foot. If you disagree with our politics, you have software freedom and can do what you like

[ryshoooo]

Just an FYI guys, this is actually really widely used out there. This is going to break quite a lot CICD pipelines, applications and whatnot, especially in a corporate setting where private CAs are a common thing. I understand your reasoning and I sort of agree. But you should likely prepare for a bunch of complaints :) for most that have to deal with private CAs, this is not a bug, it's a feature.

For the security people trying to ensure a company has secure CI/CD pipelines and can trust what you're doing, this is a significant improvement. For management that may have to deal with regulatory fines as a result of an engineer bypassing internal attempts at securing the software pipeline that can be undermined so easily, this is an improvement.

Requests bears a lot of responsibility. People are constantly trying to gain access to my accounts here and elsewhere ostensibly to take malicious control of this and other packages. Requests is so widely used that the slightest thing can be leveraged wide and far. This seems like such a tiny thing, but so many large scale attacks are combinations of things deemed too small to be worth fixing.

When we fix it we're demonized by developers for making their lives harder. When we're one link in the chain when a company's data or network is compromised, we're demonized for not having been proactive enough. Personally, I would rather do everything I can to protect anyone relying on this library whether they know it or not than I would rather protect someone's ability to bypass internal measures they feel are too difficult to comply with.

Thanks for the response :) I understand the viewpoint, and as mentioned, I sort of agree. Demonization is a strong word, I wouldn't really say that, you really are just forcing developers to do the right thing, and from this point of view, I completely agree.

There are however instances where it's just going to be tedious and unnecessary. F.e. when your compute lies in an isolated VPN, where the entire communication inside this network is already protected and quite restricted, then there is an "internal trust" between the servers inside of the VPN. Therefore no need to verify (if you can reach the machine, you trust each other essentially).

Git is also a widely used project, likely even more than requests, and they do offer the GIT_SSL_NO_VERIFY env variable. I still believe it's a feature to disable SSL verification by an env variable, I do agree that the variable shouldn't be called CURL_CA_BUNDLE, perhaps REQUESTS_SSL_NO_VERIFY instead. I also believe you shouldn't bear the responsibility of people doing dumb stuff with your library :)

Anyway, thanks for the response. I just wanted to voice opinions from my nearby community of Python developers. We will respect any decisions you make.

[Dragas]

@sigmavirus24 You're claiming that you prevent misconfigured environments by providing this fix, but you're also forgetting that the workaround is actively used by corporate developers that sit behind a firewall that acts like man in the middle proxy, where the development environment is misconfigured out of the box, as the firewall certificate is never installed in any machines, nor software that would go through that firewall. I would agree with your viewpoint that it's "more secure" to prevent this if we were forced to pin certificates, but sadly we're not.

Other than that, you could atleast provide a proper workaround, like creating your own CA bundle, and adding certificates there, and steer people to the proper solution on SO citing your status.

[sigmavirus24]

Other than that, you could atleast provide a proper workaround, like creating your own CA bundle, and adding certificates there, and steer people to the proper solution on SO citing your status.

Read the license. You aren't entitled to my time. What we do, we do with limited time and resources and generally without help at this point whilst only ever receiving criticism. I am not responsible for documenting easily Google-able instructions for problems at least a decade old.

[karlschriek]

I would also like to point out that we use non-valid certificates in development (for example, staging lets-encrypt certs which simulate the certification processs without issuing anything that is actually valid) for the microservices ecosystems we manage. We control configuration of all these services via ConfigMaps and Secrets that are attached to Deployments on Kubernetes, thereby setting variable env vars and config files.

Whenever we have to drop something in that is python-based it is an enormous chore to get this working since there is no way to globally deactivate SSL verification for anything that uses requests. I would be very happy to see something like a REQUESTS_SSL_NO_VERIFY global env var.

[brsolomon-deloitte]

Can you give evidence of this being popular?

Here's one example: authlib recommends in their docs controlling verification in this way. (See also lepture/authlib#474 requesting an update to authlib docs.) authlib in turn is widely used as a base library for building auth clients.