Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX/ENH: HttpMixin refactored and various fixes #2151

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

waldbauer-certat
Copy link
Contributor

General
Removed 'requests' MissingDependencyError, because requests is a core lib from intelmq
Removed HTTP variables from Bot class in favor of HttpMixin
Removed trying to import requests in pipeline, its a core lib from intelmq
Added additional configuration variables to HttpMixin ( from Bot class )

Bots
GitHub API is now using HttpMixin
MS Azure Collector is now using HttpMixin
DO-Portal Expert is now using HttpMixin
GeoHash using MissingDependencyError instead of ValueError (consistency)
HttpContentExpert is now using HttpMixin
HttpStatusExpert is now using HttpMixin
NationalCERTContactCertATExpert is now using HttpMixin
RDAPExpert is now using HttpMixin
RIPEExpert is now using HttpMixin
SplunkSavedSearchExpert is now using HttpMixin
TuencyExpert is now using HttpMixin
RestAPIOutput is now using HttpMixin

Bot tests
GitHub API Collector is now using requests_mock instead of MagicMock (consistency)
RestAPI Output is now using correct headers

Fixes #2150
Fixes #2137

@waldbauer-certat waldbauer-certat linked an issue Feb 4, 2022 that may be closed by this pull request
@codecov-commenter
Copy link

codecov-commenter commented Feb 7, 2022

Codecov Report

Merging #2151 (f3ae806) into develop (8d1c926) will increase coverage by 0.06%.
The diff coverage is 89.02%.

@@             Coverage Diff             @@
##           develop    #2151      +/-   ##
===========================================
+ Coverage    76.34%   76.40%   +0.06%     
===========================================
  Files          441      441              
  Lines        23652    23560      -92     
  Branches      3739     3729      -10     
===========================================
- Hits         18058    18002      -56     
+ Misses        4857     4836      -21     
+ Partials       737      722      -15     
Impacted Files Coverage Δ
intelmq/lib/bot.py 63.95% <ø> (-0.18%) ⬇️
intelmq/bots/collectors/mail/collector_mail_url.py 76.19% <50.00%> (+0.38%) ⬆️
intelmq/bots/experts/geohash/expert.py 76.00% <50.00%> (+1.00%) ⬆️
intelmq/bots/collectors/shodan/collector_stream.py 40.47% <66.66%> (+1.45%) ⬆️
intelmq/bots/experts/do_portal/expert.py 35.48% <66.66%> (+6.91%) ⬆️
...ots/experts/national_cert_contact_certat/expert.py 40.62% <66.66%> (+3.78%) ⬆️
intelmq/bots/outputs/elasticsearch/output.py 76.74% <66.66%> (-1.31%) ⬇️
intelmq/bots/experts/splunk_saved_search/expert.py 19.38% <75.00%> (-0.62%) ⬇️
intelmq/bots/outputs/restapi/output.py 75.60% <80.00%> (+4.77%) ⬆️
...ots/collectors/github_api/_collector_github_api.py 90.00% <100.00%> (+10.58%) ⬆️
... and 14 more

**General**
Removed 'requests' MissingDependencyError, because requests is a core lib from intelmq
Removed HTTP variables from Bot class in favor of HttpMixin
Removed trying to import requests in pipeline, its a core lib from intelmq
Added additional configuration variables to HttpMixin ( from Bot class )

**Bots**
GitHub API is now using HttpMixin
MS Azure Collector is now using HttpMixin
DO-Portal Expert is now using HttpMixin
GeoHash using MissingDependencyError instead of ValueError (consistency)
HttpContentExpert is now using HttpMixin
HttpStatusExpert is now using HttpMixin
NationalCERTContactCertATExpert is now using HttpMixin
RDAPExpert is now using HttpMixin
RIPEExpert is now using HttpMixin
SplunkSavedSearchExpert is now using HttpMixin
TuencyExpert is now using HttpMixin
RestAPIOutput is now using HttpMixin

**Bot tests**
GitHub API Collector is now using requests_mock instead of MagicMock (consistency)
RestAPI Output is now using correct headers

Fixes #2150
Fixes #2137

Signed-off-by: Sebastian Waldbauer <[email protected]>
@waldbauer-certat waldbauer-certat marked this pull request as ready for review February 8, 2022 09:56
@sebix sebix self-requested a review July 6, 2022 15:10
@@ -23,7 +23,7 @@
create_configuration = None # noqa


class MicrosoftAzureCollectorBot(CollectorBot, CacheMixin):
class MicrosoftAzureCollectorBot(CollectorBot, CacheMixin, HttpMixin):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it used here? The bot uses the azure library and the parameters http_proxy and https_proxy are used by direct access.

Comment on lines +44 to +45
http_username: str = None
http_password: str = None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
http_username: str = None
http_password: str = None
http_username: Optional[str] = None
http_password: Optional[str] = None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Custom headers ignored in HTTPCollectorBot Move from set_request_parameters to HttpMixin
3 participants