[#1185] Outgoing requests tracking

[vaszig]

This is an approach according to the fact that we already use requests library (same for the external libraries we use for the APIs), by adding a hook to each request (based on a session). Also this approach was discussed with Sergei and Bart at some point.

@Bartvaderkin @alextreme feel free to have a first look and if you agree I can continue with making this more stable.

[alextreme]

I think this would be a neat way to do this. Couple of thoughts:

• a Django setting to toggle this on/off (I think the default should be on if installed, but this allows it to easily be disabled if not wanted/necessary)
• Tests :P
• Split this off into a separate/external Django app to allow easy reuse (can be before or after we've tested it out in OIP)

Looking further this only works for requests when making use of requests-sessions, correct? So this does mean that plain requests.get() calls would need to be rewritten. I'd prefer to also track requests when sessions aren't used (so we have a drop-in component for this), but I don't know if that's possible and for an initial version that we can try out as long as we see the BRP and ZGW requests being sent this would be sufficient.

[vaszig]

Looking further this only works for requests when making use of requests-sessions, correct? So this does mean that plain requests.get() calls would need to be rewritten. I'd prefer to also track requests when sessions aren't used (so we have a drop-in component for this), but I don't know if that's possible and for an initial version that we can try out as long as we see the BRP and ZGW requests being sent this would be sufficient.

Requests library sends under the hood all requests through requests.Session, so a plain request with requests.get("url") is logged as well (https://github.com/psf/requests/blob/main/requests/api.py#L58) I hope this is what you mean.

[alextreme]

Compromise is to put this in src/external_api_logging/ to allow easier migration to a third-party Django app

[Bartvaderkin]

I like the header information for debugging, but I'm not sure if we can store that in logs. @alextreme can pitch in on that.

[alextreme]

The other headers should be fine, but I'd recommend to avoid logging the Authorization header

[codecov-commenter]

Codecov Report

Merging #524 (20eaf77) into develop (95cbe9d) will increase coverage by 0.04%.
The diff coverage is 96.81%.

@@             Coverage Diff             @@
##           develop     #524      +/-   ##
===========================================
+ Coverage    96.47%   96.52%   +0.04%     
===========================================
  Files          527      538      +11     
  Lines        19023    19206     +183     
===========================================
+ Hits         18353    18538     +185     
+ Misses         670      668       -2     

Impacted Files	Coverage Δ
src/log_outgoing_requests/log_requests.py	88.23% <88.23%> (ø)
src/log_outgoing_requests/handlers.py	93.33% <93.33%> (ø)
src/log_outgoing_requests/admin.py	94.73% <94.73%> (ø)
src/log_outgoing_requests/models.py	95.83% <95.83%> (ø)
src/log_outgoing_requests/apps.py	100.00% <100.00%> (ø)
src/log_outgoing_requests/formatters.py	100.00% <100.00%> (ø)
...c/log_outgoing_requests/migrations/0001_initial.py	100.00% <100.00%> (ø)
src/log_outgoing_requests/tests/test_logging.py	100.00% <100.00%> (ø)
src/open_inwoner/conf/base.py	95.23% <100.00%> (+0.17%)	⬆️

... and 11 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Bartvaderkin]

I'd put a comment here to note that .hostname is obviously part of .url but we also have it as a field so we can filter on it in the admin.

[Bartvaderkin]

This is ok for now since we only take the query, but if we'd want more url components you'd cache the urlparse() result and access parts of it like so:

@cached_property
def url_parsed(self):
    return urlparse(self.url)

@property
def query_params(self):
    return self.url_parsed.query

@property
def scheme(self):
    return self.url_parsed.scheme