How to supply Personal Access Token via the session_auth to the Client and proxy config

[raffaellof]

Maybe a stupid question, but I'm struggling trying to supply a Personal Access Token from myapp to a private github repository. Everytime app check_for_updates, it returns a 404 log error.

tufup.client - WARNING - Cannot refresh metadata: 404 Client Error: Not Found for url: https://api.github.com/repos/.../timestamp.json

I tried:

from requests.auth import AuthBase

class HTTPBearerToken(AuthBase):
    def __init__(self, token):
        self.token = token

    def __call__(self, r):
        r.headers['authorization'] = f'Bearer {self.token}'
        return r

client = Client(
            app_name=settings.APP_NAME,
            app_install_dir=settings.INSTALL_DIR,
            current_version=settings.APP_VERSION,
            metadata_dir=settings.METADATA_DIR,
            metadata_base_url=settings.METADATA_BASE_URL,
            target_dir=settings.TARGET_DIR,
            target_base_url=settings.TARGET_BASE_URL,
            refresh_required=False,
            session_auth={'https://api.github.com/': HTTPBearerToken(token)}
        )

No success... Of course, making a standard request (with requests.get) works properly (both token and repo are available). Maybe proxy configuration? Why error involves timestamp.json only? (This is a blocking error, I don't know if any other files are actually found)

Is there any way to pass proxy config to the client? Any other suggestions? Thanks

[dennisvang]

Hi @raffaellof , sorry for the late answer.

From the docs:

If you try to use a REST API endpoint without a token or with a token that has insufficient permissions, you will receive a 404 Not Found or 403 Forbidden response. Authenticating with invalid credentials will initially return a 401 Unauthorized response.

My first hunch would be an error in the url used for session_auth, viz. there should not be a trailing slash. As a result, tufup sets an empty session.auth (where session is a requests.Session object):

tufup/src/tufup/client.py

Line 401 in 0283ebb

session.auth = self.session_auth.get(key)

Could you try removing the trailing slash from your code, as follows?

session_auth={'https://api.github.com': HTTPBearerToken(token)}

Here's the relevant part of the docstring for the AuthRequestsFetcher:

tufup/src/tufup/client.py

Lines 346 to 361 in 0283ebb

	session_auth (optional):
	dict of the form {<scheme and server>: (<username>, <password>), ...}
	or {<scheme and server>: <requests.auth.AuthBase>, ...}
	or some combination of those
	where <scheme and server> can be e.g. https://example.com
	or http://localhost:8000.
	Naming follows [RFC 2396][1], which defines a generic uri as:
	<scheme>://<authority><path>?<query>
	where <authority> can be <server>.
	Also see session authentication example in requests docs: [1][2][3]

If that does not help, perhaps it has to do with the permissions for your PAT. Here's a list of required permissions for fine-grained PATs, per API endpoint, and here a list of scopes for classic PATs. Another possible cause for a 404 could arise with classic PAT and SSO, according to the docs:

If you do not authorize your personal access token (classic) for SAML SSO before you try to use it to access a single organization that enforces SAML SSO, you may receive a 404 Not Found

[raffaellof]

First of all thanks for your answer, I managed to reach the file by adding more headers (as described here https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28 ) and using another network configuration, but still having problems behind the proxy, isn't there a way to add proxy to the session_auth as well?

[dennisvang]

Good to hear you managed to reach the file.

I assume that means you removed the trailing slash from the session_auth key, as suggested above?

I'm fairly certain that's necessary for authentication to work.

[dennisvang]

... by adding more headers ... and using another network configuration ...

It would be helpful to know the details of your workaround.

[dennisvang]

... still having problems behind the proxy, ...

Could you explain which specific problems you are facing? (please be explicit here)

... isn't there a way to add proxy to the session_auth as well?

Currently, no, not out-of-the-box. I would consider that an advanced feature.

However, if you really need it, you could subclass tufup.client.AuthRequestsFetcher to set e.g. session.proxies, and then monkey-patch tufup.client.Client.__init__ to use your custom fetcher.

[raffaellof]

Here I am... much appreciated your help

I assume that means you removed the trailing slash from the session_auth key, as suggested above?

Sure, I did, it was a typo, my bad

It would be helpful to know the details of your workaround.

Subclass of requests.auth.AuthBase to add headers like so:

class HttpsGitTokenAuth(AuthBase):
    """Subclass of requests.auth.AuthBase to add custom headers"""
    def __init__(self, token):
        self.token = token

    def __call__(self, r):
        r.headers['authorization'] = f'Bearer {self.token}'
        r.headers['accept'] = 'application/vnd.github.raw+json'
        r.headers['X-GitHub-Api-Version'] = '2022-11-28'
        return r

This works for me, when the app runs in a network without proxies

Could you explain which specific problems you are facing? (please be explicit here)

Still facing an error trying downloading files from the repo, when my app runs in a network behind a proxy

tufup.client - WARNING - Cannot refresh metadata: Failed to download https://api.github.com/repos...

Then I followed your advice (thanks for putting me on the right way)

...if you really need it, you could subclass tufup.client.AuthRequestsFetcher to set e.g. session.proxies, and then monkey-patch tufup.client.Client.init to use your custom fetcher.

Subclass of tufup.client.AuthRequestsFetcher to add my custom session.proxies (overridden methods init and _get_session)

class AuthRequestsFetcherProxyMode(AuthRequestsFetcher):
    """Subclass of tufup.client.AuthRequestsFetcher to add custom proxies configuration"""
    def __init__(
            self,
            session_auth: Optional[Dict[str, Union[Tuple[str, str], AuthBase]]] = None
    ) -> None:
        super().__init__(session_auth=session_auth)

        # My custom proxies
        self.proxies = {  # Same proxy for both protocols
            'http': 'http://proxy:port',
            'https': 'http://proxy:port'
        }

    def _get_session(self, url: str) -> requests.Session:
        # set the Session.proxies attribute for the specified server, if available
        session = super()._get_session(url=url)
        session.proxies = self.proxies
        return session

...and monkey-patch the tufup.client.Client.init in the simplest way I figured out (context-manager):

        with mock.patch(target='tufup.client.AuthRequestsFetcher', new=AuthRequestsFetcherProxyMode):
            client = Client(
                app_name=settings.APP_NAME,
                app_install_dir=settings.INSTALL_DIR,
                current_version=settings.APP_VERSION,
                metadata_dir=settings.METADATA_DIR,
                metadata_base_url=settings.METADATA_BASE_URL,
                target_dir=settings.TARGET_DIR,
                target_base_url=settings.TARGET_BASE_URL,
                refresh_required=False,
                session_auth={'https://api.github.com': HttpsGitTokenAuth(git_token)}
            )

...client object initialized successfully!

Unfortunately I didn't solve the problem... and i don't know why! When my app runs behind a proxy and checks for updates, keep logging Failed Download error. I still think something goes wrong in the proxy configuration step, still looking for a fix :(

[dennisvang]

... Unfortunately I didn't solve the problem... and i don't know why! When my app runs behind a proxy and checks for updates, keep logging Failed Download error. I still think something goes wrong in the proxy configuration step, still looking for a fix :(

@raffaellof Sorry to hear that.

Could you post the relevant part of the log, showing the actual "Failed Download" error?

[dennisvang]

@raffaellof On a side note, I just found out there's an easier way to configure proxies for requests:

You can simply set the environment variables HTTP_PROXY, HTTPS_PROXY, etc.

At least this implies you should not need to subclass the AuthRequestsFetcher.

It may also help with the download issue.

[raffaellof]

I took some time to do some tests... the only way to make my app working properly, even behind a proxy, is by setting environment variables as suggested above.
Environment variables which (obviously) can be set dynamically within the app itself, so I got rid of the AuthRequestsFetcher subclass (and also monkeypatch).

os.environ['HTTP_PROXY'] = f'http://{user}:{password}@{PROXY}:{PORT}'
os.environ['HTTPS_PROXY'] = f'http://{user}:{password}@{PROXY}:{PORT}'

P.S. If (as in my case) the proxy requires authorization, credentials should be percent-encoded to avoid conflicts, to do this I used urllib

import urllib.parse

user = urllib.parse.quote(user)
password = urllib.parse.quote(password)

Problem solved. Thanks for your precious help @dennisvang

[dennisvang]

@raffaellof happy to hear the problem is solved. Thanks for the useful feedback!