Proxy support

[lbg-raghu-vennam]

Does the kubernetes deployment respect proxy environment variables ?

We tried deploying the code adding the environment variables for https_proxy and no_proxy as we have an air-gapped install and all traffic should go via our proxy including the traffic for the xxxx..live.dynatrace.com but the app does not seem to use the variables.

[mfranczak]

hi @rvennam-lbg we will implement proxy support and inform you once it's ready.

[lbg-raghu-vennam]

Thanks @mfranczak would be keen to test if you require any support with that.

[mswiatkowska]

Hi @rvennam-lbg we've just published docker image with proxy support. To deploy proper version please change in dynatrace-gcp-function.yaml:

• image: dynatrace/dynatrace-gcp-function:proxy-support-preview
• ConfigMap -> TRUST_ENV_PROXY = "true"
• add HTTPS_PROXY env variable

Please let us know if it works for you.

[lbg-raghu-vennam]

Thanks, @mswiatkowska we will be testing this in the next couple of days and will get back to you.

[lbg-raghu-vennam]

@mswiatkowska I am still seeing issues but the requests do hit the proxy now but fail to get the token. The proxy logs only show 200 responses so requests are being forwarded.

2021-01-31 20:26:08.549235  : Dynatrace function for Google Cloud Platform monitoring

Running with configured services: api,cloud_function,cloudsql_database,datastore_request,filestore_instance,gcs_bucket,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,k8s_pod,k8s_node,k8s_container,k8s_cluster,gce_instance
2021-01-31 20:26:08.549301  : Setting up...

2021-01-31 20:26:08.634298  : Missing ReadConfig/WriteConfig permission for Dynatrace API token, skipping dashboards configuration
2021-01-31 20:26:08.634970  : Trying to use default service account
2021-01-31 20:26:11.860441  : Access to following projects: mgmt-xxxx-tst-fb29
2021-01-31 20:26:14.341693  : Unable to get Dynatrace token metadata. Error details:
2021-01-31 20:26:14.341819  : Dynatrace API Token for project: 'mgmt-xxxx-tst-fb29'is not valid. Check expiration time and required token scopes: ['metrics.ingest']
2021-01-31 20:26:14.341963  : Monitoring disabled. Check your project(s) settings.

When trying exec and executing local_tests.py seeing the below errror

❯ k -n ns-kcl-dynatrace-mgmt exec $pod -it bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@dynatrace-gcp-function-5fd5949c47-brd44:/code# python3 local_test.py
2021-01-31 20:27:47.687964 [033c4d18] : Starting execution
2021-01-31 20:27:49.043011 [033c4d18] : Selected services: k8s_container,datastore_request,k8s_node,cloud_function,cloudsql_database,gcs_bucket,api,gce_instance,k8s_cluster,filestore_instance,pubsub_snapshot,pubsub_subscription,pubsub_topic,pubsublite_topic_partition,https_lb_rule,internal_http_lb_rule,internal_tcp_lb_rule,internal_udp_lb_rule,tcp_lb_rule,udp_lb_rule
2021-01-31 20:27:49.043413 [033c4d18] : Trying to use default service account
2021-01-31 20:27:49.073560 [033c4d18] : Successfully obtained access token
Traceback (most recent call last):
  File "local_test.py", line 18, in <module>
    asyncio.run(async_dynatrace_gcp_extension())
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/code/main.py", line 61, in async_dynatrace_gcp_extension
    await handle_event(data, event_context, "dynatrace-gcp-extension", project_ids)
  File "/code/main.py", line 117, in handle_event
    projects_ids = await get_all_accessible_projects(context, session, token)
  File "/code/lib/credentials.py", line 126, in get_all_accessible_projects
    response = await session.get(url, headers=headers)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 855, in _create_connection
    _, proto = await self._create_proxy_connection(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1078, in _create_proxy_connection
    raise ClientHttpProxyError(
aiohttp.client_exceptions.ClientHttpProxyError: 403, message='Forbidden', url=URL('http://proxy.<redactedurl>:3128')

[mswiatkowska]

Hi @rvennam-lbg , thanks for you feedback.
We've published new image again (with the same tag as previous one so you don't need to make any changes in the template).
Please try one more time. I fixed one more place which should also support proxy.
Hope it will work for you this time.

[lbg-raghu-vennam]

Hi @mswiatkowska I deployed the latest image and found out the communication for the cloudresourcemanager.googleapis.com is also going to proxy even after adding the NO_PROXY.

Our requirement is anything external like Dynatrace URL will use the proxies and all *.googleapis.com should not be using a proxy.

Is there any additional configuration needed to enable this setting ?

[lbg-raghu-vennam]

Hi @mswiatkowska

I accidentally clicked on close and re-opened this again.

Looks like the aiohttp libs being used currently does not support the no_proxy and a PR for this feature to be added has been pending for a while

Issue : aio-libs/aiohttp#4431
PR : aio-libs/aiohttp#4445

[mswiatkowska]

Hi @rvennam-lbg
Unfortunately I wasn't aware that aiohttp currently does not support no_proxy.
Could you tell me what environment you have - Dynatrace SaaS or Managed?

[benj5656]

Hi @rvennam-lbg
Unfortunately I wasn't aware that aiohttp currently does not support no_proxy.
Could you tell me what environment you have - Dynatrace SaaS or Managed?

Hello, we are SaaS

[lbg-raghu-vennam]

Hi @mswiatkowska, We use SaaS and deploy this on a private cluster with no egress for external connectivity so the proxy requirement for DT URL.

We do leverage the GCP VPC native routing for all the googleapis so they should not be routed via the same proxy which is where no_proxy is an inherent requirement for us.

Thanks, @benj5656 for keeping an eye on this.

[mswiatkowska]

Hi @rvennam-lbg, it looks like PR in aiohttp lib is going to be merged. If they provided support for no_proxy it would be the best solution for us.
However if PR won't be merged in next days we will try to give you update in next week - when we could do changes on our side.

[lbg-raghu-vennam]

Hi @mswiatkowska, Thanks for your response

It would be great if the aiohttp supports the no_proxy and I do see the PR is back in the discussion so hopefully will get merged soon.

Once a new release of the library is available, we can test with those changes after you are able to build using the new release.

[rohit-varshney]

hi @mswiatkowska I see the PR is not merged yet, any possibility to get it merged or provide any alternate solution. If yes, by when?

[mswiatkowska]

Hi @rohit-varshney, yes, we saw that PR in aiohttp lib is still not merged. We will provide our solution in two weeks. I'll inform you when we release changes.

[rohit-varshney]

thanks for the confirmation @mswiatkowska, any possibility of the early release of this feature?

[rohit-varshney]

hi @mswiatkowska sorry for bugging you again, but would like to know if you are still on track for this and will be able to deliver the feature in said time?

[mswiatkowska]

Hi @rohit-varshney, @Dawid-Kaszubski is working on this right now, so we should provide the solution this week. Hope we will able to give you an update tomorrow.

[Dawid-Kaszubski]

Hi @rohit-varshney, feature is currently on PR. It should be available soon.

[rohit-varshney]

hi thanks @mswiatkowska @Dawid-Kaszubski , please let me know once this is merged. And, also could you please let me know where we need to configure our Squid/Proxy URL?

[Dawid-Kaszubski]

Hi @rohit-varshney, PR is merged. To use proxy for dynatrace connections you need to set two env variables:
HTTPS_PROXY (proxy URL) and USE_PROXY (value should be set as DT_ONLY)
Values for new USE_PROXY variable could be found in https://github.com/dynatrace-oss/dynatrace-gcp-function/blob/master/HACKING.md

[jackstuart92]

Hey @Dawid-Kaszubski, Thanks for getting this in so quickly. I'm running into some issues running the code with the proxy settings enabled.

The environment variables exist

HTTPS_PROXY="http://<REDACTED>"
USE_PROXY="DT_ONLY"

However the run_docker.py script fails in due to an exception raised in the fast_check lib.

2021-03-05 09:26:57.480281 : Unable to get Dynatrace token metadata. Error details:

exception block:

async def get_dynatrace_token_metadata(dt_session: ClientSession, context: LoggingContext, dynatrace_url: str, dynatrace_api_key: str, timeout: Optional[int] = 2) -> dict:
    try:
        response = await dt_session.post(
            url=f"{dynatrace_url.rstrip('/')}/api/v1/tokens/lookup",
            headers={
                "Authorization": f"Api-Token {dynatrace_api_key}",
                "Content-Type": "application/json; charset=utf-8"
            },
            json={
                "token": dynatrace_api_key
            },
            timeout=timeout)
        if response.status != 200:
            context.log(f'Unable to get Dynatrace token metadata: {response.status}, url: {response.url}, reason: {response.reason}')
            return {}

        return await response.json()
    except Exception as e:
        context.log(f'Unable to get Dynatrace token metadata. Error details: {e}')
        return {}

If i execute the local_test.py script the gcp function respects the environment variables and is able to connect to both the Dynatrace tenant and google apis successfully. Do you have any suggestions to help me debug this futher?

Many thanks
Jack