An unknown error occurred (HTTP 504) while setting up a new connection between Shopify API (source) and Google Sheets (destination)

[cvronao]

Hi everyone!

I am new to Airbyte and is currently trying it out by fetching data from the Shopify API. My goal is to extract 1 month's worth of sales data from Shopify and save it on Google Sheets.

I've followed all the instructions (OSS Quickstart and connector instructions) to the T and I'm encountering a rather unhelpful error message when it's fetching the data source schema.

"We are fetching the schema of your data source.
This should take less than a minute, but may take a few minutes on slow internet connections or data sources with a large amount of streams."

It then gives me, "An unknown error occurred. (HTTP 504)."

The source and destination have both been tested on the UI and have passed. I'm self-hosting Airbyte on my mac, and the version I'm currently running is v0.63.1. I've also updated both source and destination to the latest versions but to no avail.

Based on my search, it seems I'm hitting a timeout limit. Not sure if it's nginx since when I checked nginx on the docker container with commands:
docker exec -it airbyte-abctl-control-plane bash systemctl is-active nginx

...it says nginx is inactive. Is this the problem?

I've also seen this Github issue where they mention about increasing timeout config on the nginx template for airbyte proxy, and here, where they mention that it can be changed on the env file. The problem is, I'm not sure how to do this exactly.

Can someone point me where/in which directory I can find the .env file? Or what else I can do to fix this?

Thanks so much in advance!

[szemek]

I've also seen this Github issue where they mention about increasing timeout config on the nginx template for airbyte proxy, and here, where they mention that it can be change on the env file. The problem is, I'm not sure how to do this exactly.

Can someone point me where/in which directory I can find the .env file? Or what else I can do to fix this?

For older way of running Airbyte locally (deprecation announcement), you were executing run-ab-platform.sh script in root of airbyte repository.
Script downloaded .env and few other configuration files, e.g. docker-compose.yaml, and started containers with docker compose.

I haven't worked with abctl yet, but for helm chart I have following values

global:
  env_vars:
    HTTP_IDLE_TIMEOUT: 1800s

and webapp ingress

webapp:
  ingress:
    annotations:
      kubernetes.io/ingress.class: internal
      nginx.ingress.kubernetes.io/proxy-body-size: 16m
      nginx.ingress.kubernetes.io/proxy-send-timeout: 1800
      nginx.ingress.kubernetes.io/proxy-read-timeout: 1800

Maybe it will be useful for your setup.

In docs here, READ_TIMEOUT environment variable was mentioned as well

[cvronao]

Hi @szemek,

Thanks for the swift reply!

I'm still lost as to what I can do to fix it. I back-tracked how exactly I installed Airbyte:

• I followed the steps on this video until before colima start, as the colima command doesn't work for me.
• I then followed the abctl route on the OSS Quickstart article.

Which means, I installed Airbyte while on the Development/testing/airbyte directory. I still can't find the .env and the docker-compose.yaml files. Based on these installation steps, can you point me where I can find these files so that I can check their contents? Or is it more advisable to reinstall Airbyte with Kubernetes via Helm since the docker-compose route has been deprecated?

Thanks again!

[cvronao]

Hi @szemek, I decided to uninstall and reinstall Airbyte on Kubernetes via Helm. It finally pushed through using your timeout values. Thank you so much!!