people-and-planet-ai.weather-forecasting.tests.predictions_tests.test_predictions: test_predictions failed

[flaky-bot]

Note: #9460 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: 7368358
buildURL: Build Status, Sponge
status: failed

Test output

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/workspace/people-and-planet-ai/conftest.py", line 363, in _run_notebook_section
    run_notebook(**kwargs)
  File "/workspace/people-and-planet-ai/conftest.py", line 329, in run_notebook
    raise RuntimeError(
RuntimeError: Error on 'notebooks/4-predictions.ipynb', section '# ☁️ Cloud Run predictions': An error occurred while executing the following cell:
------------------
import numpy as np
import requests
response = requests.get(

f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

headers={"Authorization": f"Bearer {identity_token}"},

params={"patch-size": 256, "include-inputs": True},

)
response.raise_for_status()

results = response.json()
inputs = np.array(results["inputs"], np.float32)

predictions = np.array(results["predictions"], np.uint8)

print(f"inputs      : {inputs.dtype.name} {inputs.shape}")

print(f"predictions : {predictions.dtype.name} {predictions.shape}")

HTTPError                                 Traceback (most recent call last)

Cell In[5], line 10

2 import requests

4 response = requests.get(

5     f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

6     headers={"Authorization": f"Bearer {identity_token}"},

7     params={"patch-size": 256, "include-inputs": True},

8 )

---> 10 response.raise_for_status()

11 results = response.json()

13 inputs = np.array(results["inputs"], np.float32)
File /workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/.nox/py-3-10/lib/python3.10/site-packages/requests/models.py:1021, in Response.raise_for_status(self)

1016     http_error_msg = (

1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"

1018     )

1020 if http_error_msg:

-> 1021     raise HTTPError(http_error_msg, response=self)
HTTPError: 503 Server Error: Service Unavailable for url: https://ppai-weather-predictions-81a292-g5ppuil6fq-uc.a.run.app/predict/2019-09-02T18:00/25.507,-78.322?patch-size=256&include-inputs=True
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/test_predictions.py", line 59, in test_predictions

conftest.run_notebook_parallel(

File "/workspace/people-and-planet-ai/conftest.py", line 358, in run_notebook_parallel

pool.map(_run_notebook_section, args)

File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 367, in map

return self._map_async(func, iterable, mapstar, chunksize).get()

File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 774, in get

raise self._value

RuntimeError: Error on 'notebooks/4-predictions.ipynb', section '# ☁️ Cloud Run predictions': An error occurred while executing the following cell:
import numpy as np

import requests
response = requests.get(

f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

headers={"Authorization": f"Bearer {identity_token}"},

params={"patch-size": 256, "include-inputs": True},

)
response.raise_for_status()

results = response.json()
inputs = np.array(results["inputs"], np.float32)

predictions = np.array(results["predictions"], np.uint8)

print(f"inputs      : {inputs.dtype.name} {inputs.shape}")

print(f"predictions : {predictions.dtype.name} {predictions.shape}")

HTTPError                                 Traceback (most recent call last)

Cell In[5], line 10

2 import requests

4 response = requests.get(

5     f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

6     headers={"Authorization": f"Bearer {identity_token}"},

7     params={"patch-size": 256, "include-inputs": True},

8 )

---> 10 response.raise_for_status()

11 results = response.json()

13 inputs = np.array(results["inputs"], np.float32)
File /workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/.nox/py-3-10/lib/python3.10/site-packages/requests/models.py:1021, in Response.raise_for_status(self)

1016     http_error_msg = (

1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"

1018     )

1020 if http_error_msg:

-> 1021     raise HTTPError(http_error_msg, response=self)
HTTPError: 503 Server Error: Service Unavailable for url: https://ppai-weather-predictions-81a292-g5ppuil6fq-uc.a.run.app/predict/2019-09-02T18:00/25.507,-78.322?patch-size=256&include-inputs=True

[rsamborski]

Closing as intermittent due to 503 Server Error: Service Unavailable for url

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

commit: 138219c
buildURL: Build Status, Sponge
status: failed

Test output

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/workspace/people-and-planet-ai/conftest.py", line 363, in _run_notebook_section
    run_notebook(**kwargs)
  File "/workspace/people-and-planet-ai/conftest.py", line 329, in run_notebook
    raise RuntimeError(
RuntimeError: Error on 'notebooks/4-predictions.ipynb', section '# ☁️ Cloud Run predictions': An error occurred while executing the following cell:
------------------
import numpy as np
import requests
response = requests.get(

f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

headers={"Authorization": f"Bearer {identity_token}"},

params={"patch-size": 256, "include-inputs": True},

)
response.raise_for_status()

results = response.json()
inputs = np.array(results["inputs"], np.float32)

predictions = np.array(results["predictions"], np.uint8)

print(f"inputs      : {inputs.dtype.name} {inputs.shape}")

print(f"predictions : {predictions.dtype.name} {predictions.shape}")

HTTPError                                 Traceback (most recent call last)

Cell In[5], line 10

2 import requests

4 response = requests.get(

5     f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

6     headers={"Authorization": f"Bearer {identity_token}"},

7     params={"patch-size": 256, "include-inputs": True},

8 )

---> 10 response.raise_for_status()

11 results = response.json()

13 inputs = np.array(results["inputs"], np.float32)
File /workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/.nox/py-3-10/lib/python3.10/site-packages/requests/models.py:1021, in Response.raise_for_status(self)

1016     http_error_msg = (

1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"

1018     )

1020 if http_error_msg:

-> 1021     raise HTTPError(http_error_msg, response=self)
HTTPError: 503 Server Error: Service Unavailable for url: https://ppai-weather-predictions-ab041d-g5ppuil6fq-uc.a.run.app/predict/2019-09-02T18:00/25.507,-78.322?patch-size=256&include-inputs=True
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/test_predictions.py", line 59, in test_predictions

conftest.run_notebook_parallel(

File "/workspace/people-and-planet-ai/conftest.py", line 358, in run_notebook_parallel

pool.map(_run_notebook_section, args)

File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 367, in map

return self._map_async(func, iterable, mapstar, chunksize).get()

File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 774, in get

raise self._value

RuntimeError: Error on 'notebooks/4-predictions.ipynb', section '# ☁️ Cloud Run predictions': An error occurred while executing the following cell:
import numpy as np

import requests
response = requests.get(

f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

headers={"Authorization": f"Bearer {identity_token}"},

params={"patch-size": 256, "include-inputs": True},

)
response.raise_for_status()

results = response.json()
inputs = np.array(results["inputs"], np.float32)

predictions = np.array(results["predictions"], np.uint8)

print(f"inputs      : {inputs.dtype.name} {inputs.shape}")

print(f"predictions : {predictions.dtype.name} {predictions.shape}")

HTTPError                                 Traceback (most recent call last)

Cell In[5], line 10

2 import requests

4 response = requests.get(

5     f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

6     headers={"Authorization": f"Bearer {identity_token}"},

7     params={"patch-size": 256, "include-inputs": True},

8 )

---> 10 response.raise_for_status()

11 results = response.json()

13 inputs = np.array(results["inputs"], np.float32)
File /workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/.nox/py-3-10/lib/python3.10/site-packages/requests/models.py:1021, in Response.raise_for_status(self)

1016     http_error_msg = (

1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"

1018     )

1020 if http_error_msg:

-> 1021     raise HTTPError(http_error_msg, response=self)
HTTPError: 503 Server Error: Service Unavailable for url: https://ppai-weather-predictions-ab041d-g5ppuil6fq-uc.a.run.app/predict/2019-09-02T18:00/25.507,-78.322?patch-size=256&include-inputs=True

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: 5822d8d
buildURL: Build Status, Sponge
status: failed

Test output

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/workspace/people-and-planet-ai/conftest.py", line 363, in _run_notebook_section
    run_notebook(**kwargs)
  File "/workspace/people-and-planet-ai/conftest.py", line 329, in run_notebook
    raise RuntimeError(
RuntimeError: Error on 'notebooks/4-predictions.ipynb', section '# ☁️ Cloud Run predictions': An error occurred while executing the following cell:
------------------
import numpy as np
import requests
response = requests.get(

f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

headers={"Authorization": f"Bearer {identity_token}"},

params={"patch-size": 256, "include-inputs": True},

)
response.raise_for_status()

results = response.json()
inputs = np.array(results["inputs"], np.float32)

predictions = np.array(results["predictions"], np.uint8)

print(f"inputs      : {inputs.dtype.name} {inputs.shape}")

print(f"predictions : {predictions.dtype.name} {predictions.shape}")

HTTPError                                 Traceback (most recent call last)

Cell In[5], line 10

2 import requests

4 response = requests.get(

5     f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

6     headers={"Authorization": f"Bearer {identity_token}"},

7     params={"patch-size": 256, "include-inputs": True},

8 )

---> 10 response.raise_for_status()

11 results = response.json()

13 inputs = np.array(results["inputs"], np.float32)
File /workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/.nox/py-3-10/lib/python3.10/site-packages/requests/models.py:1021, in Response.raise_for_status(self)

1016     http_error_msg = (

1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"

1018     )

1020 if http_error_msg:

-> 1021     raise HTTPError(http_error_msg, response=self)
HTTPError: 503 Server Error: Service Unavailable for url: https://ppai-weather-predictions-6d93ad-g5ppuil6fq-uc.a.run.app/predict/2019-09-02T18:00/25.507,-78.322?patch-size=256&include-inputs=True
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/test_predictions.py", line 59, in test_predictions

conftest.run_notebook_parallel(

File "/workspace/people-and-planet-ai/conftest.py", line 358, in run_notebook_parallel

pool.map(_run_notebook_section, args)

File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 367, in map

return self._map_async(func, iterable, mapstar, chunksize).get()

File "/usr/local/lib/python3.10/multiprocessing/pool.py", line 774, in get

raise self._value

RuntimeError: Error on 'notebooks/4-predictions.ipynb', section '# ☁️ Cloud Run predictions': An error occurred while executing the following cell:
import numpy as np

import requests
response = requests.get(

f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

headers={"Authorization": f"Bearer {identity_token}"},

params={"patch-size": 256, "include-inputs": True},

)
response.raise_for_status()

results = response.json()
inputs = np.array(results["inputs"], np.float32)

predictions = np.array(results["predictions"], np.uint8)

print(f"inputs      : {inputs.dtype.name} {inputs.shape}")

print(f"predictions : {predictions.dtype.name} {predictions.shape}")

HTTPError                                 Traceback (most recent call last)

Cell In[5], line 10

2 import requests

4 response = requests.get(

5     f"{model_url}/predict/2019-09-02T18:00/25.507,-78.322",

6     headers={"Authorization": f"Bearer {identity_token}"},

7     params={"patch-size": 256, "include-inputs": True},

8 )

---> 10 response.raise_for_status()

11 results = response.json()

13 inputs = np.array(results["inputs"], np.float32)
File /workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/.nox/py-3-10/lib/python3.10/site-packages/requests/models.py:1021, in Response.raise_for_status(self)

1016     http_error_msg = (

1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"

1018     )

1020 if http_error_msg:

-> 1021     raise HTTPError(http_error_msg, response=self)
HTTPError: 503 Server Error: Service Unavailable for url: https://ppai-weather-predictions-6d93ad-g5ppuil6fq-uc.a.run.app/predict/2019-09-02T18:00/25.507,-78.322?patch-size=256&include-inputs=True

[davidcavazos]

HTTPError: 503 Server Error: Service Unavailable for url

Closing as a transient error

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: bfbce77
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/people-and-planet-ai/conftest.py", line 113, in run_cmd
    p = subprocess.run(
  File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('gcloud', 'run', 'services', 'describe', 'ppai-weather-predictions-028633', '--region=us-central1', '--format=get(image)')' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/people-and-planet-ai/weather-forecasting/tests/predictions_tests/test_predictions.py", line 48, in service_name

conftest.cloud_run_cleanup(unique_name, location)

File "/workspace/people-and-planet-ai/conftest.py", line 136, in cloud_run_cleanup

run_cmd(

File "/workspace/people-and-planet-ai/conftest.py", line 130, in run_cmd

raise RuntimeError(f"{e}\n\n{e.stderr.decode('utf-8')}") from e

RuntimeError: Command '('gcloud', 'run', 'services', 'describe', 'ppai-weather-predictions-028633', '--region=us-central1', '--format=get(image)')' returned non-zero exit status 1.
ERROR: (gcloud.run.services.describe) Cannot find service [ppai-weather-predictions-028633]

[davidcavazos]

Looking more closely into the logs, it looks like it's now failing to build the image.

ERROR: (gcloud.run.deploy) Build failed; check build logs for details.

Looking into the build logs.

[builder] INFO: pip is looking at multiple versions of weather-model to determine which version is compatible with other requirements. This could take a while.
[builder] ERROR: Could not find a version that satisfies the requirement torch==1.13.1 (from weather-model) (from versions: none)
[builder] ERROR: No matching distribution found for torch==1.13.1

[davidcavazos]

It looks like Buildpacks is installing Python v3.12.0 and PyTorch is not yet available in that version (pytorch/pytorch#110436).