No projects building since 16th October

[DavidKorczynski]

There's been no projects being build since 16th October. https://oss-fuzz-build-logs.storage.googleapis.com/index.html

First I thought it was because of #14148 but since this has been fixed and base images are now building again, I assume it's either due to #14144 or #14145 or #14140

[oliverchang]

We're seeing this exception when requesting builds

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/functions_framework/execution_id.py", line 157, in wrapper
    result = view_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/functions_framework/__init__.py", line 237, in view_func
    function(data, context)
  File "/workspace/main.py", line 27, in build_project
    request_build.request_build(event, context)
  File "/workspace/request_build.py", line 122, in request_build
    run_build(
  File "/workspace/request_build.py", line 104, in run_build
    update_build_history(oss_fuzz_project, build_id, build_type)
  File "/workspace/request_build.py", line 48, in update_build_history
    project.put()
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/_options.py", line 89, in wrapper
    return wrapped(*pass_args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/utils.py", line 114, in wrapper
    return wrapped(*args, **new_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/utils.py", line 146, in positional_wrapper
    return wrapped(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 5134, in _put
    return self._put_async(_options=kwargs["_options"]).result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/tasklets.py", line 191, in result
    self.check_success()
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/tasklets.py", line 138, in check_success
    raise self._exception
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/tasklets.py", line 315, in _advance_tasklet
    yielded = self.generator.send(send_value)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 5194, in put
    ds_entity = _entity_to_ds_entity(self)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 748, in _entity_to_ds_entity
    prop._to_datastore(entity, data)
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 2082, in _to_datastore
    value = self._get_base_value_unwrapped_as_list(entity)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 1558, in _get_base_value_unwrapped_as_list
    wrapped = self._get_base_value(entity)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 1544, in _get_base_value
    return self._apply_to_values(entity, self._opt_call_to_base_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 1857, in _apply_to_values
    value[:] = map(function, value)
    ~~~~~^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 1619, in _opt_call_to_base_type
    value = _BaseValue(self._call_to_base_type(value))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 1678, in _call_to_base_type
    value = call(value)
            ^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 1824, in call
    new_value = method(self, value)
                ^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/ndb/model.py", line 2783, in _validate
    raise exceptions.BadValueError("Expected string, got {!r}".format(value))
google.cloud.ndb.exceptions.BadValueError: Expected string, got {'id': '6abc3993-f6bb-4a56-97fd-456e0761386a', 'status': 'QUEUED',

[oliverchang]

Looks like the culprit is #14140 once again.

@hunsche

3174fff#diff-d81a97b1e279fbecddeabb1f1777e1c203e9599bc6dac7c37928bde57d92f686R802

[hunsche]

FYI, I'm reverting PR #14155 as it broke the Trial Build (example: link).

The revert is in PR #14158, which should fix the build. I'm submitting a new PR with the intended (and correct) fix now.

[hunsche]

Here is the PR to fix the bug and maintain the trial build behavior: PR #14159