-
Notifications
You must be signed in to change notification settings - Fork 107
Description
Having upgrade to postgres-extra v2.0.3 bulk inserting models which contain JSON fields produces the following error:
cards_created, ids_created, references_created, notes_created = upsert_cards(batch)
File "/usr/src/app/card/bulk.py", line 100, in upsert_cards
cards = manager.on_conflict(
File "/usr/local/lib/python3.9/site-packages/psqlextra/query.py", line 170, in bulk_insert
return [
File "/usr/local/lib/python3.9/site-packages/psqlextra/query.py", line 171, in <listcomp>
self._create_model_instance(dict(row, **obj), compiler.using)
File "/usr/local/lib/python3.9/site-packages/psqlextra/query.py", line 418, in _create_model_instance
converted_field_values[field.attname] = converter(
File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/json.py", line 83, in from_db_value
return json.loads(value, cls=self.decoder)
File "/usr/local/lib/python3.9/json/__init__.py", line 339, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dictThis seems to be due to double converting JSON to be inserted into a JSON field. We have a JSON field defined on the model we're inserting: https://gitlab.developers.cam.ac.uk/uis/devops/iam/card-database/card-api/-/blob/master/card/models.py#L315.
We're bulk inserting using return_model=True which seems to cause this problem: https://gitlab.developers.cam.ac.uk/uis/devops/iam/card-database/card-api/-/blob/master/card/bulk.py#L99.
Removing return_model=True allows the bulk insert to work, but unfortunately our bulk insert logic requires that we have access to the created models.
This is likely caused by the change made in 1fccf9a#diff-cc2bc273189507c98734a1ddaddcee3a8f206c7dee9de8ef1ceffa71f71ad0d6. Seeing as adding apply_converters=False to the call to _create_model_instance made in bulk_insert stops this error from occurring.