Skip to content

bulk_insert fails when trying to insert JSON fields when used with return_model=True #158

@MontyD

Description

@MontyD

Having upgrade to postgres-extra v2.0.3 bulk inserting models which contain JSON fields produces the following error:

    cards_created, ids_created, references_created, notes_created = upsert_cards(batch)
  File "/usr/src/app/card/bulk.py", line 100, in upsert_cards
    cards = manager.on_conflict(
  File "/usr/local/lib/python3.9/site-packages/psqlextra/query.py", line 170, in bulk_insert
    return [
  File "/usr/local/lib/python3.9/site-packages/psqlextra/query.py", line 171, in <listcomp>
    self._create_model_instance(dict(row, **obj), compiler.using)
  File "/usr/local/lib/python3.9/site-packages/psqlextra/query.py", line 418, in _create_model_instance
    converted_field_values[field.attname] = converter(
  File "/usr/local/lib/python3.9/site-packages/django/db/models/fields/json.py", line 83, in from_db_value
    return json.loads(value, cls=self.decoder)
  File "/usr/local/lib/python3.9/json/__init__.py", line 339, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not dict

This seems to be due to double converting JSON to be inserted into a JSON field. We have a JSON field defined on the model we're inserting: https://gitlab.developers.cam.ac.uk/uis/devops/iam/card-database/card-api/-/blob/master/card/models.py#L315.

We're bulk inserting using return_model=True which seems to cause this problem: https://gitlab.developers.cam.ac.uk/uis/devops/iam/card-database/card-api/-/blob/master/card/bulk.py#L99.

Removing return_model=True allows the bulk insert to work, but unfortunately our bulk insert logic requires that we have access to the created models.

This is likely caused by the change made in 1fccf9a#diff-cc2bc273189507c98734a1ddaddcee3a8f206c7dee9de8ef1ceffa71f71ad0d6. Seeing as adding apply_converters=False to the call to _create_model_instance made in bulk_insert stops this error from occurring.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions