Skip to content

Commit

Permalink
Deep copy load files before iterating (#88)
Browse files Browse the repository at this point in the history
We are still seeing an error with empty payloads, the theory is that the
variable is being cleared while it is being posted to the vespa instance.
This is a further attempt to either stop this from happening, or rule it
out as a theory
  • Loading branch information
olaughter authored Jan 31, 2024
1 parent 71d295e commit 91cff61
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions src/index/vespa_.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import asyncio
import copy
import logging
from collections import defaultdict
from pathlib import Path
Expand Down Expand Up @@ -270,7 +270,8 @@ def _get_vespa_instance() -> Vespa:
def _batch_ingest(vespa: Vespa, to_process: Mapping[SchemaName, list]):
responses: list[VespaResponse] = []
for schema in _SCHEMAS_TO_PROCESS:
if documents := to_process[schema]:
if to_process[schema]:
documents = copy.deepcopy(to_process[schema])
_LOGGER.info(f"Processing {schema}, with {len(documents)} documents")
responses.extend(
vespa.feed_batch(
Expand Down

0 comments on commit 91cff61

Please sign in to comment.