Configurable multiprocessing for large data import (Ecoinvent ecospold2)

[noahebrooks-cb]

Using the ecospold2 importer:

bw2io.bw2setup()
importer: SingleOutputEcospold2Importer = bw2io.SingleOutputEcospold2Importer(
    dirpath = path,
    db_name = str_ecoinvent_version
)
importer.apply_strategies()

There is a param use_mp which is used to determine whether to use multiprocessing or not, however it doesn't appear there is any way to configure the number of cores or workers to use.

I was curious if anyone had any solution to this or strategies for doing a large data import on limited hardware.

[noahebrooks-cb]

Trying to extract full Ecoinvent dataset leads to Out of Memory Error.

RAM: 8GB
Chip: Apple M1

[cmutel]

@noahebrooks-cb Can you give some more details on exactly what you are importing? This shouldn't raise an error with 8 GB, and multiprocessing won't have much of a memory impact. You can always turn multiprocessing off to test that assumption, of course.

Check to make sure you are importing the right file: https://docs.brightway.dev/en/latest/content/faq/ecoinvent.html#which-ecoinvent-file-should-i-download