add experimental import of JSONL files, with optional multiprocessing #4855 #4858 #4918

mradamcox · 2019-06-18T16:38:59Z

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Description of Change

This pull request adds support for the import of JOSNL (json lines) files through the existing import commands.

python manage.py packages -o import_business_data -s resources.jsonl

It also adds support for the use of multiprocessing during JSONL import.

python manage.py packages -o import_business_data -s resources.jsonl --use_multiprocessing

Both of these new features should be considered "experimental". One of the main reasons for this is that the baked-in print statements that are within the actual import code (as well as the error reporting) have not yet been updated to work with these two new features. So, the console printouts during JSONL import look like this:

1 of 1 resources saved
Resources for Import: 1, Resources Saved: 1, Tiles for Import: 11, Tiles Saved: 11, Relations for Import: 0, Relations Saved: 0
1 of 1 resources saved
Resources for Import: 1, Resources Saved: 1, Tiles for Import: 58, Tiles Saved: 58, Relations for Import: 0, Relations Saved: 0
1 of 1 resources saved
Resources for Import: 1, Resources Saved: 1, Tiles for Import: 34, Tiles Saved: 34, Relations for Import: 0, Relations Saved: 0

and while using multiprocessing they get all jumbled up:

1 of 1 resources saved
1 of 1 resources saved
1 of 1 resources saved
Resources for Import: 1, Resources Saved: 1, Tiles for Import: 11, Tiles Saved: 11, Relations for Import: 0, Relations Saved: 0
1 of 1 resources saved
1 of 1 resources saved
Resources for Import: 1, Resources Saved: 1, Tiles for Import: 34, Tiles Saved: 34, Relations for Import: 0, Relations Saved: 0
1 of 1 resources saved
Resources for Import: 1, Resources Saved: 1, Tiles for Import: 33, Tiles Saved: 33, Relations for Import: 0, Relations Saved: 0
1 of 1 resources saved
Resources for Import: 1, Resources Saved: 1, Tiles for Import: 54, Tiles Saved: 54, Relations for Import: 0, Relations Saved: 0
1 of 1 resources saved
Resources for Import: 1, Resources Saved: 1, Tiles for Import: 58, Tiles Saved: 58, Relations for Import: 0, Relations Saved: 0

Another important thing to note is that while multiple processes are running, you can't just ctrl+c to stop the operation, as the processes just keep spawning.

To address these shortcomings I have added some warnings.

For JSONL:

 WARNING: Support for loading JSONL files is still experimental. Be aware that
 the format of logging and console messages has not been updated.

For multiprocessing:

WARNING: Support for multiprocessing files is still experimental. While using
multiprocessing to import resources, you will not be able to use ctrl+c (etc.)
to cancel the operation. You will need to manually kill all of the processes
with or just close the terminal. Also, be aware that print statements
will be very jumbled.

and that warning is followed by a confirmation prompt (which you can bypass by adding -y/--yes to the initial command).

Issues Solved

#4858
#4855

Checklist

Unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

Ticket Background

Sponsored by: Legion GIS
Tested by: @mradamcox
Designed by: @mradamcox

Further comments

…ect#4855 archesproject#4858

stickler-ci · 2019-06-18T16:39:12Z

arches/app/utils/data_management/resources/importer.py

+# during an import that uses multiprocessing.
+# see https://stackoverflow.com/a/49461944/3873885
+django.setup()
+from django.db import connection, connections, transaction


E402 module level import not at top of file

In fact this line must be here before the models are imported, see code comment and stack overflow link.

arches/management/commands/packages.py

coveralls · 2019-06-18T16:50:58Z

Coverage decreased (-0.05%) to 42.401% when pulling db01542 on legiongis:4855_4858_jsonl_multiprocessing_4_4_x into 8eb6fc6 on archesproject:stable/4.4.x.

stickler-ci · 2019-06-18T16:51:51Z

arches/management/commands/packages.py

@@ -828,7 +855,9 @@ def import_business_data(self, data_source, config_file=None, overwrite=None, bu
                path = utils.get_valid_path(source)
                if path is not None:
                    print 'Importing {0}. . .'.format(path)
-                    BusinessDataImporter(path, config_file).import_business_data(overwrite=overwrite, bulk=bulk_load, create_concepts=create_concepts, create_collections=create_collections)
+                    BusinessDataImporter(path, config_file).import_business_data(overwrite=overwrite,
+                            bulk=bulk_load, create_concepts=create_concepts,


E128 continuation line under-indented for visual indent

I would prefer to ignore this rule in this context. The linter would have me indent the continuation of the arguments very far to the right which in my mind is more difficult to read.

stickler-ci · 2019-06-18T16:51:51Z

arches/management/commands/packages.py

-                    BusinessDataImporter(path, config_file).import_business_data(overwrite=overwrite, bulk=bulk_load, create_concepts=create_concepts, create_collections=create_collections)
+                    BusinessDataImporter(path, config_file).import_business_data(overwrite=overwrite,
+                            bulk=bulk_load, create_concepts=create_concepts,
+                            create_collections=create_collections, use_multiprocessing=use_multiprocessing)


E128 continuation line under-indented for visual indent

I would prefer to ignore this rule in this context. The linter would have me indent the continuation of the arguments very far to the right which in my mind is more difficult to read.

…into 4855_4858_jsonl_multiprocessing_4_4_x

arches/app/utils/data_management/resources/importer.py

stickler-ci · 2019-06-19T20:00:18Z

arches/management/commands/packages.py

@@ -503,6 +509,7 @@ def load_business_data(package_dir):
                    business_data.append(os.path.join(package_dir, 'business_data', f))
            else:
                business_data += glob.glob(os.path.join(package_dir, 'business_data','*.json'))
+                business_data += glob.glob(os.path.join(package_dir, 'business_data','*.jsonl'))


E231 missing whitespace after ','

arches/management/commands/packages.py

mradamcox · 2019-06-19T21:25:07Z

This can now be tested by loading the demo HPLA package, using the branch I set up that has JSONL files in the business data directory:

python manage.py packages -o load_package -s https://github.com/legiongis/hpla_v4-pkg/archive/jsonl-business-data.zip -db true

To test multiprocessing you'll need to import those business data files individually and use the flag as described in the main body of this pull request.

mradamcox added 2 commits June 18, 2019 11:03

allow import of JSONL files, with optional multiprocessing archesproj…

6fd0a6c

…ect#4855 archesproject#4858

update multiprocessing warning message archesproject#4858

e9c8f9b

mradamcox requested a review from ryan86 June 18, 2019 16:39

stickler-ci reviewed Jun 18, 2019

View reviewed changes

some linting

c8b4fe2

stickler-ci reviewed Jun 18, 2019

View reviewed changes

arches/management/commands/packages.py Outdated Show resolved Hide resolved

more linting

54b7cd3

stickler-ci reviewed Jun 18, 2019

View reviewed changes

mradamcox added 2 commits June 19, 2019 13:52

Merge branch 'stable/4.4.x' of https://github.com/archesproject/arches …

d979643

…into 4855_4858_jsonl_multiprocessing_4_4_x

support jsonl in package loading

db01542

stickler-ci reviewed Jun 19, 2019

View reviewed changes

chiatt merged commit 4159d48 into archesproject:stable/4.4.x Jun 25, 2019

mradamcox mentioned this pull request Jul 11, 2019

Cherry-pick PR 4918 to master #4995

Merged

6 tasks

mradamcox deleted the 4855_4858_jsonl_multiprocessing_4_4_x branch August 12, 2019 18:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add experimental import of JSONL files, with optional multiprocessing #4855 #4858 #4918

add experimental import of JSONL files, with optional multiprocessing #4855 #4858 #4918

mradamcox commented Jun 18, 2019 •

edited

Loading

stickler-ci Jun 18, 2019

mradamcox Jun 18, 2019

coveralls commented Jun 18, 2019 •

edited

Loading

stickler-ci Jun 18, 2019

mradamcox Jun 18, 2019

stickler-ci Jun 18, 2019

mradamcox Jun 18, 2019

stickler-ci Jun 19, 2019

mradamcox commented Jun 19, 2019

add experimental import of JSONL files, with optional multiprocessing #4855 #4858 #4918

add experimental import of JSONL files, with optional multiprocessing #4855 #4858 #4918

Conversation

mradamcox commented Jun 18, 2019 • edited Loading

Types of changes

Description of Change

Issues Solved

Checklist

Ticket Background

Further comments

stickler-ci Jun 18, 2019

Choose a reason for hiding this comment

mradamcox Jun 18, 2019

Choose a reason for hiding this comment

coveralls commented Jun 18, 2019 • edited Loading

stickler-ci Jun 18, 2019

Choose a reason for hiding this comment

mradamcox Jun 18, 2019

Choose a reason for hiding this comment

stickler-ci Jun 18, 2019

Choose a reason for hiding this comment

mradamcox Jun 18, 2019

Choose a reason for hiding this comment

stickler-ci Jun 19, 2019

Choose a reason for hiding this comment

mradamcox commented Jun 19, 2019

mradamcox commented Jun 18, 2019 •

edited

Loading

coveralls commented Jun 18, 2019 •

edited

Loading