Fix #63. Add --input-meta parameter to explicitly specify the jsonl field dtypes

[miguelusque]

When reading jsonl files with Dask, the dataframe datatypes are inferred unless explicitly specified.

Inferring the data types can lead to several issues, such as incorrect type inference, degradation of performance and increased memory usage among others.

I think we could mitigate those issues if we would add a --input-meta parameter, which would receive a dictionary of datatypes.

That parameter would be optional, and be similar to the --meta parameter available here: https://docs.dask.org/en/latest/generated/dask.dataframe.read_json.html.

[ryantwolf]

I like this idea! I left a few changes I'd like to see made, but nothing major. Also it looks like there are some merge conflicts so you should probably resolve those.

[ryantwolf]

After sleeping on it, I do have one more nit. I think the argument for all the read_* functions should be meta. The argparse arguments can still be --input-meta. I want to try and keep the arguments and behavior as close to Dask as possible.

[miguelusque]

After sleeping on it, I do have one more nit. I think the argument for all the read_* functions should be meta. The argparse arguments can still be --input-meta. I want to try and keep the arguments and behavior as close to Dask as possible.

Hi @ryantwolf , thank you for all your feedback.

I am addressing all the comments you have made. My first thought about the name of the parameter was to name it meta, but then I thought that, in the near future, we may want to also specify the output-meta parameter, where we may want to specify specific formats for the output files.

Even if I think it is very likely that meta might work for both use-cases, I thought it would be clearer to differentiate them.

Looking forward to your thoughts. Thanks!!!

[ayushdg]

Thanks for the changes! Left a few suggestions, but generally looks good.

In addition to the changes, is it also possible to add a couple of tests, that attempt to use DocumentDataset.read_json with the input_meta param set both as a string and a dict, and verify that the result dtypes are what we'd expect.

[miguelusque]

Thanks for the changes! Left a few suggestions, but generally looks good.

In addition to the changes, is it also possible to add a couple of tests, that attempt to use DocumentDataset.read_json with the input_meta param set both as a string and a dict, and verify that the result dtypes are what we'd expect.

Done! See test_io.py file, where I think we could start adding other IO-related tests.

[ayushdg]

Thanks a lot for ddressing all my comments.