operations / transformations

[cholmes]

ToDo list (added by @m-mohr):

List of ideas:

• Add area and perimeter values: fiboa improve example.parquet -s
• Clean up geometries (applies only shapely.make_valid): fiboa improve example.parquet ... -g
• generate stats on boundary quality
• filter out columns: Can be done with fiboa merge, a single input dataset and the --include/--exclude flags (note: use -x and --crs if needed)
• harmonize data to eurocrops hcat => Harmonize data to Eurocrops HCAT #116
• reproject: fiboa improve example.parquet --crs EPSG:1234
• Convert from GeoParquet 1.0 to 1.1 (fiboa improve example.parquet) and vice versa (fiboa improve example.parquet -gp1)
• Change compression: fiboa improve example.parquet --compression brotli

---

There's been a good bit of desire to do 'extra' things in the converters, and we had a good discussion with some ideas on how to approach that in the last call, so wanted to open an issue.

The original one was #21 - add area and perimeter. But then also things like adding statistics as new columns, or filtering columns out.

Was thinking it could be ideal to keep the 'converters' very 'clean' - like they just translate from the source data to fiboa. But then there's maybe fiboa transform or something like that with a bunch of sub-commands. And ideally you could also use those sub-commands as part of the conversion process. Some of the initial ideas:

• Add area and perimeter values (converter: Add option to calculate area and perimeter if missing #21)
• Clean up geometries - automatically shift any overlapping ones. Or detect areas that are too big and remove pixels that are clearly wrong.
• generate stats on boundary quality - size, regularity, inscribed circles
• filter out columns
• subset to certain geographic areas / make test & train datasets.

I'm sure there's lots more, but basically have a set of utilities that help clean up and format data better, and harmonize it for various use cases. But every transformation is an 'opinion' to use the Varda way of thinking about it. So keep those as their own utilities, for people to choose to transform as they want.

[cholmes]

Other ideas:

• harmonize data to eurocrops hcat - add extra attributes and do the mapping from source data to the eurocrop names (seems like it would need another mini ecosystem of converters for each country, though they are likely simpler than the full-fledged fiboa converters).
• reproject - like if the source data is in a country-specific projection.

I've also been thinking about a 'merge' command for awhile - had been planning to make an issue for that, where collection-level metadata would shift to the row level. I was thinking that would be a full featured command, where you could do things like reprojection, and clean up boundaries. But it might make sense to push more to 'operations' and keep the merge pretty simple - it would just reject things that don't merge well, but the source data could be transformed more to get ready for the merge.

[m-mohr]

PR for most of the functionality: #114