EKO xgrid_reshape manipulation leaves them in an unusable state

[scarlehoff]

from eko import EKO
from eko.interpolation import XGrid
from eko.io import manipulate
from pathlib import Path
new_xgrid = XGrid([0.05, 0.5, 1.0])
target = Path("to_test.tar")
eko_op = EKO.edit(target)
for i, elem in eko_op.items():
     eko_op[i] = manipulate.xgrid_reshape(elem, eko_op.xgrid, 2, targetgrid=new_xgrid, inputgrid=new_xgrid)
eko_op.close()
eko_op = EKO.edit(target)
for i, elem in eko_op.items():
    eko_op[i] = manipulate.xgrid_reshape(elem, eko_op.xgrid, 2, targetgrid=new_xgrid, inputgrid=new_xgrid)

This is related to NNPDF/nnpdf#2181 (comment) I didn't realize that they were also going to be written down with a shape that is no longer reflected by the metadata.

(not sure whether ths is a bug or a feature that I didn't know I wanted :P but I think eko should not writed down no-longer-readable ekos?)

to_test.tar.zip

[felixhekhorn]

I'd say this is not a bug, but an abuse of interface: if you decide to manipulate the content (eko_op[i] = ) live with the consequences, i.e. a broken file. Maybe we should state more clearly that the use of EKO.edit is discouraged (unless you know exactly what you are doing)? This is particular true since the major change of mindset in v0.15 where we do no longer attempt to modify the file a posteriori, but only the in-memory object.

Why do you want to use edit in the first place?

I think eko should not writed down no-longer-readable ekos?

internally we need EKO.edit of course - but, as said, we can more bluntly discourage its use (the tutorials don't use edit ✅ )

Also let me reply to the mentioned comment here:

Besides that, (cc @felixhekhorn) I think it would be good for EKO to provide a xgrid_reshape which acts on the entire operator struct (and updates the metadata accordingly). The new interface is a bit clunky I feel and it is easy to forget steps (like updating the items in place or updating the xgrid in the operator, which were forgotten here indeed!)

Mmm ... as said, we consciously decided to put the eko into a canonical state and don't touch it afterwards. Dealing with (on any of the four axis) rotated EKO was too error prone. I'd rather leave that one additional line elem_rot = manipulate.xgrid_reshape(elem, eko_op.xgrid, 2, targetgrid=new_xgrid, inputgrid=new_xgrid) to the user - also this way they are more aware that this is potentially dangerous (meaning interpolation is complicated)

[scarlehoff]

Why do you want to use edit in the first place?

Because people was very creative with the grids they used for their EKOs which would of course fail miserably when trying to evolve a PDF.

But ok, the solution I've gone for is indeed editing in memory, with a workaround for 0.13.4 so that we can use them even if they are not supported NNPDF/nnpdf@6e80047
please have a look at my usage of rotation and dataclass.replace, maybe there's a more eko-friendly way of doing the manipulation in memory while being able to use the eko normally later.

Mmm ... as said, we consciously decided to put the eko into a canonical state and don't touch it afterwards. Dealing with (on any of the four axis) rotated EKO was too error prone. I'd rather leave that one additional line elem_rot = manipulate.xgrid_reshape(elem, eko_op.xgrid, 2, targetgrid=new_xgrid, inputgrid=new_xgrid) to the user - also this way they are more aware that this is potentially dangerous (meaning interpolation is complicated)

It is an useful feature for test so that we can create very light EKOs (with only a few points) and then let eko interpolate up to 196 points.