Improve iris.pandas cube -> data.frame

[hsteptoe]

🚀 Pull Request

Description

Improve the conversion of a Cube to a Pandas data.frame. Aims to address #4526

Aims to deal with:

• An arbitrary number of dimensions -> d8c5b40
• Missing cube dimensions -> a46c08a
• Add auxiliary coordinate information to dataframe -> 1df3072
• Add global attribute data to dataframe -> 2c2ce69
• Deal with copy issue
• Update doc strings
• Update tests (but not unit & integration testing)
• Update Whats New
• Check alignment with @trexfeathers Improvements to Pandas-to-Iris bridge #4890

Major proposed (possibly breaking) changes

• Having functions to convert to a Series and a DataFrame seems superfluous. A Series is arguably a special case of a DataFrame, so I propose having only a function to do the DataFrame conversion, and then let folk further convert to a Series if they want to via standard pandas functions. This is done in 61b801a.
  -I don't think the copy argument is valid any more. Working with 'long' format DataFrames there is no continuity between the cube.data array and the DataFrame in memory. copy=True is the default, but there is not option for copy=False any more. This is done in 071d07f. Copy ability retained.

WORK IN PROGRESS

[hsteptoe]

Good discussion with @trexfeathers about our joint work on Pandas-Iris bridging. We agreed that he would focus (#4890) on the DataFrame -> Cube, so this PR will only focus on Cube -> DataFrame

[trexfeathers]

Having learnt more about NumPy views and Pandas' copy behaviour, I think this module's original copy option is both valuable and also something we might be able to keep working. It's value is in cases where a large in-memory Cube is being converted, since copying will double the memory demand and could even blow the machine's memory. So if there is a prospect of avoiding copying I reckon it's worth exploring.

See my comments below.

[trexfeathers]

Thanks @hsteptoe! Looking good.

A few more for you...

[trexfeathers]

Thanks @hsteptoe! We're basically there, just two final comments:

• #4669 (comment)
• #4669 (comment)

[trexfeathers]

Well I'm perplexed. They all passed for me locally!

[hsteptoe]

Well I'm perplexed. They all passed for me locally!

And for me... but it looked like there was some difference in the number of cols being printed in the github tests vs my local tests... now checked the output and matched to github test version...

[trexfeathers]

Well I'm perplexed. They all passed for me locally!

And for me... but it looked like there was some difference in the number of cols being printed in the github tests vs my local tests... now checked the output and matched to github test version...

I'm guessing Pandas is sensitive to available horizontal space when running. Rather frustrating as this gives us a situation where local behaves differently to CI. I won't let this get in the way of merging and I'll discuss with the team in future.

[hsteptoe]

Well I'm perplexed. They all passed for me locally!

And for me... but it looked like there was some difference in the number of cols being printed in the github tests vs my local tests... now checked the output and matched to github test version...

I'm guessing Pandas is sensitive to available horizontal space when running. Rather frustrating as this gives us a situation where local behaves differently to CI. I won't let this get in the way of merging and I'll discuss with the team in future.

Possibly some pandas options that we could explore: https://stackoverflow.com/a/11711637

[trexfeathers]

Super! Thanks @hsteptoe. And we made it in under 100 commits 😂

[hsteptoe]

Super! Thanks @hsteptoe. And we made it in under 100 commits 😂

🎉 @trexfeathers Thanks for all your help!

[kaedonkers]

Awesome, thanks gents!