Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pandas Panel(s) are not supported in the Variable Explorer #2510

Closed
s-celles opened this issue Jul 2, 2015 · 11 comments
Closed

Pandas Panel(s) are not supported in the Variable Explorer #2510

s-celles opened this issue Jul 2, 2015 · 11 comments
Labels
component:Variable Explorer type:Enhancement

Comments

@s-celles
Copy link

s-celles commented Jul 2, 2015

Hello,

try in IPython inside Spyder

>>> #from pandas.io import data, wb
>>> from pandas_datareader import data, wb # pip install pandas_datareader
>>> panel = data.DataReader(['MSFT', 'GOOG', 'AAPL'], 'yahoo')
>>> panel

This panel doesn't appears in variables explorer.

>>> df_adj_close = panel['Adj Close']

df_adj_close appears in variables explorer.

Kind regards
@ccordoba12
Copy link
Member

We don't have support for panels, just DataFrames and Series.

@dhoegh, how hard would it be to add support for them? Could you help us with that?

@ccordoba12 ccordoba12 changed the title Pandas Panel(s) doesn't appears in variable explorer Pandas Panel(s) are not supporte in the Variable Explorer Jul 2, 2015
@dhoegh
Copy link
Contributor

dhoegh commented Jul 2, 2015

I haven't worked worked with panels, so I don't know. I do not have time for looking into the issue. By looking in the documentation panel is described as a three dimensional container, see:http://pandas.pydata.org/pandas-docs/stable/dsintro.html. This means it is not straight forward and some inspiration from the three dimensional array viewer could probably be used.

@s-celles
Copy link
Author

s-celles commented Jul 2, 2015

Maybe you should consider to display "slices" of Panel (ie DataFrame)

>>> panel.loc[:, "2015-07-01", :]

            Open        High         Low       Close    Volume   Adj Close
AAPL  126.900002  126.940002  125.989998  126.599998  30128600  126.599998
GOOG  524.729980  525.690002  518.229980  521.840027   1961000  521.840027
MSFT   44.459999   45.230000   44.099998   44.450001  28337200   44.450001

>>> panel.loc["Adj Close", :, :]
                  AAPL        GOOG       MSFT
Date                                         
2010-01-04   28.718094         NaN  26.769336
2010-01-05   28.767737         NaN  26.777984
2010-01-06   28.310151         NaN  26.613650
2010-01-07   28.257821         NaN  26.336875
2010-01-08   28.445687         NaN  26.518508
2010-01-11   28.194745         NaN  26.181189
2010-01-12   27.874037         NaN  26.008204
2010-01-13   28.267213         NaN  26.250383
2010-01-14   28.103497         NaN  26.777984
2010-01-15   27.633830         NaN  26.691493
2010-01-19   28.856306         NaN  26.899074
2010-01-20   28.412135         NaN  26.457964
2010-01-21   27.921004         NaN  25.956309
2010-01-22   26.536154         NaN  25.048140
2010-01-25   27.250048         NaN  25.359512
2010-01-26   27.635174         NaN  25.515199
2010-01-27   27.895500         NaN  25.662235
2010-01-28   26.742807         NaN  25.221125
2010-01-29   25.772608         NaN  24.373502
2010-02-01   26.130897         NaN  24.572434
2010-02-02   26.282534         NaN  24.615679
2010-02-03   26.734757         NaN  24.762716
2010-02-04   25.771266         NaN  24.079428
2010-02-05   26.228860         NaN  24.235114
2010-02-08   26.049045         NaN  23.975637
2010-02-09   26.326814         NaN  24.226465
2010-02-10   26.183238         NaN  24.209166
2010-02-11   26.659611         NaN  24.321607
2010-02-12   26.889071         NaN  24.157271
2010-02-16   27.294328         NaN  24.635203
...                ...         ...        ...
2015-05-20  130.059998  539.270020  47.580002
2015-05-21  131.389999  542.510010  47.419998
2015-05-22  132.539993  540.109985  46.900002
2015-05-26  129.619995  532.320007  46.590000
2015-05-27  132.039993  539.789978  47.610001
2015-05-28  131.779999  539.780029  47.450001
2015-05-29  130.279999  532.109985  46.860001
2015-06-01  130.539993  533.989990  47.230000
2015-06-02  129.960007  539.179993  46.919998
2015-06-03  130.119995  540.309998  46.849998
2015-06-04  129.360001  536.700012  46.360001
2015-06-05  128.649994  533.330017  46.139999
2015-06-08  127.800003  526.830017  45.730000
2015-06-09  127.419998  526.690002  45.650002
2015-06-10  128.880005  536.690002  46.610001
2015-06-11  128.589996  534.609985  46.439999
2015-06-12  127.169998  532.330017  45.970001
2015-06-15  126.919998  527.200012  45.480000
2015-06-16  127.599998  528.150024  45.830002
2015-06-17  127.300003  529.260010  45.970001
2015-06-18  127.879997  536.729980  46.720001
2015-06-19  126.599998  536.690002  46.099998
2015-06-22  127.610001  538.190002  46.230000
2015-06-23  127.029999  540.479980  45.910000
2015-06-24  128.110001  537.840027  45.639999
2015-06-25  127.500000  535.229980  45.650002
2015-06-26  126.750000  531.690002  45.259998
2015-06-29  124.529999  521.520020  44.369999
2015-06-30  125.430000  520.510010  44.150002
2015-07-01  126.599998  521.840027  44.450001

[1383 rows x 3 columns]

>>> panel.loc[:, :, "MSFT"]

                 Open       High        Low      Close     Volume  Adj Close
Date                                                                        
2010-01-04  30.620001  31.100000  30.590000  30.950001   38409100  26.769336
2010-01-05  30.850000  31.100000  30.639999  30.959999   49749600  26.777984
2010-01-06  30.879999  31.080000  30.520000  30.770000   58182400  26.613650
2010-01-07  30.629999  30.700001  30.190001  30.450001   50559700  26.336875
2010-01-08  30.280001  30.879999  30.240000  30.660000   51197400  26.518508
2010-01-11  30.709999  30.760000  30.120001  30.270000   68754700  26.181189
2010-01-12  30.150000  30.400000  29.910000  30.070000   65912100  26.008204
2010-01-13  30.260000  30.520000  30.010000  30.350000   51863500  26.250383
2010-01-14  30.309999  31.100000  30.260000  30.959999   63228100  26.777984
2010-01-15  31.080000  31.240000  30.709999  30.860001   79913200  26.691493
2010-01-19  30.750000  31.240000  30.680000  31.100000   46575700  26.899074
2010-01-20  30.809999  30.940001  30.309999  30.590000   54849500  26.457964
2010-01-21  30.610001  30.719999  30.000000  30.010000   73086700  25.956309
2010-01-22  30.000000  30.200001  28.840000  28.959999  102004600  25.048140
2010-01-25  29.240000  29.660000  29.100000  29.320000   63373000  25.359512
2010-01-26  29.200001  29.850000  29.090000  29.500000   66639900  25.515199
2010-01-27  29.350000  29.820000  29.020000  29.670000   63949500  25.662235
2010-01-28  29.840000  29.870001  28.889999  29.160000  117513700  25.221125
2010-01-29  29.900000  29.920000  27.660000  28.180000  193888500  24.373502
2010-02-01  28.389999  28.480000  27.920000  28.410000   85931100  24.572434
2010-02-02  28.370001  28.500000  28.139999  28.459999   54413700  24.615679
2010-02-03  28.260000  28.790001  28.120001  28.629999   61397900  24.762716
2010-02-04  28.379999  28.500000  27.809999  27.840000   77850000  24.079428
2010-02-05  28.000000  28.280001  27.570000  28.020000   80960100  24.235114
2010-02-08  28.010000  28.080000  27.570000  27.719999   52820600  23.975637
2010-02-09  27.969999  28.340000  27.750000  28.010000   59195800  24.226465
2010-02-10  28.030001  28.240000  27.840000  27.990000   48591300  24.209166
2010-02-11  27.930000  28.400000  27.700001  28.120001   65993700  24.321607
2010-02-12  27.809999  28.059999  27.580000  27.930000   81117200  24.157271
2010-02-16  28.129999  28.370001  28.020000  28.350000   51935600  24.635203
...               ...        ...        ...        ...        ...        ...
2015-05-20  47.389999  47.930000  47.270000  47.580002   25047900  47.580002
2015-05-21  47.279999  47.599998  47.009998  47.419998   22410700  47.419998
2015-05-22  47.299999  47.349998  46.820000  46.900002   25720600  46.900002
2015-05-26  46.830002  46.880001  46.189999  46.590000   29581900  46.590000
2015-05-27  46.820000  47.770000  46.619999  47.610001   27335600  47.610001
2015-05-28  47.500000  48.020000  47.389999  47.450001   19283700  47.450001
2015-05-29  47.430000  47.570000  46.590000  46.860001   35428100  46.860001
2015-06-01  47.060001  47.770000  46.619999  47.230000   28592900  47.230000
2015-06-02  46.930000  47.349998  46.619999  46.919998   21283400  46.919998
2015-06-03  47.369999  47.740002  46.820000  46.849998   27955200  46.849998
2015-06-04  46.790001  47.160000  46.200001  46.360001   26868000  46.360001
2015-06-05  46.310001  46.520000  45.840000  46.139999   25258900  46.139999
2015-06-08  46.299999  46.430000  45.669998  45.730000   21822300  45.730000
2015-06-09  45.759998  45.939999  45.459999  45.650002   24124700  45.650002
2015-06-10  45.790001  46.830002  45.689999  46.610001   27952300  46.610001
2015-06-11  46.660000  46.919998  46.130001  46.439999   27186700  46.439999
2015-06-12  46.220001  46.470001  45.900002  45.970001   23416200  45.970001
2015-06-15  45.450001  45.650002  45.020000  45.480000   30080100  45.480000
2015-06-16  45.349998  46.240002  45.299999  45.830002   26822500  45.830002
2015-06-17  45.730000  46.070000  45.360001  45.970001   28566300  45.970001
2015-06-18  46.220001  46.799999  46.169998  46.720001   32184500  46.720001
2015-06-19  46.790001  46.830002  45.990002  46.099998   57500600  46.099998
2015-06-22  46.330002  46.720001  46.160000  46.230000   20278600  46.230000
2015-06-23  46.130001  46.279999  45.619999  45.910000   25787000  45.910000
2015-06-24  45.669998  46.250000  45.549999  45.639999   34784600  45.639999
2015-06-25  46.029999  46.060001  45.500000  45.650002   20569500  45.650002
2015-06-26  45.650002  46.279999  45.029999  45.259998   45610900  45.259998
2015-06-29  45.040001  45.230000  44.360001  44.369999   33899600  44.369999
2015-06-30  44.709999  44.720001  43.939999  44.150002   35214100  44.150002
2015-07-01  44.459999  45.230000  44.099998  44.450001   28337200  44.450001

[1383 rows x 6 columns]

see http://pandas.pydata.org/pandas-docs/stable/indexing.html

panel.loc[item_indexer,major_indexer,minor_indexer]

Ideally panel editor should provide a way to define if slice are given as item, major_index or minor_index

@dhoegh
Copy link
Contributor

dhoegh commented Jul 2, 2015

This is exactly how the 3d array viewer works does a slice always produced a dataframe?

@s-celles
Copy link
Author

s-celles commented Jul 2, 2015

I think so but maybe @jreback or @wesm can confirm or infirm this.

@jreback
Copy link

jreback commented Jul 2, 2015

Panels can be easily converted to multi-index frames. This is a flattened version of a Panel.

In [8]: pd.set_option('max_rows',10)

In [9]: panel.to_frame()
Out[9]: 
                        Open        High         Low       Close     Volume   Adj Close
Date       minor                                                                       
2010-01-04 AAPL   213.429998  214.500023  212.380001  214.010025  123432400   28.718094
           MSFT    30.620001   31.100000   30.590000   30.950001   38409100   26.769336
2010-01-05 AAPL   214.599985  215.589994  213.250034  214.379967  150476200   28.767737
           MSFT    30.850000   31.100000   30.639999   30.959999   49749600   26.777984
2010-01-06 AAPL   214.379967  215.229986  210.749977  210.969995  138040000   28.310151
...                      ...         ...         ...         ...        ...         ...
2015-06-30 GOOG   526.020020  526.250000  520.500000  520.510010    2217200  520.510010
           MSFT    44.709999   44.720001   43.939999   44.150002   35214100   44.150002
2015-07-01 AAPL   126.900002  126.940002  125.989998  126.599998   30128600  126.599998
           GOOG   524.729980  525.690002  518.229980  521.840027    1961000  521.840027
           MSFT    44.459999   45.230000   44.099998   44.450001   28337200   44.450001

[3085 rows x 6 columns]

If you want to slice, then Panels follow the same semantics, e.g. if you specifiy for an indexer a scalar value then you would get a DataFrame

e.g.

In [11]: panel.iloc[0]
Out[11]: 
                  AAPL        GOOG       MSFT
Date                                         
2010-01-04  213.429998         NaN  30.620001
2010-01-05  214.599985         NaN  30.850000
2010-01-06  214.379967         NaN  30.879999
2010-01-07  211.750000         NaN  30.629999
2010-01-08  210.300020         NaN  30.280001
...                ...         ...        ...
2015-06-25  128.860001  538.869995  46.029999
2015-06-26  127.669998  537.260010  45.650002
2015-06-29  125.459999  525.010010  45.040001
2015-06-30  125.570000  526.020020  44.709999
2015-07-01  126.900002  524.729980  44.459999

[1383 rows x 3 columns]

In [12]: panel.iloc[:,0]
Out[12]: 
            Open        High         Low       Close     Volume  Adj Close
AAPL  213.429998  214.500023  212.380001  214.010025  123432400  28.718094
GOOG         NaN         NaN         NaN         NaN        NaN        NaN
MSFT   30.620001   31.100000   30.590000   30.950001   38409100  26.769336

In [13]: panel.iloc[:,:,0]
Out[13]: 
                  Open        High         Low       Close     Volume   Adj Close
Date                                                                             
2010-01-04  213.429998  214.500023  212.380001  214.010025  123432400   28.718094
2010-01-05  214.599985  215.589994  213.250034  214.379967  150476200   28.767737
2010-01-06  214.379967  215.229986  210.749977  210.969995  138040000   28.310151
2010-01-07  211.750000  211.999966  209.050032  210.580027  119282800   28.257821
2010-01-08  210.300020  211.999966  209.059965  211.980019  111902700   28.445687
...                ...         ...         ...         ...        ...         ...
2015-06-25  128.860001  129.199997  127.500000  127.500000   31816700  127.500000
2015-06-26  127.669998  127.989998  126.510002  126.750000   42111000  126.750000
2015-06-29  125.459999  126.470001  124.480003  124.529999   48911400  124.529999
2015-06-30  125.570000  126.120003  124.860001  125.430000   43849800  125.430000
2015-07-01  126.900002  126.940002  125.989998  126.599998   30128600  126.599998

[1383 rows x 6 columns]

multi-indexers / lists will produce a shape based on what you are sub-selecting, e.g.

In [14]: panel.iloc[0,0]
Out[14]: 
AAPL    213.429998
GOOG           NaN
MSFT     30.620001
Name: 2010-01-04 00:00:00, dtype: float64

In [15]: panel.iloc[0,0,0]
Out[15]: 213.42999800000001

In [16]: panel.iloc[[0,1],0,0]
Out[16]: 
Open    213.429998
High    214.500023
Name: AAPL, dtype: float64

note that these are all consistent with numpy slicing semantics

@ccordoba12 ccordoba12 added this to the wishlist milestone Jul 14, 2015
@s-celles
Copy link
Author

Just some code to create a sample Panel without relying on pandas_datareader

items = ['Open', 'High', 'Low', 'Close']
minor_axis = ['AAAA', 'BBBB', 'CCCC']
periods = 5
panel = pd.Panel(np.random.random((len(items), periods, len(minor_axis))), items=items,
    major_axis=pd.date_range('1/1/2000', periods=periods),
    minor_axis=minor_axis)

Panel4D can be construct using

labels = ['Label1','Label2']
items = ['Item1', 'Item2']
periods = 5
minor_axis = ['A', 'B', 'C', 'D']
p4d = pd.Panel4D(np.random.random((len(labels), len(items), periods, len(minor_axis))),
           labels=labels,
           items=items,
           major_axis=pd.date_range('1/1/2000', periods=periods),
           minor_axis=minor_axis)

but I don't think that Spyder variables editor support hyper-cubes (Numpy arrays with more than 3 dimensions)

a = np.random.random((2,3,4,5))

@jitseniesen jitseniesen mentioned this issue Dec 17, 2016
Merged
@Prikers
Copy link
Contributor

Prikers commented Sep 10, 2017

As panels are being deprecated from pandas, is it worth adding support for them now? Maybe this issue could be closed.

@ccordoba12
Copy link
Member

We could consider to support xarray's then. What do you say @Prikers?

@Prikers
Copy link
Contributor

Prikers commented Sep 10, 2017

Yes that would make sense!
I can give it a try next week. Should I open a new issue for xarray support or we use this one as a reference?

@s-celles s-celles changed the title Pandas Panel(s) are not supporte in the Variable Explorer Pandas Panel(s) are not supported in the Variable Explorer Sep 10, 2017
@goanpeca
Copy link
Member

@Prikers awesome, I think its better to open a new issue :-), you can also tag this issue for reference on that new issue.

@s-celles s-celles mentioned this issue Sep 10, 2017
Open
@ccordoba12 ccordoba12 removed this from the wishlist milestone Sep 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component:Variable Explorer type:Enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@s-celles @ccordoba12 @jreback @goanpeca @dhoegh @Prikers