Skip to content

Conversation

@jreback
Copy link
Contributor

@jreback jreback commented Jan 21, 2014

added vbenches and make most apply operations a fair bit faster

closes #6013

I can't put this one in the vbench as its cython based, but previously was about 100ms (on my machine)
0.12 clocks in at 15ms (the remaining difference in time is due to the sub-indexing, e.g. x['a'] which in theory could also be speeded but a bit tricky and not a lot more to win)

In [6]: %timeit df.apply(lambda x: integrate_f_typed(x['a'], x['b'], x['N']), axis=1)
10 loops, best of 3: 26.2 ms per loop

Here's a related benchmark from #5654 (a version of which is in the vbenches):

In [16]: s = pd.Series(np.arange(4096.))

In [17]: df = pd.DataFrame({i:s for i in range(4096)})

0.12

In [18]: %timeit df.apply((lambda x: np.corrcoef(x, s)[0, 1]), axis=1)
1 loops, best of 3: 1.11 s per loop

with this PR

In [3]: In [18]: %timeit df.apply((lambda x: np.corrcoef(x, s)[0, 1]), axis=1)

1 loops, best of 3: 1.24 s per loop
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_apply_ref_by_name                      |  15.4320 |  32.2250 |   0.4789 | like the cython example above
frame_apply_user_func                        | 166.7013 | 232.7064 |   0.7164 | bench for example right above
frame_apply_axis_1                           | 108.5210 | 109.1857 |   0.9939 |
frame_apply_pass_thru                        |   6.5626 |   6.4860 |   1.0118 |
frame_apply_np_mean                          |   3.9033 |   0.9290 |   4.2018 | this suffers from some construction overhead, not much I can do
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

PERF: allow apply to use the fast-path in mixed type frames except where datelike
      are present
jreback added a commit that referenced this pull request Jan 21, 2014
PERF: apply perf enhancements
@jreback jreback merged commit 2258efd into pandas-dev:master Jan 21, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Perf regression in 0.13+ for df.apply

1 participant