Apparent performance test regression in JS Quick Edit when Tern doesn't find results

[njx]

1. Run the JS Quick Edit Performance test (under the Performance tab) in sprint 24 and in sprint 25

Result: Major performance regression in sprint 25--from a few hundred ms in sprint 24 to >2 seconds each in sprint 25.

It looks like the case we're testing is one where Tern doesn't find anything, so is falling back to the old code. But it seems to take a long time to figure out that it can't find anything.

[njx]

@jeffkenton - hmm, looks like I can't assign this directly to you. I forgot that we had this performance test when reviewing your pull request, so we didn't notice this until we were running tests at the end of the sprint. It would be good to find out why Tern is taking so long to fail on every invocation--it seems like even if it takes awhile to start up on the first time through, we should have some cached info for the later invocations.

[njx]

Reviewed. Medium priority, to @jeffkenton for sprint 26.

[jeffkenton]

I just tried removing my recent Tern jump-to-definition code from QuickEdit and tested this. I used the brackets.js file and searched for the function "always()", which Tern doesn't find. Without the jump-to-definition code, time taken was 3 seconds. With the jump-to-definition code it was only 100ms longer. I will look further.

[njx]

Hmm. Maybe there's just something wrong with the perf test. I did try this manually and it seemed noticeably longer with Tern, but I've noticed some inconsistency in the perceived amount of time it takes to open Quick Edit with the Tern stuff so it's possible there's something else going on as well.

[jeffkenton]

One thing further. Doing the same test a second time (without closing Brackets) gets results much faster in both cases -- approximately 450 ms, using the "Show Performance Data" tools.

Question: how did you disable Tern while doing this? Did you comment out the code, or do you have a trick I didn't think of?

[njx]

I just went back to sprint 24 to compare, I think.

[jeffkenton]

I just ran the following test with newly downloaded Brackets (on a Mac), both with Sprint 24 and Sprint 25:

1. open brackets.js
2. scroll to line 200
3. position the cursor to the call to "always()".
4. QuickEdit.
5. Look at "Show Performance Data".

The performance results are about 4 seconds for QuickEdit in both cases on my machine. I don't see what you are reporting. Note that in both cases, QuickEdit reads in about 1500 extra files to find various definitions of "always()". Jump-To-Definition doesn't find "always()".

Running the performance tests, I see identical results -- about 1.25 seconds -- for both Sprint 24 and Sprint 25.

It looks like we're not doing the same thing, somehow.

[njx]

Weird. I'll spend some time today to try to verify what I thought I was seeing.

[njx]

I can definitely reproduce the performance test discrepancy (note that this is not from "Show Performance Data", but from the JSQuickEdit test in the Performance tab). In sprint 24, the first JavaScript Inline Editor Creation entry is about 600ms, and all the others further down are less than 100ms. In sprint 25, all of the JavaScript Inline Editor Creation entries are over 1800ms.

However, I'm starting to suspect that this test is misleading somehow. If I try to reproduce it manually (by opening jquery.effects.core.js and putting the cursor in the extend call on line 272, then doing Cmd-E), I get roughly similar times in both sprint-24 and sprint-25, both perceived and by looking at "Show Performance Data"--roughly 4s. And if it is actually taking 4s to open the inline editor, then it's not clear why the test should have <100ms timings in sprint 24. So it must not be actually timing what we think it's timing.

I also had to modify the test in sprint 25 to get it to run with the new code, which assumes the cursor is actually in the function name we're trying to find, whereas the old code ignored the editor contents and just looked for the specified function.

@jasonsanjose - do you know who wrote this perf test originally? Seems like we might need to figure out what's going on and try to get better timings.

[jasonsanjose]

I wrote those perf tests. I haven't looked at the new tern code yet to see what needs an update though.

[njx]

It seems like it must have been wrong even for the pre-tern version, though, because if I do a similar lookup manually in sprint 24 it takes 4s the first time, whereas the perf test shows ~600ms.

[njx]

At this point I think we should punt this out of Sprint 26 (and maybe reassign it to @jasonsanjose?). There's enough evidence that there's no significant regression when actually using Quick Edit in the product and that this might just be an artifact of the perf test.

[njx]

Moved to sprint 27.