DApp performance: Home page and Listing details page slow under load

[franckc]

Last night after the OGN announcement, we experienced some slowness on the DApp. We saw the following:

• home page tsking tens of seconds to load. As a data point, I experienced over 30sec load time from desktop. This is due to the client waiting on the graphql server response to the AllListing query. Sometimes it take over 60 sec and timeout because nginx would kill it (we have a 60sec limit on queries to the graphql backend).
• Listing details page taking tens of seconds to load. Waiting on the graphql query Listing.

[franckc]

Note: last night as a short-term measure we scaled up the number of graphql back-end instances from 1 to 3. It seemed to have helped a bit but after that change I was still seeing some long load times.

[mikeshultz]

It's not shocking we saw this last night but does this warrant an open issue(is it still occurring for you)? Why is scaling up graphql back-end instances short term?

[franckc]

Did a quick test. Wiped our all my storage data. Currently DApp is fairly responsive for me (loads within 3 sec).

But I'm worried that when Asia wakes up we'll see again the same performance issues. At least I'd love to get better instrumentation in place quickly so that we can diagnose the root cause of the issue next time it occurs.

[micahalcorn]

Closing this for now 🤞