Skip to content

chore: Upgrade replica to v0.5.5#559

Closed
ninegua wants to merge 1 commit intopaulliu/auth-transformfrom
paulliu/update-to-replica-v0.5.5
Closed

chore: Upgrade replica to v0.5.5#559
ninegua wants to merge 1 commit intopaulliu/auth-transformfrom
paulliu/update-to-replica-v0.5.5

Conversation

@ninegua
Copy link
Member

@ninegua ninegua commented Apr 10, 2020

@chenyan-dfinity reported CPU usage of replica would increase dramatically if running for longer than a couple hours for v0.5.4. This is likely related to a performance degradation problem that was already fixed in a later version. So we just cut a v0.5.5 release, and this PR is to see if everything works out for sdk.

@ninegua ninegua changed the title Upgrade replica to v0.5.5 chore: Upgrade replica to v0.5.5 Apr 10, 2020
@ninegua ninegua force-pushed the paulliu/update-to-replica-v0.5.5 branch from 90a2c0b to 34b4ef2 Compare April 10, 2020 21:41
@ninegua ninegua force-pushed the paulliu/update-to-replica-v0.5.5 branch from 34b4ef2 to a399c39 Compare April 10, 2020 21:44
@ninegua ninegua requested a review from chenyan-dfinity April 10, 2020 21:57
@chenyan-dfinity
Copy link
Contributor

After running for a few hours, the CPU can still increase to around 50%. It seems to be increasing at a slower pace than v0.5.4, and the "idle wake ups" are smaller.

Another thing I noticed is that different from v0.5.2, the replica keeps generating checkpoints even when no messages are sent. Don't know if this is expected or can cause the cpu usage.

Apr 11 05:30:50.150 INFO Created checkpoint @32758 in 16.030148ms, StateManager: 1

@ninegua
Copy link
Member Author

ninegua commented Apr 11, 2020

Execution state snapshots are made at an interval to enable purging of older state. But it shouldn't have caused increasing CPU usage. There is something else going on, I'll investigate.

@ninegua
Copy link
Member Author

ninegua commented Apr 12, 2020

Investigation of the CPU time is tracked here: https://github.com/dfinity-lab/dfinity/pull/3204

@hansl hansl mentioned this pull request Apr 20, 2020
3 tasks
@ninegua
Copy link
Member Author

ninegua commented Apr 21, 2020

A fix to replica is under view at https://github.com/dfinity-lab/dfinity/pull/3270

@hansl
Copy link
Contributor

hansl commented Apr 23, 2020

@ninegua Now that dfinity-lab/dfinity#3270 is merged, can we consider moving this PR forward? This is currently the only blocker on the release (tracking issue #581).

@ninegua
Copy link
Member Author

ninegua commented Apr 24, 2020

We need to first cut new replica release 0.5.6 before updating the reference revision here. However, there is another problem related to crypto key setup that prevents single node replica from running.

@chenyan-dfinity
Copy link
Contributor

Do we have a notion of cherry pick in the release process, where we can cut a release up to certain PR, plus a set of PRs beyond the cut point?

@ninegua
Copy link
Member Author

ninegua commented Apr 24, 2020

Do we have a notion of cherry pick in the release process, where we can cut a release up to certain PR, plus a set of PRs beyond the cut point?

That will require a separate branch (e.g. just for v0.5.5) instead of master. I actually have it ready in https://github.com/dfinity-lab/dfinity/pull/3204. But the bigger question is whether it is desirable to maintain a branch release that deviates from master. Let's see what @p-shahi has to say about it.

@p-shahi
Copy link
Contributor

p-shahi commented Apr 24, 2020

Do we have a notion of cherry pick in the release process, where we can cut a release up to certain PR, plus a set of PRs beyond the cut point?

That will require a separate branch (e.g. just for v0.5.5) instead of master. I actually have it ready in dfinity-lab/dfinity#3204. But the bigger question is whether it is desirable to maintain a branch release that deviates from master. Let's see what @p-shahi has to say about it.

update: will provide a release from branch with cherry picks for replica 0.5.6
the feature flag discussion will be concluded separately as we need buy in from crypto team on whether or not they want to add such a thing (use-case: feature flag turn off csp key generation and use a set of debug keys)

====
@ninegua it is not desire-able to have a separate branch for just one release with cherry pick fixes but to unblock SDK we can do this.

(if possible, I would suggest we wrap the crypto key setup changes around a feature flag. then, disable this feature for the replica bundled inside dfx - this should be straightforward and wouldn't require a new branch)

@hansl @chenyan-dfinity please communicate the urgency of needing to push out a new SDK release so that I know whether to immediately provide you a new tag (create a new branch with cherry picks) or if we can spend some time trying to get feature flags for this change
(alternatively, you can add this helper script into dfx: https://github.com/dfinity-lab/dfinity/pull/3318/files but is that really what dfx wants to do? @hansl )

@ninegua
Copy link
Member Author

ninegua commented Apr 24, 2020

superseded by #591

@ninegua ninegua closed this Apr 24, 2020
dfinity-bot added a commit that referenced this pull request Jan 13, 2021
mergify bot pushed a commit that referenced this pull request Jan 13, 2021
@lwshang lwshang deleted the paulliu/update-to-replica-v0.5.5 branch July 29, 2022 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants