Skip to content

Conversation

@khanhtc1202
Copy link
Member

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Failed lambda application deployment can be rolled back automatically

@pipecd-bot
Copy link
Collaborator

TODO

The following ISSUES will be created once got merged. If you want me to skip creating the issue, you can use /todo skip command.

Details

1. fix case == 1

https://github.com/pipe-cd/pipe/blob/19af962f9d86182ab043ea7cd74c68b93c9700c8/pkg/app/piped/executor/lambda/rollback.go#L107-L110

This was created by todo plugin since "TODO:" was found in 19af962 when #1462 was merged. cc: @khanhtc1202.

@pipecd-bot
Copy link
Collaborator

COVERAGE

Code coverage for golang is 32.69%. This pull request decreases coverage by -0.77%.

File Function Base Head Diff
pkg/app/piped/cloudprovider/lambda/client.go RoutingTrafficConfig.Encode -- 0.00% +0.00%
pkg/app/piped/cloudprovider/lambda/client.go percentageToPercent -- 0.00% +0.00%
pkg/app/piped/executor/lambda/deploy.go deployExecutor.Execute -- 0.00% +0.00%
pkg/app/piped/executor/lambda/deploy.go deployExecutor.ensureSync -- 0.00% +0.00%
pkg/app/piped/executor/lambda/deploy.go deployExecutor.ensurePromote -- 0.00% +0.00%
pkg/app/piped/executor/lambda/deploy.go deployExecutor.ensureRollout -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go Register -- 71.43% +71.43%
pkg/app/piped/executor/lambda/lambda.go findCloudProvider -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go loadFunctionManifest -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go sync -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go rollout -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go promote -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go configureTrafficRouting -- 100.00% +100.00%
pkg/app/piped/executor/lambda/lambda.go build -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.Execute -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.ensureRollback -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollback -- 0.00% +0.00%
pkg/app/piped/cloudprovider/lambda/client.go client.UpdateTrafficConfig 0.00% 0.00% +0.00%
pkg/config/deployment.go PipelineStage.UnmarshalJSON 45.90% 43.75% -2.15%
pkg/app/piped/cloudprovider/lambda/client.go client.GetTrafficConfig 0.00% 0.00% +0.00%

@khanhtc1202 khanhtc1202 marked this pull request as ready for review January 21, 2021 13:25
@pipecd-bot
Copy link
Collaborator

COVERAGE

Code coverage for golang is 32.59%. This pull request decreases coverage by -0.87%.

File Function Base Head Diff
pkg/app/piped/cloudprovider/lambda/client.go RoutingTrafficConfig.Encode -- 0.00% +0.00%
pkg/app/piped/cloudprovider/lambda/client.go percentageToPercent -- 0.00% +0.00%
pkg/app/piped/executor/lambda/deploy.go deployExecutor.Execute -- 0.00% +0.00%
pkg/app/piped/executor/lambda/deploy.go deployExecutor.ensureSync -- 0.00% +0.00%
pkg/app/piped/executor/lambda/deploy.go deployExecutor.ensurePromote -- 0.00% +0.00%
pkg/app/piped/executor/lambda/deploy.go deployExecutor.ensureRollout -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go Register -- 71.43% +71.43%
pkg/app/piped/executor/lambda/lambda.go findCloudProvider -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go loadFunctionManifest -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go sync -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go rollout -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go promote -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go configureTrafficRouting -- 100.00% +100.00%
pkg/app/piped/executor/lambda/lambda.go build -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.Execute -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.ensureRollback -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollback -- 0.00% +0.00%
pkg/config/deployment.go PipelineStage.UnmarshalJSON 45.90% 43.75% -2.15%
pkg/app/piped/cloudprovider/lambda/client.go client.GetTrafficConfig 0.00% 0.00% +0.00%
pkg/app/piped/cloudprovider/lambda/client.go client.UpdateTrafficConfig 0.00% 0.00% +0.00%

@nakabonne
Copy link
Member

nakabonne commented Jan 22, 2021

I was taking a walk around the implementation for lambda cloud provider, then I found out this part:
https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L274-L275

Instead it'd better be:

	for version, weight := range cfg.RoutingConfig.AdditionalVersionWeights {
		newerVersionTraffic = percentageToPercent(aws.Float64Value(weight))

Also, as you may know all around here should be replaced with aws.StringValue().
https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L266

https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L282

https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L194

I should've mentioned on previous review, but hopefully it helps you.

@khanhtc1202
Copy link
Member Author

I was taking a walk around the implementation for lambda cloud provider, then I found out this part:
https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L274-L275

Instead it'd better be:

	for version, weight := range cfg.RoutingConfig.AdditionalVersionWeights {
		newerVersionTraffic = percentageToPercent(aws.Float64Value(weight))

Also, as you may know all around here should be replaced with aws.StringValue().
https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L266

https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L282

https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L194

I should've mentioned on previous review, but hopefully it helps you.

Thanks for your comment, I will make a separate PR for those fixes 😁

}

originalTrafficCfg := provider.RoutingTrafficConfig{}
if err := json.Unmarshal([]byte(originalTrafficCfgData), &originalTrafficCfg); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we defined Encode function inside the provider package, so this Decode part should be there too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed on 2d8762f 🙏

}

// Restore original traffic config from metadata store.
originalTrafficKeyName := fmt.Sprintf("%s-%s-original", fm.Spec.Name, in.Deployment.RunningCommitHash)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the commit hash for the key is a nice idea. 👍
But do we need the function name?
Because the scope of these keys is inside the deployment. So maybe ("original-traffic-%s", commitHash) is good enough.

Copy link
Member Author

@khanhtc1202 khanhtc1202 Jan 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed on 21b648f 🙏

@khanhtc1202
Copy link
Member Author

I was taking a walk around the implementation for lambda cloud provider, then I found out this part:
https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L274-L275

Instead it'd better be:

	for version, weight := range cfg.RoutingConfig.AdditionalVersionWeights {
		newerVersionTraffic = percentageToPercent(aws.Float64Value(weight))

Also, as you may know all around here should be replaced with aws.StringValue().
https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L266

https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L282

https://github.com/pipe-cd/pipe/blob/d4d222200a9fb5df5c558f9b13c3256e51f202de/pkg/app/piped/cloudprovider/lambda/client.go#L194

I should've mentioned on previous review, but hopefully it helps you.

@nakabonne PTAL 😁
#1464

}

// Restore promoted traffic config from metadata store.
promotedTrafficKeyName := fmt.Sprintf("%s-%s-promote", fm.Spec.Name, in.Deployment.RunningCommitHash)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "latest-promote-traffic-%s", commitHash

promotedTrafficCfgData, ok := in.MetadataStore.Get(promotedTrafficKeyName)
// If there is no previous promoted traffic config, which mean no promote run previously so no need to do anything to rollback.
if !ok {
in.LogPersister.Info("No promoted traffic config found. No need to rollback current remote traffic config.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "It seems the traffic has not been changed during the deployment process. No need to rollback the traffic config."

// we need to reset any others SECONDARY created by previous (until failed) PROMOTE stages.
case 1:
// Validate stored original traffic config, since it PRIMARY ONLY, the percent must be float64(100)
primary, ok := originalTrafficCfg["primary"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "primary" should be defined as a const.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to confirm, you mean the key "primary" for those maps, is that right 👀 ( so the "secondary" as well should be defined as a const too, I guess )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the key of the map. 👍 (Because it is shared around the code.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙆‍♀️

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed on a9987d0 🙏

@pipecd-bot
Copy link
Collaborator

COVERAGE

Code coverage for golang is 32.59%. This pull request decreases coverage by -0.34%.

File Function Base Head Diff
pkg/app/piped/cloudprovider/lambda/client.go RoutingTrafficConfig.Encode -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.Execute -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.ensureRollback -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollback -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go Register 80.00% 71.43% -8.57%
pkg/app/piped/executor/lambda/lambda.go sync 0.00% 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go rollout 0.00% 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go promote 0.00% 0.00% +0.00%

@pipecd-bot
Copy link
Collaborator

GO_LINTER

The following files are not gofmt-ed. By commenting /golinter fmt, the formatted one will be appended to this pull request automatically.

pkg/app/piped/executor/lambda/lambda_test.go
--- pkg/app/piped/executor/lambda/lambda_test.go.orig
+++ pkg/app/piped/executor/lambda/lambda_test.go
@@ -17,8 +17,9 @@
 import (
 	"testing"
 
-	provider "github.com/pipe-cd/pipe/pkg/app/piped/cloudprovider/lambda"
 	"github.com/stretchr/testify/assert"
+
+	provider "github.com/pipe-cd/pipe/pkg/app/piped/cloudprovider/lambda"
 )
 
 func TestConfigureTrafficRouting(t *testing.T) {

@khanhtc1202
Copy link
Member Author

@nghialv @nakabonne PTAL 🙏

@pipecd-bot
Copy link
Collaborator

COVERAGE

Code coverage for golang is 32.58%. This pull request decreases coverage by -0.35%.

File Function Base Head Diff
pkg/app/piped/cloudprovider/lambda/client.go RoutingTrafficConfig.Encode -- 0.00% +0.00%
pkg/app/piped/cloudprovider/lambda/client.go RoutingTrafficConfig.Decode -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.Execute -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.ensureRollback -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollback -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go Register 80.00% 71.43% -8.57%
pkg/app/piped/executor/lambda/lambda.go sync 0.00% 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go rollout 0.00% 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go promote 0.00% 0.00% +0.00%


func (c *RoutingTrafficConfig) Decode(data []byte) bool {
if err := json.Unmarshal(data, c); err != nil {
return false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to return "error" instead of "bool", example for logging.
Same with the "Encode" function.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙆‍♀️ addressed on fc8fffb

@pipecd-bot
Copy link
Collaborator

COVERAGE

Code coverage for golang is 32.58%. This pull request decreases coverage by -0.35%.

File Function Base Head Diff
pkg/app/piped/cloudprovider/lambda/client.go RoutingTrafficConfig.Encode -- 0.00% +0.00%
pkg/app/piped/cloudprovider/lambda/client.go RoutingTrafficConfig.Decode -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.Execute -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollbackExecutor.ensureRollback -- 0.00% +0.00%
pkg/app/piped/executor/lambda/rollback.go rollback -- 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go promote 0.00% 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go Register 80.00% 71.43% -8.57%
pkg/app/piped/executor/lambda/lambda.go sync 0.00% 0.00% +0.00%
pkg/app/piped/executor/lambda/lambda.go rollout 0.00% 0.00% +0.00%

@nghialv
Copy link
Member

nghialv commented Jan 22, 2021

Thank you.
/lgtm

@nakabonne
Copy link
Member

🚀
/approve

@pipecd-bot
Copy link
Collaborator

APPROVE

This pull request is APPROVED by nakabonne.

Approvers can cancel the approval by writing /approve cancel in a comment. Any additional commits also will change this pull request to be not-approved.

@pipecd-bot pipecd-bot merged commit 66aa366 into master Jan 22, 2021
@pipecd-bot pipecd-bot deleted the lambda-rollback branch January 22, 2021 04:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants