Skip to content

Improve insight collector #1340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pipecd-bot merged 19 commits into from
Jan 6, 2021
Merged

Improve insight collector #1340

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
902e5f1
Improve insight collector
sanposhiho Dec 27, 2020
2d01d13
Add error hadling and fix some small issues from linter
sanposhiho Dec 27, 2020
16ac4bf
Add option to specify retryTime and retryInterval
sanposhiho Dec 27, 2020
709e19a
Remove unuse function
sanposhiho Dec 27, 2020
8abc72a
Fix handling on chunks already updated
sanposhiho Dec 27, 2020
7404079
Fix process project chunk
sanposhiho Dec 27, 2020
261b654
Fix import order
sanposhiho Dec 27, 2020
0e99357
Fix error handling
sanposhiho Dec 27, 2020
26322b5
Add descriptions for functions and rename some functions
sanposhiho Dec 27, 2020
0ba2717
Fix descriptions on functions
sanposhiho Dec 27, 2020
8bca2b1
Rename some variables and functions
sanposhiho Jan 4, 2021
06c1d3f
PutMilestone only updateErr is not nil
sanposhiho Jan 4, 2021
83958c0
Rename functions to export
sanposhiho Jan 4, 2021
b35358d
Use backoff package to retry
sanposhiho Jan 4, 2021
2b494d0
Set default value on config for insight collector
sanposhiho Jan 6, 2021
f8a5fa8
Fix logging messages
sanposhiho Jan 6, 2021
0c76835
Add successProcess flag
sanposhiho Jan 6, 2021
3fedbfd
Rename some functions and variables
sanposhiho Jan 6, 2021
0c89dc2
Merge branch 'master' into improve-insight-collector
sanposhiho Jan 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 26 additions & 12 deletions cmd/pipecd/ops.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,18 +102,32 @@ func (s *ops) run(ctx context.Context, t cli.Telemetry) error {
collector := insightcollector.NewInsightCollector(ds, fs, t.Logger)
c := cron.New(cron.WithLocation(time.UTC))
_, err := c.AddFunc(cfg.InsightCollector.Schedule, func() {
start := time.Now()
if err := collector.CollectProjectsInsight(ctx); err != nil {
t.Logger.Error("failed to run the project insight collector", zap.Error(err))
} else {
t.Logger.Info("project insight collector successfully finished", zap.Duration("duration", time.Since(start)))
}

start = time.Now()
if err := collector.CollectApplicationInsight(ctx); err != nil {
t.Logger.Error("failed to run the application insight collector", zap.Error(err))
} else {
t.Logger.Info("application insight collector successfully finished", zap.Duration("duration", time.Since(start)))
insightCollectorRetryTime := 3
retryAggregateWithCompletedAt := true
retryAggregateWithCreatedAt := true
for i := 0; i < insightCollectorRetryTime; i++ {
start := time.Now()
var err error
if retryAggregateWithCompletedAt {
err = collector.AggregateWithCompletedAt(ctx)
}
if err != nil {
t.Logger.Error("failed to aggregate with completedAt", zap.Error(err))
} else {
t.Logger.Info("aggregate with completedAt successfully finished", zap.Duration("duration", time.Since(start)))
retryAggregateWithCompletedAt = false
}

start = time.Now()
if retryAggregateWithCreatedAt {
err = collector.AggregateWithCreatedAt(ctx)
}
if err != nil {
t.Logger.Error("failed to aggregate with createdAt", zap.Error(err))
} else {
t.Logger.Info("aggregate with createdAt successfully finished", zap.Duration("duration", time.Since(start)))
retryAggregateWithCreatedAt = false
}
}
})
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ require (
github.com/prometheus/client_golang v1.6.0
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.9.1
github.com/robfig/cron/v3 v3.0.1 // indirect
github.com/robfig/cron/v3 v3.0.1
github.com/spf13/cobra v0.0.5
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.5.1
Expand Down
1 change: 0 additions & 1 deletion go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,6 @@ github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsT
github.com/prometheus/procfs v0.0.3/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDaekg4FpcdQ=
github.com/prometheus/procfs v0.0.11 h1:DhHlBtkHWPYi8O2y31JkK0TF+DGM+51OopZjH/Ia5qI=
github.com/prometheus/procfs v0.0.11/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
github.com/robfig/cron v1.2.0 h1:ZjScXvvxeQ63Dbyxy76Fj3AT3Ut0aKsyd2/tl3DTMuQ=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
Expand Down
Loading