Skip to content

Create insight collector #1287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pipecd-bot merged 42 commits into from
Dec 22, 2020
Merged

Create insight collector #1287

Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
0fc0b7d
Create insight collector
sanposhiho Dec 18, 2020
6a44216
Format unfmt-ed files
Dec 18, 2020
85e9a15
Fix by linter
sanposhiho Dec 18, 2020
dddcf8f
Merge branch 'create-insight-collector' of github.com:pipe-cd/pipe in…
sanposhiho Dec 18, 2020
26993d4
List Aplication with sort
sanposhiho Dec 18, 2020
eb9eeab
Run with cron
sanposhiho Dec 18, 2020
7014aba
Fix to get deployments from store once
sanposhiho Dec 18, 2020
f5fcc50
Fix to use split processes years and chunk
sanposhiho Dec 18, 2020
065356a
Fix to use timezone utc
sanposhiho Dec 18, 2020
b4f5d81
Add deps to BUILD.bazel
sanposhiho Dec 18, 2020
fd7d585
Fix import order and rm unuse option
sanposhiho Dec 21, 2020
185d231
Add control plane test
sanposhiho Dec 21, 2020
ca85ff5
Add error handling on insight collector
sanposhiho Dec 21, 2020
8d199dc
Format some points
sanposhiho Dec 21, 2020
21e55a4
Add function into interface
sanposhiho Dec 21, 2020
af2261e
Remove AppendDataPoint
sanposhiho Dec 21, 2020
6c3d89e
Use sorted list instead of map as updatedps
sanposhiho Dec 21, 2020
3e8419b
Add TODO comment
sanposhiho Dec 21, 2020
f32a508
Summarize page size in const valiable
sanposhiho Dec 21, 2020
b5dbab9
Fix todo comment
sanposhiho Dec 21, 2020
ff11d73
Add flag for disable insightcollector
sanposhiho Dec 21, 2020
9d3b3c1
Remove unuse func
sanposhiho Dec 21, 2020
77ff912
Fix error messages
sanposhiho Dec 21, 2020
5b352d1
Fix comment
sanposhiho Dec 21, 2020
08e8b14
Fix to use created at as fileter'
sanposhiho Dec 21, 2020
31c3187
Add option enable-insight-collector
sanposhiho Dec 21, 2020
5c288cd
Create function and split process
sanposhiho Dec 22, 2020
5065a06
Fix to process file not found pattern
sanposhiho Dec 22, 2020
c14aaec
Fix miss assignment
sanposhiho Dec 22, 2020
79f2b91
Use updatedAt instead of createdAt
sanposhiho Dec 22, 2020
be05090
Fix error handling
sanposhiho Dec 22, 2020
38c4be2
Fix testcase
sanposhiho Dec 22, 2020
dc6e7f1
Merge branch 'master' into create-insight-collector
sanposhiho Dec 22, 2020
c951140
Format unfmt-ed files
Dec 22, 2020
1945ac5
Add project level insight collector
sanposhiho Dec 22, 2020
a9e258c
Fix some variable name and comments
sanposhiho Dec 22, 2020
4641521
Merge branch 'create-insight-collector' of github.com:pipe-cd/pipe in…
sanposhiho Dec 22, 2020
d85c48b
Use ComplatedAt instead of UpdatedAt
sanposhiho Dec 22, 2020
ca698ba
Rename field
sanposhiho Dec 22, 2020
cdc6f9f
Add log message with duration
sanposhiho Dec 22, 2020
4c051d2
Fix to continue working with other chunk if fail
sanposhiho Dec 22, 2020
afde1f5
Ignore deleted application
sanposhiho Dec 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions cmd/pipecd/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ go_library(
"//pkg/app/api/service/webservice:go_default_library",
"//pkg/app/api/stagelogstore:go_default_library",
"//pkg/app/ops/handler:go_default_library",
"//pkg/app/ops/insightcollector:go_default_library",
"//pkg/cache/rediscache:go_default_library",
"//pkg/cli:go_default_library",
"//pkg/config:go_default_library",
Expand All @@ -37,6 +38,7 @@ go_library(
"//pkg/rpc:go_default_library",
"//pkg/version:go_default_library",
"@com_github_dgrijalva_jwt_go//:go_default_library",
"@com_github_robfig_cron_v3//:go_default_library",
"@com_github_nytimes_gziphandler//:go_default_library",
"@com_github_spf13_cobra//:go_default_library",
"@org_golang_x_sync//errgroup:go_default_library",
Expand Down
38 changes: 33 additions & 5 deletions cmd/pipecd/ops.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,22 +19,25 @@ import (
"net/http"
"time"

"github.com/robfig/cron/v3"
"github.com/spf13/cobra"
"go.uber.org/zap"
"golang.org/x/sync/errgroup"

"github.com/pipe-cd/pipe/pkg/admin"
"github.com/pipe-cd/pipe/pkg/app/ops/handler"
"github.com/pipe-cd/pipe/pkg/app/ops/insightcollector"
"github.com/pipe-cd/pipe/pkg/cli"
"github.com/pipe-cd/pipe/pkg/datastore"
"github.com/pipe-cd/pipe/pkg/version"
)

type ops struct {
httpPort int
adminPort int
gracePeriod time.Duration
configFile string
httpPort int
adminPort int
gracePeriod time.Duration
enableInsightCollector bool
configFile string
}

func NewOpsCommand() *cobra.Command {
Expand All @@ -51,7 +54,7 @@ func NewOpsCommand() *cobra.Command {
cmd.Flags().IntVar(&s.httpPort, "http-port", s.httpPort, "The port number used to run http server.")
cmd.Flags().IntVar(&s.adminPort, "admin-port", s.adminPort, "The port number used to run a HTTP server for admin tasks such as metrics, healthz.")
cmd.Flags().DurationVar(&s.gracePeriod, "grace-period", s.gracePeriod, "How long to wait for graceful shutdown.")

cmd.Flags().BoolVar(&s.enableInsightCollector, "enable-insight-collector", s.enableInsightCollector, "enable insight collector.")

@nghialv nghialv Dec 22, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "Enable"

cmd.Flags().StringVar(&s.configFile, "config-file", s.configFile, "The path to the configuration file.")
return cmd
}
Expand Down Expand Up @@ -82,6 +85,31 @@ func (s *ops) run(ctx context.Context, t cli.Telemetry) error {
}
}()

fs, err := createFilestore(ctx, cfg, t.Logger)
if err != nil {
t.Logger.Error("failed to create filestore", zap.Error(err))
return err
}
defer func() {
if err := fs.Close(); err != nil {
t.Logger.Error("failed to close filestore client", zap.Error(err))
}
}()

// Set insight collector

@nghialv nghialv Dec 22, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "Starting a cron job for insight collector."

if s.enableInsightCollector {
collector := insightcollector.NewInsightCollector(ds, fs, t.Logger)
c := cron.New(cron.WithLocation(time.UTC))
_, err := c.AddFunc(cfg.InsightCollector.Schedule, func() {
if err := collector.Run(ctx); err != nil {
t.Logger.Error("failed to run the insight collector", zap.Error(err))
}

@nghialv nghialv Dec 22, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, Could you add a log to show how long does each function take to run?

@sanposhiho sanposhiho Dec 22, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, added:+1:

})
if err != nil {
t.Logger.Error("failed to configure the insight collector", zap.Error(err))
}
}

// Start running HTTP server.
{
handler := handler.NewHandler(s.httpPort, datastore.NewProjectStore(ds), cfg.SharedSSOConfigs, s.gracePeriod, t.Logger)
Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ require (
github.com/prometheus/client_golang v1.6.0
github.com/prometheus/client_model v0.2.0
github.com/prometheus/common v0.9.1
github.com/robfig/cron/v3 v3.0.1 // indirect
github.com/spf13/cobra v0.0.5
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.5.1
Expand Down
3 changes: 3 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,9 @@ github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsT
github.com/prometheus/procfs v0.0.3/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDaekg4FpcdQ=
github.com/prometheus/procfs v0.0.11 h1:DhHlBtkHWPYi8O2y31JkK0TF+DGM+51OopZjH/Ia5qI=
github.com/prometheus/procfs v0.0.11/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4OA4YeYWdaU=
github.com/robfig/cron v1.2.0 h1:ZjScXvvxeQ63Dbyxy76Fj3AT3Ut0aKsyd2/tl3DTMuQ=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
Expand Down
Loading