Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(actions): support cron schedule task #26655

Merged
merged 104 commits into from
Aug 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
8650571
chore(actions): support cron schedule task.
appleboy Feb 4, 2023
f1eb0ad
chore(license): add cron v3 license
appleboy Feb 5, 2023
fffb455
refactor(actions): CreateScheduleTask
appleboy Feb 16, 2023
054873b
chore(deps): upgrade go module
appleboy Mar 10, 2023
5424793
feat: refactor schedule tasks management in actions service
appleboy Mar 14, 2023
dc1a6c1
feat: add support for schedule specs and tasks
appleboy Mar 14, 2023
4fa2416
refactor: refactor migration and update version number
appleboy Mar 14, 2023
9129de1
Merge branch 'main' into schedule
appleboy Mar 14, 2023
650434c
refactor: refactor database schema for improved organization
appleboy Mar 14, 2023
57b87fc
Merge branch 'main' into schedule
appleboy Mar 14, 2023
5a789aa
fix: refactor notifier_helper.go for improved readability and maintai…
appleboy Mar 14, 2023
756c534
refactor: refactor `ActionSchedule` struct in schedule.go and migrati…
appleboy Mar 14, 2023
257a37f
refactor: simplify error handling and rename 'jobs' to 'workflows'
appleboy Mar 14, 2023
a33777d
chore: refactor license for cron library
appleboy Mar 14, 2023
2f01a51
Update go-licenses.json
appleboy Mar 14, 2023
a0dbb87
refactor: refactor ScheduleTask functions to use passed context
appleboy Mar 14, 2023
bf44849
refactor: remove unnecessary approval function from notifier helper file
appleboy Mar 14, 2023
9ca276b
chore: add license to go-licenses.json file
appleboy Mar 14, 2023
4eae2df
style: ensure consistent line endings across all files
appleboy Mar 14, 2023
0cb21dc
refactor: improve scheduling efficiency by optimizing import statements
appleboy Mar 14, 2023
b63ba87
Merge branch 'main' into schedule
appleboy Mar 15, 2023
b14c06e
feat: refactor approval handling for workflows
appleboy Mar 17, 2023
5685ea0
refactor: refactor CreateScheduleTask function
appleboy Mar 17, 2023
737d7fe
refactor: refactor ActionScheduleSpec model struct
appleboy Mar 17, 2023
f069367
feat: refactor schedule task creation process
appleboy Mar 17, 2023
16513bb
refactor: refactor variable assignment in startTasks function
appleboy Mar 17, 2023
6c14852
refactor: refactor workflow handling function signature
appleboy Mar 17, 2023
099d9a9
Update services/cron/tasks_actions.go
appleboy Mar 17, 2023
35081d6
perf: refactor scheduling logic for more accurate timing
appleboy Mar 17, 2023
1655a8c
perf: improve performance and add new function for task starting
appleboy Mar 17, 2023
bf8cda4
refactor: simplify and optimize schedule handling function
appleboy Mar 18, 2023
0b714b6
refactor: refactor schedule task retrieval logic
appleboy Mar 18, 2023
039ead5
chore: refactor migration file naming convention
appleboy Mar 18, 2023
f12cb3e
Merge branch 'main' into schedule
appleboy Mar 18, 2023
7671a31
fix: refactor CreateScheduleTask function to handle empty rows gracef…
appleboy Mar 18, 2023
fab7c2c
feat: refactor schedule task creation and retrieval process
appleboy Mar 18, 2023
b815dac
chore: replace space with tab
appleboy Mar 18, 2023
978739d
refactor: refactor schedule task registration function
appleboy Mar 18, 2023
9efbda8
Merge branch 'main' into schedule
appleboy Mar 19, 2023
beada43
Merge branch 'main' into schedule
appleboy Mar 19, 2023
0815696
refactor: refactor file naming convention
appleboy Mar 20, 2023
5eb8947
Merge branch 'main' into schedule
appleboy Mar 20, 2023
a693fe6
chore: convert space to tab
appleboy Mar 20, 2023
5d0e0e6
chore: refactor migration file names in v1_20 directory
appleboy Mar 25, 2023
fba8dc4
Merge branch 'main' into schedule
appleboy Mar 25, 2023
ce80ce8
N/A (no changes in git diff): refactor code for improved performance …
appleboy Mar 26, 2023
d15448b
Merge branch 'main' into schedule
appleboy Mar 31, 2023
958e67b
chore: update
appleboy Mar 31, 2023
ad91c4b
chore: update
appleboy Mar 31, 2023
95320e8
Merge branch 'main' into schedule
appleboy Apr 1, 2023
513a4bf
chore: rename
appleboy Apr 11, 2023
e5257ed
chore: rename
appleboy Apr 11, 2023
ee2338b
Merge branch 'main' into schedule
appleboy Apr 11, 2023
83a5fb5
refactor: simplify schedule event detection
appleboy Apr 11, 2023
4db836d
chore: update
appleboy Apr 11, 2023
606027f
Update services/actions/schedule_tasks.go
appleboy Apr 13, 2023
07f1d6a
Merge branch 'main' into schedule
appleboy Apr 13, 2023
f972c37
bump migration name
techknowlogick Apr 17, 2023
0e5b5cf
Merge remote-tracking branch 'upstream/main' into schedule
techknowlogick Apr 17, 2023
6c7170c
fix syntax issue
techknowlogick Apr 17, 2023
a2fade1
chore: update
appleboy Apr 29, 2023
c356411
chore: update
appleboy Apr 29, 2023
463a65f
Merge branch 'main' into schedule
appleboy Apr 29, 2023
d958943
refactor: optimize scheduling and improve API usage
appleboy Apr 29, 2023
7c8b678
chore: add dashboard.start_schedule_tasks language
appleboy Apr 29, 2023
c7f89f9
chore: update
appleboy Apr 29, 2023
67e2da6
feat: improve scheduling and task handling in Gogs API
appleboy Apr 30, 2023
06fc431
feat: improve API efficiency and test coverage
appleboy Apr 30, 2023
f2c0021
fix: improve error handling and API efficiency in tests
appleboy May 6, 2023
5c9f14f
refactor: improve action scheduling and API usage
appleboy May 6, 2023
6c93d4b
refactor: improve API efficiency and test robustness
appleboy May 10, 2023
9d8f1a8
Merge branch 'main' into schedule
appleboy May 10, 2023
4c243ed
chore: update
appleboy May 10, 2023
4150af6
Merge branch 'main' into schedule
appleboy May 12, 2023
542f681
Merge branch 'main' into schedule
appleboy May 13, 2023
b07802e
Merge branch 'main' into schedule
appleboy May 14, 2023
d530efd
Merge branch 'main' into schedule
GiteaBot May 14, 2023
7ad692f
rename
appleboy Jul 25, 2023
58174b5
Merge branch 'main' into schedule
appleboy Jul 25, 2023
ab22d4d
Merge branch 'main' into schedule
appleboy Jul 25, 2023
36c58fe
rename
appleboy Jul 25, 2023
729bc69
rename
appleboy Aug 13, 2023
c1be2b5
rename
appleboy Aug 13, 2023
e0b65e2
update
appleboy Aug 13, 2023
40ed679
feat: refactor schedule handling in notifier helper
appleboy Aug 13, 2023
a12d22d
fix: improve error handling in `startTasks` function
appleboy Aug 13, 2023
c4420db
Merge branch 'main' into appleboy-schedule
lunny Aug 22, 2023
7890d91
remove gogs/cron
lunny Aug 22, 2023
c40bc37
Fix bug
lunny Aug 22, 2023
dd27c7f
Fix bug
lunny Aug 22, 2023
82694d7
Fix bug
lunny Aug 22, 2023
14d5c46
Update services/actions/schedule_tasks.go
lunny Aug 23, 2023
40ce6f7
Merge branch 'main' into appleboy-schedule
lunny Aug 23, 2023
26ce2e9
Fix bug
lunny Aug 23, 2023
8c24887
Merge branch 'main' into appleboy-schedule
lunny Aug 23, 2023
99210fc
Merge branch 'appleboy-schedule' of github.com:lunny/gitea into apple…
lunny Aug 23, 2023
83ec4c6
make code review easier
lunny Aug 23, 2023
8152ab7
Fix bug
lunny Aug 23, 2023
3e95eb6
Merge branch 'main' into appleboy-schedule
lunny Aug 23, 2023
da6ae91
Merge branch 'main' into appleboy-schedule
GiteaBot Aug 23, 2023
ed4bf1c
Merge branch 'main' into appleboy-schedule
GiteaBot Aug 23, 2023
860e2a7
Merge branch 'main' into appleboy-schedule
GiteaBot Aug 24, 2023
7e3e206
Merge branch 'main' into appleboy-schedule
GiteaBot Aug 24, 2023
23730be
Merge branch 'main' into appleboy-schedule
GiteaBot Aug 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ require (
github.com/prometheus/client_golang v1.16.0
github.com/quasoft/websspi v1.1.2
github.com/redis/go-redis/v9 v9.0.5
github.com/robfig/cron/v3 v3.0.1
github.com/santhosh-tekuri/jsonschema/v5 v5.3.1
github.com/sassoftware/go-rpmutils v0.2.0
github.com/sergi/go-diff v1.3.1
Expand Down Expand Up @@ -254,7 +255,6 @@ require (
github.com/rhysd/actionlint v1.6.25 // indirect
github.com/rivo/uniseg v0.4.4 // indirect
github.com/robfig/cron v1.2.0 // indirect
github.com/robfig/cron/v3 v3.0.1 // indirect
github.com/rogpeppe/go-internal v1.11.0 // indirect
github.com/rs/xid v1.5.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
Expand Down
120 changes: 120 additions & 0 deletions models/actions/schedule.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
// Copyright 2023 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT

package actions

import (
"context"
"time"

"code.gitea.io/gitea/models/db"
repo_model "code.gitea.io/gitea/models/repo"
user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/modules/timeutil"
webhook_module "code.gitea.io/gitea/modules/webhook"

"github.com/robfig/cron/v3"
)

// ActionSchedule represents a schedule of a workflow file
type ActionSchedule struct {
ID int64
Title string
Specs []string
RepoID int64 `xorm:"index"`
Repo *repo_model.Repository `xorm:"-"`
OwnerID int64 `xorm:"index"`
WorkflowID string
TriggerUserID int64
TriggerUser *user_model.User `xorm:"-"`
Ref string
CommitSHA string
Event webhook_module.HookEventType
EventPayload string `xorm:"LONGTEXT"`
Content []byte
Created timeutil.TimeStamp `xorm:"created"`
Updated timeutil.TimeStamp `xorm:"updated"`
}

func init() {
db.RegisterModel(new(ActionSchedule))
}

// GetSchedulesMapByIDs returns the schedules by given id slice.
func GetSchedulesMapByIDs(ids []int64) (map[int64]*ActionSchedule, error) {
schedules := make(map[int64]*ActionSchedule, len(ids))
return schedules, db.GetEngine(db.DefaultContext).In("id", ids).Find(&schedules)
}

// GetReposMapByIDs returns the repos by given id slice.
func GetReposMapByIDs(ids []int64) (map[int64]*repo_model.Repository, error) {
repos := make(map[int64]*repo_model.Repository, len(ids))
return repos, db.GetEngine(db.DefaultContext).In("id", ids).Find(&repos)
}

var cronParser = cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor)

// CreateScheduleTask creates new schedule task.
func CreateScheduleTask(ctx context.Context, rows []*ActionSchedule) error {
// Return early if there are no rows to insert
if len(rows) == 0 {
return nil
}

// Begin transaction
ctx, committer, err := db.TxContext(ctx)
if err != nil {
return err
}
defer committer.Close()

// Loop through each schedule row
for _, row := range rows {
// Create new schedule row
if err = db.Insert(ctx, row); err != nil {
return err
}

// Loop through each schedule spec and create a new spec row
now := time.Now()

for _, spec := range row.Specs {
// Parse the spec and check for errors
schedule, err := cronParser.Parse(spec)
if err != nil {
continue // skip to the next spec if there's an error
}

// Insert the new schedule spec row
if err = db.Insert(ctx, &ActionScheduleSpec{
RepoID: row.RepoID,
ScheduleID: row.ID,
Spec: spec,
Next: timeutil.TimeStamp(schedule.Next(now).Unix()),
}); err != nil {
return err
}
}
}

// Commit transaction
return committer.Commit()
}

func DeleteScheduleTaskByRepo(ctx context.Context, id int64) error {
ctx, committer, err := db.TxContext(ctx)
if err != nil {
return err
}
defer committer.Close()

if _, err := db.GetEngine(ctx).Delete(&ActionSchedule{RepoID: id}); err != nil {
return err
}

if _, err := db.GetEngine(ctx).Delete(&ActionScheduleSpec{RepoID: id}); err != nil {
return err
}

return committer.Commit()
}
94 changes: 94 additions & 0 deletions models/actions/schedule_list.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
// Copyright 2023 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT

package actions

import (
"context"

"code.gitea.io/gitea/models/db"
repo_model "code.gitea.io/gitea/models/repo"
user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/modules/container"

"xorm.io/builder"
)

type ScheduleList []*ActionSchedule

// GetUserIDs returns a slice of user's id
func (schedules ScheduleList) GetUserIDs() []int64 {
ids := make(container.Set[int64], len(schedules))
for _, schedule := range schedules {
ids.Add(schedule.TriggerUserID)
}
return ids.Values()
}

func (schedules ScheduleList) GetRepoIDs() []int64 {
ids := make(container.Set[int64], len(schedules))
for _, schedule := range schedules {
ids.Add(schedule.RepoID)
}
return ids.Values()
}

func (schedules ScheduleList) LoadTriggerUser(ctx context.Context) error {
userIDs := schedules.GetUserIDs()
users := make(map[int64]*user_model.User, len(userIDs))
if err := db.GetEngine(ctx).In("id", userIDs).Find(&users); err != nil {
return err
}
for _, schedule := range schedules {
if schedule.TriggerUserID == user_model.ActionsUserID {
schedule.TriggerUser = user_model.NewActionsUser()
} else {
schedule.TriggerUser = users[schedule.TriggerUserID]
}
}
return nil
}

func (schedules ScheduleList) LoadRepos() error {
repoIDs := schedules.GetRepoIDs()
repos, err := repo_model.GetRepositoriesMapByIDs(repoIDs)
if err != nil {
return err
}
for _, schedule := range schedules {
schedule.Repo = repos[schedule.RepoID]
}
return nil
}

type FindScheduleOptions struct {
db.ListOptions
RepoID int64
OwnerID int64
}

func (opts FindScheduleOptions) toConds() builder.Cond {
cond := builder.NewCond()
if opts.RepoID > 0 {
cond = cond.And(builder.Eq{"repo_id": opts.RepoID})
}
if opts.OwnerID > 0 {
cond = cond.And(builder.Eq{"owner_id": opts.OwnerID})
}

return cond
}

func FindSchedules(ctx context.Context, opts FindScheduleOptions) (ScheduleList, int64, error) {
e := db.GetEngine(ctx).Where(opts.toConds())
if !opts.ListAll && opts.PageSize > 0 && opts.Page >= 1 {
e.Limit(opts.PageSize, (opts.Page-1)*opts.PageSize)
}
var schedules ScheduleList
total, err := e.Desc("id").FindAndCount(&schedules)
return schedules, total, err
}

func CountSchedules(ctx context.Context, opts FindScheduleOptions) (int64, error) {
return db.GetEngine(ctx).Where(opts.toConds()).Count(new(ActionSchedule))
}
50 changes: 50 additions & 0 deletions models/actions/schedule_spec.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
// Copyright 2023 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT

package actions

import (
"context"

"code.gitea.io/gitea/models/db"
repo_model "code.gitea.io/gitea/models/repo"
"code.gitea.io/gitea/modules/timeutil"

"github.com/robfig/cron/v3"
)

// ActionScheduleSpec represents a schedule spec of a workflow file
type ActionScheduleSpec struct {
ID int64
RepoID int64 `xorm:"index"`
Repo *repo_model.Repository `xorm:"-"`
ScheduleID int64 `xorm:"index"`
Schedule *ActionSchedule `xorm:"-"`

// Next time the job will run, or the zero time if Cron has not been
// started or this entry's schedule is unsatisfiable
Next timeutil.TimeStamp `xorm:"index"`
// Prev is the last time this job was run, or the zero time if never.
Prev timeutil.TimeStamp
Spec string

Created timeutil.TimeStamp `xorm:"created"`
Updated timeutil.TimeStamp `xorm:"updated"`
}

func (s *ActionScheduleSpec) Parse() (cron.Schedule, error) {
return cronParser.Parse(s.Spec)
}

func init() {
db.RegisterModel(new(ActionScheduleSpec))
}

func UpdateScheduleSpec(ctx context.Context, spec *ActionScheduleSpec, cols ...string) error {
sess := db.GetEngine(ctx).ID(spec.ID)
if len(cols) > 0 {
sess.Cols(cols...)
}
_, err := sess.Update(spec)
return err
}
106 changes: 106 additions & 0 deletions models/actions/schedule_spec_list.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
// Copyright 2023 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT

package actions

import (
"context"

"code.gitea.io/gitea/models/db"
repo_model "code.gitea.io/gitea/models/repo"
"code.gitea.io/gitea/modules/container"

"xorm.io/builder"
)

type SpecList []*ActionScheduleSpec

func (specs SpecList) GetScheduleIDs() []int64 {
ids := make(container.Set[int64], len(specs))
for _, spec := range specs {
ids.Add(spec.ScheduleID)
}
return ids.Values()
}

func (specs SpecList) LoadSchedules() error {
scheduleIDs := specs.GetScheduleIDs()
schedules, err := GetSchedulesMapByIDs(scheduleIDs)
if err != nil {
return err
}
for _, spec := range specs {
spec.Schedule = schedules[spec.ScheduleID]
}

repoIDs := specs.GetRepoIDs()
repos, err := GetReposMapByIDs(repoIDs)
if err != nil {
return err
}
for _, spec := range specs {
spec.Repo = repos[spec.RepoID]
}

return nil
}

func (specs SpecList) GetRepoIDs() []int64 {
ids := make(container.Set[int64], len(specs))
for _, spec := range specs {
ids.Add(spec.RepoID)
}
return ids.Values()
}

func (specs SpecList) LoadRepos() error {
repoIDs := specs.GetRepoIDs()
repos, err := repo_model.GetRepositoriesMapByIDs(repoIDs)
if err != nil {
return err
}
for _, spec := range specs {
spec.Repo = repos[spec.RepoID]
}
return nil
}

type FindSpecOptions struct {
db.ListOptions
RepoID int64
Next int64
}

func (opts FindSpecOptions) toConds() builder.Cond {
cond := builder.NewCond()
if opts.RepoID > 0 {
cond = cond.And(builder.Eq{"repo_id": opts.RepoID})
}

if opts.Next > 0 {
cond = cond.And(builder.Lte{"next": opts.Next})
}

return cond
}

func FindSpecs(ctx context.Context, opts FindSpecOptions) (SpecList, int64, error) {
e := db.GetEngine(ctx).Where(opts.toConds())
if opts.PageSize > 0 && opts.Page >= 1 {
e.Limit(opts.PageSize, (opts.Page-1)*opts.PageSize)
}
var specs SpecList
total, err := e.Desc("id").FindAndCount(&specs)
if err != nil {
return nil, 0, err
}

if err := specs.LoadSchedules(); err != nil {
return nil, 0, err
}
return specs, total, nil
}

func CountSpecs(ctx context.Context, opts FindSpecOptions) (int64, error) {
return db.GetEngine(ctx).Where(opts.toConds()).Count(new(ActionScheduleSpec))
}
2 changes: 2 additions & 0 deletions models/migrations/migrations.go
Original file line number Diff line number Diff line change
Expand Up @@ -526,6 +526,8 @@ var migrations = []Migration{
NewMigration("Allow archiving labels", v1_21.AddArchivedUnixColumInLabelTable),
// v272 -> v273
NewMigration("Add Version to ActionRun table", v1_21.AddVersionToActionRunTable),
// v273 -> v274
NewMigration("Add Action Schedule Table", v1_21.AddActionScheduleTable),
}

// GetCurrentDBVersion returns the current db version
Expand Down
Loading