Skip to content

Commit

Permalink
feat: add jobs api (#751)
Browse files Browse the repository at this point in the history
* feat: add jobs api

Signed-off-by: Gaius <[email protected]>
  • Loading branch information
gaius-qi committed Jun 28, 2023
1 parent 65d24cd commit 1dccea0
Show file tree
Hide file tree
Showing 20 changed files with 644 additions and 197 deletions.
20 changes: 11 additions & 9 deletions docs/en/preheat/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,34 @@

Use preheat apis for preheating. First create a POST request for preheating, you can refer to [create preheat api document](../api-reference/api-reference.md#create-preheat)

If the `scheduler_cluster_id` does not exist, it means to preheat all scheduler clusters.
If the `scheduler_cluster_ids` does not exist, it means to preheat all scheduler clusters.

```bash
curl --request POST 'http://dragonfly-manager:8080/api/v1/preheats' \
curl --location --request POST 'http://dragonfly-manager:8080/api/v1/jobs' \
--header 'Content-Type: application/json' \
--data-raw '{
"type": "image",
"url": "https://registry-1.docker.io/v2/library/busybox/manifests/latest",
"scheduler_cluster_id": 1
"type": "preheat",
"args": {
"type": "image",
"url": "https://registry-1.docker.io/v2/library/redis/manifests/latest"
}
}'
```

If the output of command above has content like

```bash
{"id":"group_28439e0b-d4c3-43bf-945e-482b54c49dc5","status":"PENDING","create_at":"2021-10-09T11:54:50.6182794Z"}
{ "id": 1 "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", "type": "preheat", "status": "PENDING", "args": { "filter": "", "headers": null, "type": "image", "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" }}
```
Polling the preheating status with id. if status is `SUCCESS`, preheating is successful, you can refer to [get preheat api document](../api-reference/api-reference.md#get-preheat)
```bash
curl --request GET 'http://dragonfly-manager:8080/api/v1/preheats/group_28439e0b-d4c3-43bf-945e-482b54c49dc5'
curl --request GET 'http://dragonfly-manager:8080/api/v1/jobs/1'
```
If the status is `SUCCESS`, the preheating is successful.
```bash
{"id":"group_28439e0b-d4c3-43bf-945e-482b54c49dc5","status":"SUCCESS","create_at":"2021-10-09T11:54:50.5712334Z"}
```
{ "id": 1 "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", "type": "preheat", "status": "SUCCESS", "args": { "filter": "", "headers": null, "type": "image", "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" }}
```
18 changes: 10 additions & 8 deletions docs/zh-CN/user-guide/preheat/preheat.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,32 +10,34 @@ TODO

用户使用 api 进行预热。首先发送 POST 请求创建预热任务,具体 api 可以参考文档 [create preheat api document](../../api/api.md#create-preheat)

如果 `scheduler_cluster_id` 不存在,表示对所有 scheduler cluster 进行预热。
如果 `scheduler_cluster_ids` 不存在,表示对所有 scheduler cluster 进行预热。

```bash
curl --request POST 'http://dragonfly-manager:8080/api/v1/preheats' \
curl --location --request POST 'http://dragonfly-manager:8080/api/v1/jobs' \
--header 'Content-Type: application/json' \
--data-raw '{
"type": "image",
"url": "https://registry-1.docker.io/v2/library/busybox/manifests/latest",
"scheduler_cluster_id": 1
"type": "preheat",
"args": {
"type": "image",
"url": "https://registry-1.docker.io/v2/library/redis/manifests/latest"
}
}'
```

命令行日志返回预热任务 ID。

```bash
{"id":"group_28439e0b-d4c3-43bf-945e-482b54c49dc5","status":"PENDING","create_at":"2021-10-09T11:54:50.6182794Z"}
{ "id": 1 "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", "type": "preheat", "status": "PENDING", "args": { "filter": "", "headers": null, "type": "image", "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" }}
```
使用预热任务 ID 轮训查询任务是否成功,具体 api 可以参考文档 [get preheat api document](../../api/api.md#get-preheat)。
```bash
curl --request GET 'http://dragonfly-manager:8080/api/v1/preheats/group_28439e0b-d4c3-43bf-945e-482b54c49dc5'
curl --request GET 'http://dragonfly-manager:8080/api/v1/jobs/1'
```
如果返回预热任务状态为 `SUCCESS`,表示预热成功。
```bash
{"id":"group_28439e0b-d4c3-43bf-945e-482b54c49dc5","status":"SUCCESS","create_at":"2021-10-09T11:54:50.5712334Z"}
{ "id": 1 "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", "type": "preheat", "status": "SUCCESS", "args": { "filter": "", "headers": null, "type": "image", "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" }}
```
1 change: 1 addition & 0 deletions internal/dflog/logger.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ func init() {
SetStatPeerLogger(log)
SetStatSeedLogger(log)
SetDownloadLogger(log)
SetJobLogger(sugar)
}
}

Expand Down
1 change: 1 addition & 0 deletions manager/database/database.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ func newMyqsl(cfg *config.MysqlConfig) (*gorm.DB, error) {

func migrate(db *gorm.DB) error {
return db.Set("gorm:table_options", "DEFAULT CHARSET=utf8mb4 ROW_FORMAT=Dynamic").AutoMigrate(
&model.Job{},
&model.CDNCluster{},
&model.CDN{},
&model.SchedulerCluster{},
Expand Down
171 changes: 171 additions & 0 deletions manager/handlers/job.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
package handlers

import (
"net/http"

"d7y.io/dragonfly/v2/internal/job"
"d7y.io/dragonfly/v2/manager/types"
"github.com/gin-gonic/gin"
"github.com/gin-gonic/gin/binding"
)

// @Summary Create Job
// @Description create by json config
// @Tags Job
// @Accept json
// @Produce json
// @Param Job body types.CreateJobRequest true "Job"
// @Success 200 {object} model.Job
// @Failure 400
// @Failure 404
// @Failure 500
// @Router /jobs [post]
func (h *Handlers) CreateJob(ctx *gin.Context) {
var json types.CreateJobRequest
if err := ctx.ShouldBindBodyWith(&json, binding.JSON); err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": err.Error()})
return
}

switch json.Type {
case job.PreheatJob:
var json types.CreatePreheatJobRequest
if err := ctx.ShouldBindBodyWith(&json, binding.JSON); err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": err.Error()})
return
}

job, err := h.service.CreatePreheatJob(ctx.Request.Context(), json)
if err != nil {
ctx.Error(err)
return
}

ctx.JSON(http.StatusOK, job)
default:
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": "Unknow type"})
}
}

// @Summary Destroy Job
// @Description Destroy by id
// @Tags Job
// @Accept json
// @Produce json
// @Param id path string true "id"
// @Success 200
// @Failure 400
// @Failure 404
// @Failure 500
// @Router /jobs/{id} [delete]
func (h *Handlers) DestroyJob(ctx *gin.Context) {
var params types.JobParams
if err := ctx.ShouldBindUri(&params); err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": err.Error()})
return
}

if err := h.service.DestroyJob(ctx.Request.Context(), params.ID); err != nil {
ctx.Error(err)
return
}

ctx.Status(http.StatusOK)
}

// @Summary Update Job
// @Description Update by json config
// @Tags Job
// @Accept json
// @Produce json
// @Param id path string true "id"
// @Param Job body types.UpdateJobRequest true "Job"
// @Success 200 {object} model.Job
// @Failure 400
// @Failure 404
// @Failure 500
// @Router /jobs/{id} [patch]
func (h *Handlers) UpdateJob(ctx *gin.Context) {
var params types.JobParams
if err := ctx.ShouldBindUri(&params); err != nil {
ctx.Error(err)
return
}

var json types.UpdateJobRequest
if err := ctx.ShouldBindJSON(&json); err != nil {
ctx.Error(err)
return
}

job, err := h.service.UpdateJob(ctx.Request.Context(), params.ID, json)
if err != nil {
ctx.Error(err)
return
}

ctx.JSON(http.StatusOK, job)
}

// @Summary Get Job
// @Description Get Job by id
// @Tags Job
// @Accept json
// @Produce json
// @Param id path string true "id"
// @Success 200 {object} model.Job
// @Failure 400
// @Failure 404
// @Failure 500
// @Router /jobs/{id} [get]
func (h *Handlers) GetJob(ctx *gin.Context) {
var params types.JobParams
if err := ctx.ShouldBindUri(&params); err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": err.Error()})
return
}

job, err := h.service.GetJob(ctx.Request.Context(), params.ID)
if err != nil {
ctx.Error(err)
return
}

ctx.JSON(http.StatusOK, job)
}

// @Summary Get Jobs
// @Description Get Jobs
// @Tags Job
// @Accept json
// @Produce json
// @Param page query int true "current page" default(0)
// @Param per_page query int true "return max item count, default 10, max 50" default(10) minimum(2) maximum(50)
// @Success 200 {object} []model.Job
// @Failure 400
// @Failure 404
// @Failure 500
// @Router /jobs [get]
func (h *Handlers) GetJobs(ctx *gin.Context) {
var query types.GetJobsQuery
if err := ctx.ShouldBindQuery(&query); err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": err.Error()})
return
}

h.setPaginationDefault(&query.Page, &query.PerPage)
jobs, err := h.service.GetJobs(ctx.Request.Context(), query)
if err != nil {
ctx.Error(err)
return
}

totalCount, err := h.service.JobTotalCount(ctx.Request.Context(), query)
if err != nil {
ctx.Error(err)
return
}

h.setPaginationLinkHeader(ctx, query.Page, query.PerPage, int(totalCount))
ctx.JSON(http.StatusOK, jobs)
}
56 changes: 1 addition & 55 deletions manager/handlers/preheat.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,60 +23,6 @@ import (
"github.com/gin-gonic/gin"
)

// @Summary Create Preheat
// @Description create by json config
// @Tags Preheat
// @Accept json
// @Produce json
// @Param CDN body types.CreatePreheatRequest true "Preheat"
// @Success 200 {object} types.Preheat
// @Failure 400
// @Failure 404
// @Failure 500
// @Router /preheats [post]
func (h *Handlers) CreatePreheat(ctx *gin.Context) {
var json types.CreatePreheatRequest
if err := ctx.ShouldBindJSON(&json); err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": err.Error()})
return
}

preheat, err := h.service.CreatePreheat(ctx.Request.Context(), json)
if err != nil {
ctx.Error(err)
return
}

ctx.JSON(http.StatusOK, preheat)
}

// @Summary Get Preheat
// @Description Get Preheat by id
// @Tags Preheat
// @Accept json
// @Produce json
// @Param id path string true "id"
// @Success 200 {object} types.Preheat
// @Failure 400
// @Failure 404
// @Failure 500
// @Router /preheats/{id} [get]
func (h *Handlers) GetPreheat(ctx *gin.Context) {
var params types.PreheatParams
if err := ctx.ShouldBindUri(&params); err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": err.Error()})
return
}

preheat, err := h.service.GetPreheat(ctx.Request.Context(), params.ID)
if err != nil {
ctx.Error(err)
return
}

ctx.JSON(http.StatusOK, preheat)
}

// @Summary Create V1 Preheat
// @Description create by json config
// @Tags Preheat
Expand Down Expand Up @@ -116,7 +62,7 @@ func (h *Handlers) CreateV1Preheat(ctx *gin.Context) {
// @Failure 500
// @Router /preheats/{id} [get]
func (h *Handlers) GetV1Preheat(ctx *gin.Context) {
var params types.PreheatParams
var params types.V1PreheatParams
if err := ctx.ShouldBindUri(&params); err != nil {
ctx.JSON(http.StatusUnprocessableEntity, gin.H{"errors": err.Error()})
return
Expand Down
15 changes: 15 additions & 0 deletions manager/job/job.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
)

type Job struct {
*internaljob.Job
Preheat
}

Expand All @@ -43,6 +44,20 @@ func New(cfg *config.Config) (*Job, error) {
}

return &Job{
Job: j,
Preheat: p,
}, nil
}

func (j *Job) GetGroupJobState(id string) (*internaljob.GroupJobState, error) {
groupJobState, err := j.Job.GetGroupJobState(id)
if err != nil {
return nil, err
}

return &internaljob.GroupJobState{
GroupUUID: groupJobState.GroupUUID,
State: groupJobState.State,
CreatedAt: groupJobState.CreatedAt,
}, nil
}
Loading

0 comments on commit 1dccea0

Please sign in to comment.