Manager console provides clearing the cache for P2P with task id

[gaius-qi]

Feature request:

Use case:

UI Example:

[gaius-qi]

The scheduler should notify the peer to clear the cache and prevent the peer from hitting the local cache.

[IRONICBo]

Proposal: Support dirty cache deletion manager api

BACKGROUND

Currently Manager download will dispatch the download task to the peer node through scheduler, but it may not be expected on the peer download action, i.e., download failure, version error, etc., which leads to the download action on the peer node may have unexpected dirty cache.
Currently in dfcache, it also provides access to a certain dfdaemon, and then operates to delete the task on the dfdaemon and reclaim the corresponding meta data and local storage.
We need to provide a global api on the manager side to support asynchronous deletion of the cache corresponding to the task id of certain peers on the cluster, and to support getting the progress and tasks of the task.

DESIGN

The Manager provides a REST API to clear the Cache data for a P2P Task ID from all Peer nodes based on that Task ID. This requires the Manager to issue a move task to the Scheduler, and the Scheduler to call the GRPC interface of each Peer node containing that Task ID to clear the corresponding Cache.
The cache clean supports clearing the cache that has been downloaded by peers on a d7y cluster. Users can call the manager api to operate the cluster cache clean task, and can view the progress of the task.
The complete request timing diagram is as follows
ListTasksJob:

DeleteTasksJob:
Given task id, initiates a job that deletes the task cache on the peer.

Manager REST API definition

When manager starts, it registers a REST API service on 8080 by default, which internally contains a number of API methods exposed on manager at manager/router/router.go. Add a POST method to start an asynchronous task clearing the cache of the task id on peer, and a query to query the status of this asynchronous task and aggregate it into the creation logic of an existing asynchronous task.
The manager eventually collects the information needed to start the job and creates a GroupJob into the machine to send a message to the distributed task waiting to be consumed by the scheduler.

Parameter Definitions

There are existing job definitions that can reuse the job request to support new jobs.

// TODO Add auth to the following routes and fix the tests.
// Job.
job := apiv1.Group("/jobs")
job.POST("", h.CreateJob)
job.DELETE(":id", h.DestroyJob)

Support a new type job.

// internal/job/constants.go
// ListTasksJob is the name of listing tasks job.
ListTasksJob = "list_tasks"

// DeleteTasksJob is the name of deleting tasks job.
DeleteTaskJob = "delete_task"

Define a new JobRequest, response reuses the Job response, which

// For list task infos
type CreateListTasksJobRequest struct {
	BIO                 string           `json:"bio" binding:"omitempty"`
	Type                string           `json:"type" binding:"required"`
	Args                ListTasksJobArgs `json:"args" binding:"omitempty"`
	Result              map[string]any   `json:"result" binding:"omitempty"`
	UserID              uint             `json:"user_id" binding:"omitempty"`
	SchedulerClusterIDs []uint           `json:"scheduler_cluster_ids" binding:"omitempty"`
}

type ListTasksJobArgs struct {
	// TaskID is the task id for peer cache clean.
	TaskID string `json:"taskID" binding:"required"`
}

// For delete tasks
type CreateDeleteTaskJobRequest struct {
	// TODO: BIO define?
	BIO                 string         `json:"bio" binding:"omitempty"`
	Type                string         `json:"type" binding:"required"`
	Args                DeleteTasksJob `json:"args" binding:"omitempty"`
	Result              map[string]any `json:"result" binding:"omitempty"`
	UserID              uint           `json:"user_id" binding:"omitempty"`
	SchedulerClusterIDs []uint         `json:"scheduler_cluster_ids" binding:"omitempty"`
}

type DeleteTaskJob struct {
	// TaskID is the task id for deleting task cache.
	TaskID string `json:"taskID" binding:"required"`
}

In the return value definition, reuse the Job's definition manager/models/job.go to populate the Resul field with the task results corresponding to the above two jobs.

type Job struct {
    BaseModel
    ...
    Result            JSONMap            `gorm:"column:result;comment:group result" json:"result"`
    ...
}

For ListTasks, we need to return a list of clusters/nodes that contain the taskid, for the sake of returning more sufficient information, we provide the original number of related Peers to be returned here.

// reference: manager/models/peer.go
type Peer struct {
	BaseModel
	Hostname           string           `gorm:"column:host_name;type:varchar(256);index:uk_peer,unique;not null;comment:hostname" json:"host_name"`
	Type               string           `gorm:"column:type;type:varchar(256);index:idx_peer_type;comment:type" json:"type"`
	IDC                string           `gorm:"column:idc;type:varchar(1024);comment:internet data center" json:"idc"`
	Location           string           `gorm:"column:location;type:varchar(1024);comment:location" json:"location"`
	IP                 string           `gorm:"column:ip;type:varchar(256);index:uk_peer,unique;not null;comment:ip address" json:"ip"`
	Port               int32            `gorm:"column:port;not null;comment:grpc service listening port" json:"port"`
	DownloadPort       int32            `gorm:"column:download_port;not null;comment:download service listening port" json:"download_port"`
	ObjectStoragePort  int32            `gorm:"column:object_storage_port;comment:object storage service listening port" json:"object_storage_port"`
	State              string           `gorm:"column:state;type:varchar(256);default:'inactive';comment:service state" json:"state"`
	OS                 string           `gorm:"column:os;type:varchar(256);comment:os" json:"os"`
	Platform           string           `gorm:"column:platform;type:varchar(256);comment:platform" json:"platform"`
	PlatformFamily     string           `gorm:"column:platform_family;type:varchar(256);comment:platform family" json:"platform_family"`
	PlatformVersion    string           `gorm:"column:platform_version;type:varchar(256);comment:platform version" json:"platform_version"`
	KernelVersion      string           `gorm:"column:kernel_version;type:varchar(256);comment:kernel version" json:"kernel_version"`
	GitVersion         string           `gorm:"column:git_version;type:varchar(256);index:idx_peer_git_version;comment:git version" json:"git_version"`
	GitCommit          string           `gorm:"column:git_commit;type:varchar(256);index:idx_peer_git_commit;comment:git commit" json:"git_commit"`
	BuildPlatform      string           `gorm:"column:build_platform;type:varchar(256);comment:build platform" json:"build_platform"`
	SchedulerClusterID uint             `gorm:"index:uk_peer,unique;not null;comment:scheduler cluster id" json:"scheduler_cluster_id"`
	SchedulerCluster   SchedulerCluster `json:"scheduler_cluster"`
}


type ListTasksResponse struct {
    Peers []Peer `json:"peers"`
}

For DeleteTask, we need to return a list of success and failure nodes, paged at the api level.

// Reference: scheduler/storage/types.go
// Task defines the structure for task management.
type Task struct {
	ID            string `json:"id"`
	PieceLength   int    `json:"piece_length"`
	ContentLength int    `json:"content_length"`
	UploadingCount  int    `json:"uploading_count"`
	UploadedCount   int    `json:"uploaded_count"`
	UpdatedAt     int64   `json:"updated_at"`
	CreatedAt     int64   `json:"created_at"`
	PrefetchedAt  int64   `json:"prefetched_at"`
	FailedAt      int64   `json:"failed_at"`
	FinishedAt    int64   `json:"finished_at"`
}

// TaskInfo includes information about a task along with peer details and a description.
type TaskInfo struct {
	Task Task   `json:"task"`
	Peer Peer   `json:"peer"`
	Desc string `json:"desc"`
}

// DeleteTaskResponse represents the response after attempting to delete tasks,
// categorizing them into successfully and unsuccessfully deleted.
type DeleteTaskResponse struct {
	SuccessTasks []TaskInfo `json:"success_tasks"`
	FailureTasks []TaskInfo `json:"failure_tasks"`
}

Second, define interfaces in service manager/service/service.go to support calls to lower level business logic.

// manager/service/service.go
CreateListTasksJob(context.Context, types.CreateListTasksJobRequest) (*models.Job, error)
CreateDeleteTaskJob(context.Context, types.CreateDeleteTasksJobRequest) (*models.Job, error)

CreatePeerCacheCleanJob creates a Job type via manager and creates a DeleteTask and ListTasks task via SendGroupWithContext on the distributed queue, i.e., starts an asynchronous in redis task in redis, waiting for the scheduler to consume this task and execute it.

// manager/job/job.go
type Job struct {
	*internaljob.Job
    ...
    ListTasks
    DeleteTasks
}

Scheduler registers clean peer cache task

Scheduler starts a redis backend as a broker via machinery, RegisterJob way to register that clean cache task to internaljob. Finally the function is executed to get the result

Define a job function in scheduler/job/job.go:

// scheduler/job/job.go
// listTasks is a job to list tasks.
func (j *job) listTasks(ctx context.Context, data string) error {
	// TODO:
	// 1. query all peers with task id

	return nil
}

// deleteTasks is a job to delete dirty tasks.
func (j *job) deleteTasks(ctx context.Context, data string) error {
	// TODO:
	// 1. query all peers with task id
	// 2. delete task on all peers

	return nil
}

In deleteTasks, it tries to retrieve all peer ids that contain the task id and sends a delete task request to all peer ids (host) and returns the status of the task.

1. Multiple batch calls to lower level api when calling interface

Peer-side execution of delete task

On the specific Peer side, the rpc request for delete task of scheduler will be executed (you can consider synchronous/asynchronous deletion), the main tasks to be done on the peer side are:

1. delete local meta data and cache
2. Execute a broadcast to notify the Peer that the deletion was successful.

After defining proto, there are interfaces for defining delete task in dfdaemon(rust client), the core of which is the interface defined below:

// DfdaemonDownload represents download service of dfdaemon.
service DfdaemonDownload{
  ...
  // DeleteTask deletes task from p2p network.
  rpc DeleteTask(DeleteTaskRequest) returns(google.protobuf.Empty);
}

There is no implementation in the impl for the time being. Therefore, you can refer to the semantics of the go end and its own peer clean task to complete, specific to this implementation, the function needs to complete the work can refer to the following implementation:

1. update the meta data in rocksDB
2. reclaim local storage

// dragonfly-client/src/grpc/dfdaemon_download.rs
// delete_task calls the dfdaemon to delete the task.
#[instrument(skip_all)]
async fn delete_task(
    &self,
    request: Request<DeleteTaskRequest>,
) -> Result<Response<()>, Status> {
    println!("delete_task: {:?}", request);
    // TODO:
    // 1. Delete meta data
    // 2. Reclaim local storage
    Err(Status::unimplemented("not implemented"))
}

Deleting metadata and taskdir from the local filesystem is already implemented in storage, which is kind of simplified compared to the go side, i.e. no marking of reclaim and direct deletion of the whole task directory.

In addition, the delete task here only cleans up in rocksdb, and the metadata is updated twice in dragon-client/src/task/mod.rs, so if you clean up the metadata in the delete task, the download task overwrites the contents of the delete task.

Currently, meta data is directly written to the db, if you need to support delete behaviour, you need to judge whether the put operation is existed or not after the first put operation is started, if it is not existed, then it means it is deleted, and delete the dirty content generated by the current operation.

Another thing is that delete task needs to support announce task, there are GCs in rust that do delete task, you should have to aggregate announce task (or should be called boardcast) into delete_task implementation within Storage.

// dragonfly-client-storage/src/lib.rs
// delete_task deletes the task metadatas, task content and piece metadatas.
pub async fn delete_task(&self, id: &str) -> Result<()> {
    self.metadata.delete_task(id)?;
    self.metadata.delete_pieces(id)?;
    self.content.delete_task(id).await?;
    Ok(())
}

Unit Test & E2E Test

Unit Test

Align unit test with other jobs in manager and scheduler, add mock job type:

// manager/handlers/job_test.go
var (
	mockDeleteTasksReqBody = ...
    ...
)

Define the DeleteTasks and ListTasks mocks in manager/job/mocks, refer to manager/job/mocks/preheat_mock.go.
There is no need to add a new unit test in the scheduler for now.

• manager rest api mock and test
• scheduler local job mock and test
• delete_task test

E2E Test

Referring to preheat's e2e test, test/e2e/v2/manager/preheat.go, define the test case for the ginkgo test, which needs to include the query and delete It() test points.

• create a delete task and remove peer cache

REFERENCE

1. Meetup1 https://www.bilibili.com/video/BV1TV4y1j7Yk
2. meetup2 https://www.bilibili.com/video/BV12S4y1J7LG
3. quick start https://d7y.io/zh/docs/next/getting-started/quick-start/kubernetes/

[IRONICBo]

Reference request and response example:

create a job

url: /jobs
request/response body:

1. list task request

{
  "args": {
    "task_id": "string",
    "page": "int",
    "per_page": "int",
  },
  "bio": "",
  "result": {
  },
  "scheduler_cluster_ids": [
    0
  ],
  "seed_peer_cluster_ids": [
    0
  ],
  "type": "list_tasks",
  "user_id": 0
}

2. list task response

{
  "args": {
    "task_id": "string",
    "page": "int",
    "per_page": "int",
  },
  "bio": "string",
  "created_at": "string",
  "id": 0,
  "is_del": 0,
  "result": {
  },
  "scheduler_clusters": [
    {
      "bio": "string",
      "client_config": {
      },
      "config": {
      },
      "created_at": "string",
      "id": 0,
      "is_default": true,
      "is_del": 0,
      "jobs": [
        "string"
      ],
      "name": "string",
      "peers": [
        {
          "build_platform": "string",
          "created_at": "string",
          "download_port": 0,
          "git_commit": "string",
          "git_version": "string",
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "kernel_version": "string",
          "location": "string",
          "object_storage_port": 0,
          "os": "string",
          "platform": "string",
          "platform_family": "string",
          "platform_version": "string",
          "port": 0,
          "scheduler_cluster": "string",
          "scheduler_cluster_id": 0,
          "state": "string",
          "type": "string",
          "updated_at": "string"
        }
      ],
      "schedulers": [
        {
          "created_at": "string",
          "features": [
            "string"
          ],
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "location": "string",
          "models": [
            {
              "bio": "string",
              "created_at": "string",
              "evaluation": {
              },
              "id": 0,
              "is_del": 0,
              "name": "string",
              "scheduler": "string",
              "scheduler_id": 0,
              "state": "string",
              "type": "string",
              "updated_at": "string",
              "version": "string"
            }
          ],
          "port": 0,
          "scheduler_cluster": "string",
          "scheduler_cluster_id": 0,
          "state": "string",
          "updated_at": "string"
        }
      ],
      "scopes": {
      },
      "seed_peer_clusters": [
        {
          "bio": "string",
          "config": {
          },
          "created_at": "string",
          "id": 0,
          "is_del": 0,
          "jobs": [
            "string"
          ],
          "name": "string",
          "scheduler_clusters": [
            "string"
          ],
          "seed_peer": [
            {
              "created_at": "string",
              "download_port": 0,
              "host_name": "string",
              "id": 0,
              "idc": "string",
              "ip": "string",
              "is_del": 0,
              "location": "string",
              "object_storage_port": 0,
              "port": 0,
              "seed_peer_cluster": "string",
              "seed_peer_cluster_id": 0,
              "state": "string",
              "type": "string",
              "updated_at": "string"
            }
          ],
          "updated_at": "string"
        }
      ],
      "updated_at": "string"
    }
  ],
  "seed_peer_clusters": [
    {
      "bio": "string",
      "config": {
      },
      "created_at": "string",
      "id": 0,
      "is_del": 0,
      "jobs": [
        "string"
      ],
      "name": "string",
      "scheduler_clusters": [
        "string"
      ],
      "seed_peer": [
        {
          "created_at": "string",
          "download_port": 0,
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "location": "string",
          "object_storage_port": 0,
          "port": 0,
          "seed_peer_cluster": "string",
          "seed_peer_cluster_id": 0,
          "state": "string",
          "type": "string",
          "updated_at": "string"
        }
      ],
      "updated_at": "string"
    }
  ],
  "state": "string",
  "task_id": "string",
  "type": "string",
  "updated_at": "string",
  "user": {
    "avatar": "string",
    "bio": "string",
    "configs": [
      {
        "bio": "string",
        "created_at": "string",
        "id": 0,
        "is_del": 0,
        "name": "string",
        "updated_at": "string",
        "user": "string",
        "user_id": 0,
        "value": "string"
      }
    ],
    "created_at": "string",
    "email": "string",
    "id": 0,
    "is_del": 0,
    "location": "string",
    "name": "string",
    "phone": "string",
    "state": "string",
    "updated_at": "string"
  },
  "user_id": 0
}

3. delete task request

{
  "args": {
    "task_id": "string",
  },
  "bio": "string",
  "result": {
  },
  "scheduler_cluster_ids": [
    0
  ],
  "seed_peer_cluster_ids": [
    0
  ],
  "type": "delete_task",
  "user_id": 0
}

4. delete task response

{
  "args": {
    "task_id": "string",
  },
  "bio": "string",
  "created_at": "string",
  "id": 0,
  "is_del": 0,
  "result": {
  },
  "scheduler_clusters": [
    {
      "bio": "string",
      "client_config": {
      },
      "config": {
      },
      "created_at": "string",
      "id": 0,
      "is_default": true,
      "is_del": 0,
      "jobs": [
        "string"
      ],
      "name": "string",
      "peers": [
        {
          "build_platform": "string",
          "created_at": "string",
          "download_port": 0,
          "git_commit": "string",
          "git_version": "string",
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "kernel_version": "string",
          "location": "string",
          "object_storage_port": 0,
          "os": "string",
          "platform": "string",
          "platform_family": "string",
          "platform_version": "string",
          "port": 0,
          "scheduler_cluster": "string",
          "scheduler_cluster_id": 0,
          "state": "string",
          "type": "string",
          "updated_at": "string"
        }
      ],
      "schedulers": [
        {
          "created_at": "string",
          "features": [
            "string"
          ],
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "location": "string",
          "models": [
            {
              "bio": "string",
              "created_at": "string",
              "evaluation": {
              },
              "id": 0,
              "is_del": 0,
              "name": "string",
              "scheduler": "string",
              "scheduler_id": 0,
              "state": "string",
              "type": "string",
              "updated_at": "string",
              "version": "string"
            }
          ],
          "port": 0,
          "scheduler_cluster": "string",
          "scheduler_cluster_id": 0,
          "state": "string",
          "updated_at": "string"
        }
      ],
      "scopes": {
      },
      "seed_peer_clusters": [
        {
          "bio": "string",
          "config": {
          },
          "created_at": "string",
          "id": 0,
          "is_del": 0,
          "jobs": [
            "string"
          ],
          "name": "string",
          "scheduler_clusters": [
            "string"
          ],
          "seed_peer": [
            {
              "created_at": "string",
              "download_port": 0,
              "host_name": "string",
              "id": 0,
              "idc": "string",
              "ip": "string",
              "is_del": 0,
              "location": "string",
              "object_storage_port": 0,
              "port": 0,
              "seed_peer_cluster": "string",
              "seed_peer_cluster_id": 0,
              "state": "string",
              "type": "string",
              "updated_at": "string"
            }
          ],
          "updated_at": "string"
        }
      ],
      "updated_at": "string"
    }
  ],
  "seed_peer_clusters": [
    {
      "bio": "string",
      "config": {
      },
      "created_at": "string",
      "id": 0,
      "is_del": 0,
      "jobs": [
        "string"
      ],
      "name": "string",
      "scheduler_clusters": [
        "string"
      ],
      "seed_peer": [
        {
          "created_at": "string",
          "download_port": 0,
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "location": "string",
          "object_storage_port": 0,
          "port": 0,
          "seed_peer_cluster": "string",
          "seed_peer_cluster_id": 0,
          "state": "string",
          "type": "string",
          "updated_at": "string"
        }
      ],
      "updated_at": "string"
    }
  ],
  "state": "string",
  "task_id": "string",
  "type": "string",
  "updated_at": "string",
  "user": {
    "avatar": "string",
    "bio": "string",
    "configs": [
      {
        "bio": "string",
        "created_at": "string",
        "id": 0,
        "is_del": 0,
        "name": "string",
        "updated_at": "string",
        "user": "string",
        "user_id": 0,
        "value": "string"
      }
    ],
    "created_at": "string",
    "email": "string",
    "id": 0,
    "is_del": 0,
    "location": "string",
    "name": "string",
    "phone": "string",
    "state": "string",
    "updated_at": "string"
  },
  "user_id": 0
}

query job result

url: /jobs/{id}
response body:

1. list task request response，the response data is in result property.

{
  "args": {
    "task_id": "string",
    "page": "int",
    "per_page": "int",
  },
  "bio": "string",
  "created_at": "string",
  "id": 0,
  "is_del": 0,
  "result": {
    "peers": [
      {
        "host_name": "server1.example.com",
        "type": "primary",
        "idc": "IDC1",
        "location": "New York, NY",
        "ip": "192.168.1.1",
        "port": 8080,
        "download_port": 9090,
        "object_storage_port": 10000,
        "state": "active",
        "os": "Linux",
        "platform": "x86_64",
        "platform_family": "debian",
        "platform_version": "10",
        "kernel_version": "4.19.0-12-amd64",
        "git_version": "v1.2.3",
        "git_commit": "a1b2c3d4",
        "build_platform": "linux/amd64",
        "scheduler_cluster_id": 1,
        "scheduler_cluster": {
          "id": 1,
          "name": "Cluster A",
          "state": "active",
          "region": "us-east-1"
        }
      },
      {
        "host_name": "server2.example.com",
        "type": "secondary",
        "idc": "IDC2",
        "location": "San Francisco, CA",
        "ip": "192.168.1.2",
        "port": 8081,
        "download_port": 9091,
        "object_storage_port": 10001,
        "state": "inactive",
        "os": "Linux",
        "platform": "x86_64",
        "platform_family": "ubuntu",
        "platform_version": "18.04",
        "kernel_version": "5.4.0-42-generic",
        "git_version": "v1.2.4",
        "git_commit": "e5f6g7h8",
        "build_platform": "linux/amd64",
        "scheduler_cluster_id": 2,
        "scheduler_cluster": {
          "id": 2,
          "name": "Cluster B",
          "state": "inactive",
          "region": "us-west-2"
        }
      }
    ]
  },
  "scheduler_clusters": [
    {
      "bio": "string",
      "client_config": {
      },
      "config": {
      },
      "created_at": "string",
      "id": 0,
      "is_default": true,
      "is_del": 0,
      "jobs": [
        "string"
      ],
      "name": "string",
      "peers": [
        {
          "build_platform": "string",
          "created_at": "string",
          "download_port": 0,
          "git_commit": "string",
          "git_version": "string",
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "kernel_version": "string",
          "location": "string",
          "object_storage_port": 0,
          "os": "string",
          "platform": "string",
          "platform_family": "string",
          "platform_version": "string",
          "port": 0,
          "scheduler_cluster": "string",
          "scheduler_cluster_id": 0,
          "state": "string",
          "type": "string",
          "updated_at": "string"
        }
      ],
      "schedulers": [
        {
          "created_at": "string",
          "features": [
            "string"
          ],
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "location": "string",
          "models": [
            {
              "bio": "string",
              "created_at": "string",
              "evaluation": {
              },
              "id": 0,
              "is_del": 0,
              "name": "string",
              "scheduler": "string",
              "scheduler_id": 0,
              "state": "string",
              "type": "string",
              "updated_at": "string",
              "version": "string"
            }
          ],
          "port": 0,
          "scheduler_cluster": "string",
          "scheduler_cluster_id": 0,
          "state": "string",
          "updated_at": "string"
        }
      ],
      "scopes": {
      },
      "seed_peer_clusters": [
        {
          "bio": "string",
          "config": {
          },
          "created_at": "string",
          "id": 0,
          "is_del": 0,
          "jobs": [
            "string"
          ],
          "name": "string",
          "scheduler_clusters": [
            "string"
          ],
          "seed_peer": [
            {
              "created_at": "string",
              "download_port": 0,
              "host_name": "string",
              "id": 0,
              "idc": "string",
              "ip": "string",
              "is_del": 0,
              "location": "string",
              "object_storage_port": 0,
              "port": 0,
              "seed_peer_cluster": "string",
              "seed_peer_cluster_id": 0,
              "state": "string",
              "type": "string",
              "updated_at": "string"
            }
          ],
          "updated_at": "string"
        }
      ],
      "updated_at": "string"
    }
  ],
  "seed_peer_clusters": [
    {
      "bio": "string",
      "config": {
      },
      "created_at": "string",
      "id": 0,
      "is_del": 0,
      "jobs": [
        "string"
      ],
      "name": "string",
      "scheduler_clusters": [
        "string"
      ],
      "seed_peer": [
        {
          "created_at": "string",
          "download_port": 0,
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "location": "string",
          "object_storage_port": 0,
          "port": 0,
          "seed_peer_cluster": "string",
          "seed_peer_cluster_id": 0,
          "state": "string",
          "type": "string",
          "updated_at": "string"
        }
      ],
      "updated_at": "string"
    }
  ],
  "state": "string",
  "task_id": "string",
  "type": "string",
  "updated_at": "string",
  "user": {
    "avatar": "string",
    "bio": "string",
    "configs": [
      {
        "bio": "string",
        "created_at": "string",
        "id": 0,
        "is_del": 0,
        "name": "string",
        "updated_at": "string",
        "user": "string",
        "user_id": 0,
        "value": "string"
      }
    ],
    "created_at": "string",
    "email": "string",
    "id": 0,
    "is_del": 0,
    "location": "string",
    "name": "string",
    "phone": "string",
    "state": "string",
    "updated_at": "string"
  },
  "user_id": 0
}

2. delete task response, the response is in result property

{
  "args": {
    "task_id": "string",
  },
  "bio": "string",
  "created_at": "string",
  "id": 0,
  "is_del": 0,
  "result": {
    "success_tasks": [
      {
        "task": {
          "id": "task1",
          "piece_length": 1024,
          "content_length": 20480,
          "uploading_count": 5,
          "uploaded_count": 5,
          "updated_at": 1599557760,
          "created_at": 1599554160,
          "prefetched_at": 1599554460,
          "failed_at": 0,
          "finished_at": 1599557760
        },
        "peer": {
          "id": "peer1",
          "address": "192.168.1.100",
          "port": 8080
        },
        "desc": "Task completed successfully."
      },
      {
        "task": {
          "id": "task2",
          "piece_length": 2048,
          "content_length": 40960,
          "uploading_count": 3,
          "uploaded_count": 3,
          "updated_at": 1599657760,
          "created_at": 1599654160,
          "prefetched_at": 1599654460,
          "failed_at": 0,
          "finished_at": 1599657760
        },
        "peer": {
          "id": "peer2",
          "address": "192.168.1.101",
          "port": 8081
        },
        "desc": "Task completed without issues."
      }
    ],
    "failure_tasks": [
      {
        "task": {
          "id": "task3",
          "piece_length": 512,
          "content_length": 10240,
          "uploading_count": 2,
          "uploaded_count": 1,
          "updated_at": 1599757760,
          "created_at": 1599754160,
          "prefetched_at": 0,
          "failed_at": 1599755000,
          "finished_at": 0
        },
        "peer": {
          "id": "peer3",
          "address": "192.168.1.102",
          "port": 8082
        },
        "desc": "Task failed due to network issues."
      }
    ]
  },
  "scheduler_clusters": [
    {
      "bio": "string",
      "client_config": {
      },
      "config": {
      },
      "created_at": "string",
      "id": 0,
      "is_default": true,
      "is_del": 0,
      "jobs": [
        "string"
      ],
      "name": "string",
      "peers": [
        {
          "build_platform": "string",
          "created_at": "string",
          "download_port": 0,
          "git_commit": "string",
          "git_version": "string",
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "kernel_version": "string",
          "location": "string",
          "object_storage_port": 0,
          "os": "string",
          "platform": "string",
          "platform_family": "string",
          "platform_version": "string",
          "port": 0,
          "scheduler_cluster": "string",
          "scheduler_cluster_id": 0,
          "state": "string",
          "type": "string",
          "updated_at": "string"
        }
      ],
      "schedulers": [
        {
          "created_at": "string",
          "features": [
            "string"
          ],
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "location": "string",
          "models": [
            {
              "bio": "string",
              "created_at": "string",
              "evaluation": {
              },
              "id": 0,
              "is_del": 0,
              "name": "string",
              "scheduler": "string",
              "scheduler_id": 0,
              "state": "string",
              "type": "string",
              "updated_at": "string",
              "version": "string"
            }
          ],
          "port": 0,
          "scheduler_cluster": "string",
          "scheduler_cluster_id": 0,
          "state": "string",
          "updated_at": "string"
        }
      ],
      "scopes": {
      },
      "seed_peer_clusters": [
        {
          "bio": "string",
          "config": {
          },
          "created_at": "string",
          "id": 0,
          "is_del": 0,
          "jobs": [
            "string"
          ],
          "name": "string",
          "scheduler_clusters": [
            "string"
          ],
          "seed_peer": [
            {
              "created_at": "string",
              "download_port": 0,
              "host_name": "string",
              "id": 0,
              "idc": "string",
              "ip": "string",
              "is_del": 0,
              "location": "string",
              "object_storage_port": 0,
              "port": 0,
              "seed_peer_cluster": "string",
              "seed_peer_cluster_id": 0,
              "state": "string",
              "type": "string",
              "updated_at": "string"
            }
          ],
          "updated_at": "string"
        }
      ],
      "updated_at": "string"
    }
  ],
  "seed_peer_clusters": [
    {
      "bio": "string",
      "config": {
      },
      "created_at": "string",
      "id": 0,
      "is_del": 0,
      "jobs": [
        "string"
      ],
      "name": "string",
      "scheduler_clusters": [
        "string"
      ],
      "seed_peer": [
        {
          "created_at": "string",
          "download_port": 0,
          "host_name": "string",
          "id": 0,
          "idc": "string",
          "ip": "string",
          "is_del": 0,
          "location": "string",
          "object_storage_port": 0,
          "port": 0,
          "seed_peer_cluster": "string",
          "seed_peer_cluster_id": 0,
          "state": "string",
          "type": "string",
          "updated_at": "string"
        }
      ],
      "updated_at": "string"
    }
  ],
  "state": "string",
  "task_id": "string",
  "type": "string",
  "updated_at": "string",
  "user": {
    "avatar": "string",
    "bio": "string",
    "configs": [
      {
        "bio": "string",
        "created_at": "string",
        "id": 0,
        "is_del": 0,
        "name": "string",
        "updated_at": "string",
        "user": "string",
        "user_id": 0,
        "value": "string"
      }
    ],
    "created_at": "string",
    "email": "string",
    "id": 0,
    "is_del": 0,
    "location": "string",
    "name": "string",
    "phone": "string",
    "state": "string",
    "updated_at": "string"
  },
  "user_id": 0
}

[IRONICBo]

GetTask and DeleteTask request and response example:

create a get task job

url: /jobs
request/response body:

1. get task request

{
  "args": {
    "task_id": "string",
  },
  "type": "get_task",
}

2. get task response

{
  "args": {
    "task_id": "string",
  },
  "bio": "string",
  "created_at": "string",
  "id": 0,
  "is_del": 0,
  "result": {
  },
  "scheduler_clusters": [
  ],
  "seed_peer_clusters": [
  ],
  "state": "string",
  "task_id": "string",
  "type": "string",
  "updated_at": "string",
  "user": {
  },
  "user_id": 0
}

create a delete task job

1. delete task request

{
  "args": {
    "task_id": "string",
  },
  "type": "delete_task",
}

2. delete task response

{
  "args": {
    "task_id": "string",
  },
  "bio": "string",
  "created_at": "string",
  "id": 0,
  "is_del": 0,
  "result": {
  },
  "scheduler_clusters": [
  ],
  "seed_peer_clusters": [
  ],
  "state": "string",
  "task_id": "string",
  "type": "string",
  "updated_at": "string",
  "user": {
  },
  "user_id": 0
}

query job result

url: /jobs/{id}
response body:

1. query get task response

1. get task request response，the response data is in result property.

{
  "args": {
    "task_id": "string",
  },
  "bio": "string",
  "created_at": "string",
  "id": 0,
  "is_del": 0,
  "result": {
    "peers": [
      {
        "host_name": "server1.example.com",
        "type": "primary",
        "idc": "IDC1",
        "location": "New York, NY",
        "ip": "192.168.1.1",
        "port": 8080,
        "download_port": 9090,
        "object_storage_port": 10000,
        "state": "active",
        "os": "Linux",
        "platform": "x86_64",
        "platform_family": "debian",
        "platform_version": "10",
        "kernel_version": "4.19.0-12-amd64",
        "git_version": "v1.2.3",
        "git_commit": "a1b2c3d4",
        "build_platform": "linux/amd64",
        "scheduler_cluster_id": 1,
        "scheduler_cluster": {
          "id": 1,
          "name": "Cluster A",
          "state": "active",
          "region": "us-east-1"
        }
      },
      {
        "host_name": "server2.example.com",
        "type": "secondary",
        "idc": "IDC2",
        "location": "San Francisco, CA",
        "ip": "192.168.1.2",
        "port": 8081,
        "download_port": 9091,
        "object_storage_port": 10001,
        "state": "inactive",
        "os": "Linux",
        "platform": "x86_64",
        "platform_family": "ubuntu",
        "platform_version": "18.04",
        "kernel_version": "5.4.0-42-generic",
        "git_version": "v1.2.4",
        "git_commit": "e5f6g7h8",
        "build_platform": "linux/amd64",
        "scheduler_cluster_id": 2,
        "scheduler_cluster": {
          "id": 2,
          "name": "Cluster B",
          "state": "inactive",
          "region": "us-west-2"
        }
      }
    ],
  },
  "scheduler_clusters": [
  ],
  "seed_peer_clusters": [
  ],
  "state": "string",
  "task_id": "string",
  "type": "get_task",
  "updated_at": "string",
  "user": {
  },
  "user_id": 0
}

2. delete task response, the response is in result property

{
  "args": {
    "task_id": "string"
  },
  "bio": "string",
  "created_at": "string",
  "id": 0,
  "is_del": 0,
  "result": {
    "success_peers": [
      {
        "peer": {
          "host_name": "server2.example.com",
          "type": "secondary",
          "idc": "IDC2",
          "location": "San Francisco, CA",
          "ip": "192.168.1.2",
          "port": 8081,
          "download_port": 9091,
          "object_storage_port": 10001,
          "state": "inactive",
          "os": "Linux",
          "platform": "x86_64",
          "platform_family": "ubuntu",
          "platform_version": "18.04",
          "kernel_version": "5.4.0-42-generic",
          "git_version": "v1.2.4",
          "git_commit": "e5f6g7h8",
          "build_platform": "linux/amd64",
          "scheduler_cluster_id": 2,
          "scheduler_cluster": {
            "id": 2,
            "name": "Cluster B",
            "state": "inactive",
            "region": "us-west-2"
          }
        },
        "description": ""
      }
    ],
    "failure_peers": [
      {
        "peer": {
          "host_name": "server2.example.com",
          "type": "secondary",
          "idc": "IDC2",
          "location": "San Francisco, CA",
          "ip": "192.168.1.2",
          "port": 8081,
          "download_port": 9091,
          "object_storage_port": 10001,
          "state": "inactive",
          "os": "Linux",
          "platform": "x86_64",
          "platform_family": "ubuntu",
          "platform_version": "18.04",
          "kernel_version": "5.4.0-42-generic",
          "git_version": "v1.2.4",
          "git_commit": "e5f6g7h8",
          "build_platform": "linux/amd64",
          "scheduler_cluster_id": 2,
          "scheduler_cluster": {
            "id": 2,
            "name": "Cluster B",
            "state": "inactive",
            "region": "us-west-2"
            }
        },
        "description": "Error message."
      }
    ],
  "scheduler_clusters": [
  ],
  "seed_peer_clusters": [
  ],
  "state": "string",
  "task_id": "string",
  "type": "delete_task",
  "updated_at": "string",
  "user": {
  },
  "user_id": 0
}