diff --git a/.github/workflows/base.yml b/.github/workflows/base.yml index c5501bd..49f0f3d 100644 --- a/.github/workflows/base.yml +++ b/.github/workflows/base.yml @@ -4,54 +4,54 @@ # # -on: [ push, pull_request ] +# on: [ push, pull_request ] -name: Base GitHub Action for Check, Test and Lints +# name: Base GitHub Action for Check, Test and Lints -jobs: - check: - name: Check - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - uses: actions-rs/toolchain@v1 - with: - profile: minimal - toolchain: stable - override: true - - uses: actions-rs/cargo@v1 - with: - command: check - - uses: actions/setup-node@v3 - with: - node-version: 16 +# jobs: +# check: +# name: Check +# runs-on: ubuntu-latest +# steps: +# - uses: actions/checkout@v2 +# - uses: actions-rs/toolchain@v1 +# with: +# profile: minimal +# toolchain: stable +# override: true +# - uses: actions-rs/cargo@v1 +# with: +# command: check +# - uses: actions/setup-node@v3 +# with: +# node-version: 16 - test: - name: Test Suite - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - uses: actions-rs/toolchain@v1 - with: - profile: minimal - toolchain: stable - override: true - - uses: actions-rs/cargo@v1 - with: - command: test +# test: +# name: Test Suite +# runs-on: ubuntu-latest +# steps: +# - uses: actions/checkout@v2 +# - uses: actions-rs/toolchain@v1 +# with: +# profile: minimal +# toolchain: stable +# override: true +# - uses: actions-rs/cargo@v1 +# with: +# command: test - clippy: - name: Clippy - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v2 - - uses: actions-rs/toolchain@v1 - with: - profile: minimal - toolchain: stable - override: true - - run: rustup component add clippy - - uses: actions-rs/cargo@v1 - with: - command: clippy - args: -- -D warnings \ No newline at end of file +# clippy: +# name: Clippy +# runs-on: ubuntu-latest +# steps: +# - uses: actions/checkout@v2 +# - uses: actions-rs/toolchain@v1 +# with: +# profile: minimal +# toolchain: stable +# override: true +# - run: rustup component add clippy +# - uses: actions-rs/cargo@v1 +# with: +# command: clippy +# args: -- -D warnings \ No newline at end of file diff --git a/.gitignore b/.gitignore index 311329f..d3b40d2 100644 --- a/.gitignore +++ b/.gitignore @@ -16,3 +16,5 @@ Cargo.lock # Exclude execute log /*.log + +.vscode diff --git a/.vscode/settings.json b/.vscode/settings.json deleted file mode 100644 index 39a5ca1..0000000 --- a/.vscode/settings.json +++ /dev/null @@ -1,3 +0,0 @@ -{ - "rust-analyzer.cargo.features": "all" -} diff --git a/README.md b/README.md index 6e66712..e02d422 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,22 @@ `dagrs` are suitable for the execution of multiple tasks with graph-like dependencies. `dagrs` has the characteristics of high performance and asynchronous execution. It provides users with a convenient programming interface. +The development of `dagrs` follows the concept of Flow-based Programming. + +## Flow-based Programming + +Flow-based Programming (FBP) was invented by J. Paul Morrison in the early 1970s. It was initially implemented in software for a Canadian bank. Over the years, it’s had various names but has always maintained its core principles of reducing development time and managing processes efficiently. + +FBP treats applications as networks of 'black box' processes that communicate by sending and receiving data, referred to as Information Packets, over predefined connections. It’s a component-oriented approach that fits well with modular software architecture. + +FBP features + +- Encapsulated processes and information packets. +- The external definition of connections +- Asynchronous +- Information packets with unique ownership and defined lifetimes +- Bounded connections with a finite capacity and reserve pressure. + ## What can `dagrs` do `dagrs` allows users to easily execute multiple sets of tasks with complex graph dependencies. It only requires: @@ -40,385 +56,9 @@ This graph represents the dependencies between tasks, and the graph composed of Among them, each task may produce output, and may also require the output of some tasks as its input. -## Try using `dagrs` - -`dagrs` provides two basic task definition methods, which are programming to implement the logic of the task and defining the yaml configuration file. Programmatically implementing the definition of tasks will make the logic of tasks more flexible, and it is also the main method of using `dagrs`. Next, we will introduce the usage of the two methods in detail. - -*Make sure the Rust compilation environment is available.* - -### Programmatically implement task definition - -Users need to program to implement the `Action` trait to define the specific logic of the task, and then build a series of `DefaultTask`. - -First, users need to define some specific task logic. There are two ways to define task logic: - -- Create a closure whose type is `Simple`, which is suitable for simple scenarios. -- Create a type and implement the `Complex` trait, which is suitable for more complex situations. For example, if the logic of the task is to execute a system command, the command string needs to be recorded in some way. You can create a `Command` structure with a string attribute inside to store the command string. - -You can refer to examples:`examples/actions.rs`. - -In the second step, you need to use the defined task logic to create specific tasks. Here you may need to use the `DefaultTask` type, which provides users with several ways to create `Task`. `DefaultTask` allows you to specify specific task logic for the task and give the task a name. Please refer to the documentation for specific function functions. - -In the third step, you need to specify dependencies for the defined series of tasks. Here you need to use the `set_predecessors` function of `DefaultTask`. This function requires you to specify a series of predecessor tasks for the current task. - -The fourth step is to create a `Dag` and put all the defined tasks into the `Dag` scheduler. - -Optional step: You can specify an environment variable for `Dag`. This environment variable is available in all tasks. In some specific tasks, this behavior can be useful. - -Finally, you can [initialize a logger](https://github.com/rust-lang/log#in-executables), and then you can call the `start` function of `Dag` to start executing all tasks. - -You can refer to an example for the above complete steps: `examples/compute_dag.rs` - - -Here is the `examples/compute_dag.rs` example: - -```rust -//! Only use Dag, execute a job. The graph is as follows: -//! -//! ↱----------↴ -//! B -→ E --→ G -//! ↗ ↗ ↗ -//! A --→ C / -//! ↘ ↘ / -//! D -→ F -//! -//! The final execution result is 272. - -extern crate dagrs; - -use std::sync::Arc; -use dagrs::{Complex, Dag, DefaultTask, EnvVar, Input, Output}; - -struct Compute(usize); - -impl Complex for Compute { - fn run(&self, input: Input, env: Arc) -> Output { - let base = env.get::("base").unwrap(); - let mut sum = self.0; - input - .get_iter() - .for_each(|i| sum += i.get::().unwrap() * base); - Output::new(sum) - } -} - -fn main() { - // initialization log. - env_logger::init(); - - // generate some tasks. - let a = DefaultTask::with_action("Compute A", Compute(1)); - - let mut b = DefaultTask::with_action("Compute B", Compute(2)); - - let mut c = DefaultTask::new("Compute C"); - c.set_action(Compute(4)); - - let mut d = DefaultTask::new("Compute D"); - d.set_action(Compute(8)); - - let mut e = DefaultTask::with_closure("Compute E", |input, env| { - let base = env.get::("base").unwrap(); - let mut sum = 16; - input - .get_iter() - .for_each(|i| sum += i.get::().unwrap() * base); - Output::new(sum) - }); - let mut f = DefaultTask::with_closure("Compute F", |input, env| { - let base = env.get::("base").unwrap(); - let mut sum = 32; - input - .get_iter() - .for_each(|i| sum += i.get::().unwrap() * base); - Output::new(sum) - }); - - let mut g = DefaultTask::new("Compute G"); - g.set_closure(|input, env| { - let base = env.get::("base").unwrap(); - let mut sum = 64; - input - .get_iter() - .for_each(|i| sum += i.get::().unwrap() * base); - Output::new(sum) - }); - - // Set up task dependencies. - b.set_predecessors(&[&a]); - c.set_predecessors(&[&a]); - d.set_predecessors(&[&a]); - e.set_predecessors(&[&b, &c]); - f.set_predecessors(&[&c, &d]); - g.set_predecessors(&[&b, &e, &f]); - // Create a new Dag. - let mut dag = Dag::with_tasks(vec![a, b, c, d, e, f, g]); - // Set a global environment variable for this dag. - let mut env = EnvVar::new(); - env.set("base", 2usize); - dag.set_env(env); - // Start executing this dag - assert!(dag.start().unwrap()); - // Get execution result. - let res = dag.get_result::().unwrap(); - println!("The result is {}.", res); -} -``` - -**explain:** - -First, we initialize the logger, declare the `Compute` type, and implement the `Complex` trait for it. In the rewritten run function, we simply get the output value of the predecessor task and multiply it by the environment variable `base`. Then accumulate the multiplied result to itself self.0. - -Next, we define 6 tasks and show the usage of some functions in the `DefaultTask` type. Set predecessor tasks for each task. - -Then, create a `Dag`, set a base environment variable for it, and use the start method to start executing all tasks. - -Finally we call the `start` function of `Dag` to execute all tasks. After the task is executed, call the `get_result` function to obtain the final execution result of the task. - -The graph formed by the task is shown below: - -```mermaid -flowchart LR - A-->B - A-->C - B-->D - B-->F - C-->D - C-->E - D-->F - E-->F -``` - -The execution order is a->c->b->d. - -```bash -$ cargo run --example compute_dag -[Start] -> Compute A -> Compute B -> Compute D -> Compute C -> Compute F -> Compute E -> Compute G -> [End] -Executing task [name: Compute A, id: 1] -Execution succeed [name: Compute A, id: 1] -Executing task [name: Compute C, id: 3] -Executing task [name: Compute B, id: 2] -Executing task [name: Compute D, id: 4] -Execution succeed [name: Compute C, id: 3] -Execution succeed [name: Compute B, id: 2] -Execution succeed [name: Compute D, id: 4] -Executing task [name: Compute F, id: 6] -Executing task [name: Compute E, id: 5] -Execution succeed [name: Compute F, id: 6] -Execution succeed [name: Compute E, id: 5] -Executing task [name: Compute G, id: 7] -Execution succeed [name: Compute G, id: 7] -The result is 272. -``` - -### `Yaml` configuration file - -A standard yaml configuration file format is given below: - -```yaml -dagrs: - a: - name: "Task 1" - after: [ b, c ] - cmd: echo a - b: - name: "Task 2" - after: [ c, f, g ] - cmd: echo b - c: - name: "Task 3" - after: [ e, g ] - cmd: echo c - d: - name: "Task 4" - after: [ c, e ] - cmd: echo d - e: - name: "Task 5" - after: [ h ] - cmd: echo e - f: - name: "Task 6" - after: [ g ] - cmd: python3 ./tests/config/test.py - g: - name: "Task 7" - after: [ h ] - cmd: node ./tests/config/test.js - h: - name: "Task 8" - cmd: echo h -``` - -These yaml-defined task items form a complex dependency graph. In the yaml configuration file: - -- The file starts with `dagrs` -- Similar to `a`, `b`, `c`... is the unique identifier of the task -- `name` is a required attribute, which is the name of the task -- `after` is an optional attribute (only the first executed task does not have this attribute), which represents which tasks are executed after the task, that is, specifies dependencies for tasks -- `cmd` is a optional attribute. You need to point out the command to be executed, such as the basic shell command: `echo hello`, execute the python script `python test.py`, etc. The user must ensure that the interpreter that executes the script exists in the environment variable. `CommandAction` is the implementation of the specific execution logic of the script, which is put into a specific `Task` type. - If users want to customize other types of script tasks, or implement their own script execution logic, they can implement the "Action" feature through programming, and when parsing the configuration file, provide the parser with a specific type that implements the `Action` feature, and the method should be in the form of a key-value pair: . Although this is more troublesome, this method will be more flexible. - -To parse the yaml configured file, you need to compile this project, requiring rust version >= 1.70: - -```bash -$ cargo build --release --features=yaml -$ ./target/release/dagrs.exe --help -Usage: dagrs.exe [OPTIONS] --yaml - -Options: - --log-path Log output file, the default is to print to the terminal - --yaml yaml configuration file path - --log-level Log level, the default is Info - -h, --help Print help - -V, --version Print version -``` - -**parameter explanation:** - -- The parameter yaml represents the path of the yaml configuration file and is a required parameter. -- The parameter log-path represents the path of the log output file and is an optional parameter. If not specified, the log is printed on the console by default. -- The parameter log-level represents the log output level, which is an optional parameter and defaults to info. - -We can try an already defined file at `tests/config/correct.yaml` - -```bash -$ ./target/release/dagrs --yaml=./tests/config/correct.yaml --log-path=./dagrs.log --log-level=info -[Start] -> Task 8 -> Task 5 -> Task 7 -> Task 6 -> Task 3 -> Task 2 -> Task 1 -> Task 4 -> [End] -Executing Task[name: Task 8] -Executing Task[name: Task 5] -Executing Task[name: Task 7] -Executing Task[name: Task 6] -Executing Task[name: Task 3] -Executing Task[name: Task 2] -Executing Task[name: Task 4] -Executing Task[name: Task 1] -``` - -You can see an example: `examples/yaml_dag.rs`. In fact, you can also programmatically read the yaml configuration file generation task, which is very simple, just use the `with_yaml` function provided by `Dag` to parse the configuration file. - --------------------------------------- - -**In addition to these two methods, `dagrs` also supports advanced task custom configuration.** - -- `DefaultTask` is a default implementation of the `Task` trait. Users can also customize tasks and add more functions and attributes to tasks, but they still need to have the four necessary attributes in `DefaultTask`. `YamlTask` is another example of `Task` concrete implementation, its source code is available for reference. No matter how you customize the task type, the customized task type must have the following attributes: - - `id`: uniquely identifies the task assigned by the global ID assigner - - `name`: the name of the task - - `predecessor_tasks`: the predecessor tasks of this task - - `action`: is a dynamic type that implements the Action trait in user programming, and it is the specific logic to be executed by the task - -- In addition to yaml-type configuration files, users can also provide other types of configuration files, but in order to allow other types of configuration files to be parsed as tasks, users need to implement the `Parser` trait. `YamlParser` source code is available for reference. - -`examples/custom_parser_and_task.rs` is an example of a custom task type and a custom configuration file parser - -## Analyze the logic of task execution - -**The execution process of Dag is roughly as follows:** - -- The user gives a list of tasks `tasks`. These tasks can be parsed from configuration files, or provided by user programming implementations. - -- Internally generate `Graph` based on task dependencies, and generate execution sequences based on* `rely_graph`. - - ```mermaid - flowchart TD - subgraph tasks - direction LR - A-->B - A-->C - B-->D - B-->F - C-->D - C-->E - D-->F - E-->F - end - subgraph seq - direction LR - a(A)-->b(B)-->c(C)-->d(D)-->e(E)-->f(F) - end - tasks==Generate execution sequence based on topological sort==>seq - ``` - - - -- The task is scheduled to start executing asynchronously. - -- The task will wait to get the result`execute_states`generated by the execution of the predecessor task. - - ```mermaid - --- - title: data flow - --- - flowchart LR - A-->oa((out)) - oa--input-->B - oa--input-->C - B-->ob((out)) - ob--input-->D - ob--input-->F - C-->oc((out)) - oc--input-->D - oc--input-->E - D-->od((out)) - od--input-->F - E-->oe((out)) - oe--input-->F - F-->of((out)) - ``` - - - -- If the result of the predecessor task can be obtained, check the continuation status`can_continue`, if it is true, continue to execute the defined logic, if it is false, trigger`handle_error`, and cancel the execution of the subsequent task. - -- After all tasks are executed, set the continuation status to false, which means that the tasks of the `dag` cannot be scheduled for execution again. - -The task execution mode of `dagrs` is parallel. In the figure, the execution sequence is divided into four intervals by the vertical dividing line. During the overall execution of the task, it will go through four parallel execution stages. As shown in the figure: first task A is executed, and tasks B and C obtain the output of A as the input of their own tasks and start to execute in parallel; similarly, tasks D and E must wait until they obtain the output of their predecessors before starting to execute in parallel; finally, Task F must wait for the execution of tasks B, D, and E to complete before it can start executing. - -```mermaid -gantt - dateFormat X - axisFormat %s - title Execution timing - section Step1 - Task A:0,10 - Task B:0,1 - Task C:0,1 - Task D:0,1 - Task E:0,1 - Task F:0,1 - section Step2 - Task B:10,19 - Task C:10,19 - section Step3 - Task D:19,28 - Task E:19,28 - section Step4 - Task F:28,37 -``` - - - -## The examples - -### Basic function usage - -`examples/compute_dag.rs`: A complete usage example of dagrs. - -`examples/action.rs`: Two ways to define the specific logic of a task. - -`examples/yaml_dag.rs`: Example of reading yaml configuration file (needs to enable `yaml` features). - -`examples/engine.rs`: Using `Engine` to manage multiple dags with different task types. - -### Advanced Features - -`examples/custom_parser_and_task.rs`: Custom task types and configuration file parsers. -`examples/custom_log.rs`: Implement the `Logger` trait to define your own global logger. -`examples/derive_task.rs`:Use `CustomTask` derived macros to help customize task types. -`examples/dependencies.rs`:Use the `dependencies!` macro to specify dependencies in an intuitive way and define a series of tasks. ## Contribution diff --git a/benches/compute_dag_bench.rs b/archived/benches/compute_dag_bench.rs similarity index 100% rename from benches/compute_dag_bench.rs rename to archived/benches/compute_dag_bench.rs diff --git a/benches/profile.bash b/archived/benches/profile.bash old mode 100755 new mode 100644 similarity index 100% rename from benches/profile.bash rename to archived/benches/profile.bash diff --git a/derive/Cargo.toml b/archived/derive/Cargo.toml similarity index 100% rename from derive/Cargo.toml rename to archived/derive/Cargo.toml diff --git a/derive/src/lib.rs b/archived/derive/src/lib.rs similarity index 100% rename from derive/src/lib.rs rename to archived/derive/src/lib.rs diff --git a/derive/src/relay.rs b/archived/derive/src/relay.rs similarity index 100% rename from derive/src/relay.rs rename to archived/derive/src/relay.rs diff --git a/derive/src/task.rs b/archived/derive/src/task.rs similarity index 100% rename from derive/src/task.rs rename to archived/derive/src/task.rs diff --git a/examples/actions.rs b/archived/examples/actions.rs similarity index 100% rename from examples/actions.rs rename to archived/examples/actions.rs diff --git a/examples/compute_dag.rs b/archived/examples/compute_dag.rs similarity index 100% rename from examples/compute_dag.rs rename to archived/examples/compute_dag.rs diff --git a/examples/custom_log.rs b/archived/examples/custom_log.rs similarity index 100% rename from examples/custom_log.rs rename to archived/examples/custom_log.rs diff --git a/examples/custom_parser_and_task.rs b/archived/examples/custom_parser_and_task.rs similarity index 100% rename from examples/custom_parser_and_task.rs rename to archived/examples/custom_parser_and_task.rs diff --git a/examples/dependencies.rs b/archived/examples/dependencies.rs similarity index 100% rename from examples/dependencies.rs rename to archived/examples/dependencies.rs diff --git a/examples/derive_task.rs b/archived/examples/derive_task.rs similarity index 100% rename from examples/derive_task.rs rename to archived/examples/derive_task.rs diff --git a/examples/engine.rs b/archived/examples/engine.rs similarity index 100% rename from examples/engine.rs rename to archived/examples/engine.rs diff --git a/examples/yaml_dag.rs b/archived/examples/yaml_dag.rs similarity index 100% rename from examples/yaml_dag.rs rename to archived/examples/yaml_dag.rs diff --git a/src/bin/dagrs.rs b/archived/src/bin/dagrs.rs similarity index 100% rename from src/bin/dagrs.rs rename to archived/src/bin/dagrs.rs diff --git a/src/engine/dag.rs b/archived/src/engine/dag.rs similarity index 100% rename from src/engine/dag.rs rename to archived/src/engine/dag.rs diff --git a/src/engine/graph.rs b/archived/src/engine/graph.rs similarity index 100% rename from src/engine/graph.rs rename to archived/src/engine/graph.rs diff --git a/src/engine/mod.rs b/archived/src/engine/mod.rs similarity index 100% rename from src/engine/mod.rs rename to archived/src/engine/mod.rs diff --git a/src/lib.rs b/archived/src/lib.rs similarity index 100% rename from src/lib.rs rename to archived/src/lib.rs diff --git a/src/task/action.rs b/archived/src/task/action.rs similarity index 100% rename from src/task/action.rs rename to archived/src/task/action.rs diff --git a/src/task/cmd.rs b/archived/src/task/cmd.rs similarity index 100% rename from src/task/cmd.rs rename to archived/src/task/cmd.rs diff --git a/src/task/default_task.rs b/archived/src/task/default_task.rs similarity index 100% rename from src/task/default_task.rs rename to archived/src/task/default_task.rs diff --git a/src/task/mod.rs b/archived/src/task/mod.rs similarity index 100% rename from src/task/mod.rs rename to archived/src/task/mod.rs diff --git a/src/task/state.rs b/archived/src/task/state.rs similarity index 100% rename from src/task/state.rs rename to archived/src/task/state.rs diff --git a/src/utils/env.rs b/archived/src/utils/env.rs similarity index 100% rename from src/utils/env.rs rename to archived/src/utils/env.rs diff --git a/src/utils/file.rs b/archived/src/utils/file.rs similarity index 100% rename from src/utils/file.rs rename to archived/src/utils/file.rs diff --git a/src/utils/mod.rs b/archived/src/utils/mod.rs similarity index 100% rename from src/utils/mod.rs rename to archived/src/utils/mod.rs diff --git a/src/utils/parser.rs b/archived/src/utils/parser.rs similarity index 100% rename from src/utils/parser.rs rename to archived/src/utils/parser.rs diff --git a/src/yaml/mod.rs b/archived/src/yaml/mod.rs similarity index 100% rename from src/yaml/mod.rs rename to archived/src/yaml/mod.rs diff --git a/src/yaml/yaml_parser.rs b/archived/src/yaml/yaml_parser.rs similarity index 100% rename from src/yaml/yaml_parser.rs rename to archived/src/yaml/yaml_parser.rs diff --git a/src/yaml/yaml_task.rs b/archived/src/yaml/yaml_task.rs similarity index 100% rename from src/yaml/yaml_task.rs rename to archived/src/yaml/yaml_task.rs diff --git a/tests/config/correct.yaml b/archived/tests/config/correct.yaml similarity index 100% rename from tests/config/correct.yaml rename to archived/tests/config/correct.yaml diff --git a/tests/config/custom_file_task.txt b/archived/tests/config/custom_file_task.txt similarity index 100% rename from tests/config/custom_file_task.txt rename to archived/tests/config/custom_file_task.txt diff --git a/tests/config/empty_file.yaml b/archived/tests/config/empty_file.yaml similarity index 100% rename from tests/config/empty_file.yaml rename to archived/tests/config/empty_file.yaml diff --git a/tests/config/illegal_content.yaml b/archived/tests/config/illegal_content.yaml similarity index 100% rename from tests/config/illegal_content.yaml rename to archived/tests/config/illegal_content.yaml diff --git a/tests/config/loop_error.yaml b/archived/tests/config/loop_error.yaml similarity index 100% rename from tests/config/loop_error.yaml rename to archived/tests/config/loop_error.yaml diff --git a/tests/config/no_run.yaml b/archived/tests/config/no_run.yaml similarity index 100% rename from tests/config/no_run.yaml rename to archived/tests/config/no_run.yaml diff --git a/tests/config/no_script.yaml b/archived/tests/config/no_script.yaml similarity index 100% rename from tests/config/no_script.yaml rename to archived/tests/config/no_script.yaml diff --git a/tests/config/no_start_with_dagrs.yaml b/archived/tests/config/no_start_with_dagrs.yaml similarity index 100% rename from tests/config/no_start_with_dagrs.yaml rename to archived/tests/config/no_start_with_dagrs.yaml diff --git a/tests/config/no_task_name.yaml b/archived/tests/config/no_task_name.yaml similarity index 100% rename from tests/config/no_task_name.yaml rename to archived/tests/config/no_task_name.yaml diff --git a/tests/config/no_type.yaml b/archived/tests/config/no_type.yaml similarity index 100% rename from tests/config/no_type.yaml rename to archived/tests/config/no_type.yaml diff --git a/tests/config/precursor_not_found.yaml b/archived/tests/config/precursor_not_found.yaml similarity index 100% rename from tests/config/precursor_not_found.yaml rename to archived/tests/config/precursor_not_found.yaml diff --git a/tests/config/script_run_failed.yaml b/archived/tests/config/script_run_failed.yaml similarity index 100% rename from tests/config/script_run_failed.yaml rename to archived/tests/config/script_run_failed.yaml diff --git a/tests/config/self_loop_error.yaml b/archived/tests/config/self_loop_error.yaml similarity index 100% rename from tests/config/self_loop_error.yaml rename to archived/tests/config/self_loop_error.yaml diff --git a/tests/config/test.js b/archived/tests/config/test.js similarity index 100% rename from tests/config/test.js rename to archived/tests/config/test.js diff --git a/tests/config/test.py b/archived/tests/config/test.py similarity index 100% rename from tests/config/test.py rename to archived/tests/config/test.py diff --git a/tests/config/test.sh b/archived/tests/config/test.sh similarity index 100% rename from tests/config/test.sh rename to archived/tests/config/test.sh diff --git a/tests/dag_job_test.rs b/archived/tests/dag_job_test.rs similarity index 100% rename from tests/dag_job_test.rs rename to archived/tests/dag_job_test.rs diff --git a/tests/env_test.rs b/archived/tests/env_test.rs similarity index 100% rename from tests/env_test.rs rename to archived/tests/env_test.rs diff --git a/tests/yaml_parser_test.rs b/archived/tests/yaml_parser_test.rs similarity index 100% rename from tests/yaml_parser_test.rs rename to archived/tests/yaml_parser_test.rs diff --git a/docs/upgrade.md b/docs/upgrade.md new file mode 100644 index 0000000..aac4a37 --- /dev/null +++ b/docs/upgrade.md @@ -0,0 +1,58 @@ +# Upgrade Plan + +Our Goals: + +- Encapsulated processes and information packets.(Nov. 2024) +- The external definition of connections.(Oct. 2024) +- Asynchronous.(Nov. 2024) +- Information packets with unique ownership and defined lifetimes.(Rust features) +- Bounded connections with a finite capacity. (tokio) +- Reserve pressure. (tokio) +- Parser.(Nov. or Dec. 2024) + + + +## Encapsulated processes and information packets + +- [ ] Provide a new trait `Node` that defines unique identifiers for nodes, input channels for receiving packets, output channels for sending packets, and an interface to start the workload, replacing the trait `Task` in the old version. +- [ ] Enable asynchronous operations inside and between processes in trait `Action`. +- [ ] Use `Content` as encapsulation of information packet. + +## The external definition of connections + +- [ ] Provide asynchronous channels encapsulating the tokio channels and provide a unified interface. + +## Asynchronous + +- [ ] Provide a struct `Graph`, replacing `Dag` in the old version. + - [ ] Remove field `rely_graph`. + - [ ] Automatically create channels and assign corresponding senders and receivers to the nodes when building the graph. + - [ ] Modify the logic of error-handling: If one node fails, the graph will not stop running, and users can then handle exceptions or errors at a successor node. + +## Bounded connections with a finite capacity + +[tokio]: https://tokio.rs/tokio/tutorial "tokio" + + provides four different channel implementations. + +- `one-shot` is a channel with a single producer and a single consumer. Only one value can be sent at the same time. + +- `mpsc` is a channel that supports multiple producers and a single consumer. It is different from one-shot in that it supports sending multiple messages. + +- `broadcast` is a channel that supports multiple producers and multiple consumers. Multiple values can be sent. + +- `watch` is a variant of broadcast. The receiver can only see whether the latest value has changed. + +We pick `mpsc` and `broadcast` as the channels used in Dagrs because they meet the requirements for connections in FBP: they are bounded, have finite capacity and support reserve pressure. + +## Reserve pressure + +`tokio` provides congestion handling mechanisms for both `mpsc`and `broadcast` to deal with insufficient channel capacity or the consumer is too slow. + +- `mpsc` will block the sender when the capacity is full. +- `broadcast` will drop the oldest value, and the slow consumers will get an exception the next time it calls the receive function. + +## Parser + +- [ ] Support defining custom nodes via macros. +- [ ] Modify the macro `dependencies!` to add dependencies to the already defined nodes (implemented with the `Node` trait) and return with a `Graph` ready to start. \ No newline at end of file