support custom config

[skyzh]

Now every task has its own configuration, and the mirror-clone program has its global configuration. I suggest the following config scheme.

The final config used by mirror-clone is composed of three parts. The configuration provided in toml format, default config, and the command line configs.

For example, we have config.toml

[global]
io-limit = 16 # only a total of 16 concurrent downloads are allowed
cpu-limit = 16 # only 16 concurrent CPU-bound tasks are allowed
io-thread-pool = 4
cpu-thread-pool = 4

[global.log]
log-format = "json"
log-level = "warning"

[opam]
use-cache = false

[conda]

[[conda.repo]]
name = "anaconda/pkgs/main/win-64"
url = "balahbalah"

And now, we call mirror-clone with the following arguments.

The basic usage of mirror-clone is mirror-clone <task> <base_dir> <config>

mirror-clone --config config.toml conda /data/conda --all-repos # clone all repos specified in config
mirror-clone --config config.toml conda /data/conda --repos=anaconda/pkgs/main/win-64,anaconda/pkgs/main/linux-64 # use pre-defined repo in config
mirror-clone --config config.toml conda /data/conda/pkgs/main/win-64 --url=mirrors.sjtug.sjtu.edu.cn/anaconda/pkgs/main/win-64

Command-line arguments take precedence. For example, we could override use-cache in opam.

mirror-clone --config config.toml opam --use-cache=true # clone all repos specified in config

If we do not specify cpu-thread-pool in both config.toml and command-line arguments, mirror-clone will use its default value specified in program.