Command Line Interface for IBM Aspera products

Version : 4.8.0.pre

Laurent/2016-2022

This gem provides the ascli Command Line Interface to IBM Aspera software.

ascli is a also great tool to learn Aspera APIs.

Ruby Gem: https://rubygems.org/gems/aspera-cli

Ruby Doc: https://www.rubydoc.info/gems/aspera-cli

Minimum required Ruby version: >= 2.4. Deprecation notice: the minimum will be 2.5 in a future version.

Aspera APIs on IBM developer Link 2

When to use and when not to use

ascli is designed to be used as a command line tool to:

execute commands on Aspera products
transfer to/from Aspera products

So it is designed for:

Interactive operations on a text terminal (typically, VT100 compatible)
Batch operations in (shell) scripts (e.g. cron job)

ascli can be seen as a command line tool integrating:

a configuration file (config.yaml)
advanced command line options
cURL (for REST calls)
Aspera transfer (ascp)

One might be tempted to use it as an integration element, e.g. by building a command line programmatically, and then executing it. It is generally not a good idea. For such integration cases, e.g. performing operations and transfer to aspera products, it is preferred to use Aspera APIs:

Product APIs (REST) : e.g. AoC, Faspex, node
Transfer SDK : with gRPC interface and language stubs (C, C++, Python, .NET/C#, java, ruby, etc...)

Using APIs (application REST API and transfer SDK) will prove to be easier to develop and maintain.

For scripting and ad'hoc command line operations, ascli is perfect.

Notations, Shell and Command line parsing

In examples, command line operations are shown using a shell such: bash or zsh.

Command line parameters in examples beginning with my_, like my_param_value are user-provided value and not fixed value commands.

ascli is typically executed in a shell, either interactively or in a script. ascli receives its arguments from this shell.

On Linux and Unix environments, this is typically a POSIX shell (bash, zsh, ksh, sh). In this environment shell command line parsing applies before ascli (Ruby) is executed, e.g. bash shell operation. Ruby receives a list parameters and gives it to ascli. So special character handling (quotes, spaces, env vars, ...) is done in the shell.

On Windows, cmd.exe is typically used. Windows process creation does not receive the list of arguments but just the whole line. It's up to the program to parse arguments. Ruby follows the Microsoft C/C++ parameter parsing rules.

In case of doubt of argument values after parsing test like this:

ascli conf echo "Hello World" arg2 3

"Hello World"
ERROR: Argument: unprocessed values: ["arg2", "3"]

echo displays the value of the first argument using ruby syntax (strings get double quotes) after command line parsing (shell) and extended value parsing (ascli), next command line arguments are shown in the error message.

Quick Start

This section guides you from installation, first use and advanced use.

First, follow the section: Installation (Ruby, Gem, FASP) to start using ascli.

Once the gem is installed, ascli shall be accessible:

ascli --version

4.8.0.pre

First use

Once installation is completed, you can proceed to the first use with a demo server:

If you want to test with Aspera on Cloud, jump to section: Wizard

To test with Aspera demo transfer server, setup the environment and then test:

ascli config initdemo

ascli server browse /

:............:...........:......:........:...........................:.......................:
:   zmode    :   zuid    : zgid :  size  :           mtime           :         name          :
:............:...........:......:........:...........................:.......................:
: dr-xr-xr-x : asperaweb : fasp : 4096   : 2014-04-10 19:44:05 +0200 : aspera-test-dir-tiny  :
: drwxr-xr-x : asperaweb : fasp : 176128 : 2018-03-15 12:20:10 +0100 : Upload                :
: dr-xr-xr-x : asperaweb : fasp : 4096   : 2015-04-01 00:37:22 +0200 : aspera-test-dir-small :
: dr-xr-xr-x : asperaweb : fasp : 4096   : 2018-05-04 14:26:55 +0200 : aspera-test-dir-large :
:............:...........:......:........:...........................:.......................:

If you want to use ascli with another server, and in order to make further calls more convenient, it is advised to define a option preset for the server's authentication options. The following example will:

create a option preset
define it as default for server plugin
list files in a folder
download a file

ascli config preset update myserver --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_pass_here_

updated: myserver

ascli config preset set default server myserver

updated: default&rarr;server to myserver

ascli server browse /aspera-test-dir-large

:............:...........:......:..............:...........................:............................:
:   zmode    :   zuid    : zgid :     size     :           mtime           :            name            :
:............:...........:......:..............:...........................:............................:
: -rw-rw-rw- : asperaweb : fasp : 10133504     : 2018-05-04 14:16:24 +0200 : ctl_female_2.fastq.partial :
: -rw-r--r-- : asperaweb : fasp : 209715200    : 2014-04-10 19:49:27 +0200 : 200MB                      :
: -rw-r--r-- : asperaweb : fasp : 524288000    : 2014-04-10 19:44:15 +0200 : 500MB                      :
: -rw-r--r-- : asperaweb : fasp : 5368709120   : 2014-04-10 19:45:52 +0200 : 5GB                        :
: -rw-r--r-- : asperaweb : fasp : 500000000000 : 2017-06-14 20:09:57 +0200 : 500GB                      :
: -rw-rw-rw- : asperaweb : fasp : 13606912     : 2018-05-04 14:20:21 +0200 : ctl_male_2.fastq.partial   :
: -rw-rw-rw- : asperaweb : fasp : 76           : 2018-05-04 14:13:18 +0200 : ctl_female_2.fastq.haspx   :
: -rw-rw-rw- : asperaweb : fasp : 647348       : 2018-05-04 14:26:39 +0200 : ctl_female_2.gz            :
: -rw-rw-rw- : asperaweb : fasp : 74           : 2018-05-04 14:16:00 +0200 : ctl_male_2.fastq.haspx     :
: -rw-r--r-- : asperaweb : fasp : 1048576000   : 2014-04-10 19:49:23 +0200 : 1GB                        :
: -rw-r--r-- : asperaweb : fasp : 104857600    : 2014-04-10 19:49:29 +0200 : 100MB                      :
: -rw-r--r-- : asperaweb : fasp : 10737418240  : 2014-04-10 19:49:04 +0200 : 10GB                       :
:............:...........:......:..............:...........................:............................:

ascli server download /aspera-test-dir-large/200MB

Time: 00:00:02 ========================================================================================================== 100% 100 Mbps Time: 00:00:00
complete

Going further

Get familiar with configuration, options, commands : Command Line Interface.

Then, follow the section relative to the product you want to interact with ( Aspera on Cloud, Faspex, ...) : Application Plugins

Installation

It is possible to install either directly on the host operating system (Linux, Windows, macOS) or as a docker container.

The direct installation is recommended and consists in installing:

Ruby (version: >= 2.4. Deprecation notice: the minimum will be 2.5 in a future version)
aspera-cli
Aspera SDK (ascp)

The following sections provide information on the various installation methods.

An internet connection is required for the installation. If you don't have internet for the installation, refer to section Installation without internet access.

Docker container

Use this method only if you know what you do, else use the standard recommended method as described here above.

This method installs a docker image that contains: Ruby, ascli and the FASP sdk.

The image is: https://hub.docker.com/r/martinlaurent/ascli

Ensure that you have Docker installed.

docker --version

Download the wrapping script:

curl -o ascli https://raw.githubusercontent.com/IBM/aspera-cli/develop/bin/dascli

chmod a+x ascli

Install the container image:

./ascli install

Start using it !

Note that the tool is run in the container, so transfers are also executed in the container, not calling host.

The wrapping script maps the container folder /usr/src/app/config to configuration folder $HOME/.aspera/ascli on host.

To transfer to/from the native host, you will need to map a volume in docker or use the config folder (already mapped). To add local storage as a volume edit the script: ascli and add a --volume stanza.

Ruby

Use this method to install on the native host.

A ruby interpreter is required to run the tool or to use the gem and tool.

Required Ruby version: >= 2.4. Deprecation notice: the minimum will be 2.5 in a future version.

Ruby can be installed using any method : rpm, yum, dnf, rvm, brew, windows installer, ... .

Refer to the following sections for a proposed method for specific operating systems.

The recommended installation method is rvm for systems with "bash-like" shell (Linux, macOS, Windows with cygwin, etc...). If the generic install is not suitable (e.g. Windows, no cygwin), you can use one of OS-specific install method. If you have a simpler better way to install Ruby : use it ! (version: >= 2.4. Deprecation notice: the minimum will be 2.5 in a future version)

Generic: RVM: single user installation (not root)

Use this method which provides more flexibility.

Install "rvm": follow https://rvm.io/ :

Install the 2 keys

gpg2 --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB

Execute the shell/curl command. As regular user, it install in the user's home: ~/.rvm .

\curl -sSL https://get.rvm.io | bash -s stable

If you keep the same terminal (not needed if re-login):

source ~/.rvm/scripts/rvm

It is advised to get one of the pre-compiled ruby version, you can list with:

rvm list --remote

Install the chosen pre-compiled Ruby version:

rvm install 2.7.2 --binary

Ruby is now installed for the user, go on to Gem installation.

Generic: RVM: global installation (as root)

Follow the same method as single user install, but execute as "root".

As root, it installs by default in /usr/local/rvm for all users and creates /etc/profile.d/rvm.sh. One can install in another location with :

curl -sSL https://get.rvm.io | bash -s -- --path /usr/local

As root, make sure this will not collide with other application using Ruby (e.g. Faspex). If so, one can rename the login script: mv /etc/profile.d/rvm.sh /etc/profile.d/rvm.sh.ok. To activate ruby (and ascli) later, source it:

source /etc/profile.d/rvm.sh.ok

rvm version

Windows: Installer

Install Latest stable Ruby using https://rubyinstaller.org/ :

Go to "Downloads".
Select the Ruby 2 version "without devkit", x64 corresponding to the one recommended "with devkit". Devkit is not needed.
At the end of the installer uncheck the box to skip the installation of "MSys2": not needed.

macOS: pre-installed or `brew`

macOS 10.13+ (High Sierra) comes with a recent Ruby. So you can use it directly. You will need to install aspera-cli using sudo :

sudo gem install aspera-cli --pre

Alternatively, if you use Homebrew already you can install Ruby with it:

brew install ruby

Linux: package

If your Linux distribution provides a standard ruby package, you can use it provided that the version is compatible (check at beginning of section).

Example:

yum install -y ruby rubygems ruby-json

One can cleanup the whole yum-installed ruby environment like this to uninstall:

gem uninstall $(ls $(gem env gemdir)/gems/|sed -e 's/-[^-]*$//'|sort -u)

yum remove -y ruby ruby-libs

Other Unixes (AIX)

Ruby is sometimes made available as installable package through third party providers. For example for AIX, one can look at:

https://www.ibm.com/support/pages/aix-toolbox-open-source-software-downloads-alpha#R

If your Unix does not provide a pre-built ruby, you can get it using one of those methods.

For instance to build from source, and install in /opt/ruby :

wget https://cache.ruby-lang.org/pub/ruby/2.7/ruby-2.7.2.tar.gz

gzip -d ruby-2.7.2.tar.gz

tar xvf ruby-2.7.2.tar

cd ruby-2.7.2

./configure --prefix=/opt/ruby

make ruby.imp

make

make install

If you already have a Java JVM on your system (java), it is possible to use jruby:

https://www.jruby.org/download

Note that using jruby the startup time is longer than the native ruby, but transfer speed is not impacted (executed by ascp binary).

`aspera-cli` gem

Once you have Ruby and rights to install gems: Install the gem and its dependencies:

gem install aspera-cli --pre

To upgrade to the latest version:

gem update aspera-cli

ascli checks every week if a new version is available and notify the user in a WARN log. To de-activate this feature set the option version_check_days to 0, or specify a different period in days.

To check manually:

ascli conf check_update

FASP Protocol

Most file transfers will be done using the FASP protocol, using ascp. Only two additional files are required to perform an Aspera Transfer, which are part of Aspera SDK:

ascp
aspera-license (in same folder, or ../etc)

This can be installed either be installing an Aspera transfer software, or using an embedded command:

ascli conf ascp install

If a local SDK installation is preferred instead of fetching from internet: one can specify the location of the SDK file:

curl -Lso SDK.zip https://ibm.biz/aspera_sdk

ascli conf ascp install --sdk-url=file:///SDK.zip

The format is: file:///<path>, where <path> can be either a relative path (not starting with /), or an absolute path.

If the embedded method is not used, the following packages are also suitable:

IBM Aspera Connect Client (Free)
IBM Aspera Desktop Client (Free)
IBM Aspera CLI (Free)
IBM Aspera High Speed Transfer Server (Licensed)
IBM Aspera High Speed Transfer EndPoint (Licensed)

For instance, Aspera Connect Client can be installed by visiting the page: https://www.ibm.com/aspera/connect/.

ascli will detect most of Aspera transfer products in standard locations and use the first one found. Refer to section FASP for details on how to select a client or set path to the FASP protocol.

Several methods are provided to start a transfer. Use of a local client (direct transfer agent) is one of them, but other methods are available. Refer to section: Transfer Agents

Installation in air gapped environment

Note that currently no pre-packaged version exist yet. A method to build one is provided here:

The procedure:

Follow the non-root installation procedure with RVM, including gem
Archive (zip, tar) the main RVM folder (includes ascli):

cd $HOME && tar zcvf rvm-ascli.tgz .rvm

Get the Aspera SDK.

ascli conf --show-config --fields=sdk_url

Download the SDK archive from that URL.

curl -Lso SDK.zip https://ibm.biz/aspera_sdk

Transfer those 2 files to the target system
On target system

cd $HOME

tar zxvf rvm-ascli.tgz

source ~/.rvm/scripts/rvm

ascli conf ascp install --sdk-url=file:///SDK.zip

Add those lines to shell init (.profile)

source ~/.rvm/scripts/rvm

Command Line Interface: `ascli`

The aspera-cli Gem provides a command line interface (CLI) which interacts with Aspera Products (mostly using REST APIs):

IBM Aspera High Speed Transfer Server (FASP and Node)
IBM Aspera on Cloud (including ATS)
IBM Aspera Faspex
IBM Aspera Shares
IBM Aspera Console
IBM Aspera Orchestrator
and more...

ascli provides the following features:

Supports most Aspera server products (on-premise and SaaS)
Any command line options (products URL, credentials or any option) can be provided on command line, in configuration file, in env var, in files
Supports Commands, Option values and Parameters shortcuts
FASP Transfer Agents can be: local ascp, or Connect Client, or any transfer node
Transfer parameters can be altered by modification of transfer-spec, this includes requiring multi-session
Allows transfers from products to products, essentially at node level (using the node transfer agent)
Supports FaspStream creation (using Node API)
Supports Watchfolder creation (using Node API)
Additional command plugins can be written by the user
Supports download of faspex and Aspera on Cloud "external" links
Supports "legacy" ssh based FASP transfers and remote commands (ascmd)

Basic usage is displayed by executing:

ascli -h

Refer to sections: Usage and Sample Commands.

Not all ascli features are fully documented here, the user may explore commands on the command line.

Arguments : Commands and options

Arguments are the units of command line, as parsed by the shell, typically separated by spaces (and called "argv").

There are two types of command line arguments: Commands and Options. Example :

ascli command subcommand --option-name=VAL1 VAL2

executes command: command subcommand
with one option: option_name
this option is given a value of: VAL1
the command has one additional argument: VAL2

When the value of a command, option or argument is constrained by a fixed list of values, it is possible to use the first letters of the value only, provided that it uniquely identifies a value. For example ascli conf ov is the same as ascli config overview.

The value of options and arguments is evaluated with the Extended Value Syntax.

Options

All options, e.g. --log-level=debug, are command line arguments that:

start with --
have a name, in lowercase, using - as word separator in name (e.g. --log-level=debug)
have a value, separated from name with a =
can be used by prefix, provided that it is unique. E.g. --log-l=debug is the same as --log-level=debug

Exceptions:

some options accept a short form, e.g. -Ptoto is equivalent to --preset=toto, refer to the manual or -h.
some options (flags) don't take a value, e.g. -r
the special option -- stops option processing and is ignored, following command line arguments are taken as arguments, including the ones starting with a -. Example:

ascli config echo -- --sample

"--sample"

Note that here, --sample is taken as an argument, and not as an option, due to --.

Options can be optional or mandatory, with or without (hardcoded) default value. Options can be placed anywhere on command line and evaluated in order.

The value for any options can come from the following locations (in this order, last value evaluated overrides previous value):

Configuration file.
Environment variable
Command line

Environment variable starting with prefix: ASCLI_ are taken as option values, e.g. ASCLI_OPTION_NAME is for --option-name.

Options values can be displayed for a given command by providing the --show-config option: ascli node --show-config

Commands and Arguments

Command line arguments that are not options are either commands or arguments. If an argument must begin with -, then either use the @val: syntax (see Extended Values), or use the -- separator (see above).

Interactive Input

Some options and parameters are mandatory and other optional. By default, the tool will ask for missing mandatory options or parameters for interactive execution.

The behavior can be controlled with:

--interactive=<yes|no> (default=yes if STDIN is a terminal, else no)
- yes : missing mandatory parameters/options are asked to the user
- no : missing mandatory parameters/options raise an error message
--ask-options=<yes|no> (default=no)
- optional parameters/options are asked to user

Output

Command execution will result in output (terminal, stdout/stderr). The information displayed depends on the action.

Types of output data

Depending on action, the output will contain:

single_object : displayed as a 2 dimensional table: one line per attribute, first column is attribute name, and second is attribute value. Nested hashes are collapsed.
object_list : displayed as a 2 dimensional table: one line per item, one column per attribute.
value_list : a table with one column.
empty : nothing
status : a message
other_struct : a complex structure that cannot be displayed as an array

Format of output

By default, result of type single_object and object_list are displayed using format table. The table style can be customized with parameter: table_style (horizontal, vertical and intersection characters) and is :.: by default.

In a table format, when displaying "objects" (single, or list), by default, sub object are flattened (option flat_hash). So, object {"user":{"id":1,"name":"toto"}} will have attributes: user.id and user.name. Setting flat_hash to false will only display one field: "user" and value is the sub hash table. When in flatten mode, it is possible to filter fields by "dotted" field name.

Object lists are displayed one per line, with attributes as columns. Single objects are transposed: one attribute per line. If transposition of single object is not desired, use option: transpose_single set to no.

The style of output can be set using the format parameter, supporting:

table : Text table
ruby : Ruby code
json : JSON code
jsonpp : JSON pretty printed
yaml : YAML
csv : Comma Separated Values

Option: `select`: Filter on columns values for `object_list`

Table output can be filtered using the select parameter. Example:

ascli aoc admin res user list --fields=name,email,ats_admin --query=@json:'{"sort":"name"}' --select=@json:'{"ats_admin":true}'

:...............................:..................................:...........:
:             name              :              email               : ats_admin :
:...............................:..................................:...........:
: John Curtis                   : [email protected]                 : true      :
: Laurent Martin                : [email protected]              : true      :
:...............................:..................................:...........:

Note that select filters selected elements from the result of API calls, while the query parameters gives filtering parameters to the API when listing elements.

Verbosity of output

Output messages are categorized in 3 types:

info output contain additional information, such as number of elements in a table
data output contain the actual output of the command (object, or list of objects)
erroroutput contain error messages

The option display controls the level of output:

info displays all messages
data display data and error messages
error display only error messages.

By default, secrets are shown on output. To hide secrets from results, set option show_secrets to no.

Selection of output object properties

By default, a table output will display one line per entry, and columns for each entries. Depending on the command, columns may include by default all properties, or only some selected properties. It is possible to define specific columns to be displayed, by setting the fields option to one of the following value:

DEF : default display of columns (that's the default, when not set)
ALL : all columns available
a,b,c : the list of attributes specified by the comma separated list
Array extended value: for instance, @json:'["a","b","c"]' same as above
+a,b,c : add selected properties to the default selection.
-a,b,c : remove selected properties from the default selection.

Extended Value Syntax

Usually, values of options and arguments are specified by a simple string. But sometime it is convenient to read a value from a file, or decode it, or have a value more complex than a string (e.g. Hash table).

The extended value syntax is:

<0 or more decoders><0 or 1 reader><nothing or some text value>

The difference between reader and decoder is order and ordinality. Both act like a function of value on right hand side. Decoders are at the beginning of the value, followed by a single optional reader, followed by the optional value.

The following "readers" are supported (returns value in []):

@val:VALUE : [String] prevent further special prefix processing, e.g. --username=@val:laurent sets the option username to value laurent.
@file:PATH : [String] read value from a URL, e.g. --fpac=@uri:http://serv/f.pac
@uri:URL : [String] read value from a file (prefix ~/ is replaced with the users home folder), e.g. --key=@file:~/.ssh/mykey
@path:PATH : [String] performs path expansion (prefix ~/ is replaced with the users home folder), e.g. --config-file=@path:~/sample_config.yml
@env:ENVVAR : [String] read from a named env var, e.g.--password=@env:MYPASSVAR
@stdin: : [String] read from stdin (no value on right)
@preset:NAME : [Hash] get whole option preset value by name. Subvalues can also be used using . as separator. e.g. foo.bar is conf[foo][bar]

In addition it is possible to decode a value, using one or multiple decoders :

@base64: [String] decode a base64 encoded string
@json: [any] decode JSON values (convenient to provide complex structures)
@zlib: [String] uncompress data
@ruby: [any] execute ruby code
@csvt: [Array] decode a titled CSV value
@lines: [Array] split a string in multiple lines and return an array
@list: [Array] split a string in multiple items taking first character as separator and return an array
@incps: [Hash] include values of presets specified by key incps in input hash

To display the result of an extended value, use the config echo command.

Example: read the content of the specified file, then, base64 decode, then unzip:

ascli config echo @zlib:@base64:@file:myfile.dat

Example: create a value as a hash, with one key and the value is read from a file:

ascli config echo @ruby:'{"token_verification_key"=>File.read("pubkey.txt")}'

Example: read a csv file and create a list of hash for bulk provisioning:

cat test.csv

name,email
lolo,[email protected]
toto,[email protected]

ascli config echo @csvt:@file:test.csv

:......:.....................:
: name :        email        :
:......:.....................:
: lolo : [email protected] :
: toto : [email protected]      :
:......:.....................:

Example: create a hash and include values from preset named "config" of config file in this hash

ascli config echo @incps:@json:'{"hello":true,"incps":["config"]}'

{"version"=>"0.9", "hello"=>true}

Note that @incps:@json:'{"incps":["config"]}' or @incps:@ruby:'{"incps"=>["config"]}' is equivalent to: @preset:config

Structured Value

Some options and parameters expect a Structured Value, i.e. a value more complex than a simple string. This is usually a Hash table or an Array, which could also contain sub structures.

For instance, a transfer-spec is expected to be a Structured Value.

Structured values shall be described using the Extended Value Syntax. A convenient way to specify a Structured Value is to use the @json: decoder, and describe the value in JSON format. The @ruby: decoder can also be used. For an array of hash tables, the @csvt: decoder can be used.

It is also possible to provide a Structured Value in a file using @json:@file:<path>

Configuration and Persistency Folder

ascli configuration and other runtime files (token cache, file lists, persistency files, SDK) are stored [config folder]: [User's home folder]/.aspera/ascli.

Note: [User's home folder] is found using ruby's Dir.home (rb_w32_home_dir). It uses the HOME env var primarily, and on MS Windows it also looks at %HOMEDRIVE%%HOMEPATH% and %USERPROFILE%. ascli sets the env var %HOME% to the value of %USERPROFILE% if set and exists. So, on Windows %USERPROFILE% is used as it is more reliable than %HOMEDRIVE%%HOMEPATH%.

The [config folder] can be displayed using :

ascli config folder

/Users/kenji/.aspera/ascli

It can be overridden using the environment variable ASCLI_HOME.

Example (Windows):

set ASCLI_HOME=C:\Users\Kenji\.aspera\ascli

ascli config folder

C:\Users\Kenji\.aspera\ascli

When OAuth is used (AoC, Faspex4 apiv4, Faspex5) ascli keeps a cache of generated bearer tokens in [config folder]/persist_store by default. Option cache_tokens (yes/no) allows to control if Oauth tokens are cached on file system, or generated for each request. The command config flush_tokens deletes all existing tokens. Tokens are kept on disk for a maximum of 30 minutes (TOKEN_CACHE_EXPIRY_SEC) and garbage collected after that. Tokens that can be refreshed are refreshed. Else tokens are re-generated if expired.

Configuration file

On the first execution of ascli, an empty configuration file is created in the configuration folder. Nevertheless, there is no mandatory information required in this file, the use of it is optional as any option can be provided on the command line.

Although the file is a standard YAML file, ascli provides commands to read and modify it using the config command.

All options for ascli can be set on command line, or by env vars, or using option presets in the configuration file.

A configuration file provides a way to define default values, especially for authentication parameters, thus avoiding to always having to specify those parameters on the command line.

The default configuration file is: $HOME/.aspera/ascli/config.yaml (this can be overridden with option --config-file=path or equivalent env var).

The configuration file is simply a catalog of pre-defined lists of options, called: option presets. Then, instead of specifying some common options on the command line (e.g. address, credentials), it is possible to invoke the ones of a option preset (e.g. mypreset) using the option: -Pmypreset or --preset=mypreset.

Option preset

A option preset is simply a collection of parameters and their associated values in a named section in the configuration file.

A named option preset can be modified directly using ascli, which will update the configuration file :

ascli config preset set|delete|show|initialize|update <option preset>

The command update allows the easy creation of option preset by simply providing the options in their command line format, e.g. :

ascli config preset update demo_server --url=ssh://demo.asperasoft.com:33001 --username=asperaweb --password=_pass_here_ --ts=@json:'{"precalculate_job_size":true}'

This creates a option preset demo_server with all provided options.

The command set allows setting individual options in a option preset.

ascli config preset set demo_server password _pass_here_

The command initialize, like update allows to set several parameters at once, but it deletes an existing configuration instead of updating it, and expects a Structured Value.

ascli config preset initialize demo_server @json:'{"url":"ssh://demo.asperasoft.com:33001","username":"asperaweb","password":"_pass_here_","ts":{"precalculate_job_size":true}}'

A full terminal based overview of the configuration can be displayed using:

ascli config preset over

A list of option preset can be displayed using:

ascli config preset list

A good practice is to not manually edit the configuration file and use modification commands instead. If necessary, the configuration file can opened in a text editor with:

ascli config open

Older format for commands are still supported:

ascli config id <name> set|delete|show|initialize|update
ascli config over
ascli config list

Special Option preset: config

This preset name is reserved and contains a single key: version. This is the version of ascli which created the file.

Special Option preset: default

This preset name is reserved and contains an array of key-value , where the key is the name of a plugin, and the value is the name of another preset.

When a plugin is invoked, the preset associated with the name of the plugin is loaded, unless the option --no-default (or -N) is used.

Note that special plugin name: config can be associated with a preset that is loaded initially, typically used for default values.

Operations on this preset are done using regular config operations:

ascli config preset set default _plugin_name_ _default_preset_for_plugin_

ascli config preset get default _plugin_name_

"_default_preset_for_plugin_"

Special Plugin: config

Plugin config (not to be confused with Option preset config) is used to configure ascli but it also contains global options.

When ascli starts, it looks for the default Option preset and if there is a value for config, if so, it loads the option values for any plugin used.

If no global default is set by the user, the tool will use global_common_defaults when setting global parameters (e.g. conf ascp use)

Format of file

The configuration file is a hash in a YAML file. Example:

config:
  version: 0.3.7
default:
  config: cli_default
  server: demo_server
cli_default:
  interactive: no
demo_server:
  url: ssh://demo.asperasoft.com:33001
  username: asperaweb
  password: _pass_here_

We can see here:

The configuration was created with CLI version 0.3.7
the default option preset to load for server plugin is : demo_server
the option preset demo_server defines some parameters: the URL and credentials
the default option preset to load in any case is : cli_default

Two option presets are reserved:

config contains a single value: version showing the CLI version used to create the configuration file. It is used to check compatibility.
default is reserved to define the default option preset name used for known plugins.

The user may create as many option presets as needed. For instance, a particular option preset can be created for a particular application instance and contain URL and credentials.

Values in the configuration also follow the Extended Value Syntax.

Note: if the user wants to use the Extended Value Syntax inside the configuration file, using the config preset update command, the user shall use the @val: prefix. Example:

ascli config preset set my_aoc_org private_key @val:@file:"$HOME/.aspera/ascli/my_private_key"

This creates the option preset:

...
my_aoc_org:
  private_key: @file:"/Users/laurent/.aspera/ascli/my_private_key"
...

So, the key file will be read only at execution time, but not be embedded in the configuration file.

Options evaluation order

Some options are global, some options are available only for some plugins. (the plugin is the first level command).

Options are loaded using this algorithm:

If option --no-default (or -N) is specified, then no default value is loaded is loaded for the plugin
else it looks for the name of the plugin as key in section default, the value is the name of the default option preset for it, and loads it.
If option --preset=<name or extended value hash> is specified (or -Pxxxx), this reads the option preset specified from the configuration file, or of the value is a Hash, it uses it as options values.
Environment variables are evaluated
Command line options are evaluated

Parameters are evaluated in the order of command line.

To avoid loading the default option preset for a plugin, use: -N

On command line, words in parameter names are separated by a dash, in configuration file, separator is an underscore. E.g. --xxx-yyy on command line gives xxx_yyy in configuration file.

The main plugin name is config, so it is possible to define a default option preset for the main plugin with:

ascli config preset set cli_default interactive no

ascli config preset set default config cli_default

A option preset value can be removed with unset:

ascli config preset unset cli_default interactive

Example: Define options using command line:

ascli -N --url=_url_here_ --password=_pass_here_ --username=_name_here_ node --show-config

Example: Define options using a hash:

ascli -N --preset=@json:'{"url":"_url_here_","password":"_pass_here_","username":"_name_here_"}' node --show-config

Shares Examples

For Faspex, Shares, Node (including ATS, Aspera Transfer Service), Console, only username/password and url are required (either on command line, or from config file). Those can usually be provided on the command line:

ascli shares repo browse / --url=https://10.25.0.6 --username=john --password=_pass_here_

This can also be provisioned in a config file:

Build option preset

ascli config preset set shares06 url https://10.25.0.6
ascli config preset set shares06 username john
ascli config preset set shares06 password _pass_here_

This can also be done with one single command:

ascli config preset init shares06 @json:'{"url":"https://10.25.0.6","username":"john","password":"_pass_here_"}'

or

ascli config preset update shares06 --url=https://10.25.0.6 --username=john --password=_pass_here_

Define this option preset as the default option preset for the specified plugin (shares)

ascli config preset set default shares shares06

Display the content of configuration file in table format

ascli config overview

Execute a command on the shares application using default parameters

ascli shares repo browse /

Secret Vault

When a secret or password is needed, it is possible to store in the secret vault.

By default the vault is defined using option secrets, which can be stored in the configuration file.

Using system keychain

Only on macOS.

It is possible to store secrets in macOS keychain (only read supported currently).

Set option secrets to value system to use the default keychain or use value system:[name] to use a custom keychain.

Modern config file format: encrypted in config file

It is possible to store and use secrets encrypted. For this use the config vault command.

The vault can be initialized with config vault init

Then secrets can be manipulated using commands:

set
get
list
delete

Secrets must be uniquely identified by url and username. An optional description can be provided using option value.

Legacy config file format

The value provided can be a Hash, where keys are usernames (or access key id), and values are the associated password or secrets in clear.

For example, choose a repository name, for example my_secrets, and populate it like this:

ascli conf id my_secrets set 'access_key1' 'secret1'

ascli conf id my_secrets set 'access_key2' 'secret2'

ascli conf id default get config

cli_default

Here above, one has already set a config global preset to preset cli_default (refer to earlier in documentation). So the repository can be read by default like this (note the prefix @val: to avoid the evaluation of prefix @preset:):

ascli conf id cli_default set secrets @val:@preset:my_secrets

A secret repository can always be selected at runtime using --secrets=@preset:xxxx, or --secrets=@json:'{"accesskey1":"secret1"}'

To test if a secret can be found use:

ascli conf vault get --username=access_key1

Private Key

Some applications allow the user to be authenticated using a private key (Server, AoC, Faspex5...). It consists in generating a private key, or using a previouly generated key. The same key can be used for multiple applications. Technically, a private key contains the public key, which can be extracted. Currently, only private key not protected by a passphrase are supported. (TODO: add passphrase protection as option for aspera apps).

Several methods can be used to generate a key pair:

ascli

The generated key is of type RSA 4096 bit. For convenience, the public key is also extracted.

ascli config genkey ~/.aspera/ascli/my_private_key

ssh-keygen

ssh-keygen -t rsa -f ~/.aspera/ascli/my_private_key -N ''

openssl

openssl is sometimes compiled to support option -nodes (no DES, i.e. no passphrase, e.g. on macOS). To generate a privatekey pait without passphrase the following shall work on any system:

APIKEY=~/.aspera/ascli/my_private_key
openssl genrsa -passout pass:dummypassword -out ${APIKEY}.protected 2048
openssl rsa -passin pass:dummypassword -in ${APIKEY}.protected -out ${APIKEY}
openssl rsa -pubout -in ${APIKEY} -out ${APIKEY}.pub
rm -f ${APIKEY}.protected

SSL CA certificate bundle

ascli uses ruby openssl gem, which uses the openssl library. Certificates are checked against the ruby default certificates OpenSSL::X509::DEFAULT_CERT_FILE, which are typically the ones of openssl on Unix systems (Linux, macOS, etc..). The environment variables SSL_CERT_FILE and SSL_CERT_DIR are used if defined.

ascp also needs to validate certificates when using WSS. By default, ascp uses primarily certificates from hard-coded path (e.g. on macOS: /Library/Aspera/ssl). ascli overrides and sets the default ruby certificate path as well for ascp using -i switch. So to update certificates, update ruby's openssl gem, or use env vars SSL_CERT_*.

Plugins

The CLI tool uses a plugin mechanism. The first level command (just after ascli on the command line) is the name of the concerned plugin which will execute the command. Each plugin usually represents commands sent to a specific application. For instance, the plugin faspex allows operations on the application "Aspera Faspex".

Available plugins can be found using command:

ascli conf plugin list

+--------------+--------------------------------------------------------+
| plugin       | path                                                   |
+--------------+--------------------------------------------------------+
| shares       | ..../aspera-cli/lib/aspera/cli/plugins/shares.rb       |
| node         | ..../aspera-cli/lib/aspera/cli/plugins/node.rb         |
...
+--------------+--------------------------------------------------------+

Create your own plugin

By default plugins are looked-up in folders specifed by (multi-value) option plugin_folder:

ascli --show-config --select=@json:'{"key":"plugin_folder"}'

You can create the skeleton of a new plugin like this:

ascli conf plugin create foo .

Created ./foo.rb

ascli --plugin-folder=. foo

Plugins: Application URL and Authentication

ascli comes with several Aspera application plugins.

REST APIs of Aspera legacy applications (Aspera Node, Faspex, Shares, Console, Orchestrator, Server) use simple username/password authentication: HTTP Basic Authentication.

Those are using options:

url
username
password

Those can be provided using command line, parameter set, env var, see section above.

Aspera on Cloud relies on Oauth, refer to the Aspera on Cloud section.

Logging, Debugging

The gem is equipped with traces. By default logging level is warn. To increase debug level, use parameter log_level (e.g. using command line --log-level=xx, env var ASCLI_LOG_LEVEL, or parameter in con file).

It is also possible to activate traces before initialization using env var AS_LOG_LEVEL.

By default passwords and secrets are removed from logs. Use option log_secrets set to yes to reveal secrets in logs.

Learning Aspera Product APIs (REST)

This CLI uses REST APIs. To display HTTP calls, use argument -r or --rest-debug, this is useful to display exact content of HTTP requests and responses.

In order to get traces of execution, use argument : --log-level=debug

HTTP socket parameters

If the server does not provide a valid certificate, use option: --insecure=yes.

Ruby HTTP socket parameters can be adjusted.

parameter	default
`read_timeout`	60
`write_timeout`	60
`open_timeout`	60
`keep_alive_timeout`	2

Values are in set seconds and can be of type either integer or float. Default values are the ones of Ruby. For details refer to the Ruby library: Net::HTTP.

Like any other option, those can be set either on command line, or in config file, either in a global preset or server-specific one.

Example:

ascli aoc admin res package list --http-options=@json:'{"read_timeout":10.0}'

Graphical Interactions: Browser and Text Editor

Some actions may require the use of a graphical tool:

a browser for Aspera on Cloud authentication (web auth method)
a text editor for configuration file edition

By default the CLI will assume that a graphical environment is available on windows, and on other systems, rely on the presence of the "DISPLAY" environment variable. It is also possible to force the graphical mode with option --ui :

--ui=graphical forces a graphical environment, a browser will be opened for URLs or a text editor for file edition.
--ui=text forces a text environment, the URL or file path to open is displayed on terminal.

Proxy

There are several types of network connections, each of them use a different mechanism to define a proxy.

HTTP proxy for REST calls and transfers using HTTP gateway

To specify a HTTP proxy when ruby HTTP is used, set the http_proxy environment variable (lower case, preferred, or upper case). See Ruby findproxy.

export http_proxy=http://myproxy.org.net:3128

Note that ruby expects a URL and myproxy.org.net:3128 alone is not accepted.

FASP proxy (forward) for transfers

To specify a FASP proxy (forward), set the transfer-spec parameter: EX_fasp_proxy_url (only supported with the direct agent).

HTTP proxy legacy Aspera HTTP fallback transfers

To specify a proxy for legacy HTTP fallback, set the transfer-spec parameter: EX_http_proxy_url (only supported with the direct agent). (It is also possible to use EX_ascp_args and native options in direct)

Proxy auto config

The fpac option allows use of a Proxy Auto Configuration (PAC) script defined as javascript value. To read the script from a URL (http:, https: and file:), use: @uri:. A minimal script can be specified, for example like this, to define the use of a local proxy:

export ASCLI_FPAC='function FindProxyForURL(url, host){return "PROXY localhost:3128"}'

The PAC file will be used for any HTTP/HTTPS/REST connection, but not other (e.g. FASP, HTTP fallback, HTTPGW)

The PAC file can be tested with command: config proxy_check. Example, using command line option:

ascli conf proxy_check --fpac='function FindProxyForURL(url, host) {return "PROXY proxy.example.com:1234;DIRECT";}' http://example.com
PROXY proxy.example.com:1234;DIRECT

ascli config proxy_check --fpac=@file:./proxy.pac http://www.example.com
PROXY proxy.example.com:8080

ascli config proxy_check --fpac=@uri:http://server/proxy.pac http://www.example.com
PROXY proxy.example.com:8080

FASP configuration

The config plugin also allows specification for the use of a local FASP client. It provides the following commands for ascp subcommand:

show : shows the path of ascp used
use : list,download connect client versions available on internet
products : list Aspera transfer products available locally
connect : list,download connect client versions available on internet

Show path of currently used `ascp`

ascli config ascp show

/Users/laurent/.aspera/ascli/sdk/ascp

ascli config ascp info

+--------------------+-----------------------------------------------------------+
| key                | value                                                     |
+--------------------+-----------------------------------------------------------+
| ascp               | /Users/laurent/.aspera/ascli/sdk/ascp                     |
...

Selection of `ascp` location for `direct` agent

By default, ascli uses any found local product with ascp, including SDK.

To temporarily use an alternate ascp path use option ascp_path (--ascp-path=)

For a permanent change, the command config ascp use sets the same parameter for the global default.

Using a POSIX shell:

ascli config ascp use @path:'~/Applications/Aspera CLI/bin/ascp'

ascp version: 4.0.0.182279
Updated: global_common_defaults: ascp_path <- /Users/laurent/Applications/Aspera CLI/bin/ascp
Saved to default global preset global_common_defaults

Windows:

ascli config ascp use C:\Users\admin\.aspera\ascli\sdk\ascp.exe

ascp version: 4.0.0.182279
Updated: global_common_defaults: ascp_path <- C:\Users\admin\.aspera\ascli\sdk\ascp.exe
Saved to default global preset global_common_defaults

If the path has spaces, read section: Shell and Command line parsing.

List locally installed Aspera Transfer products

Locally installed Aspera products can be listed with:

ascli config ascp products list

:.........................................:................................................:
:                  name                   :                    app_root                    :
:.........................................:................................................:
: Aspera Connect                          : /Users/laurent/Applications/Aspera Connect.app :
: IBM Aspera CLI                          : /Users/laurent/Applications/Aspera CLI         :
: IBM Aspera High-Speed Transfer Endpoint : /Library/Aspera                                :
: Aspera Drive                            : /Applications/Aspera Drive.app                 :
:.........................................:................................................:

Selection of local client for `ascp` for `direct` agent

If no ascp is selected, this is equivalent to using option: --use-product=FIRST.

Using the option use_product finds the ascp binary of the selected product.

To permanently use the ascp of a product:

ascli config ascp products use 'Aspera Connect'
saved to default global preset /Users/laurent/Applications/Aspera Connect.app/Contents/Resources/ascp

Installation of Connect Client on command line

ascli config ascp connect list

+-----------------------------------------------+--------------------------------------+-----------+
| id                                            | title                                | version   |
+-----------------------------------------------+--------------------------------------+-----------+
| urn:uuid:589F9EE5-0489-4F73-9982-A612FAC70C4E | Aspera Connect for Windows           | 3.11.2.63 |
| urn:uuid:A3820D20-083E-11E2-892E-0800200C9A66 | Aspera Connect for Windows 64-bit    | 3.11.2.63 |
| urn:uuid:589F9EE5-0489-4F73-9982-A612FAC70C4E | Aspera Connect for Windows XP        | 3.11.2.63 |
| urn:uuid:55425020-083E-11E2-892E-0800200C9A66 | Aspera Connect for Windows XP 64-bit | 3.11.2.63 |
| urn:uuid:D8629AD2-6898-4811-A46F-2AF386531BFF | Aspera Connect for Mac Intel         | 3.11.2.63 |
| urn:uuid:97F94DF0-22B1-11E2-81C1-0800200C9A66 | Aspera Connect for Linux 64          | 3.11.2.63 |
+-----------------------------------------------+--------------------------------------+-----------+

ascli config ascp connect version 'Aspera Connect for Mac Intel' list

+-------------------------------------------+--------------------------+-----------------------------------------------------------------------------------------+----------+---------------------+
| title                                     | type                     | href                                                                                    | hreflang | rel                 |
+-------------------------------------------+--------------------------+-----------------------------------------------------------------------------------------+----------+---------------------+
| Mac Intel Installer                       | application/octet-stream | bin/IBMAsperaConnectInstaller-3.11.2.63.dmg                                             | en       | enclosure           |
| Mac Intel Installer                       | application/octet-stream | bin/IBMAsperaConnectInstallerOneClick-3.11.2.63.dmg                                     | en       | enclosure-one-click |
| Aspera Connect for Mac HTML Documentation | text/html                | https://www.ibm.com/docs/en/aspera-connect/3.11?topic=aspera-connect-user-guide-macos   | en       | documentation       |
| Aspera Connect for Mac Release Notes      | text/html                | https://www.ibm.com/docs/en/aspera-connect/3.11?topic=notes-release-aspera-connect-3112 | en       | release-notes       |
+-------------------------------------------+--------------------------+-----------------------------------------------------------------------------------------+----------+---------------------+

ascli config ascp connect version 'Aspera Connect for Mac Intel' download enclosure --to-folder=.

Time: 00:00:02 ======================================================================= 100% 27766 KB/sec Time: 00:00:02
Downloaded: IBMAsperaConnectInstaller-3.11.2.63.dmg

Transfer Agents

Some of the actions on Aspera Applications lead to file transfers (upload and download) using the FASP protocol (ascp).

When a transfer needs to be started, a transfer-spec has been internally prepared. This transfer-spec will be executed by a transfer client, here called "Transfer Agent".

There are currently 3 agents:

direct : a local execution of ascp
connect : use of a local Connect Client
node : use of an Aspera Transfer Node (potentially remote).
httpgw : use of an Aspera HTTP Gateway
trsdk : use of Aspera Transfer SDK

Note that all transfer operation are seen from the point of view of the agent. For instance, a node agent making an "upload", or "package send" operation, will effectively push files to the related server from the agent node.

ascli standardizes on the use of a transfer-spec instead of raw ascp options to provide parameters for a transfer session, as a common method for those three Transfer Agents.

Direct

The direct agent directly executes a local ascp. This is the default for ascli. This is equivalent to specifying --transfer=direct. ascli will detect locally installed Aspera products, including SDK. Refer to section FASP.

The transfer-info accepts the following optional parameters to control multi-session, WSS

Name	Type	Description
wss	Bool	Web Socket Session Enable use of web socket session in case it is available Default: false
spawn_timeout_sec	Float	Multi session Verification time that ascp is running Default: 3
spawn_delay_sec	Float	Multi session Delay between startup of sessions Default: 2
multi_incr_udp	Bool	Multi Session Increment UDP port on multi-session If true, each session will have a different UDP port starting at `fasp_port` (or default 33001) Else, each session will use `fasp_port` (or `ascp` default) Default: true
resume	Hash	Resume parameters See below
resume.iter_max	int	Resume Max number of retry on error Default: 7
resume.sleep_initial	int	Resume First Sleep before retry Default: 2
resume.sleep_factor	int	Resume Multiplier of sleep period between attempts Default: 2
resume.sleep_max	int	Resume Default: 60

Resume: In case of transfer interruption, the agent will resume a transfer up to iter_max time. Sleep between iterations is:

max( sleep_max , sleep_initial * sleep_factor ^ (iter_index-1) )

Some transfer errors are considered "retryable" (e.g. timeout) and some other not (e.g. wrong password).

Examples:

ascli ... --transfer-info=@json:'{"wss":true,"resume":{"iter_max":10}}'
ascli ... --transfer-info=@json:'{"spawn_delay_sec":2.5,"multi_incr_udp":false}'

IBM Aspera Connect Client GUI

By specifying option: --transfer=connect, ascli will start transfers using the locally installed Aspera Connect Client. There are no option for transfer_info.

Aspera Node API : Node to node transfers

By specifying option: --transfer=node, the CLI will start transfers in an Aspera Transfer Server using the Node API, either on a local or remote node. Parameters provided in option transfer_info are:

NameTypeDescription urlstringURL of the node API
Mandatory usernamestringnode api user or access key
Mandatory passwordstringpassword, secret or bearer token
Mandatory root_idstringpassword or secret
Mandatory only for bearer token

Like any other option, transfer_info can get its value from a pre-configured option preset : --transfer-info=@preset:<psetname> or be specified using the extended value syntax : --transfer-info=@json:'{"url":"https://...","username":"theuser","password":"_pass_here_"}'

If transfer_info is not specified and a default node has been configured (name in node for section default) then this node is used by default.

If the password value begins with Bearer then the username is expected to be an access key and the parameter root_id is mandatory and specifies the root file id on the node. It can be either the access key's root file id, or any authorized file id underneath it.

HTTP Gateway

If it possible to send using a HTTP gateway, in case FASP is not allowed. transfer_info shall have a single mandatory parameter: url.

Example:

ascli faspex package recv --id=323 --transfer=httpgw --transfer-info=@json:'{"url":"https://asperagw.example.com:9443/aspera/http-gwy/v1"}'

Note that the gateway only supports transfers authorized with a token.

Transfer SDK

Another possibility is to use the Transfer SDK daemon (asperatransferd).

By default it will listen on local port 55002 on 127.0.0.1.

Transfer Specification

Some commands lead to file transfer (upload/download), all parameters necessary for this transfer is described in a transfer-spec (Transfer Specification), such as:

server address
transfer user name
credentials
file list
etc...

ascli builds a default transfer-spec internally, so it is not necessary to provide additional parameters on the command line for this transfer.

If needed, it is possible to modify or add any of the supported transfer-spec parameter using the ts option. The ts option accepts a Structured Value containing one or several transfer-spec parameters. Multiple ts options on command line are cumulative.

It is possible to specify ascp options when the transfer option is set to direct using the special transfer-spec parameter: EX_ascp_args. Example: --ts=@json:'{"EX_ascp_args":["-l","100m"]}'. This is especially useful for ascp command line parameters not supported yet in the transfer spec.

The use of a transfer-spec instead of ascp parameters has the advantage of:

common to all Transfer Agent
not dependent on command line limitations (special characters...)

A transfer-spec is a Hash table, so it is described on the command line with the Extended Value Syntax.

Transfer Parameters

All standard transfer-spec parameters can be specified. transfer-spec can also be saved/overridden in the config file.

References:

Aspera Node API Documentation→/opt/transfers
Aspera Transfer SDK Documentation→Guides→API Ref→Transfer Spec V1
Aspera Connect SDK → search The parameters for starting a transfer.

Parameters can be displayed with commands:

ascli config ascp spec
ascli config ascp spec --select=@json:'{"d":"Y"}' --fields=-d,n,c

Columns:

D=Direct (local ascp execution)
N=Node API
C=Connect Client

ascp argument or environment variable is provided in description.

Fields with EX_ prefix are extensions to transfer agent direct. (only in ascli).

Field	Type	D	N	C	Description
cipher	string	Y	Y	Y	In transit encryption type. Allowed values: none, aes-128, aes-192, aes-256, aes-128-cfb, aes-192-cfb, aes-256-cfb, aes-128-gcm, aes-192-gcm, aes-256-gcm (-c)
content_protection	string	Y	Y	Y	Enable client-side encryption at rest. (CSEAR, content protection) Allowed values: encrypt, decrypt (--file-crypt)
content_protection_password	string	Y	Y	Y	Specifies CSEAR password. (content protection) (env:ASPERA_SCP_FILEPASS)
cookie	string	Y	Y	Y	Metadata for transfer specified by application (env:ASPERA_SCP_COOKIE)
create_dir	bool	Y	Y	Y	Specifies whether to create new directories. (-d)
delete_before_transfer	bool	Y	Y	Y	Before transfer, delete files that exist at the destination but not at the source. The source and destination arguments must be directories that have matching names. Objects on the destination that have the same name but different type or size as objects on the source are not deleted. (--delete-before-transfer)
delete_source	bool	Y	Y		Remove SRC files after transfer success (--remove-after-transfer)
destination_root	string	Y	Y	Y	Destination root directory.
direction	string	Y	Y	Y	Direction of transfer (on client side) Allowed values: send, receive (--mode)
exclude_newer_than	int	Y			skip src files with mtime > arg (--exclude-newer-than)
exclude_older_than	int	Y			skip src files with mtime < arg (--exclude-older-than)
fasp_port	int	Y	Y	Y	Specifies fasp (UDP) port. (-O)
file_checksum	string	Y	Y		Enable checksum reporting for transferred files by specifying the hash to use. Allowed values: sha-512, sha-384, sha-256, sha1, md5, none
http_fallback	bool string	Y	Y	Y	When true(1), attempts to perform an HTTP transfer if a FASP transfer cannot be performed. (-y)
http_fallback_port	int	Y			Specifies http port when no cipher is used (-t)
https_fallback_port	int	Y	Y	Y	Specifies https port when cipher is used (-t)
move_after_transfer	string	Y	Y	Y	(--move-after-transfer)
multi_session	int	Y	Y	Y	Use multi-session transfer. max 128. Each participant on one host needs an independent UDP (-O) port. Large files are split between sessions only when transferring with resume_policy=none.
multi_session_threshold	int	Y	Y		Split files across multiple ascp sessions if their size in bytes is greater than or equal to the specified value. (0=no file is split) (--multi-session-threshold)
overwrite	string	Y	Y	Y	Overwrite destination files with the source files of the same name. Allowed values: never, always, diff, older, diff+older (--overwrite)
paths	array	Y	Y	Y	Array of path to the source (required) and a path to the destination (optional).
precalculate_job_size	bool	Y	Y	Y	Specifies whether to precalculate the job size. (--precalculate-job-size)
preserve_access_time	bool	Y	Y	Y	(--preserve-access-time)
preserve_creation_time	bool	Y	Y	Y	(--preserve-creation-time)
preserve_modification_time	bool	Y	Y	Y	(--preserve-modification-time)
preserve_times	bool	Y	Y	Y	(--preserve-times)
rate_policy	string	Y	Y	Y	The transfer rate policy to use when sharing bandwidth. Allowed values: low, fair, high, fixed (--policy)
remote_host	string	Y	Y	Y	IP or fully qualified domain name of the remote server (--host)
remote_user	string	Y	Y	Y	Remote user. Default value is "xfer" on node or connect. (--user)
remote_password	string	Y	Y	Y	SSH session password (env:ASPERA_SCP_PASS)
remove_after_transfer	bool	Y	Y		Remove SRC files after transfer success (--remove-after-transfer)
remove_empty_directories	bool	Y	Y		Specifies whether to remove empty directories. (--remove-empty-directories)
remove_skipped	bool	Y	Y	Y	Must also have remove_after_transfer set to true, Defaults to false, if true, skipped files will be removed as well. (--remove-skipped)
proxy	string	Y			Specify the address of the Aspera high-speed proxy server. dnat(s)://[user[:password]@]server:port Default ports for DNAT and DNATS protocols are 9091 and 9092. Password, if specified here, overrides the value of environment variable ASPERA_PROXY_PASS. (--proxy)
resume_policy	string	Y	Y	Y	If a transfer is interrupted or fails to finish, resume without re-transferring the whole files. Allowed values: none, attrs, sparse_csum, full_csum (-k)
retry_duration	string int		Y	Y	Specifies how long to wait before retrying transfer. (e.g. "5min")
source_root_id	string		Y		The file ID of the source root directory. Required when using Bearer token auth for the source node.
ssh_port	int	Y	Y	Y	Specifies SSH (TCP) port. Default: local:22, other:33001 (-P)
ssh_private_key	string	Y			Private key used for SSH authentication. Shall look like: -----BEGIN RSA PRIV4TE KEY-----\nMII... Note the JSON encoding: \n for newlines. (env:ASPERA_SCP_KEY)
ssh_private_key_passphrase	string	Y			The passphrase associated with the transfer user's SSH private key. Available as of 3.7.2. (env:ASPERA_SCP_PASS)
symlink_policy	string	Y	Y	Y	Handle source side symbolic links Allowed values: follow, copy, copy+force, skip (--symbolic-links)
tags	hash	Y	Y	Y	Metadata for transfer as JSON (--tags64)
target_rate_cap_kbps	int			Y	Returned by upload/download_setup node API.
target_rate_kbps	int	Y	Y	Y	Specifies desired speed for the transfer. (-l)
title	string		Y	Y	Title of the transfer
token	string	Y	Y	Y	Authorization token: Bearer, Basic or ATM (Also arg -W) (env:ASPERA_SCP_TOKEN)
use_ascp4	bool	Y	Y		specify version of protocol
source_root	string	Y	Y	Y	Path to be prepended to each source path. This is either a conventional path or it can be a URI but only if there is no root defined. (--source-prefix64)
min_rate_cap_kbps	int	Y	Y	Y
lock_rate_policy	bool	Y	Y	Y
lock_target_rate_kbps	bool	Y	Y	Y
lock_min_rate_kbps	bool	Y	Y	Y
apply_local_docroot	bool	Y			(--apply-local-docroot)
preserve_acls	string	Y			Preserve access control lists. Allowed values: none, native, metafile (--preserve-acls)
preserve_remote_acls	string	Y			Preserve remote access control lists. Allowed values: none, native, metafile (--remote-preserve-acls)
preserve_file_owner_uid	bool	Y			Preserve the user ID for a file owner (--preserve-file-owner-uid)
preserve_file_owner_gid	bool	Y			Preserve the group ID for a file owner (--preserve-file-owner-gid)
preserve_source_access_time	bool	Y			Preserve the time logged for when the source file was accessed (--preserve-source-access-time)
remove_empty_source_directory	bool	Y			Remove empty source subdirectories and remove the source directory itself, if empty (--remove-empty-source-directory)
EX_at_rest_password	string	Y			Passphrase used for at rest encryption or decryption. Prefer to use standard: content_protection_password (env:ASPERA_SCP_FILEPASS)
EX_proxy_password	string	Y			Password used for Aspera proxy server authentication. May be overridden by password in URL EX_fasp_proxy_url. (env:ASPERA_PROXY_PASS)
EX_license_text	string	Y			License file text override. By default ascp looks for license file near executable. (env:ASPERA_SCP_LICENSE)
dgram_size	int	Y	Y	Y	UDP datagram size in bytes (-Z)
min_rate_kbps	int	Y	Y	Y	Set the minimum transfer rate in kilobits per second. (-m)
sshfp	string	Y	Y	Y	Check it against server SSH host key fingerprint (--check-sshfp)
EX_http_proxy_url	string	Y			Specify the proxy server address used by HTTP Fallback (-x)
EX_ssh_key_paths	array	Y			Use public key authentication for SSH and specify the private key file paths (-i)
EX_http_transfer_jpeg	int	Y			HTTP transfers as JPEG file (-j)
EX_no_read	bool	Y			no read source (--no-read)
EX_no_write	bool	Y			no write on destination (--no-write)
target_rate_percentage	string	Y	Y	Y
rate_policy_allowed	string			Y	Specifies most aggressive rate policy that is allowed. Returned by node API. Allowed values: low, fair, high, fixed
lock_min_rate	bool	Y	Y	Y
lock_target_rate	bool	Y	Y	Y
authentication	string			Y	value=token for SSH bypass keys, else password asked if not provided.
cipher_allowed	string	Y	Y	Y	returned by node API. Valid literals include "aes-128" and "none".
EX_file_list	string	Y			source file list
EX_file_pair_list	string	Y			source file pair list
EX_ascp_args	array	Y			Add native command line arguments to ascp
wss_enabled	bool	Y	Y	Y
wss_port	int	Y	Y	Y	TCP port used for websocket service feed.

Destination folder for transfers

The destination folder is set by ascli by default to:

. for downloads
/ for uploads

It is specified by the transfer-spec parameter destination_root. As such, it can be modified with option: --ts=@json:'{"destination_root":"<path>"}'. The option to_folder provides an equivalent and convenient way to change this parameter: --to-folder=<path> .

List of files for transfers

When uploading, downloading or sending files, the user must specify the list of files to transfer. The option to specify the list of files is sources, the default value is @args, which means: take remain non used arguments (not starting with - as list of files. So, by default, the list of files to transfer will be simply specified on the command line:

ascli server upload ~/mysample.file secondfile

This is equivalent to:

ascli server upload --sources=@args ~/mysample.file secondfile

More advanced options are provided to adapt to various cases. In fact, list of files to transfer are normally conveyed using the transfer-spec using the field: "paths" which is a list (array) of pairs of "source" (mandatory) and "destination" (optional).

Note that this is different from the "ascp" command line. The paradigm used by ascli is: all transfer parameters are kept in transfer-spec so that execution of a transfer is independent of the transfer agent. Note that other IBM Aspera interfaces use this: connect, node, transfer sdk.

For ease of use and flexibility, the list of files to transfer is specified by the option sources. Accepted values are:

@args : (default value) the list of files is directly provided at the end of the command line (see at the beginning of this section).
an Extended Value holding an Array of String. Examples:

--sources=@json:'["file1","file2"]'

--sources=@lines:@stdin:

--sources=@ruby:'File.read("myfilelist").split("\n")'

@ts : the user provides the list of files directly in the ts option, in its paths field. Example:

--sources=@ts --ts=@json:'{"paths":[{"source":"file1"},{"source":"file2"}]}'

providing a file list directly to ascp:

... --sources=@ts --ts=@json:'{"paths":[],"EX_file_list":"filelist.txt"}'

Not recommended: It is possible to specify bare ascp arguments using the pseudo transfer-spec parameter EX_ascp_args.

--sources=@ts --ts=@json:'{"paths":[{"source":"dummy"}],"EX_ascp_args":["--file-list","myfilelist"]}'

This method avoids creating a copy of the file list, but has drawbacks: it applies only to the direct transfer agent (i.e. bare ascp) and not for Aspera on Cloud. One must specify a dummy list in the transfer-spec, which will be overridden by the bare ascp command line provided. (TODO) In next version, dummy source paths can be removed.

In case the file list is provided on the command line i.e. using --sources=@args or --sources=<Array> (but not --sources=@ts), then the list of files will be used either as a simple file list or a file pair list depending on the value of the option: src_type:

list : (default) the path of destination is the same as source
pair : in that case, the first element is the first source, the second element is the first destination, and so on.

Example:

ascli server upload --src-type=pair ~/Documents/Samples/200KB.1 /Upload/sample1

Internally, when transfer agent direct is used, a temporary file list (or pair) file is generated and provided to ascp, unless --file-list or --file-pair-list is provided in ts in EX_ascp_args.

Note the special case when the source files are located on "Aspera on Cloud", i.e. using access keys and the file id API:

All files must be in the same source folder.
If there is a single file : specify the full path
For multiple files, specify the source folder as first item in the list followed by the list of file names.

Source files are located on "Aspera on cloud", when :

the server is Aspera on Cloud, and making a download / recv
the agent is Aspera on Cloud, and making an upload / send

Support of multi-session

Multi session, i.e. starting a transfer of a file set using multiple sessions (one ascp process per session) is supported on "direct" and "node" agents, not yet on connect.

when agent=node :

--ts=@json:'{"multi_session":10,"multi_session_threshold":1}'

Multi-session is directly supported by the node daemon.

when agent=direct :

--ts=@json:'{"multi_session":5,"multi_session_threshold":1,"resume_policy":"none"}'

Note: resume policy of "attr" may cause problems. "none" or "sparse_csum" shall be preferred.

Multi-session spawn is done by ascli.

When multi-session is used, one separate UDP port is used per session (refer to ascp manual page).

Content protection

Also known as Client-side encryption at reast (CSEAR), content protection allows a client to send files to a server which will store them encrypted (upload), and decrypt files as they are being downloaded from a server, both using a passphrase, only known by users sharing files. Files stay encrypted on server side.

activating CSEAR consists in using transfer spec parameters:

content_protection : activate encryption (encrypt for upload) or decryption (decrypt for download)
content_protection_password : the passphrase to be used.

Example: parameter to download a faspex package and decrypt on the fly

--ts=@json:'{"content_protection":"decrypt","content_protection_password":"_pass_here_"}'

Note that up to version 4.6.0, the following parameters should be used for agent direct:

--ts=@json:'{"EX_ascp_args":["--file-crypt=decrypt"],"EX_at_rest_password":"_secret_here_"}'

Transfer Spec Examples

Change target rate

--ts=@json:'{"target_rate_kbps":500000}'

Override the FASP SSH port to a specific TCP port:

--ts=@json:'{"ssh_port":33002}'

Force http fallback mode:

--ts=@json:'{"http_fallback":"force"}'

Activate progress when not activated by default on server

--ts=@json:'{"precalculate_job_size":true}'

Lock for exclusive execution

In some conditions, it may be desirable to ensure that ascli is not executed several times in parallel.

For instance when ascli is executed automatically on a schedule basis, one generally desire that a new execution is not started if a previous execution is still running because an on-going operation may last longer than the scheduling period:

Executing instances may pile-up and kill the system
The same file may be transferred by multiple instances at the same time.
preview may generate the same files in multiple instances.

Usually the OS native scheduler already provides some sort of protection against parallel execution:

The Windows scheduler does this by default
Linux cron can leverage the utility flock to do the same:

/usr/bin/flock -w 0 /var/cron.lock ascli ...

ascli natively supports a locking mechanism with option lock_port. (Technically, this opens a local TCP server port, and fails if this port is already used, providing a local lock. Lock is released when process exits).

Example:

Run this same command in two separate terminals within less than 30 seconds:

ascli config echo @ruby:'sleep(30)' --lock-port=12345

The first instance will sleep 30 seconds, the second one will immediately exit like this:

WARN -- : Another instance is already running (Address already in use - bind(2) for "127.0.0.1" port 12345).

"Provençale"

ascp, the underlying executable implementing Aspera file transfer using FASP, has a capability to not only access the local file system (using system's open,read,write,close primitives), but also to do the same operations on other data storage such as S3, Hadoop and others. This mechanism is call PVCL. Several PVCL adapters are available, some are embedded in ascp , some are provided om shared libraries and must be activated. (e.g. using trapd)

The list of supported PVCL adapters can be retrieved with command:

ascli conf ascp info

+--------------------+-----------------------------------------------------------+
| key                | value                                                     |
+--------------------+-----------------------------------------------------------+
-----8<-----snip-----8<-----
| product_name       | IBM Aspera SDK                                            |
| product_version    | 4.0.1.182389                                              |
| process            | pvcl                                                      |
| shares             | pvcl                                                      |
| noded              | pvcl                                                      |
| faux               | pvcl                                                      |
| file               | pvcl                                                      |
| stdio              | pvcl                                                      |
| stdio-tar          | pvcl                                                      |
+--------------------+-----------------------------------------------------------+

Here we can see the adapters: process, shares, noded, faux, file, stdio, stdio-tar.

Those adapters can be used wherever a file path is used in ascp including configuration. They act as a pseudo "drive".

The simplified format is:

<adapter>:///<sub file path>?<arg1>=<val1>&...

One of the adapters, used in this manual, for testing, is faux. It is a pseudo file system allowing generation of file data without actual storage (on source or destination).

`faux:` for testing

This is an extract of the man page of ascp. This feature is a feature of ascp, not ascli.

This adapter can be used to simulate a file or a directory.

To send uninitialized data in place of an actual source file, the source file is replaced with an argument of the form:

faux:///filename?filesize

where:

filename is the name that will be assigned to the file on the destination
filesize is the number of bytes that will be sent (in decimal).

Note: characters ? and & are shell special characters (wildcard and backround), so faux file specification on command line should be protected (using quotes or \). If not, the shell may give error: no matches found or equivalent.

For all sizes, a suffix can be added (case insensitive) to the size: k,m,g,t,p,e (values are power of 2, e.g. 1M is 2²⁰, i.e. 1 mebibyte, not megabyte). The maximum allowed value is 8*2⁶⁰. Very large faux file sizes (petabyte range and above) will likely fail due to lack of destination storage unless destination is faux://.

To send uninitialized data in place of a source directory, the source argument is replaced with an argument of the form:

faux:///dirname?<arg1>=<val1>&...

where:

dirname is the folder name and can contain / to specify a subfolder.
supported arguments are:

Name	Type	Description
count	int	mandatory
file	string	Basename for files Default: "file"
size	int	Size of first file. Default: 0
inc	int	Increment applied to determine next file size Default: 0
seq	enum	Sequence in determining next file size Values: random, sequential Default: sequential
buf_init	enum	How source data is initialized Option 'none' is not allowed for downloads. Values:none, zero, random Default:zero

The sequence parameter is applied as follows:

If seq is random then each file size is:
- size +/- (inc * rand())
- Where rand is a random number between 0 and 1
- Note that file size must not be negative, inc will be set to size if it is greater than size
- Similarly, overall file size must be less than 82⁶⁰. If size + inc is greater, inc will be reduced to limit size + inc to 72⁶⁰.
If seq is sequential then each file size is:
- size + ((fileindex - 1) * inc)
- Where first file is index 1
- So file1 is size bytes, file2 is size + inc bytes, file3 is size + inc * 2 bytes, etc.
- As with random, inc will be adjusted if size + (count * inc) is not less then 8*2⁶⁰.

Filenames generated are of the form: <file>_<00000 ... count>_<filesize>

To discard data at the destination, the destination argument is set to faux:// .

Examples:

Upload 20 gibibytes of random data to file myfile to directory /Upload

ascli server upload faux:///myfile\?20g --to-folder=/Upload

Upload a file /tmp/sample but do not save results to disk (no docroot on destination)

ascli server upload /tmp/sample --to-folder=faux://

Upload a faux directory mydir containing 1 million files, sequentially with sizes ranging from 0 to 2 Mebibyte - 2 bytes, with the basename of each file being testfile to /Upload

ascli server upload "faux:///mydir?file=testfile&count=1m&size=0&inc=2&seq=sequential" --to-folder=/Upload

Sample Commands

A non complete list of commands used in unit tests:

ascli
ascli -h
ascli aoc -N remind --username=my_aoc_user_email
ascli aoc -N servers
ascli aoc admin analytics transfers --query=@json:'{"status":"completed","direction":"receive"}' --notif-to=my_recipient_email --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: <%=ev["files_completed"]%> files received\n\n<%=ev.to_yaml%>}'
ascli aoc admin ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
ascli aoc admin ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"akibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
ascli aoc admin ats access_key delete akibmcloud
ascli aoc admin ats access_key list --fields=name,id
ascli aoc admin ats access_key node akibmcloud --secret=somesecret browse /
ascli aoc admin ats cluster clouds
ascli aoc admin ats cluster list
ascli aoc admin ats cluster show --cloud=aws --region=eu-west-1
ascli aoc admin ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
ascli aoc admin res application list
ascli aoc admin res client list
ascli aoc admin res client_access_key list
ascli aoc admin res client_registration_token create @json:'{"data":{"name":"test_client_reg1","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}'
ascli aoc admin res client_registration_token delete my_clt_reg_id
ascli aoc admin res client_registration_token list
ascli aoc admin res contact list
ascli aoc admin res dropbox list
ascli aoc admin res dropbox_membership list
ascli aoc admin res group list
ascli aoc admin res kms_profile list
ascli aoc admin res node list
ascli aoc admin res operation list
ascli aoc admin res organization show
ascli aoc admin res package list --http-options=@json:'{"read_timeout":120.0}'
ascli aoc admin res saml_configuration list
ascli aoc admin res self show
ascli aoc admin res short_link list
ascli aoc admin res user list
ascli aoc admin res workspace_membership list
ascli aoc admin resource node --name=my_aoc_node1_name --secret=my_aoc_node1_secret v3 access_key create --value=@json:'{"id":"testsub1","storage":{"path":"/folder1"}}'
ascli aoc admin resource node --name=my_aoc_node1_name --secret=my_aoc_node1_secret v3 events
ascli aoc admin resource node --name=my_aoc_node1_name --secret=my_aoc_node1_secret v4 browse /
ascli aoc admin resource node --name=my_aoc_node1_name --secret=my_aoc_node1_secret v4 delete /folder1
ascli aoc admin resource node --name=my_aoc_node1_name --secret=my_aoc_node1_secret v4 mkdir /folder1
ascli aoc admin resource node v3 name my_aoc_node1_name --secret=my_aoc_node1_secret access_key delete testsub1
ascli aoc admin resource workspace list
ascli aoc admin resource workspace_membership list --fields=ALL --query=@json:'{"page":1,"per_page":50,"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'
ascli aoc automation workflow action my_wf_id create --value=@json:'{"name":"toto"}'
ascli aoc automation workflow create --value=@json:'{"name":"test_workflow"}'
ascli aoc automation workflow delete my_wf_id
ascli aoc automation workflow list
ascli aoc automation workflow list --select=@json:'{"name":"test_workflow"}' --fields=id --format=csv --display=data
ascli aoc automation workflow list --value=@json:'{"show_org_workflows":"true"}' --scope=admin:all
ascli aoc bearer_token --display=data --scope=user:all
ascli aoc faspex
ascli aoc files bearer /
ascli aoc files bearer_token_node / --cache-tokens=no
ascli aoc files browse /
ascli aoc files browse / -N --link=my_aoc_publink_folder
ascli aoc files delete /testsrc
ascli aoc files download --transfer=connect /200KB.1
ascli aoc files file show my_file_id
ascli aoc files find / --value='\.partial$'
ascli aoc files http_node_download --to-folder=. /200KB.1
ascli aoc files mkdir /testsrc
ascli aoc files rename /somefolder testdst
ascli aoc files short_link create --to-folder=/testdst --value=private
ascli aoc files short_link create --to-folder=/testdst --value=public
ascli aoc files short_link list --value=@json:'{"purpose":"shared_folder_auth_link"}'
ascli aoc files transfer --from-folder=/testsrc --to-folder=/testdst testfile.bin
ascli aoc files upload --to-folder=/testsrc testfile.bin
ascli aoc files upload -N --to-folder=/ testfile.bin --link=my_aoc_publink_folder
ascli aoc files upload Test.pdf --transfer=node --transfer-info=@json:@stdin:
ascli aoc files v3 info
ascli aoc org -N --link=my_aoc_publink_recv_from_aocuser
ascli aoc organization
ascli aoc packages list
ascli aoc packages list --query=@json:'{"dropbox_name":"my_aoc_shbx_name","sort":"-received_at","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false}'
ascli aoc packages recv "my_package_id" --to-folder=.
ascli aoc packages recv ALL --to-folder=. --once-only=yes --lock-port=12345
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_external_user"]}' --new-user-option=@json:'{"package_contact":true}' testfile.bin
ascli aoc packages send --value=@json:'{"name":"Important files delivery","recipients":["my_email_internal_user"],"note":"my note"}' testfile.bin
ascli aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' testfile.bin
ascli aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"],"metadata":{"Project Id":"456","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' testfile.bin
ascli aoc packages send --workspace="my_aoc_shbx_ws" --value=@json:'{"name":"Important files delivery","recipients":["my_aoc_shbx_name"]}' testfile.bin
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_aoc_user --password=my_aoc_publink_send_use_pass
ascli aoc packages send -N --value=@json:'{"name":"Important files delivery"}' testfile.bin --link=my_aoc_publink_send_shd_inbox
ascli aoc packages shared_inboxes list
ascli aoc user profile modify @json:'{"name":"dummy change"}'
ascli aoc user profile show
ascli aoc user workspaces current
ascli aoc user workspaces list
ascli ats access_key cluster akibmcloud --secret=somesecret
ascli ats access_key create --cloud=aws --region=my_aws_bucket_region --params=@json:'{"id":"ak_aws","name":"my test key AWS","storage":{"type":"aws_s3","bucket":"my_aws_bucket_name","credentials":{"access_key_id":"my_aws_bucket_key","secret_access_key":"my_aws_bucket_secret"},"path":"/"}}'
ascli ats access_key create --cloud=softlayer --region=my_icos_bucket_region --params=@json:'{"id":"akibmcloud","secret":"somesecret","name":"my test key","storage":{"type":"ibm-s3","bucket":"my_icos_bucket_name","credentials":{"access_key_id":"my_icos_bucket_key","secret_access_key":"my_icos_bucket_secret"},"path":"/"}}'
ascli ats access_key delete ak_aws
ascli ats access_key delete akibmcloud
ascli ats access_key list --fields=name,id
ascli ats access_key node akibmcloud browse / --secret=somesecret
ascli ats api_key create
ascli ats api_key instances
ascli ats api_key list
ascli ats cluster clouds
ascli ats cluster list
ascli ats cluster show --cloud=aws --region=eu-west-1
ascli ats cluster show 1f412ae7-869a-445c-9c05-02ad16813be2
ascli conf flush_tokens
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath= --username=my_aoc_user_email --test-mode=yes
ascli conf wiz --url=https://my_aoc_org.ibmaspera.com --config-file=SAMPLE_CONFIG_FILE --pkeypath= --username=my_aoc_user_email --test-mode=yes --use-generic-client=yes
ascli config ascp connect info 'Aspera Connect for Windows'
ascli config ascp connect list
ascli config ascp connect version 'Aspera Connect for Windows' download 'Windows Installer' --to-folder=.
ascli config ascp connect version 'Aspera Connect for Windows' list
ascli config ascp connect version 'Aspera Connect for Windows' open documentation
ascli config ascp errors
ascli config ascp info
ascli config ascp install
ascli config ascp products list
ascli config ascp show
ascli config ascp spec
ascli config check_update
ascli config detect --url=https://my_aoc_org.ibmaspera.com
ascli config detect --url=my_faspex_url
ascli config doc
ascli config doc transfer-parameters
ascli config email_test --notif-to=my_recipient_email
ascli config export
ascli config genkey mykey
ascli config plugin create mycommand T
ascli config plugin list
ascli config proxy_check --fpac=@file:examples/proxy.pac https://eudemo.asperademo.com
ascli console transfer current list 
ascli console transfer smart list 
ascli console transfer smart sub my_job_id @json:'{"source":{"paths":["my_file_name"]},"source_type":"user_selected"}'
ascli cos -N --bucket=my_icos_bucket_name --endpoint=my_icos_bucket_endpoint --apikey=my_icos_bucket_apikey --crn=my_icos_resource_instance_id node info
ascli cos -N --bucket=my_icos_bucket_name --region=my_icos_bucket_region --service-credentials=@json:@file:service_creds.json node info
ascli cos node access_key show self
ascli cos node download testfile.bin --to-folder=.
ascli cos node info
ascli cos node upload testfile.bin
ascli faspex dropbox list --recipient="*my_faspex_dbx"
ascli faspex dropbox list --recipient="*my_faspex_wkg"
ascli faspex health
ascli faspex package list
ascli faspex package list --box=sent --fields=package_id --format=csv --display=data --query=@json:'{"max":1}'
ascli faspex package list --fields=package_id --format=csv --display=data --query=@json:'{"max":1}'
ascli faspex package list --recipient="*my_faspex_dbx" --format=csv --fields=package_id --query=@json:'{"max":1}'
ascli faspex package list --recipient="*my_faspex_wkg" --format=csv --fields=package_id --query=@json:'{"max":1}'
ascli faspex package recv "my_package_id" --to-folder=.
ascli faspex package recv "my_package_id" --to-folder=. --box=sent
ascli faspex package recv --to-folder=. --link="my_faspex_publink_recv_from_fxuser"
ascli faspex package recv ALL --to-folder=. --once-only=yes
ascli faspex package recv my_pkgid --recipient="*my_faspex_dbx" --to-folder=.
ascli faspex package recv my_pkgid --recipient="*my_faspex_wkg" --to-folder=.
ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["*my_faspex_dbx"]}' testfile.bin
ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["*my_faspex_wkg"]}' testfile.bin
ascli faspex package send --delivery-info=@json:'{"title":"Important files delivery","recipients":["my_email_internal_user","my_faspex_username"]}' testfile.bin
ascli faspex package send --link="my_faspex_publink_send_to_dropbox" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
ascli faspex package send --link="my_faspex_publink_send_to_fxuser" --delivery-info=@json:'{"title":"Important files delivery"}' testfile.bin
ascli faspex source name "Server Files" node br /
ascli faspex v4 dmembership list
ascli faspex v4 dropbox list
ascli faspex v4 metadata_profile list
ascli faspex v4 user list
ascli faspex v4 wmembership list
ascli faspex v4 workgroup list
ascli faspex5 admin res accounts list
ascli faspex5 admin res contacts list
ascli faspex5 admin res jobs list
ascli faspex5 admin res node list --value=@json:'{"type":"received","subtype":"mypackages"}'
ascli faspex5 admin res oauth_clients list
ascli faspex5 admin res registrations list
ascli faspex5 admin res saml_configs list
ascli faspex5 admin res shared_inboxes list
ascli faspex5 admin res workgroups list
ascli faspex5 package list --value=@json:'{"mailbox":"inbox","state":["released"]}'
ascli faspex5 package receive "my_package_id" --to-folder=.
ascli faspex5 package send --value=@json:'{"title":"test title","recipients":[{"name":"my_f5_user"}]}' testfile.bin
ascli mycommand --plugin-folder=T
ascli node -N -Ptst_node_preview access_key create --value=@json:'{"id":"aoc_1","storage":{"type":"local","path":"/"}}'
ascli node -N -Ptst_node_preview access_key delete aoc_1
ascli node async bandwidth 1
ascli node async counters 1
ascli node async files 1
ascli node async list
ascli node async show 1
ascli node async show ALL
ascli node basic_token
ascli node browse / -r
ascli node delete folder_1/10MB.1
ascli node delete testfile.bin
ascli node download testfile.bin --to-folder=.
ascli node download testfile.bin --to-folder=. --token-type=hybrid
ascli node health
ascli node info --fpac='function FindProxyForURL(url,host){return "DIRECT"}'
ascli node search / --value=@json:'{"sort":"mtime"}'
ascli node service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
ascli node service delete service1
ascli node service list
ascli node transfer list --value=@json:'{"active_only":true}'
ascli node upload --to-folder="folder_1" --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.1"}],"precalculate_job_size":true}' --transfer=node --transfer-info=@json:'{"url":"my_node_url","username":"my_node_user","password":"my_node_pass"}'
ascli node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000}'
ascli node upload testfile.bin --to-folder=folder_1 --ts=@json:'{"target_rate_cap_kbps":10000}' --token-type=hybrid
ascli orchestrator info
ascli orchestrator plugins
ascli orchestrator processes
ascli orchestrator workflow inputs my_orch_workflow_id
ascli orchestrator workflow list
ascli orchestrator workflow start my_orch_workflow_id --params=@json:'{"Param":"world !"}'
ascli orchestrator workflow start my_orch_workflow_id --params=@json:'{"Param":"world !"}' --result=ResultStep:Complete_status_message
ascli orchestrator workflow status ALL
ascli orchestrator workflow status my_orch_workflow_id
ascli preview check --skip-types=office
ascli preview folder 1 --skip-types=office --log-level=info --file-access=remote --ts=@json:'{"target_rate_kbps":1000000}'
ascli preview scan --skip-types=office --log-level=info
ascli preview test --case=test mp4 my_file_mxf --video-conversion=blend --log-level=debug
ascli preview test --case=test mp4 my_file_mxf --video-conversion=clips --log-level=debug
ascli preview test --case=test mp4 my_file_mxf --video-conversion=reencode --log-level=debug
ascli preview test --case=test png my_file_dcm --log-level=debug
ascli preview test --case=test png my_file_docx --log-level=debug
ascli preview test --case=test png my_file_mxf --video-png-conv=animated --log-level=debug
ascli preview test --case=test png my_file_mxf --video-png-conv=fixed --log-level=debug
ascli preview test --case=test png my_file_pdf --log-level=debug
ascli preview trevents --once-only=yes --skip-types=office --log-level=info
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user ctl all:status
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user health app_services --format=nagios
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user health asctlstatus --format=nagios --cmd-prefix='sudo '
ascli server -N -Ptst_hstsfaspex_ssh -Plocal_user nodeadmin -- -l
ascli server -N -Ptst_server_bykey -Plocal_user br /
ascli server browse /
ascli server browse folder_1/target_hot
ascli server cp NEW_SERVER_FOLDER/testfile.bin folder_1/200KB.2
ascli server delete NEW_SERVER_FOLDER
ascli server delete folder_1/target_hot
ascli server delete folder_1/to.delete
ascli server df
ascli server download NEW_SERVER_FOLDER/testfile.bin --to-folder=. --transfer-info=@json:'{"wss":false,"resume":{"iter_max":1}}'
ascli server download NEW_SERVER_FOLDER/testfile.bin --to-folder=folder_1 --transfer=node
ascli server du /
ascli server health transfer --to-folder=folder_1 --format=nagios 
ascli server info
ascli server md5sum NEW_SERVER_FOLDER/testfile.bin
ascli server mkdir NEW_SERVER_FOLDER --logger=stdout
ascli server mkdir folder_1/target_hot
ascli server mv folder_1/200KB.2 folder_1/to.delete
ascli server upload --sources=@ts --ts=@json:'{"paths":[{"source":"testfile.bin","destination":"NEW_SERVER_FOLDER/othername"}]}'
ascli server upload --src-type=pair --sources=@json:'["testfile.bin","NEW_SERVER_FOLDER/othername"]'
ascli server upload --src-type=pair testfile.bin NEW_SERVER_FOLDER/othername --notif-to=my_recipient_email
ascli server upload --src-type=pair testfile.bin folder_1/with_options --ts=@json:'{"cipher":"aes-192-gcm","content_protection":"encrypt","content_protection_password":"_secret_here_","cookie":"biscuit","create_dir":true,"delete_before_transfer":false,"delete_source":false,"exclude_newer_than":1,"exclude_older_than":10000,"fasp_port":33001,"http_fallback":false,"multi_session":0,"overwrite":"diff+older","precalculate_job_size":true,"preserve_access_time":true,"preserve_creation_time":true,"rate_policy":"fair","resume_policy":"sparse_csum","symlink_policy":"follow"}'
ascli server upload --to-folder=folder_1/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}' source_hot
ascli server upload testfile.bin --to-folder=NEW_SERVER_FOLDER --ts=@json:'{"multi_session":3,"multi_session_threshold":1,"resume_policy":"none","target_rate_kbps":1500}' --transfer-info=@json:'{"spawn_delay_sec":2.5}' --progress=multi
ascli shares admin share list
ascli shares repository browse /
ascli shares repository delete my_shares_upload/testfile.bin
ascli shares repository download --to-folder=. my_shares_upload/testfile.bin
ascli shares repository download --to-folder=. my_shares_upload/testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy/v1"}'
ascli shares repository upload --to-folder=my_shares_upload testfile.bin
ascli shares repository upload --to-folder=my_shares_upload testfile.bin --transfer=httpgw --transfer-info=@json:'{"url":"https://my_http_gw_fqdn/aspera/http-gwy/v1"}'
ascli shares2 appinfo
ascli shares2 organization list
ascli shares2 project list --organization=Sport
ascli shares2 repository browse /
ascli shares2 userinfo
ascli sync start --parameters=@json:'{"sessions":[{"name":"test","reset":true,"remote_dir":"/sync_test","local_dir":"contents","host":"my_remote_host","tcp_port":33001,"user":"my_remote_user","private_key_path":"my_local_user_key"}]}'
...and more

Usage

ascli -h
NAME
	ascli -- a command line tool for Aspera Applications (v4.8.0.pre)

SYNOPSIS
	ascli COMMANDS [OPTIONS] [ARGS]

DESCRIPTION
	Use Aspera application to perform operations on command line.
	Documentation and examples: https://rubygems.org/gems/aspera-cli
	execute: ascli conf doc
	or visit: https://www.rubydoc.info/gems/aspera-cli
	source repo: https://github.com/IBM/aspera-cli

ENVIRONMENT VARIABLES
	ASCLI_HOME config folder, default: $HOME/.aspera/ascli
	Any option can be set as an environment variable, refer to the manual

COMMANDS
	To list first level commands, execute: ascli
	Note that commands can be written shortened (provided it is unique).

OPTIONS
	Options begin with a '-' (minus), and value is provided on command line.
	Special values are supported beginning with special prefix @pfx:, where pfx is one of:
	base64, json, zlib, ruby, csvt, lines, list, incps, val, file, path, env, uri, stdin, preset
	Dates format is 'DD-MM-YY HH:MM:SS', or 'now' or '-<num>h'

ARGS
	Some commands require mandatory arguments, e.g. a path.

OPTIONS: global
        --interactive=ENUM           use interactive input of missing params: yes, [no]
        --ask-options=ENUM           ask even optional options: yes, [no]
        --format=ENUM                output format: [table], ruby, json, jsonpp, yaml, csv, nagios
        --display=ENUM               output only some information: [info], data, error
        --fields=VALUE               comma separated list of fields, or ALL, or DEF
        --select=VALUE               select only some items in lists, extended value: hash (column, value)
        --table-style=VALUE          table display style
        --flat-hash=ENUM             display hash values as additional keys: [yes], no
        --transpose-single=ENUM      single object fields output vertically: [yes], no
        --show-secrets=ENUM          show secrets on command output: [yes], no
    -h, --help                       Show this message.
        --bash-comp                  generate bash completion for command
        --show-config                Display parameters used for the provided action.
    -r, --rest-debug                 more debug for HTTP calls
    -v, --version                    display version
    -w, --warnings                   check for language warnings
        --ui=ENUM                    method to start browser: text, [graphical]
        --log-level=ENUM             Log level: debug, info, [warn], error, fatal, unknown
        --logger=ENUM                log method: [stderr], stdout, syslog
        --lock-port=VALUE            prevent dual execution of a command, e.g. in cron
        --query=VALUE                additional filter for API calls (extended value) (some commands)
        --http-options=VALUE         options for http socket (extended value)
        --insecure=ENUM              do not validate HTTPS certificate: [yes], no
        --once-only=ENUM             process only new items (some commands): yes, [no]
        --log-secrets=ENUM           show passwords in logs: yes, [no]
        --cache-tokens=ENUM          save and reuse Oauth tokens: [yes], no

COMMAND: config
SUBCOMMANDS: list overview id preset open documentation genkey gem plugin flush_tokens echo wizard export_to_cli detect coffee ascp email_test smtp_settings proxy_check folder file check_update initdemo vault
OPTIONS:
        --value=VALUE                extended value for create, update, list filter
        --property=VALUE             name of property to set
        --id=VALUE                   resource identifier (modify,delete,show)
        --config-file=VALUE          read parameters from file in YAML format, current=/usershome/.aspera/ascli/config.yaml
    -N, --no-default                 do not load default configuration for plugin
        --override=ENUM              Wizard: override existing value: yes, [no]
        --use-generic-client=ENUM    Wizard: AoC: use global or org specific jwt client id: yes, [no]
        --default=ENUM               Wizard: set as default configuration for specified plugin (also: update): yes, [no]
        --test-mode=ENUM             Wizard: skip private key check step: yes, [no]
    -P, --presetVALUE                load the named option preset from current config file
        --pkeypath=VALUE             Wizard: path to private key for JWT
        --ascp-path=VALUE            path to ascp
        --use-product=VALUE          use ascp from specified product
        --smtp=VALUE                 smtp configuration (extended value: hash)
        --fpac=VALUE                 proxy auto configuration script
        --secret=VALUE               default secret
        --secrets=VALUE              secret vault
        --sdk-url=VALUE              URL to get SDK
        --sdk-folder=VALUE           SDK folder path
        --notif-to=VALUE             email recipient for notification of transfers
        --notif-template=VALUE       email ERB template for notification of transfers
        --version-check-days=VALUE   period in days to check new version (zero to disable)
        --plugin-folder=VALUE        folder where to find additional plugins
        --ts=VALUE                   override transfer spec values (Hash, use @json: prefix), current={"create_dir"=>true}
        --local-resume=VALUE         set resume policy (Hash, use @json: prefix), current=
        --to-folder=VALUE            destination folder for downloaded files
        --sources=VALUE              list of source files (see doc)
        --transfer-info=VALUE        parameters for transfer agent
        --src-type=ENUM              type of file list: list, pair
        --transfer=ENUM              type of transfer agent: direct, node, connect, httpgw, trsdk
        --progress=ENUM              type of progress bar: none, native, multi


COMMAND: shares
SUBCOMMANDS: repository admin
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password


COMMAND: node
SUBCOMMANDS: postprocess stream transfer cleanup forward access_key watch_folder service async central asperabrowser basic_token browse upload download api_details health events space info license mkdir mklink mkfile rename delete search
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --validator=VALUE            identifier of validator (optional for central)
        --asperabrowserurl=VALUE     URL for simple aspera web ui
        --sync-name=VALUE            sync name
        --token-type=ENUM            Type of token used for transfers: aspera, basic, hybrid


COMMAND: orchestrator
SUBCOMMANDS: info workflow plugins processes
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --params=VALUE               parameters hash table, use @json:{"param":"value"}
        --result=VALUE               specify result value as: 'work step:parameter'
        --synchronous=ENUM           work step:parameter expected as result: yes, [no]
        --ret-style=ENUM             how return type is requested in api: header, arg, ext
        --auth-style=ENUM            authentication type: arg_pass, head_basic, apikey


COMMAND: bss
SUBCOMMANDS: subscription
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password


COMMAND: alee
SUBCOMMANDS: entitlement
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password


COMMAND: ats
SUBCOMMANDS: cluster access_key api_key aws_trust_policy
OPTIONS:
        --ibm-api-key=VALUE          IBM API key, see https://cloud.ibm.com/iam/apikeys
        --instance=VALUE             ATS instance in ibm cloud
        --ats-key=VALUE              ATS key identifier (ats_xxx)
        --ats-secret=VALUE           ATS key secret
        --params=VALUE               Parameters access key creation (@json:)
        --cloud=VALUE                Cloud provider
        --region=VALUE               Cloud region


COMMAND: faspex5
SUBCOMMANDS: package admin
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --client-id=VALUE            OAuth client identifier
        --client-secret=VALUE        OAuth client secret
        --redirect-uri=VALUE         OAuth redirect URI
        --auth=ENUM                  OAuth type of authentication: web, jwt, boot
        --private-key=VALUE          Oauth RSA private key PEM value for JWT (prefix file path with @val:@file:)


COMMAND: cos
SUBCOMMANDS: node
OPTIONS:
        --bucket=VALUE               IBM Cloud Object Storage bucket name
        --endpoint=VALUE             storage endpoint url
        --apikey=VALUE               storage API key
        --crn=VALUE                  ressource instance id
        --service-credentials=VALUE  IBM Cloud service credentials (Hash)
        --region=VALUE               IBM Cloud Object storage region


COMMAND: faspex
SUBCOMMANDS: health package source me dropbox v4 address_book login_methods
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --link=VALUE                 public link for specific operation
        --delivery-info=VALUE        package delivery information (extended value)
        --source-name=VALUE          create package from remote source (by name)
        --storage=VALUE              Faspex local storage definition
        --recipient=VALUE            use if recipient is a dropbox (with *)
        --box=ENUM                   package box: inbox, archive, sent


COMMAND: preview
SUBCOMMANDS: scan events trevents check test
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --skip-format=ENUM           skip this preview format (multiple possible): png, mp4
        --folder-reset-cache=ENUM    force detection of generated preview by refresh cache: [no], header, read
        --skip-types=VALUE           skip types in comma separated list
        --previews-folder=VALUE      preview folder in storage root
        --temp-folder=VALUE          path to temp folder
        --skip-folders=VALUE         list of folder to skip
        --case=VALUE                 basename of output for for test
        --scan-path=VALUE            subpath in folder id to start scan in (default=/)
        --scan-id=VALUE              forder id in storage to start scan in, default is access key main folder id
        --mimemagic=ENUM             use Mime type detection of gem mimemagic: yes, [no]
        --overwrite=ENUM             when to overwrite result file: always, never, [mtime]
        --file-access=ENUM           how to read and write files in repository: [local], remote
        --max-size=VALUE             maximum size (in bytes) of preview file
        --thumb-vid-scale=VALUE      png: video: size (ffmpeg scale argument)
        --thumb-vid-fraction=VALUE   png: video: position of snapshot
        --thumb-img-size=VALUE       png: non-video: height (and width)
        --thumb-text-font=VALUE      png: plaintext: font to render text with image magick convert, list with: identify -list font
        --video-conversion=ENUM      mp4: method for preview generation: [reencode], blend, clips
        --video-png-conv=ENUM        mp4: method for thumbnail generation: [fixed], animated
        --video-start-sec=VALUE      mp4: start offset (seconds) of video preview
        --video-scale=VALUE          mp4: video scale (ffmpeg)
        --blend-keyframes=VALUE      mp4: blend: # key frames
        --blend-pauseframes=VALUE    mp4: blend: # pause frames
        --blend-transframes=VALUE    mp4: blend: # transition blend frames
        --blend-fps=VALUE            mp4: blend: frame per second
        --clips-count=VALUE          mp4: clips: number of clips
        --clips-length=VALUE         mp4: clips: length in seconds of each clips


COMMAND: sync
SUBCOMMANDS: start admin
OPTIONS:
        --parameters=VALUE           extended value for session set definition
        --session-name=VALUE         name of session to use for admin commands, by default first one


COMMAND: aoc
SUBCOMMANDS: reminder servers bearer_token organization tier_restrictions user packages files admin automation gateway
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --auth=ENUM                  OAuth type of authentication: web, jwt
        --operation=ENUM             client operation for transfers: push, pull
        --client-id=VALUE            OAuth API client identifier in application
        --client-secret=VALUE        OAuth API client passcode
        --redirect-uri=VALUE         OAuth API client redirect URI
        --private-key=VALUE          OAuth JWT RSA private key PEM value (prefix file path with @val:@file:)
        --workspace=VALUE            name of workspace
        --name=VALUE                 resource name
        --path=VALUE                 file or folder path
        --link=VALUE                 public link to shared resource
        --new-user-option=VALUE      new user creation option
        --from-folder=VALUE          share to share source folder
        --scope=VALUE                OAuth scope for AoC API calls
        --bulk=ENUM                  bulk operation: yes, [no]
        --default-ports=ENUM         use standard FASP ports or get from node api: yes, [no]


COMMAND: server
SUBCOMMANDS: health nodeadmin userdata configurator ctl download upload browse delete rename ls rm mv du info mkdir cp df md5sum
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --ssh-keys=VALUE             ssh key path list (Array or single)
        --ssh-options=VALUE          ssh options (Hash)
        --cmd-prefix=VALUE           prefix to add for as cmd execution, e.g. sudo or /opt/aspera/bin 


COMMAND: console
SUBCOMMANDS: transfer health
OPTIONS:
        --url=VALUE                  URL of application, e.g. https://org.asperafiles.com
        --username=VALUE             username to log in
        --password=VALUE             user's password
        --filter-from=DATE           only after date
        --filter-to=DATE             only before date

Note that actions and parameter values can be written in short form.

Plugin: Aspera on Cloud

Aspera on Cloud uses the more advanced Oauth v2 mechanism for authentication (HTTP Basic authentication is not supported).

It is recommended to use the wizard to set it up, but manual configuration is also possible.

Configuration: using Wizard

ascli provides a configuration wizard. Here is a sample invocation :

ascli config wizard
option: url> https://myorg.ibmaspera.com
Detected: Aspera on Cloud
Preparing preset: aoc_myorg
Please provide path to your private RSA key, or empty to generate one:
option: pkeypath>
using existing key:
/Users/myself/.aspera/ascli/aspera_aoc_key
Using global client_id.
option: username> [email protected]
Updating profile with new key
creating new config preset: aoc_myorg
Setting config preset as default for aspera
saving config file
Done.
You can test with:
ascli aoc user profile show

Optionally, it is possible to create a new organization-specific "integration", i.e. client application identification. For this, specify the option: --use-generic-client=no.

This will guide you through the steps to create.

If the wizard does not detect the application but you know the application, you can force it using option value:

ascli config wizard --value=aoc

Configuration: using manual setup

If you used the wizard (recommended): skip this section.

Configuration details

Several types of OAuth authentication are supported:

JSON Web Token (JWT) : authentication is secured by a private key (recommended for CLI)
Web based authentication : authentication is made by user using a browser
URL Token : external users authentication with url tokens (public links)

The authentication method is controlled by option auth.

For a quick start, follow the mandatory and sufficient section: API Client Registration (auth=web) as well as [option preset for Aspera on Cloud](#aocpreset).

For a more convenient, browser-less, experience follow the JWT section (auth=jwt) in addition to Client Registration.

In Oauth, a "Bearer" token are generated to authenticate REST calls. Bearer tokens are valid for a period of time.ascli saves generated tokens in its configuration folder, tries to re-use them or regenerates them when they have expired.

Optional: API Client Registration

If you use the built-in client_id and client_secret, skip this and do not set them in next section.

Else you can use a specific OAuth API client_id, the first step is to declare ascli in Aspera on Cloud using the admin interface.

(official documentation: https://ibmaspera.com/help/admin/organization/registering_an_api_client ).

Let's start by a registration with web based authentication (auth=web):

Open a web browser, log to your instance: e.g. https://myorg.ibmaspera.com/
Go to Apps→Admin→Organization→Integrations
Click "Create New"
- Client Name: ascli
- Redirect URIs: http://localhost:12345
- Origins: localhost
- uncheck "Prompt users to allow client to access"
- leave the JWT part for now
Save

Note: for web based authentication, ascli listens on a local port (e.g. specified by the redirect_uri, in this example: 12345), and the browser will provide the OAuth code there. For ``ascli`, HTTP is required, and 12345 is the default port.

Once the client is registered, a "Client ID" and "Secret" are created, these values will be used in the next step.

option preset for Aspera on Cloud

If you did not use the wizard, you can also manually create a option preset for ascli in its configuration file.

Lets create an option preset called: my_aoc_org using ask interactive input (client info from previous step):

ascli config preset ask my_aoc_org url client_id client_secret
option: url> https://myorg.ibmaspera.com/
option: client_id> my_BJbQiFw
option: client_secret> yFS1mu-crbKuQhGFtfhYuoRW...
updated: my_aoc_org

(This can also be done in one line using the command config preset update my_aoc_org --url=...)

Define this option preset as default configuration for the aspera plugin:

ascli config preset set default aoc my_aoc_org

Note: Default auth method is web and default redirect_uri is http://localhost:12345. Leave those default values.

Activation of JSON Web Token (JWT) for direct authentication

For a Browser-less, Private Key-based authentication, use the following steps.

In order to use JWT for Aspera on Cloud API client authentication, a private/public key pair must be used (without passphrase)

API Client JWT activation

If you are not using the built-in client_id and secret, JWT needs to be authorized in Aspera on Cloud. This can be done in two manners:

Graphically
- Open a web browser, log to your instance: https://myorg.ibmaspera.com/
- Go to Apps→Admin→Organization→Integrations
- Click on the previously created application
- select tab : "JSON Web Token Auth"
- Modify options if necessary, for instance: activate both options in section "Settings"
- Click "Save"
Using command line

ascli aoc admin res client list

:............:...............:
:     id     :  name         :
:............:...............:
: my_BJbQiFw : my-client-app :
:............:...............:

ascli aoc admin res client modify my_BJbQiFw @json:'{"jwt_grant_enabled":true,"explicit_authorization_required":false}'

modified

User key registration

The public key must be assigned to your user. This can be done in two manners:

Graphically

Open the previously generated public key located here: $HOME/.aspera/ascli/my_private_key.pub

Open a web browser, log to your instance: https://myorg.ibmaspera.com/
Click on the user's icon (top right)
Select "Account Settings"
Paste the Public Key in the "Public Key" section
Click on "Submit"

Using command line

ascli aoc admin res user list

:........:................:
:   id   :      name      :
:........:................:
: 109952 : Tech Support   :
: 109951 : LAURENT MARTIN :
:........:................:

ascli aoc user profile modify @ruby:'{"public_key"=>File.read(File.expand_path("~/.aspera/ascli/my_private_key.pub"))}'

modified

Note: the aspera user info show command can be used to verify modifications.

option preset modification for JWT

To activate default use of JWT authentication for ascli using the option preset, do the following:

change auth method to JWT
provide location of private key
provide username to login as (OAuth "subject")

Execute:

ascli config preset update my_aoc_org --auth=jwt --private-key=@val:@file:~/.aspera/ascli/my_private_key [email protected]

Note: the private key argument represents the actual PEM string. In order to read the content from a file, use the @file: prefix. But if the @file: argument is used as is, it will read the file and set in the config file. So to keep the "@file" tag in the configuration file, the @val: prefix is added.

After this last step, commands do not require web login anymore.

First Use

Once client has been registered and option preset created: ascli can be used:

ascli aoc files br /

Current Workspace: Default Workspace (default)
empty

Administration

The admin command allows several administrative tasks (and require admin privilege).

It allows actions (create, update, delete) on "resources": users, group, nodes, workspace, etc... with the admin resource command.

Bulk creation and deletion of resource

Bulk creation and deletion of resources are possible using option bulk (yes,no(default)). In that case, the operation expects an Array of Hash instead of a simple Hash using the Extended Value Syntax.

Listing resources

The command aoc admin res <type> list lists all entities of given type. It uses paging and multiple requests if necessary.

The option query can be optionally used. It expects a Hash using Extended Value Syntax, generally provided using: --query=@json:{...}. Values are directly sent to the API call and used as a filter on server side.

The following parameters are supported:

q : a filter on name of resource (case insensitive, matches if value is contained in name)
sort: name of fields to sort results, prefix with - for reverse order.
max : maximum number of items to retrieve (stop pages when the maximum is passed)
pmax : maximum number of pages to request (stop pages when the maximum is passed)
page : native api parameter, in general do not use (added by
per_page : native api parameter, number of items par api call, in general do not use
Other specific parameters depending on resource type.

Both max and pmax are processed internally in ascli, not included in actual API call and limit the number of successive pages requested to API. ascli will return all values using paging if not provided.

Other parameters are directly sent as parameters to the GET request on API.

page and per_page are normally added by ascli to build successive API calls to get all values if there are more than 1000. (AoC allows a maximum page size of 1000).

q and sort are available on most resource types.

Other parameters depend on the type of entity (refer to AoC API).

Examples:

List users with laurent in name:

ascli aoc admin res user list --query=--query=@json:'{"q":"laurent"}'

List users who logged-in before a date:

ascli aoc admin res user list --query=@json:'{"q":"last_login_at:<2018-05-28"}'

List external users and sort in reverse alphabetical order using name:

ascli aoc admin res user list --query=@json:'{"member_of_any_workspace":false,"sort":"-name"}'

Refer to the AoC API for full list of query parameters, or use the browser in developer mode with the web UI.

Note the option select can also be used to further refine selection, refer to section earlier.

Selecting a resource

Resources are identified by a unique id, as well as a unique name (case insensitive).

To execute an action on a specific resource, select it using one of those methods:

recommended: give id directly on command line after the action: aoc admin res node show 123
give name on command line after the action: aoc admin res node show name abc
provide option id : aoc admin res node show --id=123
provide option name : aoc admin res node show --name=abc

Access Key secrets

In order to access some administrative actions on "nodes" (in fact, access keys), the associated secret is required. It is usually provided using the secret option. For example in a command like:

ascli aoc admin res node --id=123 --secret="secret1" v3 info

It is also possible to provide a set of secrets used on a regular basis using the secret vault.

Activity

The activity app can be queried with:

ascli aoc admin analytics transfers

It can also support filters and send notification using option notif_to. a template is defined using option notif_template :

mytemplate.erb:

From: <%=from_name%> <<%=from_email%>>
To: <<%=ev['user_email']%>>
Subject: <%=ev['files_completed']%> files received

Dear <%=ev[:user_email.to_s]%>,
We received <%=ev['files_completed']%> files for a total of <%=ev['transferred_bytes']%> bytes, starting with file:
<%=ev['content']%>

Thank you.

The environment provided contains the following additional variable:

ev : all details on the transfer event

Example:

ascli aoc admin analytics transfers --once-only=yes --lock-port=12345 \
--query=@json:'{"status":"completed","direction":"receive"}' \
--notif-to=active --notif-template=@file:mytemplate.erb

Options:

once_only keep track of last date it was called, so next call will get only new events
query filter (on API call)
notify send an email as specified by template, this could be places in a file with the @file modifier.

Note this must not be executed in less than 5 minutes because the analytics interface accepts only a period of time between 5 minutes and 6 months. The period is [date of previous execution]..[now].

Transfer: Using specific transfer ports

By default transfer nodes are expected to use ports TCP/UDP 33001. The web UI enforces that. The option default_ports ([yes]/no) allows ascli to retrieve the server ports from an API call (download_setup) which reads the information from aspera.conf on the server.

Using ATS

Refer to section "Examples" of ATS and substitute command ats with aoc admin ats.

Example: Bulk creation of users

ascli aoc admin res user create --bulk=yes @json:'[{"email":"[email protected]"},{"email":"[email protected]"}]'

:.......:.........:
:  id   : status  :
:.......:.........:
: 98398 : created :
: 98399 : created :
:.......:.........:

Example: Find with filter and delete

ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id,email

:.......:........................:
:  id   :         email          :
:.......:........................:
: 98398 : [email protected] :
: 98399 : [email protected] :
:.......:........................:

thelist=$(ascli aoc admin res user list --query='@json:{"q":"dummyuser"}' --fields=id --format=json --display=data|jq -cr 'map(.id)')

echo $thelist

["113501","354061"]

ascli aoc admin res user --bulk=yes --id=@json:"$thelist" delete

:.......:.........:
:  id   : status  :
:.......:.........:
: 98398 : deleted :
: 98399 : deleted :
:.......:.........:

Example: Find deactivated users since more than 2 years

ascli aoc admin res user list --query=@ruby:'{"deactivated"=>true,"q"=>"last_login_at:<#{(DateTime.now.to_time.utc-2*365*86400).iso8601}"}'

To delete them use the same method as before

Example: Display current user's workspaces

ascli aoc user workspaces list

:......:............................:
:  id  :            name            :
:......:............................:
: 16   : Engineering                :
: 17   : Marketing                  :
: 18   : Sales                      :
:......:............................:

Example: Create a sub access key in a "node"

Creation of a sub-access key is like creation of access key with the following difference: authentication to node API is made with accesskey (master access key) and only the path parameter is provided: it is relative to the storage root of the master key. (id and secret are optional)

ascli aoc admin resource node --name=_node_name_ --secret=_secret_ v4 access_key create --value=@json:'{"storage":{"path":"/folder1"}}'

Example: Display transfer events (ops/transfer)

ascli aoc admin res node --secret=_secret_ v3 transfer list --value=@json:'[["q","*"],["count",5]]'

Examples of query (TODO: cleanup):

{"q":"type(file_upload OR file_delete OR file_download OR file_rename OR folder_create OR folder_delete OR folder_share OR folder_share_via_public_link)","sort":"-date"}

{"tag":"aspera.files.package_id=LA8OU3p8w"}

Example: Display node events (events)

ascli aoc admin res node --secret=_secret_ v3 events

Example: Display members of a workspace

ascli aoc admin res workspace_membership list --fields=member_type,manager,member.email --query=@json:'{"embed":"member","inherited":false,"workspace_id":11363,"sort":"name"}'

:.............:.........:..................................:
: member_type : manager :           member.email           :
:.............:.........:..................................:
: user        : true    : [email protected]            :
: user        : false   : [email protected] :
: user        : false   : [email protected]               :
: user        : false   : [email protected]         :
: group       : false   :                                  :
: user        : false   : [email protected]            :
:.............:.........:..................................:

other query parameters:

{"workspace_membership_through":true,"include_indirect":true}

Example: add all members of a workspace to another workspace

a- Get id of first workspace

WS1='First Workspace'
WS1ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS1"'"}' --select=@json:'{"name":"'"$WS1"'"}' --fields=id --format=csv)

b- Get id of second workspace

WS2='Second Workspace'
WS2ID=$(ascli aoc admin res workspace list --query=@json:'{"q":"'"$WS2"'"}' --select=@json:'{"name":"'"$WS2"'"}' --fields=id --format=csv)

c- Extract membership information

ascli aoc admin res workspace_membership list --fields=manager,member_id,member_type,workspace_id --query=@json:'{"workspace_id":'"$WS1ID"'}' --format=jsonpp > ws1_members.json

d- Convert to creation data for second workspace:

grep -Eve '(direct|effective_manager|_count|storage|"id")' ws1_members.json|sed '/workspace_id/ s/"'"$WS1ID"'"/"'"$WS2ID"'"/g' > ws2_members.json

or, using jq:

jq '[.[] | {member_type,member_id,workspace_id,manager,workspace_id:"'"$WS2ID"'"}]' ws1_members.json > ws2_members.json

e- Add members to second workspace

ascli aoc admin res workspace_membership create --bulk=yes @json:@file:ws2_members.json

Example: Get users who did not log since a date

ascli aoc admin res user list --fields=email --query=@json:'{"q":"last_login_at:<2018-05-28"}'

:...............................:
:             email             :
:...............................:
: [email protected]          :
: [email protected]      :
:...............................:

Example: List "Limited" users

ascli aoc admin res user list --fields=email --select=@json:'{"member_of_any_workspace":false}'

Example: Perform a multi Gbps transfer between two remote shared folders

In this example, a user has access to a workspace where two shared folders are located on different sites, e.g. different cloud regions.

First, setup the environment (skip if already done)

ascli conf wizard --url=https://sedemo.ibmaspera.com [email protected]

Detected: Aspera on Cloud
Preparing preset: aoc_sedemo
Using existing key:
/Users/laurent/.aspera/ascli/aspera_aoc_key
Using global client_id.
Please Login to your Aspera on Cloud instance.
Navigate to your "Account Settings"
Check or update the value of "Public Key" to be:
-----BEGIN PUBLIC KEY-----
SOME PUBLIC KEY PEM DATA HERE
-----END PUBLIC KEY-----
Once updated or validated, press enter.

creating new config preset: aoc_sedemo
Setting config preset as default for aspera
saving config file
Done.
You can test with:
ascli aoc user profile show

This creates the option preset "aoc_<org name>" to allow seamless command line access and sets it as default for aspera on cloud.

Then, create two shared folders located in two regions, in your files home, in a workspace.

Then, transfer between those:

ascli -Paoc_show aoc files transfer --from-folder='IBM Cloud SJ' --to-folder='AWS Singapore' 100GB.file --ts=@json:'{"target_rate_kbps":"1000000","multi_session":10,"multi_session_threshold":1}'

Example: create registration key to register a node

ascli aoc admin res client create @json:'{"data":{"name":"laurentnode","client_subject_scopes":["alee","aejd"],"client_subject_enabled":true}}' --fields=token --format=csv

jfqslfdjlfdjfhdjklqfhdkl

Example: delete all registration keys

ascli aoc admin res client list --fields=id --format=csv|ascli aoc admin res client delete --bulk=yes --id=@lines:@stdin:

+-----+---------+
| id  | status  |
+-----+---------+
| 99  | deleted |
| 100 | deleted |
| 101 | deleted |
| 102 | deleted |
+-----+---------+

Example: Create a node

AoC nodes as actually composed with two related entities:

An access key created on the Transfer Server (HSTS/ATS)
a node resource in the AoC application.

The web UI allows creation of both entities in one shot but not the CLI for more flexibility. Note that when selecting "Use existing access key" in the web UI, this actually skips access key creation.

So, for example, the creation of a node using ATS in IBM Cloud looks like (see other example in this manual):

create the access key on ATS

ascli aoc admin ats access_key create --cloud=softlayer --region=eu-de --params=@json:'{"storage":{"type":"ibm-s3","bucket":"mybucket","credentials":{"access_key_id":"mykey","secret_access_key":"mysecret"},"path":"/"}}'

Take a note of the randomly generated id and secret.

Retrieve the ATS node address

ascli aoc admin ats cluster show --cloud=softlayer --region=eu-de --fields=transfer_setup_url --format=csv --transpose-single=no

Create the node entity

ascli aoc admin res node create @json:'{"name":"myname","access_key":"*accesskeyid*","ats_access_key":true,"ats_storage_type":"ibm-s3","url":"https://ats-sl-fra-all.aspera.io"}'

Creation of a node with a self-managed node is similar, but the command aoc admin ats access_key create is replaced with node access_key create on the private node itself.

Packages

The webmail-like application.

Send a Package

General syntax:

ascli aoc packages send --value=[package extended value] [other parameters such as file list and transfer parameters]

Notes:

The value option can contain any supported package creation parameter. Refer to the AoC package creation API, or display an existing package in JSON to list attributes.
List allowed shared inbox destinations with: ascli aoc packages shared_inboxes list
Use fields: recipients and/or bcc_recipients to provide the list of recipients: user or shared inbox.
- Provide either ids as expected by API: "recipients":[{"type":"dropbox","id":"1234"}]
- or just names: "recipients":[{"The Dest"}] . ascli will resolve the list of email addresses and dropbox names to the expected type/id list, based on case insensitive partial match.
If a user recipient (email) is not already registered and the workspace allows external users, then the package is sent to an external user, and
- if the option new_user_option is @json:{"package_contact":true} (default), then a public link is sent and the external user does not need to create an account
- if the option new_user_option is @json:{}, then external users are invited to join the workspace

Example: Send a package with one file to two users, using their email

ascli aoc package send --value=@json:'{"name":"my title","note":"my note","recipients":["[email protected]","[email protected]"]}' my_file.dat

Example: Send a package to a shared inbox with metadata

ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":["Shared Inbox With Meta"],"metadata":{"Project Id":"123","Type":"Opt2","CheckThose":["Check1","Check2"],"Optional Date":"2021-01-13T15:02:00.000Z"}}' ~/Documents/Samples/200KB.1

It is also possible to use identifiers and API parameters:

ascli aoc package send --workspace=eudemo --value=@json:'{"name":"my pack title","recipients":[{"type":"dropbox","id":"12345"}],"metadata":[{"input_type":"single-text","name":"Project Id","values":["123"]},{"input_type":"single-dropdown","name":"Type","values":["Opt2"]},{"input_type":"multiple-checkbox","name":"CheckThose","values":["Check1","Check2"]},{"input_type":"date","name":"Optional Date","values":["2021-01-13T15:02:00.000Z"]}]}' ~/Documents/Samples/200KB.1

Example: List packages in a given shared inbox

When user packages are listed, the following query is used:

{"archived":false,"exclude_dropbox_packages":true,"has_content":true,"received":true}

To list packages in a shared inbox, the query has to be specified with withe the shared inbox by name or its identifier. Additionnal parameters can be specified, as supported by the API (to find out available filters, consult the API definition, or use the web interface in developer mode). The current workspace is added unless specified in the query.

Using shared inbox name:

ascli aoc packages list --query=@json:'{"dropbox_name":"My Shared Inbox","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'

Using shared inbox identifier: first retrieve the id of the shared inbox, and then list packages with the appropriate filter.

shbxid=$(ascli aoc packages shared_inboxes show name 'My Shared Inbox' --format=csv --display=data --fields=id --transpose-single=no)

ascli aoc packages list --query=@json:'{"dropbox_id":"'$shbxid'","archived":false,"received":true,"has_content":true,"exclude_dropbox_packages":false,"include_draft":false,"sort":"-received_at"}'

Receive new packages only (Cargo)

It is possible to automatically download new packages, like using Aspera Cargo:

ascli aoc packages recv --id=ALL --once-only=yes --lock-port=12345

--id=ALL (case sensitive) will download all packages
--once-only=yes keeps memory of any downloaded package in persistency files located in the configuration folder
--lock-port=12345 ensures that only one instance is started at the same time, to avoid running two downloads in parallel

Typically, one would execute this command on a regular basis, using the method of your choice:

Windows: Task Scheduler
Linux/Unix: cron
etc...

Files

Folder sharing app.

Download Files

Download of files is straightforward with a specific syntax for the aoc files download action: Like other commands the source file list is provided as a list with the sources option. Nevertheless, consider this:

if only one source is provided, it is downloaded
if multiple sources must be downloaded, then the first in list is the path of the source folder, and the remaining items are the file names in this folder (without path).

Shared folders

list shared folders in node

ascli aoc admin res node --id=8669 shared_folders

list shared folders in workspace

ascli aoc admin res workspace --id=10818 shared_folders

list members of shared folder

ascli aoc admin res node --id=8669 v4 perm 82 show

Cross Organization transfers

It is possible to transfer files directly between organizations without having to first download locally and then upload...

Although optional, the creation of option preset is recommended to avoid placing all parameters in the command line.

Procedure to send a file from org1 to org2:

Get access to Organization 1 and create a option preset: e.g. org1, for instance, use the Wizard
Check that access works and locate the source file e.g. mysourcefile, e.g. using command files browse
Get access to Organization 2 and create a option preset: e.g. org2
Check that access works and locate the destination folder mydestfolder
execute the following:

ascli -Porg1 aoc files node_info /mydestfolder --format=json --display=data | ascli -Porg2 aoc files upload mysourcefile --transfer=node --transfer-info=@json:@stdin:

Explanation:

-Porg1 aoc use Aspera on Cloud plugin and load credentials for org1
files node_info /mydestfolder generate transfer information including node api credential and root id, suitable for the next command
--format=json format the output in JSON (instead of default text table)
--display=data display only the result, and remove other information, such as workspace name
| the standard output of the first command is fed into the second one
-Porg2 aoc use Aspera on Cloud plugin and load credentials for org2
files upload mysourcefile upload the file named mysourcefile (located in org1)
--transfer=node use transfer agent type node instead of default direct
--transfer-info=@json:@stdin: provide node transfer agent information, i.e. node API credentials, those are expected in JSON format and read from standard input

Find Files

The command aoc files find [--value=expression] will recursively scan storage to find files matching the expression criteria. It works also on node resource using the v4 command. (see examples)

The expression can be of 3 formats:

empty (default) : all files, equivalent to value: exec:true
not starting with exec: : the expression is a regular expression, using Ruby Regex syntax. equivalent to value: exec:f['name'].match(/expression/)

For instance, to find files with a special extension, use --value='\.myext$'

starting with exec: : the Ruby code after the prefix is executed for each entry found. The entry variable name is f. The file is displayed if the result of the expression is true;

Examples of expressions: (using like this: --value=exec:'<expression>')

Find files more recent than 100 days

f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100

Find files older than 1 year on a given node and store in file list

ascli aoc admin res node --name='my node name' --secret='_secret_here_' v4 find / --fields=path --value='exec:f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100' --format=csv > my_file_list.txt

Delete the files, one by one

cat my_file_list.txt|while read path;do echo ascli aoc admin res node --name='my node name' --secret='_secret_here_' v4 delete "$path" ;done

Delete the files in bulk

cat my_file_list.txt | ascli aoc admin res node --name='my node name' --secret='_secret_here_' v3 delete @lines:@stdin:

Plugin: Aspera Transfer Service

ATS is usable either :

from an AoC subscription : ascli aoc admin ats : use AoC authentication
or from an IBM Cloud subscription : ascli ats : use IBM Cloud API key authentication

IBM Cloud ATS : creation of api key

This section is about using ATS with an IBM cloud subscription. If you are using ATS as part of AoC, then authentication is thropugh AoC, not IBM Cloud.

First get your IBM Cloud APIkey. For instance, it can be created using the IBM Cloud web interface, or using command line:

ibmcloud iam api-key-create mykeyname -d 'my sample key'

OK
API key mykeyname was created

Please preserve the API key! It cannot be retrieved after it's created.

Name          mykeyname
Description   my sample key
Created At    2019-09-30T12:17+0000
API Key       my_secret_api_key_here_8f8d9fdakjhfsashjk678
Locked        false
UUID          ApiKey-05b8fadf-e7fe-4bc4-93a9-6fd348c5ab1f

References:

Then, to register the key by default for the ats plugin, create a preset. Execute:

ascli config preset update my_ibm_ats --ibm-api-key=my_secret_api_key_here_8f8d9fdakjhfsashjk678

ascli config preset set default ats my_ibm_ats

ascli ats api_key instances

+--------------------------------------+
| instance                             |
+--------------------------------------+
| aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee |
+--------------------------------------+

ascli config preset update my_ibm_ats --instance=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee

ascli ats api_key create

+--------+----------------------------------------------+
| key    | value                                        |
+--------+----------------------------------------------+
| id     | ats_XXXXXXXXXXXXXXXXXXXXXXXX                 |
| secret | YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY |
+--------+----------------------------------------------+
ascli config preset update my_ibm_ats --ats-key=ats_XXXXXXXXXXXXXXXXXXXXXXXX --ats-secret=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

Misc. Examples

Example: create access key on IBM Cloud (softlayer):

ascli ats access_key create --cloud=softlayer --region=ams --params=@json:'{"storage":{"type":"softlayer_swift","container":"_container_name_","credentials":{"api_key":"_secret_here_","username":"_name_:_usr_name_"},"path":"/"},"id":"_optional_id_","name":"_optional_name_"}'

Example: create access key on AWS:

ascli ats access_key create --cloud=aws --region=eu-west-1 --params=@json:'{"id":"testkey3","name":"laurent key AWS","storage":{"type":"aws_s3","bucket":"my-bucket","credentials":{"access_key_id":"AKIA_MY_API_KEY","secret_access_key":"_secret_here_"},"path":"/laurent"}}'

Example: create access key on Azure SAS:

ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure_sas","credentials":{"shared_access_signature":"https://containername.blob.core.windows.net/blobname?sr=c&..."},"path":"/"}}'

(Note that the blob name is mandatory after server address and before parameters. and that parameter sr=c is mandatory.)

Example: create access key on Azure:

ascli ats access_key create --cloud=azure --region=eastus --params=@json:'{"id":"testkeyazure","name":"laurent key azure","storage":{"type":"azure","credentials":{"account":"myaccount","key":"myaccesskey","storage_endpoint":"myblob"},"path":"/"}}'

delete all my access keys:

for k in $(ascli ats access_key list --field=id --format=csv);do ascli ats access_key id $k delete;done

The parameters provided to ATS for access key creation are the ones of ATS API for the POST /access_keys endpoint.

Plugin: IBM Aspera High Speed Transfer Server (transfer)

This plugin uses SSH as a session protocol (using commands ascp and ascmd) and does not use the node API. It is the legacy way of accessing an Aspera Server, often used for server to server transfers. Modern mode is to use the node API and transfer tokens.

Authentication

Both password and SSH keys auth are supported.

If username is not provided, the default transfer user xfer is used.

If no SSH password or key is provided, and a token is provided in transfer spec, then standard bypass keys are used:

ascli server --url=ssh://... --ts=@json:'{"token":"Basic abc123"}'

Multiple SSH key paths can be provided. The value of the parameter ssh_keys can be a single value or an array. Each value is a path to a private key and is expanded (~ is replaced with the user's home folder).

Examples:

ascli server --ssh-keys=~/.ssh/id_rsa
ascli server --ssh-keys=@list:,~/.ssh/id_rsa
ascli server --ssh-keys=@json:'["~/.ssh/id_rsa"]'

The underlying ssh library net::ssh provides several options that may be used depending on environment. By default the ssh library expect that an ssh-agent is running.

On Linux, if you get an error message such as:

ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: Agent not configured

or on Windows:

ERROR -- net.ssh.authentication.agent: could not connect to ssh-agent: pageant process not running

This means that you don't have such an ssh agent running, then:

check env var: SSH_AGENT_SOCK
check if the ssh key is protected with a passphrase
check the manual
To disable use of ssh-agent, use the option ssh_option like this:

ascli server --ssh-options=@ruby:'{use_agent: false}' ...

This can also be set as default using a global preset.

Example

One can test the server application using the well known demo server:

ascli config initdemo
ascli server browse /aspera-test-dir-large
ascli server download /aspera-test-dir-large/200MB

initdemo creates a option preset demoserver and set it as default for plugin server.

Plugin: IBM Aspera High Speed Transfer Server (node)

This plugin gives access to capabilities provided by HSTS node API.

File Operations

It is possible to:

browse
transfer (upload / download)
...

For transfers, it is possible to control how transfer is authorized using option: token_type:

aspera : api <upload|download>_setup is called to create the transfer spec including the Aspera token, used as is.
hybrid : same as aspera, but token is replaced with basic token like basic
basic : transfer spec is created like this:

{
  "remote_host": address of node url,
  "remote_user": "xfer",
  "ssh_port": 33001,
  "token": "Basic <base 64 encoded user/pass>",
  "direction": send/receive
}

Note that the port is assumed to be the default SSH port 33001 and transfer user is assumed to be xfer.

Central

The central subcommand uses the "reliable query" API (session and file). It allows listing transfer sessions and transferred files.

Filtering can be applied:

ascli node central file list

by providing the validator option, offline transfer validation can be done.

FASP Stream

It is possible to start a FASPStream session using the node API:

Use the "node stream create" command, then arguments are provided as a transfer-spec.

ascli node stream create --ts=@json:'{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"_pass_here_"}' --preset=stream

Watchfolder

Refer to Aspera documentation for watch folder creation.

ascli supports remote operations through the node API. Operations are:

Start watchd and watchfolderd services running as a system user having access to files
configure a watchfolder to define automated transfers

ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"user1"}}'
ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"user1"}}'
ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"user1","pass":"mypassword"}}'

Out of Transfer File Validation

Follow the Aspera Transfer Server configuration to activate this feature.

ascli node central file list --validator=ascli --data=@json:'{"file_transfer_filter":{"max_result":1}}'

:..............:..............:............:......................................:
: session_uuid :    file_id   :   status   :              path                    :
:..............:..............:............:......................................:
: 1a74444c-... : 084fb181-... : validating : /home/xfer.../PKG - my title/200KB.1 :
:..............:..............:............:......................................:

ascli node central file update --validator=ascli --data=@json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'

updated

Example: SHOD to ATS

Scenario: Access to a "Shares on Demand" (SHOD) server on AWS is provided by a partner. We need to transfer files from this third party SHOD instance into our Azure BLOB storage. Simply create an "Aspera Transfer Service" instance (https://ts.asperasoft.com), which provides access to the node API. Then create a configuration for the "SHOD" instance in the configuration file: in section "shares", a configuration named: awsshod. Create another configuration for the Azure ATS instance: in section "node", named azureats. Then execute the following command:

ascli node download /share/sourcefile --to-folder=/destinationfolder --preset=awsshod --transfer=node --transfer-info=@preset:azureats

This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files.

Create access key

ascli node access_key create --value=@json:'{"id":"eudemo-sedemo","secret":"mystrongsecret","storage":{"type":"local","path":"/data/asperafiles"}}'

Plugin: IBM Aspera Faspex5

This is currently in beta, limited operations are supported.

This was tested with version Beta 5.

The API is listed in Faspex 5 API Reference under "IBM Aspera Faspex API".

Faspex 5 authentication

3 authentication methods are supported:

jwt
web
boot

Faspex 5 using JWT

This is the preferred method to use.

For JWT, create an API client in Faspex with JWT support:

Identify a private key, if you don't have any refer to section Private Key
Navigate to the web UI: Admin → Configurations → API Clients → Create
Activate JWT
Paste public key in the appropriate section
Click on Create Button
Take note of Client Id and Secret

Then use options:

--auth=jwt
--client-id=xxx
--client-secret=xxx
--username=xxx
--password=xxx
--private-key=@file:../path/to/key.pem

Faspex 5 using web browser

For web method, create an API client in Faspex without JWT:

Navigate to the web UI: Admin → Configurations → API Clients → Create
Do not Activate JWT
enter https://127.0.0.1:8888 in the redirect URI
Click on Create Button
Take note of Client Id

Then use options:

--auth=web
--client-id=xxx
--redirect-uri=https://127.0.0.1:8888

Faspex 5 using bootstrap

For boot method: (will be removed in future)

open a browser
start developer mode
login to faspex 5
find the first API call with Authorization token, and copy it (kind of base64 long string)

Use it as password and use --auth=boot.

ascli conf id f5boot update --url=https://localhost/aspera/faspex --auth=boot --password=ABC.DEF.GHI...

Plugin: IBM Aspera Faspex (4.x)

Notes:

The command "v4" requires the use of APIv4, refer to the Faspex Admin manual on how to activate.
For full details on Faspex API, refer to: Reference on Developer Site

Listing Packages

Command: faspex package list

Option `box`

By default it looks in box inbox, but the following boxes are also supported: archive and sent, selected with option box.

Option `recipient`

A user can receive a package because the recipient is:

the user himself (default)
the user is member of a dropbox/workgroup: filter using option recipient set with value *<name of dropbox/workgroup>

Option `query`

As inboxes may be large, it is possible to use the following query parameters:

count : (native) number items in one API call (default=0, equivalent to 10)
page : (native) id of page in call (default=0)
startIndex : (native) index of item to start, default=0, oldest index=0
max : maximum number of items
pmax : maximum number of pages

(SQL query is LIMIT <startIndex>, <count>)

The API is listed in Faspex 4 API Reference under "Services (API v.3)".

If no parameter max or pmax is provided, then all packages will be listed in the inbox, which result in paged API calls (using parameter: count and page). By default page is 0 (10), it can be increased to have less calls.

Example: list packages in dropbox

ascli faspex package list --box=inbox --recipient='*my_dropbox' --query=@json:'{"max":20,"pmax":2,"count":20}'

List a maximum of 20 items grouped by pages of 20, with maximum 2 pages in received box (inbox) when received in dropbox *my_dropbox.

Receiving a Package

The command is package recv, possible methods are:

provide a package id with option id
provide a public link with option link
provide a faspe: URI with option link

ascli faspex package recv --id=12345
ascli faspex package recv --link=faspe://...

If the package is in a specific dropbox/workgroup, add option recipient for both the list and recv commands.

ascli faspex package list --recipient='*thedropboxname'
ascli faspex package recv 125 --recipient='*thedropboxname'

if id is set to ALL, then all packages are downloaded, and if option once_onlyis used, then a persistency file is created to keep track of already downloaded packages.

Sending a Package

The command is faspex package send. Package information (title, note, metadata, options) is provided in option delivery_info. (Refer to Faspex API).

Example:

ascli faspex package send --delivery-info=@json:'{"title":"my title","recipients":["[email protected]"]}' --url=https://faspex.corp.com/aspera/faspex --username=foo --password=bar /tmp/file1 /home/bar/file2

If the recipient is a dropbox or workgroup: provide the name of the dropbox or workgroup preceded with * in the recipients field of the delivery_info option: "recipients":["*MyDropboxName"]

Additional optional parameters in delivery_info:

Package Note: : "note":"note this and that"
Package Metadata: "metadata":{"Meta1":"Val1","Meta2":"Val2"}

Email notification on transfer

Like for any transfer, a notification can be sent by email using parameters: notif_to and notif_template .

Example:

ascli faspex package send --delivery-info=@json:'{"title":"test pkg 1","recipients":["[email protected]"]}' ~/Documents/Samples/200KB.1 [email protected] --notif-template=@ruby:'%Q{From: <%=from_name%> <<%=from_email%>>\nTo: <<%=to%>>\nSubject: Package sent: <%=ts["tags"]["aspera"]["faspex"]["metadata"]["_pkg_name"]%> files received\n\nTo user: <%=ts["tags"]["aspera"]["faspex"]["recipients"].first["email"]%>}'

In this example the notification template is directly provided on command line. Package information placed in the message are directly taken from the tags in transfer spec. The template can be placed in a file using modifier: @file:

Operation on dropboxes

Example:

ascli faspex v4 dropbox create --value=@json:'{"dropbox":{"e_wg_name":"test1","e_wg_desc":"test1"}}'
ascli faspex v4 dropbox list
ascli faspex v4 dropbox delete --id=36

Remote sources

Faspex lacks an API to list the contents of a remote source (available in web UI). To workaround this, the node API is used, for this it is required to add a section ":storage" that links a storage name to a node config and sub path.

Example:

my_faspex_conf:
  url: https://10.25.0.3/aspera/faspex
  username: admin
  password: MyUserPassword
  storage:
    testlaurent:
      node: "@preset:my_faspex_node"
      path: /myfiles
my_faspex_node:
  url: https://10.25.0.3:9092
  username: node_faspex
  password: MyNodePassword

In this example, a faspex storage named "testlaurent" exists in Faspex, and is located under the docroot in "/myfiles" (this must be the same as configured in Faspex). The node configuration name is "my_faspex_node" here.

Note: the v4 API provides an API for nodes and shares.

Automated package download (cargo)

It is possible to tell ascli to download newly received packages, much like the official cargo client, or drive. Refer to the same section in the Aspera on Cloud plugin:

ascli faspex packages recv --id=ALL --once-only=yes --lock-port=12345

Plugin: IBM Aspera Shares

Aspera Shares supports the "node API" for the file transfer part. (Shares 1 and 2)

Plugin: IBM Cloud Object Storage

The IBM Cloud Object Storage provides the possibility to execute transfers using FASP. It uses the same transfer service as Aspera on Cloud, called Aspera Transfer Service (ATS). Available ATS regions: https://status.aspera.io

There are two possibilities to provide credentials. If you already have the endpoint, apikey and CRN, use the first method. If you don't have credentials but have access to the IBM Cloud console, then use the second method.

Using endpoint, apikey and Resource Instance ID (CRN)

If you have those parameters already, then following options shall be provided:

bucket bucket name
endpoint storage endpoint url, e.g. https://s3.hkg02.cloud-object-storage.appdomain.cloud
apikey API Key
crn resource instance id

For example, let us create a default configuration:

ascli conf id mycos update --bucket=mybucket --endpoint=https://s3.us-east.cloud-object-storage.appdomain.cloud --apikey=abcdefgh --crn=crn:v1:bluemix:public:iam-identity::a/xxxxxxx
ascli conf id default set cos mycos

Then, jump to the transfer example.

Using service credential file

If you are the COS administrator and don't have yet the credential: Service credentials are directly created using the IBM cloud web ui. Navigate to:

Navigation Menu → Resource List → Storage → Cloud Object Storage → Service Credentials → <select or create credentials> → view credentials → copy

Then save the copied value to a file, e.g. : $HOME/cos_service_creds.json

or using the IBM Cloud CLI:

ibmcloud resource service-keys
ibmcloud resource service-key aoclaurent --output JSON|jq '.[0].credentials'>$HOME/service_creds.json

(if you don't have jq installed, extract the structure as follows)

It consists in the following structure:

{
  "apikey": "_api_key_here_",
  "cos_hmac_keys": {
    "access_key_id": "_access_key_here_",
    "secret_access_key": "_secret_here_"
  },
  "endpoints": "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints",
  "iam_apikey_description": "my description _here_ ...",
  "iam_apikey_name": "my key name _here_",
  "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
  "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/xxxxxxx.....",
  "resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/xxxxxxx....."
}

The field resource_instance_id is for option crn

The field apikey is for option apikey

(If needed: endpoints for regions can be found by querying the endpoints URL.)

The required options for this method are:

bucket bucket name
region bucket region, e.g. eu-de
service_credentials see below

For example, let us create a default configuration:

ascli conf id mycos update --bucket=laurent --service-credentials=@val:@json:@file:~/service_creds.json --region=us-south
ascli conf id default set cos mycos

Operations, transfers

Let's assume you created a default configuration from once of the two previous steps (else specify the access options on command lines).

A subset of node plugin operations are supported, basically node API:

ascli cos node info
ascli cos node upload 'faux:///sample1G?1g'

Note: we generate a dummy file sample1G of size 2GB using the faux PVCL (man ascp and section above), but you can of course send a real file by specifying a real file instead.

Plugin: IBM Aspera Sync

A basic plugin to start an "async" using ascli. The main advantage is the possibility to start from ma configuration file, using ascli standard options.

Plugin: Preview

The preview generates "previews" of graphical files, i.e. thumbnails (office, images, video) and video previews on storage for use primarily in the Aspera on Cloud application. This is based on the "node API" of Aspera HSTS when using Access Keys only inside it's "storage root". Several parameters can be used to tune several aspects:

methods for detection of new files needing generation
methods for generation of video preview
parameters for video handling

Aspera Server configuration

Specify the previews folder as shown in:

https://ibmaspera.com/help/admin/organization/installing_the_preview_maker

By default, the preview plugin expects previews to be generated in a folder named previews located in the storage root. On the transfer server execute:

PATH=/opt/aspera/bin:$PATH

asconfigurator -x "server;preview_dir,previews"
asnodeadmin --reload

Note: the configuration preview_dir is relative to the storage root, no need leading or trailing /. In general just set the value to previews

If another folder is configured on the HSTS, then specify it to ascli using the option previews_folder.

The HSTS node API limits any preview file to a parameter: max_request_file_create_size_kb (1 KB is 1024 bytes). This size is internally capped to 1<<24 Bytes (16777216) , i.e. 16384 KBytes.

To change this parameter in aspera.conf, use asconfigurator. To display the value, use asuserdata:

asuserdata -a | grep max_request_file_create_size_kb

  max_request_file_create_size_kb: "1024"

asconfigurator -x "server; max_request_file_create_size_kb,16384"

If you use a value different than 16777216, then specify it using option max_size.

Note: the HSTS parameter (max_request_file_create_size_kb) is in kiloBytes while the generator parameter is in Bytes (factor of 1024).

External tools: Linux

The tool requires the following external tools available in the PATH:

ImageMagick : convert composite
OptiPNG : optipng
FFmpeg : ffmpeg ffprobe
Libreoffice : libreoffice

Here shown on Redhat/CentOS.

Other OSes should work as well, but are note tested.

To check if all tools are found properly, execute:

ascli preview check

Image: ImageMagick and optipng

yum install -y ImageMagick optipng

You may also install ghostscript which adds fonts to ImageMagick. Available fonts, used to generate png for text, can be listed with magick identify -list font. Prefer ImageMagick version >=7.

Video: FFmpeg

The easiest method is to download and install the latest released version of ffmpeg with static libraries from https://johnvansickle.com/ffmpeg/

curl -s https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz|(mkdir -p /opt && cd /opt && rm -f ffmpeg /usr/bin/{ffmpeg,ffprobe} && rm -fr ffmpeg-*-amd64-static && tar xJvf - && ln -s ffmpeg-* ffmpeg && ln -s /opt/ffmpeg/{ffmpeg,ffprobe} /usr/bin)

Office: Unoconv and Libreoffice

If you don't want to have preview for office documents or if it is too complex you can skip office document preview generation by using option: --skip-types=office

The generation of preview in based on the use of unoconv and libreoffice

CentOS 8

dnf install unoconv

Amazon Linux

amazon-linux-extras enable libreoffice
yum clean metadata
yum install libreoffice-core libreoffice-calc libreoffice-opensymbol-fonts libreoffice-ure libreoffice-writer libreoffice-pyuno libreoffice-impress
wget https://raw.githubusercontent.com/unoconv/unoconv/master/unoconv
mv unoconv /usr/bin
chmod a+x /usr/bin/unoconv

Configuration

The preview generator is run as a user, preferably a regular user (not root). When using object storage, any user can be used, but when using local storage it is usually better to use the user xfer, as uploaded files are under this identity: this ensures proper access rights. (we will assume this)

Like any ascli commands, parameters can be passed on command line or using a configuration option preset. The configuration file must be created with the same user used to run so that it is properly used on runtime.

Note that the xfer user has a special protected shell: aspshell, so changing identity requires specification of alternate shell:

su -s /bin/bash - xfer

ascli config preset update previewconf --url=https://localhost:9092 --username=my_access_key --password=my_secret --skip-types=office --lock-port=12346

ascli config preset set default preview previewconf

Here we assume that Office file generation is disabled, else remove this option. lock_port prevents concurrent execution of generation when using a scheduler.

One can check if the access key is well configured using:

ascli -Ppreviewconf node browse /

This shall list the contents of the storage root of the access key.

Execution

The tool intentionally supports only a "one shot" mode (no infinite loop) in order to avoid having a hanging process or using too many resources (calling REST api too quickly during the scan or event method). It needs to be run on a regular basis to create or update preview files. For that use your best reliable scheduler. For instance use "CRON" on Linux or Task Scheduler on Windows.

Typically, for "Access key" access, the system/transfer is xfer. So, in order to be consistent have generate the appropriate access rights, the generation process should be run as user xfer.

Lets do a one shot test, using the configuration previously created:

su -s /bin/bash - xfer

ascli preview scan --overwrite=always

When the preview generator is first executed it will create a file: .aspera_access_key in the previews folder which contains the access key used. On subsequent run it reads this file and check that previews are generated for the same access key, else it fails. This is to prevent clash of different access keys using the same root.

Configuration for Execution in scheduler

Here is an example of configuration for use with cron on Linux. Adapt the scripts to your own preference.

We assume here that a configuration preset was created as shown previously.

Lets first setup a script that will be used in the scheduler and sets up the environment.

Example of startup script cron_ascli, which sets the Ruby environment and adds some timeout protection:

 #!/bin/bash
 # set a timeout protection, just in case
case "$*" in *trev*) tmout=10m ;; *) tmout=30m ;; esac
. /etc/profile.d/rvm.sh
rvm use 2.6 --quiet
exec timeout ${tmout} ascli "${@}"

Here the cronjob is created for user xfer.

crontab<<EOF
0    * * * *  /home/xfer/cron_ascli preview scan --logger=syslog --display=error
2-59 * * * *  /home/xfer/cron_ascli preview trev --logger=syslog --display=error
EOF

Note that the logging options are kept in the cronfile instead of conf file to allow execution on command line with output on command line.

Candidate detection for creation or update (or deletion)

The tool generates preview files using those commands:

trevents : only recently uploaded files will be tested (transfer events)
events : only recently uploaded files will be tested (file events: not working)
scan : recursively scan all files under the access key's "storage root"
test : test using a local file

Once candidate are selected, once candidates are selected, a preview is always generated if it does not exist already, else if a preview already exist, it will be generated using one of three values for the overwrite option:

always : preview is always generated, even if it already exists and is newer than original
never : preview is generated only if it does not exist already
mtime : preview is generated only if the original file is newer than the existing

Deletion of preview for deleted source files: not implemented yet (TODO).

If the scan or events detection method is used, then the option : skip_folders can be used to skip some folders. It expects a list of path relative to the storage root (docroot) starting with slash, use the @json: notation, example:

ascli preview scan --skip-folders=@json:'["/not_here"]'

The option folder_reset_cache forces the node service to refresh folder contents using various methods.

When scanning the option value has the same behavior as for the node find command.

For instance to filter out files beginning with ._ do:

... --value='exec:!f["name"].start_with?("._") or f["name"].eql?(".DS_Store")'

Preview File types

Two types of preview can be generated:

png: thumbnail
mp4: video preview (only for video)

Use option skip_format to skip generation of a format.

Supported input Files types

The preview generator supports rendering of those file categories:

image
pdf
plaintext
office
video

To avoid generation for some categories, specify a list using option skip_types.

Each category has a specific rendering method to produce the png thumbnail.

The mp4 video preview file is only for category video

File type is primarily based on file extension detected by the node API and translated info a mime type returned by the node API.

The tool can also locally detect the mime type using option mimemagic.

To use it, set option mimemagic to yes: --mimemagic=yes

If not used, Mime type used for conversion is the one provided by the node API.

If used, the preview command will first analyze the file content using mimemagic, and if no match, will try by extension.

Access to original files and preview creation

Standard open source tools are used to create thumbnails and video previews. Those tools require that original files are accessible in the local file system and also write generated files on the local file system. The tool provides 2 ways to read and write files with the option: file_access

If the preview generator is run on a system that has direct access to the file system, then the value local can be used. In this case, no transfer happen, source files are directly read from the storage, and preview files are directly written to the storage.

If the preview generator does not have access to files on the file system (it is remote, no mount, or is an object storage), then the original file is first downloaded, then the result is uploaded, use method remote.

SMTP for email notifications

Aspera CLI can send email, for that setup SMTP configuration. This is done with option smtp.

The smtp option is a hash table (extended value) with the following fields:

field	default	example	description
`server`	-	smtp.gmail.com	SMTP server address
`tls`	true	false	use of TLS
`port`	587 for tls 25 else	587	port for service
`domain`	domain of server	gmail.com	email domain of user
`username`	-	[email protected]	user to authenticate on SMTP server, leave empty for open auth.
`password`	-	MyP@ssword	password for above username
`from_email`	username if defined	[email protected]	address used if received replies
`from_name`	same as email	John Wayne	display name of sender

Example of configuration:

ascli config preset set smtp_google server smtp.google.com
ascli config preset set smtp_google username [email protected]
ascli config preset set smtp_google password _pass_here_

or

ascli config preset init smtp_google @json:'{"server":"smtp.google.com","username":"[email protected]","password":"_pass_here_"}'

or

ascli config preset update smtp_google --server=smtp.google.com [email protected] --password=_pass_here_

Set this configuration as global default, for instance:

ascli config preset set cli_default smtp @val:@preset:smtp_google
ascli config preset set default config cli_default

Email templates

Sent emails are built using a template that uses the ERB syntax.

The template is the full SMTP message, including headers.

The following variables are defined by default:

from_name
from_email
to

Other variables are defined depending on context.

Test

Check settings with smtp_settings command. Send test email with email_test.

ascli config --smtp=@preset:smtp_google smtp
ascli config --smtp=@preset:smtp_google email [email protected]

Notifications for transfer status

An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).

To activate, use option notif_to.

A default e-mail template is used, but it can be overridden with option notif_template.

The environment provided contains the following additional variables:

subject
body
global_transfer_status
ts

Example of template:

From: <%=from_name%> <<%=from_email%>>
To: <<%=to%>>
Subject: <%=subject%>

Transfer is: <%=global_transfer_status%>

Tool: `asession`

This gem comes with a second executable tool providing a simplified standardized interface to start a FASP session: asession.

It aims at simplifying the startup of a FASP session from a programmatic stand point as formatting a transfer-spec is:

common to Aspera Node API (HTTP POST /ops/transfer)
common to Aspera Connect API (browser javascript startTransfer)
easy to generate by using any third party language specific JSON library

Hopefully, IBM integrates this diectly in ascp, and this tool is made redundant.

This makes it easy to integrate with any language provided that one can spawn a sub process, write to its STDIN, read from STDOUT, generate and parse JSON.

The tool expect one single argument: a transfer-spec.

If not argument is provided, it assumes a value of: @json:@stdin:, i.e. a JSON formatted transfer-spec on stdin.

Note that if JSON is the format, one has to specify @json: to tell the tool to decode the hash using JSON.

During execution, it generates all low level events, one per line, in JSON format on stdout.

Note that there are special "extended" transfer-spec parameters supported by asession:

EX_loglevel to change log level of the tool
EX_file_list_folder to set the folder used to store (exclusively, because of garbage collection) generated file lists. By default it is [system tmp folder]/[username]_asession_filelists

Note that in addition, many "EX_" transfer-spec parameters are supported for the direct transfer agent (used by asession), refer to section transfer-spec.

Comparison of interfaces

feature/tool	asession	ascp	FaspManager	Transfer SDK
language integration	any	any	C/C++ C#/.net Go Python java	any
additional components to ascp	Ruby Aspera	-	library (headers)	daemon
startup	JSON on stdin (standard APIs: JSON.generate Process.spawn)	command line arguments	API	daemon
events	JSON on stdout	none by default or need to open management port and proprietary text syntax	callback	callback
platforms	any with ruby and ascp	any with ascp	any with ascp	any with ascp and transferdaemon

Simple session

MY_TSPEC='{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"_pass_here_","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}'

echo "${MY_TSPEC}"|asession

Asynchronous commands and Persistent session

asession also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be "snake" style, for example: LongParameter -> long_parameter

This is particularly useful for a persistent session ( with the transfer-spec parameter: "keepalive":true )

asession
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"_pass_here_","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
{"type":"DONE"}

(events from FASP are not shown in above example. They would appear after each command)

Example of language wrapper

Nodejs: https://www.npmjs.com/package/aspera

Help

asession -h
USAGE
    asession
    asession -h|--help
    asession <transfer spec extended value>
    
    If no argument is provided, default will be used: @json:@stdin
    -h, --help display this message
    <transfer spec extended value> a JSON value for transfer_spec, using the prefix: @json:
    The value can be either:
       the JSON description itself, e.g. @json:'{"xx":"yy",...}'
       @json:@stdin, if the JSON is provided from stdin
       @json:@file:<path>, if the JSON is provided from a file
    Asynchronous commands can be provided on STDIN, examples:
       {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
       {"type":"START","source":"xx","destination":"yy"}
       {"type":"DONE"}
Note: debug information can be placed on STDERR, using the "EX_loglevel" parameter in transfer spec (debug=0)
EXAMPLES
    asession @json:'{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"demoaspera","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}]}'
    echo '{"remote_host":...}'|asession @json:@stdin

Hot folder

Requirements

ascli maybe used as a simple hot folder engine. A hot folder being defined as a tool that:

locally (or remotely) detects new files in a top folder
send detected files to a remote (respectively, local) repository
only sends new files, do not re-send already sent files
optionally: sends only files that are not still "growing"
optionally: after transfer of files, deletes or moves to an archive

In addition: the detection should be made "continuously" or on specific time/date.

Setup procedure

The general idea is to rely on :

existing ascp features for detection and transfer
take advantage of ascli configuration capabilities and server side knowledge
the OS scheduler for reliability and continuous operation

ascp features

Interesting ascp features are found in its arguments: (see ascp manual):

ascp already takes care of sending only "new" files: option -k 1,2,3, or transfer_spec: resume_policy
ascp has some options to remove or move files after transfer: --remove-after-transfer, --move-after-transfer, --remove-empty-directories
ascp has an option to send only files not modified since the last X seconds: --exclude-newer-than (--exclude-older-than)
--src-base if top level folder name shall not be created on destination

Note that:

ascli takes transfer parameters exclusively as a transfer_spec, with --ts parameter.
most, but not all native ascp arguments are available as standard transfer_spec parameters
native ascp arguments can be provided with the transfer-spec parameter: EX_ascp_args (array), only for the direct transfer agent (not connect or node)

server side and configuration

Virtually any transfer on a "repository" on a regular basis might emulate a hot folder. Note that file detection is not based on events (inotify, etc...), but on a stateless scan on source side.

Note: parameters may be saved in a option preset and used with -P.

Scheduling

Once ascli parameters are defined, run the command using the OS native scheduler, e.g. every minutes, or 5 minutes, etc... Refer to section Scheduling.

Example: upload folder

ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"EX_ascp_args":["--remove-after-transfer","--remove-empty-directories","--exclude-newer-than=-8","--src-base","source_hot"]}'

The local folder (here, relative path: source_hot) is sent (upload) to basic fasp server, source files are deleted after transfer. growing files will be sent only once they don't grow anymore (based on an 8-second cooloff period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock-port).

Health check and Nagios

Most plugin provide a health command that will check the health status of the application. Example:

ascli console health

+--------+-------------+------------+
| status | component   | message    |
+--------+-------------+------------+
| ok     | console api | accessible |
+--------+-------------+------------+

Typically, the health check uses the REST API of the application with the following exception: the server plugin allows checking health by:

issuing a transfer to the server
checking web app status with asctl all:status
checking daemons process status

ascli can be called by Nagios to check the health status of an Aspera server. The output can be made compatible to Nagios with option --format=nagios :

ascli server health transfer --to-folder=/Upload --format=nagios --progress=none

OK - [transfer:ok]

ascli server health asctlstatus --cmd_prefix='sudo ' --format=nagios

OK - [NP:running, MySQL:running, Mongrels:running, Background:running, DS:running, DB:running, Email:running, Apache:running]

Ruby Module: `Aspera`

Main components:

Aspera generic classes for REST and OAuth
Aspera::Fasp: starting and monitoring transfers. It can be considered as a FASPManager class for Ruby.
Aspera::Cli: ascli.

A working example can be found in the gem, example:

ascli config gem path

cat $(ascli config gem path)/../examples/transfer.rb

This sample code shows some example of use of the API as well as REST API. Note: although nice, it's probably a good idea to use RestClient for REST.

Example of use of the API of Aspera on Cloud:

require 'aspera/aoc'

aoc=Aspera::AoC.new(url: 'https://sedemo.ibmaspera.com',auth: :jwt, scope: 'user:all', private_key: File.read(File.expand_path('~/.aspera/ascli/aspera_on_cloud_key')),username: '[email protected]',subpath: 'api/v1')

aoc.read('self')

https://github.com/IBM/aspera-cli/blob/main/examples/aoc.rb

History

When I joined Aspera, there was only one CLI: ascp, which is the implementation of the FASP protocol, but there was no CLI to access the various existing products (Server, Faspex, Shares). Once, Serban (founder) provided a shell script able to create a Faspex Package using Faspex REST API. Since all products relate to file transfers using FASP (ascp), I thought it would be interesting to have a unified CLI for transfers using FASP. Also, because there was already the ascp tool, I thought of an extended tool : eascp.pl which was accepting all ascp options for transfer but was also able to transfer to Faspex and Shares (destination was a kind of URI for the applications).

There were a few pitfalls:

The tool was written in the aging perl language while most Aspera application products (but the Transfer Server) are written in ruby.
The tool was only for transfers, but not able to call other products APIs

So, it evolved into ascli:

portable: works on platforms supporting ruby (and ascp)
easy to install with the gem utility
supports transfers with multiple Transfer Agents, that's why transfer parameters moved from ascp command line to transfer-spec (more reliable , more standard)
ruby is consistent with other Aspera products

Changes (Release notes)

4.8.0.pre *
4.7.0
- new: option to specify font used to generate image of text file in preview
- new: #66 improvement for content protection (support standard transfer spec options for direct agent)
- new: option fpac is now applicable to all ruby based HTTP connections, i.e. API calls
- new: option show_secrets to reveal secrets in command output
- new: added and updated commands for Faspex 5
- new: option cache_tokens
- new: Faspex4 dropbox packages can now be received by id
- change: (break) command conf gem path replaces conf gem_path
- change: (break) option fpac expects a value instead of URL
- change: (break) option cipher in transfer spec must have hyphen
- change: (break) renamed option log_passwords to log_secrets
- change: (break) removed plugin shares2 as products is now EOL
- fix: After AoC version update, wizard did not detect AoC properly
4.6.0
- new: command conf plugin create
- new: global option plugin_folder
- new: global option transpose_single
- new: simplified metadata passing for shared inbox package creation in AoC
- change: (break) command aoc packages shared_inboxes list replaces aoc user shared_inboxes
- change: (break) command aoc user profile replaces aoc user info
- change: (break) command aoc user workspaces list replaces aoc user workspaces
- change: (break) command aoc user workspaces current replaces aoc workspace
- change: (break) command conf plugin list replaces conf plugins
- change: (break) command conf connect simplified
- fix: #60 ascli executable was not installed by default in 4.5.0
- fix: add password hiding case in logs
4.5.0
- new: support transfer agent: Transfer SDK
- new: support http socket options
- new: logs hide passwords and secrets, option log_passwords to enable logging secrets
- new: config vault supports encrypted passwords, also macos keychain
- new: config preset command for consistency with id
- new: identifier can be provided using either option id or directly after the command, e.g. delete 123 is the same as delete --id=123
- change: when using wss, use ruby's CA certs
- change: unexpected parameter makes exit code not zero
- change: (break) options id and name cannot be specified at the same time anymore, use positional identifer or name selection
- change: (break) aoc admin res node does not take workspace main node as default node if no id specified.
- change: (break): orchestrator workflow status requires id, and supports special id ALL
- fix: various smaller fixes and renaming of some internal classes (transfer agents and few other)
4.4.0
- new: aoc packages list add possibility to add filter with option query
- new: aoc admin res xxx list now get all items by default #50
- new: preset option can specify name or hash value
- new: node plugin accepts bearer token and access key as credential
- new: node option token_type allows using basic token in addition to aspera type.
- change: server: option username not mandatory anymore: xfer user is by default. If transfer spec token is provided, password or keys are optional, and bypass keys are used by default.
- change: (break) resource apps_new of aoc replaced with application (more clear)
4.3.0
- new: parameter multi_incr_udp for option transfer_info: control if UDP port is incremented when multi-session is used on direct transfer agent.
- new: command aoc files node_info to get node information for a given folder in the Files application of AoC. Allows cross-org or cross-workspace transfers.
4.2.2
- new: faspex package list retrieves the whole list, not just first page
- new: support web based auth to aoc and faspex 5 using HTTPS, new dependency on gem webrick
- new: the error "Remote host is not who we expected" displays a special remediation message
- new: conf ascp spec displays supported transfer spec
- new: options notif_to and notif_template to send email notifications on transfer (and other events)
- fix: space character in faspe: url are precent encoded if needed
- fix: preview scan: if file_id is unknown, ignore and continue scan
- change: for commands that potentially execute several transfers (package recv --id=ALL), if one transfer fails then ascli exits with code 1 (instead of zero=success)
- change: (break) option notify or aoc replaced with notif_to and notif_template
4.2.1
- new: command faspex package recv supports link of type: faspe:
- new: command faspex package recv supports option recipient to specify dropbox with leading *
4.2.0
- new: command aoc remind to receive organization membership by email
- new: in preview option value to filter out on file name
- new: initdemo to initialize for demo server
- new: direct transfer agent options: spawn_timeout_sec and spawn_delay_sec
- fix: on Windows conf ascp use expects ascp.exe
- fix: (break) multi_session_threshold is Integer, not String
- fix: conf ascp install renames sdk folder if it already exists (leftover shared lib may make fail)
- fix: removed replace_illegal_chars from default aspera.conf causing "Error creating illegal char conversion table"
- change: (break) aoc apiinfo is removed, use aoc servers to provide the list of cloud systems
- change: (break) parameters for resume in transfer-info for direct are now in sub-key "resume"
4.1.0
- fix: remove keys from transfer spec and command line when not needed * fix: default to create_dir:true so that sending single file to a folder does not rename file if folder does not exist
- new: update documentation with regard to offline and docker installation
- new: renamed command nagios_check to health
- new: agent http_gw now supports upload
- new: added option sdk_url to install SDK from local file for offline install
- new: check new gem version periodically
- new: the --fields= option, support -fieldname to remove a field from default fields
- new: Oauth tokens are discarded automatically after 30 minutes (useful for COS delegated refresh tokens)
- new: mimemagic is now optional, needs manual install for preview, compatible with version 0.4.x
- new: AoC a password can be provided for a public link
- new: conf doc take an optional parameter to go to a section
- new: initial support for Faspex 5 Beta 1
4.0.0
- now available as open source at https://github.com/IBM/aspera-cli with general cleanup
- changed default tool name from mlia to ascli
- changed aspera command to aoc
- changed gem name from asperalm to aspera-cli
- changed module name from Asperalm to Aspera
- removed command folder in preview, merged to scan
- persistency files go to sub folder instead of main folder
- added possibility to install SDK: config ascp install
0.11.8
- Simplified to use unoconv instead of bare libreoffice for office conversion, as unoconv does not require a X server (previously using Xvfb
0.11.7
- rework on rest call error handling
- use option display with value data to remove out of extraneous information
- fixed option lock_port not working
- generate special icon if preview failed
- possibility to choose transfer progress bar type with option progress
- AoC package creation now output package id
0.11.6
- orchestrator : added more choice in auth type
- preview: cleanup in generator (removed and renamed parameters)
- preview: better documentation
- preview: animated thumbnails for video (option: video_png_conv=animated)
- preview: new event trigger: trevents (events seems broken)
- preview: unique tmp folder to avoid clash of multiple instances
- repo: added template for secrets used for testing
0.11.5
- added option default_ports for AoC (see manual)
- allow bulk delete in aspera files with option bulk=yes
- fix getting connect versions
- added section for Aix
- support all ciphers for direct agent (including gcm, etc..)
- added transfer spec param apply_local_docroot for direct
0.11.4
- possibility to give shared inbox name when sending a package (else use id and type)
0.11.3
- minor fixes on multi-session: avoid exception on progress bar
0.11.2
- fixes on multi-session: progress bat and transfer spec param for "direct"
0.11.1
- enhanced short_link creation commands (see examples)
0.11
- add transfer spec option (agent direct only) to provide file list directly to ascp: EX_file_list.
0.10.18
- new option in. server : ssh_options
0.10.17
- fixed problem on server for option ssh_keys, now accepts both single value and list.
- new modifier: @list:<separator>val1<separator>...
0.10.16
- added list of shared inboxes in workspace (or global), use --query=@json:'{}'
0.10.15
- in case of command line error, display the error cause first, and non-parsed argument second
- AoC : Activity / Analytics
0.10.14
- added missing bss plugin
0.10.13
- added Faspex5 (use option value to give API arguments)
0.10.12
- added support for AoC node registration keys
- replaced option : local_resume with transfer_info for agent direct
- Transfer agent is no more a Singleton instance, but only one is used in CLI
- @incps : new extended value modifier
- ATS: no more provides access keys secrets: now user must provide it
- begin work on "aoc" transfer agent
0.10.11
- minor refactor and fixes
0.10.10
- fix on documentation
0.10.9.1
- add total number of items for AoC resource list
- better gem version dependency (and fixes to support Ruby 2.0.0)
- removed aoc search_nodes
0.10.8
- removed option: fasp_proxy, use pseudo transfer spec parameter: EX_fasp_proxy_url
- removed option: http_proxy, use pseudo transfer spec parameter: EX_http_proxy_url
- several other changes..
0.10.7
- fix: ascli fails when username cannot be computed on Linux.
0.10.6
- FaspManager: transfer spec authentication no more needed for local transfer to use Aspera public keys. public keys will be used if there is a token and no key or password is provided.
- gem version requirements made more open
0.10.5
- fix faspex package receive command not working
0.10.4
- new options for AoC : secrets
- ACLI-533 temp file list folder to use file lists is set by default, and used by asession
0.10.3
- included user name in oauth bearer token cache for AoC when JWT is used.
0.10.2
- updated search_nodes to be more generic, so it can search not only on access key, but also other queries.
- added doc for "cargo" like actions
- added doc for multi-session
0.10.1
- AoC and node v4 "browse" works now on non-folder items: file, link
- initial support for AoC automation (do not use yet)
0.10
- support for transfer using IBM Cloud Object Storage
- improved find action using arbitrary expressions
0.9.36
- added option to specify file pair lists
0.9.35
- updated plugin preview , changed parameter names, added documentation
- fix in ats plugin : instance id needed in request header
0.9.34
- parser "@preset" can be used again in option "transfer_info"
- some documentation re-organizing
0.9.33
- new command to display basic token of node
- new command to display bearer token of node in AoC
- the --fields= option, support +fieldname to add a field to default fields
- many small changes
0.9.32
- all Faspex public links are now supported
- removed faspex operation recv_publink
- replaced with option link (consistent with AoC)
0.9.31
- added more support for public link: receive and send package, to user or dropbox and files view.
- delete expired file lists
- changed text table gem from text-table to terminal-table because it supports multiline values
0.9.27
- basic email support with SMTP
- basic proxy auto config support
0.9.26
- table display with --fields=ALL now includes all column names from all lines, not only first one
- unprocessed argument shows error even if there is an error beforehand
0.9.25
- the option value of command find, to filter on name, is not optional
- find now also reports all types (file, folder, link)
- find now is able to report all fields (type, size, etc...)
0.9.24
- fix bug where AoC node to node transfer did not work
- fix bug on error if ED25519 private key is defined in .ssh
0.9.23
- defined REST error handlers, more error conditions detected
- commands to select specific ascp location
0.9.21
- supports simplified wizard using global client
- only ascp binary is required, other SDK (keys) files are now generated
0.9.20
- improved wizard (prepare for AoC global client id)
- preview generator: addedoption : --skip-format=<png,mp4>
- removed outdated pictures from this doc
0.9.19
- added command aspera bearer --scope=xx
0.9.18
- enhanced aspera admin events to support query
0.9.16
- AoC transfers are now reported in activity app
- new interface for Rest class authentication (keep backward compatibility)
0.9.15
- new feature: "find" command in aspera files
- sample code for transfer API
0.9.12
- add nagios commands
- support of ATS for IBM Cloud, removed old version based on aspera id
0.9.11
- Breaking change: @stdin is now @stdin:
- support of ATS for IBM Cloud, removed old version based on aspera id
0.9.10
- Breaking change: parameter transfer-node becomes more generic: transfer-info
- Display SaaS storage usage with command: aspera admin res node --id=nn info
- cleaner way of specifying source file list for transfers
- Breaking change: replaced download_mode option with http_download action
0.9.9
- Breaking change: "aspera package send" parameter deprecated, use the --value option instead with "recipients" value. See example.
- Now supports "cargo" for Aspera on Cloud (automatic package download)
0.9.8
- Faspex: use option once_only set to yes to enable cargo like function. id=NEW deprecated.
- AoC: share to share transfer with command "transfer"
0.9.7
- homogeneous transfer-spec for node and direct transfer agents
- preview persistency goes to unique file by default
- catch mxf extension in preview as video
- Faspex: possibility to download all packages by specifying id=ALL
- Faspex: to come: cargo-like function to download only new packages with id=NEW
0.9.6
- Breaking change: @param:is now @preset: and is generic
- AoC: added command to display current workspace information
0.9.5
- new parameter: new_user_option used to choose between public_link and invite of external users.
- fixed bug in wizard, and wizard uses now product detection
0.9.4
- Breaking change: onCloud file list follow --source convention as well (plus specific case for download when first path is source folder, and other are source file names).
- AoC Package send supports external users
- new command to export AoC config to Aspera CLI config
0.9.3
- REST error message show host and code
- option for quiet display
- modified transfer interface and allow token re-generation on error
- async add admin command
- async add db parameters
- Breaking change: new option "sources" to specify files to transfer
0.9.2
- Breaking change: changed AoC package creation to match API, see AoC section
0.9.1
- Breaking change: changed faspex package creation to match API, see Faspex section
0.9
- Renamed the CLI from aslmcli to ascli
- Automatic rename and conversion of former config folder from aslmcli to ascli
0.7.6
- add "sync" plugin
0.7
- Breaking change: AoC package recv take option if for package instead of argument.
- Breaking change: Rest class and Oauth class changed init parameters
- AoC: receive package from public link
- select by col value on output
- added rename (AoC, node)
0.6.19
- change: (break) ats server list provisioned → ats cluster list
- change: (break) ats server list clouds → ats cluster clouds
- change: (break) ats server list instance --cloud=x --region=y → ats cluster show --cloud=x --region=y
- change: (break) ats server id xxx → ats cluster show --id=xxx
- change: (break) ats subscriptions → ats credential subscriptions
- change: (break) ats api_key repository list → ats credential cache list
- change: (break) ats api_key list → ats credential list
- change: (break) ats access_key id xxx → ats access_key --id=xxx
0.6.18
- some commands take now --id option instead of id command.
0.6.15
- Breaking change: "files" application renamed to "aspera" (for "Aspera on Cloud"). "repository" renamed to "files". Default is automatically reset, e.g. in config files and change key "files" to "aspera" in option preset "default".

Common problems

Error "Remote host is not who we expected"

Cause: ascp >= 4.x checks fingerprint of highest server host key, including ECDSA. ascp < 4.0 (3.9.6 and earlier) support only to RSA level (and ignore ECDSA presented by server). aspera.conf supports a single fingerprint.

Workaround on client side: To ignore the certificate (SSH fingerprint) add option on client side (this option can also be added permanently to the config file):

--ts=@json:'{"sshfp":null}'

Workaround on server side: Either remove the fingerprint from aspera.conf, or keep only RSA host keys in sshd_config.

References: ES-1944 in release notes of 4.1 and to HSTS admin manual section "Configuring Transfer Server Authentication With a Host-Key Fingerprint".

ED255519 key not supported

ED25519 keys are deactivated since version 0.9.24 so this type of key will just be ignored.

Without this deactivation, if such key was present the following error was generated:

OpenSSH keys only supported if ED25519 is available

Which meant that you do not have ruby support for ED25519 SSH keys. You may either install the suggested Gems, or remove your ed25519 key from your .ssh folder to solve the issue.

BUGS, FEATURES, CONTRIBUTION

For issues reports or feature requests use the Github repository and issues.

You can also contribute to this open source project.

One can also create one's own plugin.

Miscellaneous

replace rest and oauth classes with ruby standard gems:
- https://github.com/rest-client/rest-client
- https://github.com/oauth-xx/oauth2
use gem Thor http://whatisthor.com/ (or other standard Ruby CLI manager)
Package with https://github.com/pmq20/ruby-packer (rubyc)

Name		Name	Last commit message	Last commit date
Latest commit History 1,919 Commits
.github/workflows		.github/workflows
.vscode		.vscode
bin		bin
docs		docs
etc		etc
examples		examples
lib/aspera		lib/aspera
spec		spec
tests		tests
.gitignore		.gitignore
.rspec		.rspec
.rubocop.yml		.rubocop.yml
.secrets.baseline		.secrets.baseline
.solargraph.yml		.solargraph.yml
Dockerfile		Dockerfile
Gemfile		Gemfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
Rakefile		Rakefile
aspera-cli.gemspec		aspera-cli.gemspec
common.make		common.make

License

josefinarcasanova/aspera-cli

Folders and files

Latest commit

History

Repository files navigation

Command Line Interface for IBM Aspera products

When to use and when not to use

Notations, Shell and Command line parsing

Quick Start

First use

Going further

Installation

Docker container

Ruby

Generic: RVM: single user installation (not root)

Generic: RVM: global installation (as root)

Windows: Installer

macOS: pre-installed or brew

Linux: package

Other Unixes (AIX)

aspera-cli gem

FASP Protocol

Installation in air gapped environment

Command Line Interface: ascli

Arguments : Commands and options

Options

Commands and Arguments

Interactive Input

Output

Types of output data

Format of output

Option: select: Filter on columns values for object_list

Verbosity of output

Selection of output object properties

Extended Value Syntax

Structured Value

Configuration and Persistency Folder

Configuration file

Option preset

Special Option preset: config

Special Option preset: default

Special Plugin: config

Format of file

Options evaluation order

Shares Examples

Secret Vault

Using system keychain

Modern config file format: encrypted in config file

Legacy config file format

Private Key

SSL CA certificate bundle

Plugins

Create your own plugin

Plugins: Application URL and Authentication

Logging, Debugging

Learning Aspera Product APIs (REST)

HTTP socket parameters

Graphical Interactions: Browser and Text Editor

Proxy

HTTP proxy for REST calls and transfers using HTTP gateway

FASP proxy (forward) for transfers

HTTP proxy legacy Aspera HTTP fallback transfers

Proxy auto config

FASP configuration

Show path of currently used ascp

Selection of ascp location for direct agent

List locally installed Aspera Transfer products

Selection of local client for ascp for direct agent

Installation of Connect Client on command line

Transfer Agents

Direct

IBM Aspera Connect Client GUI

Aspera Node API : Node to node transfers

HTTP Gateway

Transfer SDK

Transfer Specification

Transfer Parameters

Destination folder for transfers

List of files for transfers

macOS: pre-installed or `brew`

`aspera-cli` gem

Command Line Interface: `ascli`

Option: `select`: Filter on columns values for `object_list`

Show path of currently used `ascp`

Selection of `ascp` location for `direct` agent

Selection of local client for `ascp` for `direct` agent

`faux:` for testing