Variables

Variables in a workflow

Drake lets you define and use variables throughout your workflow.

Here's a simple example of defining, then using, a variable in a simple workflow:

MYVAR=some_value

out.txt <- in.txt
  echo $[MYVAR]

The first line defines the value of MYVAR. The $[MYVAR] syntax tells Drake to substitute the value of MYVAR before interpreting the rest of the step.

There is also a conditional definition, :=, for variables. For example:

MYVAR:=some_value

The := operator tells Drake to only set the value of MYVAR to some_value if MYVAR is not already defined. (We'll talk later about ways to define variables outside of the workflow.)

Here's another example, this time setting up a kind of profiles concept for our workflow:

PROFILE:=default_profile
%include $[PROFILE]

out.txt <- $[INFILE]
  echo $[MYVAR]

The above example assumes we've organized a set of variables into profile files, such as a file called default_profile. The workflow uses %include to pull that file in, thereby allowing that file to define its custom values for our variables. Our workflow then makes use of those variables, such as INFILE and MYVAR.

An example default_profile file would be like:

INFILE=in.json
MYVAR=some_other_value

Defining variables outside the workflow

There are two ways to pass variables to a Drake workflow:

Way #1: Define environment variables before running the workflow

When Drake runs a workflow, it will pull in variable values from the environment. So if you've defined the environment variable MYVAR, for example like this...

export MYVAR=my_value

... your workflow can refer to that variable as $[MYVAR].

Way #2: Set variables via the --vars command line option

You can set variables when you run drake on the command line by using the --vars switch, then supplying a comma delimited list of variable names and values separated by =. For example:

drake --vars MYVAR=my-value,ANOTHERVAR=aValue -w mywork.Drakefile

This way takes precedence over any variables set up in the shell environment.

Note that this approach is severely limited by syntax considerations. Your variable values cannot contain syntax that would confuse the the command line, such as spaces or = symbols.

Using the $ syntax in shell commands

Before running shell steps, Drake makes sure to load the shell environment with any workflow variables that have been defined. This means your shell commands can refer to those variables with the traditional $ shell syntax. For example:

MYVAR=some_value

out.txt <- in.txt
  echo $MYVAR

Note that a big difference here is that Drake is not doing the substituting. Drake is only prepping the shell environment with MYVAR. When the echo statement runs, it literally runs on the shell as echo $MYVAR, and it's the shell that does the substitution.

Requiring variables to be defined

If Drake encounters a variable reference such as $[MYVAR] in the workflow, that variable must be defined by that point in the workflow or else Drake will error out with a message telling you that MYVAR is not defined.

Note that Drake does not provide any checking for variables you refer to in your shell commands using the $ syntax. If you refer to a variable in such a way and it is not defined when the workflow runs, the behaviour depends on shell.

Further reading

Please see the "Variables" section of the official spec for more details on the use of variables with Drake.