The base task package

The ScriptEngine base task package collects essential tasks that are commonly needed to write scripts. The base task package is pre-installed in ScriptEngine and any task within the base.* namespace can be used without further installation or setup. Many useful ScriptEngine scripts can be written with just the base task package.

The base task package contains the following tasks, described in more detail below:

base.echo, base.chdir, base.command, base.context, base.context.from,
base.copy, base.exit, base.find, base.getenv, base.include, base.link,
base.make_dir, base.move, base.remove, base.setenv, base.task_timer,
base.template, base.time, base.unsetenv

`base.echo`

This is a very basic task that can be used to write customised messages:

base.echo:
    msg: <MESSAGE>

for example:

base.echo:
    msg: Hello, world!

or, provided that planet is set in the context (base.context):

base.echo:
    msg: "Hello, {{ planet }}!"

Working with the SE context

`base.context`

Stores or updates data (one or more, possibly nested, pairs of names and values) in the ScriptEngine context:

base.context:
    <NAME>: <VALUE>
    [...]

For example:

base.context:
    planet: Earth
    some_countries:
        - Norway
        - Sweden
        - Finnland
        - Danmark
    number: 4

The arguments of base.context will be merged into the ScriptEngine context. This means that data can be added to existing data structures, such as lists or dictionaries:

- base.context:
    mylist:
        - one
        - two
# mylist is ['one', 'two']
- base.context:
    mylist:
        - 3
        - 4
# mylist is now ['one', 'two', 3, 4]

In the case of conflicts, the arguments of base.context will overwrite the current values in the ScriptEngine context. This can be used to clean data:

- base.context:
    mylist:
        - one
        - two
# mylist is ['one', 'two']
- base.context:
    mylist: null  # "remove" the value of mylist
- base.context:
    mylist:
        - 3
        - 4
# mylist is now [3, 4]

Since the ScriptEngine context will usually hold a lot of information, it is most of the time helpful to structure the parameters in nested levels. However, this can lead to overly verbose scripts, such as:

- base.context:
    my:
      deeply:
        nested:
          parameter: value

It is therefore allowed in ScriptEngine scripts to refer to context parameters using “dotted keys”, as in:

- base.context:
    my.deeply.nested.parameter: value

This feature allows a shorter notation wherever context parameters are accessed by their names.

Added in version 1.0: Allow for dotted keys for nested context parameters.

`base.context.load`

This is an extension of base.context. It updates the ScriptEngine context in the same way, but it allows to load the context update from another source instead of explicitly specifying the name-value pairs as task arguments. In particular, base.context.load accepts one of two arguments, dict or file. The arguments are mutual exclusive:

base.context.load:
    # exactly one of the two arguments:
    dict: <DICTIONARY>  # optional, mutual exclusive
    file: <FILE_NAME>  # optional, mutual exclusive

If given the dict argument, the context update is loaded from the argument value, which must be a dictionary. This may sound rather similar to the standard base.context, but it allows greater flexibility because the argument value can be taken from the context itself. For example, one could implement overwriteable default settings using this feature:

# Let the user set preferred values
- base.context:
    user_config:
        foo: 5
# [... later (could be in another script) ...]
# Set default values
- base.context:
    foo: 1
    bar: 2
# Overwrite defaults with user preferences
- base.context.load:
    dict: "{{ user_config  }}"
# result: foo==5, bar==2

The file argument of base.context.load can be used to load context values from a YAML file:

# data.yml
foo: 4
bar: 5

# script.yml
- base.context.load:
    file: data.yml

When running the script with se script.yml, the context will contain foo==4 and bar==5, provided that the file data.yml can be found in the current directory.

The only supported file format for the time being is YAML. The content of the file must be a, possibly nested, dictionary (i.e. single values or lists are not allowed).

Added in version 1.0: This task has been renamed to base.context.load (the old name was base.context.from).

Control flow

`base.include`

Reads and executes another ScriptEngine script. This is done if (see Conditionals) and when the base.include task is executed:

base.include:
    src: <PATH>
    ignore_not_found: <true or false>  # optional, default false

The script to be included is given by the src argument, which must be a path relative to

the current working directory at the moment base.include is run,
the original working directory when the se command was run, or
any of the directories that the scripts given to the se command were in.

If ignore_not_found is true, only a warning is written in case the include script is not found. If it is false (the default) an error is raised in this case.

`base.command`

Executes an external command, with optional arguments:

- base.command:
    name: <COMMAND_NAME>
    args: <LIST_OF_ARGS>  # optional
    cwd: <PATH>  # optional
    stdout: [true|false|<STRING>]  # optional
    stderr: [true|false|<STRING>]  # optional
    ignore_error: [true|false]  # optional

When cwd (current work directory) is specified, the command is executed in the given directory:

- base.command:
    name: ls
    args: [-l]
    cwd: /tmp

When the stdout is given, it can be either true, false, or a string that makes for a valid name in the ScriptEngine context. If stdout is set to true (the default), then the standard output of the command is printed as log messages on the INFO level. When stdout is false, the standard output of the command is ignored.

When stdout is a name, the standard output of the command is stored, under that name, in the ScriptEngine context, for example:

- base.command:
    name: echo
    args: [ Hello, World! ]
    stdout: message
- base.echo:
    msg: "Command returned: {{message}}"

Note that the standard output is always returned as a list of lines, even if there is only one line (as in the example above). This is often desired, for example when using the command output in a loop. However, if one wanted to extract the first (and only) line in the example above, Jinja2 syntax could be used:

- echo:
    msg: "Command returned: {{message|first}}"

Note

When tasks update the ScriptEngine context, the changes are always merged (see base.context). This implies, among other things, that list items are appended if the list is already defined in the context. This mechanism applies also to base.command and consequently output lines are appended to the context variable if it already exists.

The stderr argument works exactly as stdout, but for standard error output.

If the command returns a non-zero exit code, ScriptEngine writes the exit code as log message (on the ERROR level) and stops with an error. However, if ignore_error is true and the command returns a non-zero exit code, the exit code of the command is logged at the WARNING level instead and ScriptEngine continues. The default value for ignore_error is false.

`base.exit`

Requests ScriptEngine to stop, optionally displaying a customised message:

- base.exit:
    msg: <MESSAGE>  # optional

If the msg argument is not given, a default message is printed.

Shell environment

`base.chdir`

This task changes the current working directory:

base.chdir:
    path: <PATH>

for example:

- base.getenv:
    home: HOME
- base.chdir:
    path: "{{ home }}"

`base.getenv`

Reads one or more environment variables and stores the values in the ScriptEngine context:

- base.getenv:
    <CONTEXT_PARAMETER>: <ENV_VAR_NAME>
    [...]

for example:

- base.getenv:
    name: USER
    home: HOME
- base.echo:
    msg: "I am {{ name }} and {{ home }} is my castle."

When a requested environment variable does not exist, a warning is given and no corresponding context changes are made.

Added in version 1.0: Allow dotted keys for nested context parameters.

`base.setenv`

Sets one or more environment variables from values of the ScriptEngine context:

- base.setenv:
    <ENV_VAR_NAME>: <CONTEXT_PARAMETER>
    [...]

The following example:

- base.context:
    libs: /path/to/libraries
- base.setenv:
    LD_LIBRARY_PATH:  "{{ libs }}"
    FOO: 1
    bar: two

will set the environment variables $LD_LIBRARY_PATH to "/path/to/libraries", $FOO to "1" and $bar to "two".

Note

Environment variables are always strings! Thus, all values are converted to strings before they are assigned. In the above example, the number 1 is converted to the string "1" before it is assigned to the environment variable $FOO.

Added in version 1.0: Allow dotted keys for nested context parameters.

`base.unsetenv`

Unsets one or more environment variables:

- base.unsetenv:
    vars:
      - <ENV_VAR_NAME_1>
      - <ENV_VAR_NAME_2>
      [...]

If just one environment variable should be unset, no list is required, i.e.:

- base.unsetenv:
    vars: FOO

The following example:

- base.unsetenv:
    vars:
      - SLURM_HOSTFILE
      - LD_LIBRARY_PATH

will unset the environment variables $SLURM_HOSTFILE and $LD_LIBRARY_PATH.

Added in version 1.2.0: New base task base.unsetenv added.

Basic file operations

The ScriptEngine base task package provides tasks to create, copy/move/link and remove files and directories, as described in detail below in this section.

Whenever it makes sense, the tasks will accept as their argument values single file or directory names, as well as YAML lists of such. Furthermore, instead of full names, also Unix shell wildcard expressions are accepted.

`base.make_dir`

Creates a new directory at the given path:

base.make_dir:
    path: <PATH>

If path already exists, an info message is displayed (no warning or error). This task creates parent directories as needed. An error is raised when path is a file or symbolic link.

A list of names is accepted for the path argument.

`base.copy`

This task copies the file or directory given by src to dst:

- base.copy:
    src: <PATH>
    dst: <PATH>
    ignore_not_found: <true or false>  # optional, default false

If src is a file and dst is a directory, the src file is copied into the dst/ directory. If src is a directory, dst must be a directory as well and src is copied recursively into dst. When a directory is copied, symbolic links are preserved.

When copying a file and the dst exists already, it is overwritten and a warning is issued. Copying a directory when dst already exists results in an error. An error occurs if src does not exist, unless ignore_not_found is true.

A list of names or wildcard expressions is accepted for the src argument, provided that dst is a directory.

`base.link`

Creates a symbolic link with name given by link, which is pointing to the path given by target:

base.link:
    target: <PATH>
    link: <PATH>

When the target does not exist, the link is still created and a warning is issued.

When link is a directory, a symbolic link with the same base name as target is created within the link directory, pointing to target.

A list of names or wildcard expressions is accepted for the target argument, provided that link is a directory. A symbolic link for each target is created in that case in the link directory.

Warning

In ScriptEngine versions up to 0.13.1 the arguments of base.link were named src (now target) and dst (now link). The use of these argument names is deprecated and will be invalid in future versions of ScriptEngine. The use of directories for dst or lists and wildcards for src was not available in those versions.

`base.move`

Moves files or directories (the latter recursively) from src to dst:

base.move:
    src: <PATH>
    dst: <PATH>
    ignore_not_found: <true or false>  # optional, default false

If dst is a directory, src is moved into dst. If src does not exists, an error occurs unless ignore_not_found is true.

Provided that dst is a directory, src may be a list or wildcard expression.

`base.remove`

Removes a file, link, or directory:

base.remove:
    path: <PATH>
    ignore_not_found: <true or false>  # optional, default false

Directories are recursively deleted, effectively removing all files and subdirectories that it contains. When path does not exist, an error occurs, unless ignore_not_found is true.

A list of names or wildcard expressions is accepted for path.

Other file operations

`base.template`

Runs the template file given by src through the Jinja2 Template Engine and saves the result as a file at dst:

base.template:
    src: <PATH>
    dst: <PATH>
    executable: [true|false]  # optional

ScriptEngine searches for the template file (src) in the following directories, in the order given:

.
./templates
{{ se.cli.cwd }}
{{ se.cli.cwd }}/templates

where . is the current directory at the time when the template task is executed and {{se.cli.cwd}} is the original working directory, the working directory at the time when the ScriptEngine command line tool was called.

The ScriptEngine context is passed to the Jinja2 template engine when the template is rendered, which means that all context parameters can be referred to in the template.

If the executable argument is true (the default being false), the destination file will get executable permissions. The setting of permissions will respect the user’s umask.

`base.find`

This task can be used to find files or directories in the file system:

base.find:
    path: <PATH>
    pattern: <SEARCH_PATTERN>  # optional, default "*"
    type: <FILE_OR_DIR>  # optional, default "file"
    depth: <NUMBER>  # optional, default -1
    set: <CONTEXT_PARAMETER>  # optional, default "result"

Files and directories are searched starting at path and descending at most depth levels down the directory hierarchy. If depth is less than zero, no limit is used for the search. Files and directories are matched against the Unix shell-style wildcards pattern, which may include:

*       matches everything
?       matches any single character
[seq]   matches any character in seq
[!seq]  matches any character not in seq

If type is file (default) or dir, then base.find will search for files or directories, respectively.

Note

base.find supports Unix shell-type wildcards, not regular expressions.

Timing

`base.time`

This task can be used to measure absolute or elapsed time in ScriptEngine scripts:

base.time:
    set: <CONTEXT_NAME>
    since: <DATETIME>  # optional

The set argument specifies a name under which the time is stored in the context. Only simple names, without dots, may be used.

If the since argument is used, it must represent a datetime object and the elapsed time since this reference is measured:

- base.time:
    set: start
- base.echo:
    msg: Hello, world!
- base.time:
    set: elapsed_time
    since: "{{ start }}"

`base.task_timer`

This task can be used to control ScriptEngine’s build-in timing feature. It is used as:

base.task_timer:
    mode: <TIMING_MODE>
    logging: <LOGLEVEL>  # optional

where:

TIMING_MODE is one of
    false:       Timing is switched off.
    'basic':     Each task is timed, log messages are written according to
                 'logging' argument.
    'classes':   As for 'basic', plus times are accumulated for task
                 classes.
    'instances': As for 'classes', plus times are accumulated for each
                 individual task instance.

and:

LOGLEVEL is one of
    false:   No time logging after each task
    'info':  Logging to the info logger
    'debug': Logging to the debug logger

Note

Switching off logging (LOGLEVEL: false) does not affect the collection of timing data for the tasks.

Context task

Stores or updates data (one or more, possibly nested, pairs of names and values) in the ScriptEngine context.

Usage:

base.context:
    <NAME>: <VALUE>
    [...]

Example:

base.context:
    planet: Earth
    some_countries:
        - Norway
        - Sweden
        - Finnland
        - Danmark
    number: 4

The arguments of base.context will be merged into the ScriptEngine context. This means that data can be added to existing data structures, such as lists or dictionaries:

- base.context:
    mylist:
        - one
        - two
# mylist is ['one', 'two']
- base.context:
    mylist:
        - 3
        - 4
# mylist is now ['one', 'two', 3, 4]

In the case of conflicts, the arguments of base.context will overwrite the current values in the ScriptEngine context. This can be used to clean data:

- base.context:
    mylist:
        - one
        - two
# mylist is ['one', 'two']
- base.context:
    mylist: null  # "remove" the value if mylist
- base.context:
    mylist:
        - 3
        - 4
# mylist is now [3, 4]

The base task package

base.echo

Working with the SE context

base.context

base.context.load

Control flow

base.include

base.command

base.exit

Shell environment

base.chdir

base.getenv

base.setenv

base.unsetenv

Basic file operations

base.make_dir

base.copy

base.link

base.move

base.remove

Other file operations

base.template

base.find

Timing

base.time

base.task_timer

Context task

`base.echo`

`base.context`

`base.context.load`

`base.include`

`base.command`

`base.exit`

`base.chdir`

`base.getenv`

`base.setenv`

`base.unsetenv`

`base.make_dir`

`base.copy`

`base.link`

`base.move`

`base.remove`

`base.template`

`base.find`

`base.time`

`base.task_timer`